Files
xiaoxia-saas/tests/unit/test_task_discard_guard_1714.py
T
saas-backend-bot df99305dd6
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 1s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 2s
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 29s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 29s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 49s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 1m39s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m44s
CI/CD Pipeline / Validate - Style (pull_request) Successful in 2m19s
AI Code Review / AI Code Review (pull_request) Failing after 2m52s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 2m54s
CI/CD Pipeline / Validate - Security (pull_request) Successful in 4m11s
CI/CD Pipeline / Unit Tests (pull_request) Failing after 6m23s
CI/CD Pipeline / Build Production API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI/CD Pipeline / CI Gate (pull_request) Failing after 1s
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / Canary Release to Production (pull_request) Has been skipped
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 3m39s
feat(worker): celery 队列隔离 + 孤儿任务消息作废 (#1714)
问题:素材转码与视频生成共用 celery 默认队列、worker 单进程消费,
20+ 转码积压会把用户生成任务堵 40 分钟以上;孤儿清理把任务标 failed
后 Redis 队列消息未作废,消息被重投导致 failed→running 非法转换,
worker 打印 ERROR 后继续产出半成品。

队列隔离:
- 新增 packages/shared/celery_queues.py:generation/transcode/celery
  三队列与 task_routes(generate_video→generation;ingest_asset/
  classify_asset/duplication→transcode),apply_queue_settings()
- worker 入口改双进程:generation worker 独占队列并内嵌 beat
  (prefetch=1, GENERATION_CONCURRENCY 默认 2),transcode worker
  消费 transcode,celery(并发=总-2,最小 1),任一退出则整体终止
- compose/部署脚本/ps1 同步新增 GENERATION_CONCURRENCY 与健康检查

消息作废:
- 新增 packages/shared/celery_orphan_guard.py:终态守卫
  ensure_task_claimable、Redis 队列消息物理清理(JSON 信封解析,
  按业务 id + celery headers.id 双匹配,未命中 rpush 保序)、
  revoke_and_purge(control.revoke + 物理清队列双保险)
- 入队点(生成/上传/分片/重试)send_task 后持久化 celery_task_id
  到 generation_tasks/ingest_jobs(新列,067 迁移,失败仅 warning)
- generate_video/ingest_asset 执行前校验 DB 状态:终态直接 discarded
  不进业务逻辑;mark_processing 返回 False(非法转换)安全中止
- 孤儿/超时清理标 failed 时同时 revoke + 清队列消息
- pending 超时阈值 15→45 分钟,与 running 孤儿(20min)区分

测试:新增 22 个单测(路由表/真实 Redis 消息清理/终态守卫/
非法转换中止/标 failed 后消息不重投/入队持久化),全量
14301 passed;067 迁移隔离 DDL 验证 upgrade/downgrade 通过。
2026-09-05 19:07:47 +08:00

244 lines
8.8 KiB
Python
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
"""Issue #1714:任务执行前状态守卫 — 已作废消息必须丢弃,禁止非法转换后继续跑。
覆盖:
- ingest_assetjob 已 failed/completed 时直接返回 discarded,不下载、不转码、不回写
- generate_videoGenerationTask 已 failed 时返回 discarded,不进入渲染
- generate_videopending → running 标记失败(非法转换)时安全中止
"""
from __future__ import annotations
import sys
from pathlib import Path
from types import SimpleNamespace
from unittest.mock import MagicMock
# ── worker 模块标准加载方式 ──
# 显式保存将要覆盖的注入键旧值:全量收集时更早的测试文件(如
# test_ingest_validation.py)可能已向 sys.modules 注入 worker_app.* mock
# 导入完成后必须精确恢复旧值,否则本文件的 bind 感知透传装饰器会残留,
# 污染后续懒加载短路径 worker_app.celery_app 的 worker 测试。
_INJECTED_KEYS = ("worker_app.db", "worker_app.core.config", "worker_app.celery_app")
_SAVED_MODULE_VALUES = {k: sys.modules.get(k) for k in _INJECTED_KEYS}
_SAVED_MODULES_KEYS = set(sys.modules.keys())
_mock_db_module = MagicMock()
_mock_db_module.SessionLocal = MagicMock()
sys.modules["worker_app.db"] = _mock_db_module
sys.modules["worker_app.core.config"] = MagicMock()
_mock_celery_module = MagicMock()
def _passthrough_decorator(*args, **kwargs):
if len(args) == 1 and callable(args[0]):
return args[0]
bind = kwargs.get("bind", False)
def _wrap(f):
if bind:
# 模拟 celery bind=Truetask(task_id) 调用时注入 selfMagicMock
return lambda *a, **kw: f(MagicMock(), *a, **kw)
return f
return _wrap
_mock_celery_module.celery_app.task = MagicMock(side_effect=_passthrough_decorator)
sys.modules["worker_app.celery_app"] = _mock_celery_module
_WORKER_PATH = str(Path(__file__).resolve().parents[2] / "apps" / "worker")
sys.path.insert(0, _WORKER_PATH)
import pytest # noqa: E402
from worker_app.tasks import ingest as ingest_mod # noqa: E402
# video_processing 相关 mockgeneration 模块导入链)
for _mod_name in [
"video_processing",
"video_processing.ffmpeg_utils",
"video_processing.oss_helpers",
]:
sys.modules.setdefault(_mod_name, MagicMock())
from worker_app.tasks import generation as gen_mod # noqa: E402
from packages.domain import IngestJobStatus # noqa: E402
# 模块导入完成后立即清理:删除本次 import 新引入的模块缓存(本模块已通过名字绑定
# 持有 ingest_mod/gen_mod/IngestJobStatus,删除缓存不影响调用),再把三个注入键
# 精确恢复为注入前的旧值(旧值不存在则移除),杜绝 mock 残留污染其他 worker 测试。
for _key in list(sys.modules.keys()):
if _key not in _SAVED_MODULES_KEYS and not _key.startswith("video_processing"):
del sys.modules[_key]
for _k, _v in _SAVED_MODULE_VALUES.items():
if _v is None:
sys.modules.pop(_k, None)
else:
sys.modules[_k] = _v
del _SAVED_MODULES_KEYS, _SAVED_MODULE_VALUES
# ── ingest 守卫 ────────────────────────────────────────────────────────
class _FakeJobRepo:
def __init__(self, job):
self.job = job
def get(self, job_id):
return self.job
def _make_ingest_job(status):
job = MagicMock()
job.id = "job-stale-1"
job.storage_key = "uploads/proj/stale.mov"
job.status = status
job.file_hash = "h"
job.asset_id = ""
return job
def test_ingest_discards_failed_job_message():
"""job 已 failed:消息丢弃,不进入下载/转码/回写。"""
job = _make_ingest_job(IngestJobStatus.FAILED)
fake_session = MagicMock()
_mock_db_module.SessionLocal = MagicMock(return_value=fake_session)
# SQLAlchemy 仓储构造返回 fake
fake_job_repo = _FakeJobRepo(job)
fake_asset_repo = MagicMock()
orig_job_repo = ingest_mod.SQLAlchemyIngestJobRepository
orig_asset_repo = ingest_mod.SQLAlchemyAssetRepository
ingest_mod.SQLAlchemyIngestJobRepository = MagicMock(return_value=fake_job_repo)
ingest_mod.SQLAlchemyAssetRepository = MagicMock(return_value=fake_asset_repo)
try:
result = ingest_mod.ingest_asset("job-stale-1")
finally:
ingest_mod.SQLAlchemyIngestJobRepository = orig_job_repo
ingest_mod.SQLAlchemyAssetRepository = orig_asset_repo
assert result["status"] == "discarded"
# 没有任何 update / commit / 下载动作
fake_session.commit.assert_not_called()
fake_asset_repo.create.assert_not_called()
def test_ingest_discards_completed_job_message():
job = _make_ingest_job(IngestJobStatus.COMPLETED)
fake_session = MagicMock()
_mock_db_module.SessionLocal = MagicMock(return_value=fake_session)
fake_job_repo = _FakeJobRepo(job)
orig = ingest_mod.SQLAlchemyIngestJobRepository
ingest_mod.SQLAlchemyIngestJobRepository = MagicMock(return_value=fake_job_repo)
ingest_mod.SQLAlchemyAssetRepository = MagicMock(return_value=MagicMock())
try:
result = ingest_mod.ingest_asset("job-stale-1")
finally:
ingest_mod.SQLAlchemyIngestJobRepository = orig
assert result["status"] == "discarded"
# ── generation 守卫 ────────────────────────────────────────────────────
def _make_gen_task(status_value: str):
from packages.domain import GenerationTask
task = GenerationTask.create(project_id="p", asset_library_id="l", created_by_user_id="u")
task.status = type(task.status)(status_value)
return task
def test_generate_video_discards_failed_task(monkeypatch):
"""GenerationTask 已 failed:直接 discarded,不加载渲染数据。"""
failed_task = _make_gen_task("failed")
fake_repo = MagicMock()
fake_repo.get.return_value = failed_task
fake_session = MagicMock()
_mock_db_module.SessionLocal = MagicMock(return_value=fake_session)
import packages.adapters.sqlalchemy_impl.generation_task_repository as gen_repo_mod
orig = gen_repo_mod.SQLAlchemyGenerationTaskRepository
gen_repo_mod.SQLAlchemyGenerationTaskRepository = MagicMock(return_value=fake_repo)
update_status_mock = MagicMock(return_value=False)
monkeypatch.setattr(gen_mod, "_update_task_status", update_status_mock)
monkeypatch.setattr(
gen_mod,
"_load_task_info",
lambda task_id: {
"project_id": "p",
"template_id": "",
"task_asset_ids": [],
"batch_id": "",
"user_id": "u",
"mode": "one_take",
},
)
monkeypatch.setattr(gen_mod, "_flush_logs", lambda *a, **k: None)
task_fn = gen_mod.generate_video
if hasattr(task_fn, "__wrapped__"):
task_fn = task_fn.__wrapped__
try:
result = task_fn("task-stale-1")
finally:
gen_repo_mod.SQLAlchemyGenerationTaskRepository = orig
assert result["status"] == "discarded"
# 状态守卫命中终态,根本不应尝试 mark_processing
update_status_mock.assert_not_called()
def test_generate_video_aborts_when_claim_fails(monkeypatch):
"""pending 但 mark_processing 返回 False(状态机非法转换)时安全中止。"""
pending_task = _make_gen_task("pending")
fake_repo = MagicMock()
fake_repo.get.return_value = pending_task
fake_session = MagicMock()
_mock_db_module.SessionLocal = MagicMock(return_value=fake_session)
import packages.adapters.sqlalchemy_impl.generation_task_repository as gen_repo_mod
orig = gen_repo_mod.SQLAlchemyGenerationTaskRepository
gen_repo_mod.SQLAlchemyGenerationTaskRepository = MagicMock(return_value=fake_repo)
monkeypatch.setattr(
gen_mod,
"_load_task_info",
lambda task_id: {
"project_id": "p",
"template_id": "",
"task_asset_ids": [],
"batch_id": "",
"user_id": "u",
"mode": "one_take",
},
)
monkeypatch.setattr(gen_mod, "_flush_logs", lambda *a, **k: None)
# 模拟 mark_processing 失败(failed→running 非法转换被 _update_task_status 吞掉返回 False
update_status_mock = MagicMock(return_value=False)
monkeypatch.setattr(gen_mod, "_update_task_status", update_status_mock)
render_mock = MagicMock(side_effect=AssertionError("must not render"))
monkeypatch.setattr(gen_mod, "_render_from_edit_plan", render_mock)
task_fn = gen_mod.generate_video
if hasattr(task_fn, "__wrapped__"):
task_fn = task_fn.__wrapped__
try:
result = task_fn("task-claim-fail")
finally:
gen_repo_mod.SQLAlchemyGenerationTaskRepository = orig
assert result["status"] == "discarded"
render_mock.assert_not_called()