df99305dd6
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 1s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 2s
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 29s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 29s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 49s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 1m39s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m44s
CI/CD Pipeline / Validate - Style (pull_request) Successful in 2m19s
AI Code Review / AI Code Review (pull_request) Failing after 2m52s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 2m54s
CI/CD Pipeline / Validate - Security (pull_request) Successful in 4m11s
CI/CD Pipeline / Unit Tests (pull_request) Failing after 6m23s
CI/CD Pipeline / Build Production API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI/CD Pipeline / CI Gate (pull_request) Failing after 1s
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / Canary Release to Production (pull_request) Has been skipped
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 3m39s
问题:素材转码与视频生成共用 celery 默认队列、worker 单进程消费, 20+ 转码积压会把用户生成任务堵 40 分钟以上;孤儿清理把任务标 failed 后 Redis 队列消息未作废,消息被重投导致 failed→running 非法转换, worker 打印 ERROR 后继续产出半成品。 队列隔离: - 新增 packages/shared/celery_queues.py:generation/transcode/celery 三队列与 task_routes(generate_video→generation;ingest_asset/ classify_asset/duplication→transcode),apply_queue_settings() - worker 入口改双进程:generation worker 独占队列并内嵌 beat (prefetch=1, GENERATION_CONCURRENCY 默认 2),transcode worker 消费 transcode,celery(并发=总-2,最小 1),任一退出则整体终止 - compose/部署脚本/ps1 同步新增 GENERATION_CONCURRENCY 与健康检查 消息作废: - 新增 packages/shared/celery_orphan_guard.py:终态守卫 ensure_task_claimable、Redis 队列消息物理清理(JSON 信封解析, 按业务 id + celery headers.id 双匹配,未命中 rpush 保序)、 revoke_and_purge(control.revoke + 物理清队列双保险) - 入队点(生成/上传/分片/重试)send_task 后持久化 celery_task_id 到 generation_tasks/ingest_jobs(新列,067 迁移,失败仅 warning) - generate_video/ingest_asset 执行前校验 DB 状态:终态直接 discarded 不进业务逻辑;mark_processing 返回 False(非法转换)安全中止 - 孤儿/超时清理标 failed 时同时 revoke + 清队列消息 - pending 超时阈值 15→45 分钟,与 running 孤儿(20min)区分 测试:新增 22 个单测(路由表/真实 Redis 消息清理/终态守卫/ 非法转换中止/标 failed 后消息不重投/入队持久化),全量 14301 passed;067 迁移隔离 DDL 验证 upgrade/downgrade 通过。
244 lines
8.8 KiB
Python
244 lines
8.8 KiB
Python
"""Issue #1714:任务执行前状态守卫 — 已作废消息必须丢弃,禁止非法转换后继续跑。
|
||
|
||
覆盖:
|
||
- ingest_asset:job 已 failed/completed 时直接返回 discarded,不下载、不转码、不回写
|
||
- generate_video:GenerationTask 已 failed 时返回 discarded,不进入渲染
|
||
- generate_video:pending → running 标记失败(非法转换)时安全中止
|
||
"""
|
||
|
||
from __future__ import annotations
|
||
|
||
import sys
|
||
from pathlib import Path
|
||
from types import SimpleNamespace
|
||
from unittest.mock import MagicMock
|
||
|
||
# ── worker 模块标准加载方式 ──
|
||
# 显式保存将要覆盖的注入键旧值:全量收集时更早的测试文件(如
|
||
# test_ingest_validation.py)可能已向 sys.modules 注入 worker_app.* mock,
|
||
# 导入完成后必须精确恢复旧值,否则本文件的 bind 感知透传装饰器会残留,
|
||
# 污染后续懒加载短路径 worker_app.celery_app 的 worker 测试。
|
||
_INJECTED_KEYS = ("worker_app.db", "worker_app.core.config", "worker_app.celery_app")
|
||
_SAVED_MODULE_VALUES = {k: sys.modules.get(k) for k in _INJECTED_KEYS}
|
||
_SAVED_MODULES_KEYS = set(sys.modules.keys())
|
||
|
||
_mock_db_module = MagicMock()
|
||
_mock_db_module.SessionLocal = MagicMock()
|
||
sys.modules["worker_app.db"] = _mock_db_module
|
||
sys.modules["worker_app.core.config"] = MagicMock()
|
||
|
||
_mock_celery_module = MagicMock()
|
||
|
||
|
||
def _passthrough_decorator(*args, **kwargs):
|
||
if len(args) == 1 and callable(args[0]):
|
||
return args[0]
|
||
bind = kwargs.get("bind", False)
|
||
|
||
def _wrap(f):
|
||
if bind:
|
||
# 模拟 celery bind=True:task(task_id) 调用时注入 self(MagicMock)
|
||
return lambda *a, **kw: f(MagicMock(), *a, **kw)
|
||
return f
|
||
|
||
return _wrap
|
||
|
||
|
||
_mock_celery_module.celery_app.task = MagicMock(side_effect=_passthrough_decorator)
|
||
sys.modules["worker_app.celery_app"] = _mock_celery_module
|
||
|
||
_WORKER_PATH = str(Path(__file__).resolve().parents[2] / "apps" / "worker")
|
||
sys.path.insert(0, _WORKER_PATH)
|
||
|
||
import pytest # noqa: E402
|
||
from worker_app.tasks import ingest as ingest_mod # noqa: E402
|
||
|
||
# video_processing 相关 mock(generation 模块导入链)
|
||
for _mod_name in [
|
||
"video_processing",
|
||
"video_processing.ffmpeg_utils",
|
||
"video_processing.oss_helpers",
|
||
]:
|
||
sys.modules.setdefault(_mod_name, MagicMock())
|
||
|
||
from worker_app.tasks import generation as gen_mod # noqa: E402
|
||
|
||
from packages.domain import IngestJobStatus # noqa: E402
|
||
|
||
# 模块导入完成后立即清理:删除本次 import 新引入的模块缓存(本模块已通过名字绑定
|
||
# 持有 ingest_mod/gen_mod/IngestJobStatus,删除缓存不影响调用),再把三个注入键
|
||
# 精确恢复为注入前的旧值(旧值不存在则移除),杜绝 mock 残留污染其他 worker 测试。
|
||
for _key in list(sys.modules.keys()):
|
||
if _key not in _SAVED_MODULES_KEYS and not _key.startswith("video_processing"):
|
||
del sys.modules[_key]
|
||
for _k, _v in _SAVED_MODULE_VALUES.items():
|
||
if _v is None:
|
||
sys.modules.pop(_k, None)
|
||
else:
|
||
sys.modules[_k] = _v
|
||
del _SAVED_MODULES_KEYS, _SAVED_MODULE_VALUES
|
||
|
||
|
||
# ── ingest 守卫 ────────────────────────────────────────────────────────
|
||
|
||
|
||
class _FakeJobRepo:
|
||
def __init__(self, job):
|
||
self.job = job
|
||
|
||
def get(self, job_id):
|
||
return self.job
|
||
|
||
|
||
def _make_ingest_job(status):
|
||
job = MagicMock()
|
||
job.id = "job-stale-1"
|
||
job.storage_key = "uploads/proj/stale.mov"
|
||
job.status = status
|
||
job.file_hash = "h"
|
||
job.asset_id = ""
|
||
return job
|
||
|
||
|
||
def test_ingest_discards_failed_job_message():
|
||
"""job 已 failed:消息丢弃,不进入下载/转码/回写。"""
|
||
job = _make_ingest_job(IngestJobStatus.FAILED)
|
||
fake_session = MagicMock()
|
||
_mock_db_module.SessionLocal = MagicMock(return_value=fake_session)
|
||
|
||
# SQLAlchemy 仓储构造返回 fake
|
||
fake_job_repo = _FakeJobRepo(job)
|
||
fake_asset_repo = MagicMock()
|
||
|
||
orig_job_repo = ingest_mod.SQLAlchemyIngestJobRepository
|
||
orig_asset_repo = ingest_mod.SQLAlchemyAssetRepository
|
||
ingest_mod.SQLAlchemyIngestJobRepository = MagicMock(return_value=fake_job_repo)
|
||
ingest_mod.SQLAlchemyAssetRepository = MagicMock(return_value=fake_asset_repo)
|
||
try:
|
||
result = ingest_mod.ingest_asset("job-stale-1")
|
||
finally:
|
||
ingest_mod.SQLAlchemyIngestJobRepository = orig_job_repo
|
||
ingest_mod.SQLAlchemyAssetRepository = orig_asset_repo
|
||
|
||
assert result["status"] == "discarded"
|
||
# 没有任何 update / commit / 下载动作
|
||
fake_session.commit.assert_not_called()
|
||
fake_asset_repo.create.assert_not_called()
|
||
|
||
|
||
def test_ingest_discards_completed_job_message():
|
||
job = _make_ingest_job(IngestJobStatus.COMPLETED)
|
||
fake_session = MagicMock()
|
||
_mock_db_module.SessionLocal = MagicMock(return_value=fake_session)
|
||
fake_job_repo = _FakeJobRepo(job)
|
||
|
||
orig = ingest_mod.SQLAlchemyIngestJobRepository
|
||
ingest_mod.SQLAlchemyIngestJobRepository = MagicMock(return_value=fake_job_repo)
|
||
ingest_mod.SQLAlchemyAssetRepository = MagicMock(return_value=MagicMock())
|
||
try:
|
||
result = ingest_mod.ingest_asset("job-stale-1")
|
||
finally:
|
||
ingest_mod.SQLAlchemyIngestJobRepository = orig
|
||
|
||
assert result["status"] == "discarded"
|
||
|
||
|
||
# ── generation 守卫 ────────────────────────────────────────────────────
|
||
|
||
|
||
def _make_gen_task(status_value: str):
|
||
from packages.domain import GenerationTask
|
||
|
||
task = GenerationTask.create(project_id="p", asset_library_id="l", created_by_user_id="u")
|
||
task.status = type(task.status)(status_value)
|
||
return task
|
||
|
||
|
||
def test_generate_video_discards_failed_task(monkeypatch):
|
||
"""GenerationTask 已 failed:直接 discarded,不加载渲染数据。"""
|
||
failed_task = _make_gen_task("failed")
|
||
|
||
fake_repo = MagicMock()
|
||
fake_repo.get.return_value = failed_task
|
||
|
||
fake_session = MagicMock()
|
||
_mock_db_module.SessionLocal = MagicMock(return_value=fake_session)
|
||
|
||
import packages.adapters.sqlalchemy_impl.generation_task_repository as gen_repo_mod
|
||
|
||
orig = gen_repo_mod.SQLAlchemyGenerationTaskRepository
|
||
gen_repo_mod.SQLAlchemyGenerationTaskRepository = MagicMock(return_value=fake_repo)
|
||
|
||
update_status_mock = MagicMock(return_value=False)
|
||
monkeypatch.setattr(gen_mod, "_update_task_status", update_status_mock)
|
||
monkeypatch.setattr(
|
||
gen_mod,
|
||
"_load_task_info",
|
||
lambda task_id: {
|
||
"project_id": "p",
|
||
"template_id": "",
|
||
"task_asset_ids": [],
|
||
"batch_id": "",
|
||
"user_id": "u",
|
||
"mode": "one_take",
|
||
},
|
||
)
|
||
monkeypatch.setattr(gen_mod, "_flush_logs", lambda *a, **k: None)
|
||
|
||
task_fn = gen_mod.generate_video
|
||
if hasattr(task_fn, "__wrapped__"):
|
||
task_fn = task_fn.__wrapped__
|
||
try:
|
||
result = task_fn("task-stale-1")
|
||
finally:
|
||
gen_repo_mod.SQLAlchemyGenerationTaskRepository = orig
|
||
|
||
assert result["status"] == "discarded"
|
||
# 状态守卫命中终态,根本不应尝试 mark_processing
|
||
update_status_mock.assert_not_called()
|
||
|
||
|
||
def test_generate_video_aborts_when_claim_fails(monkeypatch):
|
||
"""pending 但 mark_processing 返回 False(状态机非法转换)时安全中止。"""
|
||
pending_task = _make_gen_task("pending")
|
||
|
||
fake_repo = MagicMock()
|
||
fake_repo.get.return_value = pending_task
|
||
fake_session = MagicMock()
|
||
_mock_db_module.SessionLocal = MagicMock(return_value=fake_session)
|
||
|
||
import packages.adapters.sqlalchemy_impl.generation_task_repository as gen_repo_mod
|
||
|
||
orig = gen_repo_mod.SQLAlchemyGenerationTaskRepository
|
||
gen_repo_mod.SQLAlchemyGenerationTaskRepository = MagicMock(return_value=fake_repo)
|
||
|
||
monkeypatch.setattr(
|
||
gen_mod,
|
||
"_load_task_info",
|
||
lambda task_id: {
|
||
"project_id": "p",
|
||
"template_id": "",
|
||
"task_asset_ids": [],
|
||
"batch_id": "",
|
||
"user_id": "u",
|
||
"mode": "one_take",
|
||
},
|
||
)
|
||
monkeypatch.setattr(gen_mod, "_flush_logs", lambda *a, **k: None)
|
||
# 模拟 mark_processing 失败(failed→running 非法转换被 _update_task_status 吞掉返回 False)
|
||
update_status_mock = MagicMock(return_value=False)
|
||
monkeypatch.setattr(gen_mod, "_update_task_status", update_status_mock)
|
||
render_mock = MagicMock(side_effect=AssertionError("must not render"))
|
||
monkeypatch.setattr(gen_mod, "_render_from_edit_plan", render_mock)
|
||
|
||
task_fn = gen_mod.generate_video
|
||
if hasattr(task_fn, "__wrapped__"):
|
||
task_fn = task_fn.__wrapped__
|
||
try:
|
||
result = task_fn("task-claim-fail")
|
||
finally:
|
||
gen_repo_mod.SQLAlchemyGenerationTaskRepository = orig
|
||
|
||
assert result["status"] == "discarded"
|
||
render_mock.assert_not_called()
|