saas-backend-bot
df99305dd6
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 1s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 2s
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 29s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 29s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 49s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 1m39s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m44s
CI/CD Pipeline / Validate - Style (pull_request) Successful in 2m19s
AI Code Review / AI Code Review (pull_request) Failing after 2m52s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 2m54s
CI/CD Pipeline / Validate - Security (pull_request) Successful in 4m11s
CI/CD Pipeline / Unit Tests (pull_request) Failing after 6m23s
CI/CD Pipeline / Build Production API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI/CD Pipeline / CI Gate (pull_request) Failing after 1s
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / Canary Release to Production (pull_request) Has been skipped
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 3m39s
问题：素材转码与视频生成共用 celery 默认队列、worker 单进程消费， 20+ 转码积压会把用户生成任务堵 40 分钟以上；孤儿清理把任务标 failed 后 Redis 队列消息未作废，消息被重投导致 failed→running 非法转换， worker 打印 ERROR 后继续产出半成品。 队列隔离： - 新增 packages/shared/celery_queues.py：generation/transcode/celery 三队列与 task_routes（generate_video→generation；ingest_asset/ classify_asset/duplication→transcode），apply_queue_settings() - worker 入口改双进程：generation worker 独占队列并内嵌 beat （prefetch=1, GENERATION_CONCURRENCY 默认 2），transcode worker 消费 transcode,celery（并发=总-2，最小 1），任一退出则整体终止 - compose/部署脚本/ps1 同步新增 GENERATION_CONCURRENCY 与健康检查 消息作废： - 新增 packages/shared/celery_orphan_guard.py：终态守卫 ensure_task_claimable、Redis 队列消息物理清理（JSON 信封解析， 按业务 id + celery headers.id 双匹配，未命中 rpush 保序）、 revoke_and_purge（control.revoke + 物理清队列双保险） - 入队点（生成/上传/分片/重试）send_task 后持久化 celery_task_id 到 generation_tasks/ingest_jobs（新列，067 迁移，失败仅 warning） - generate_video/ingest_asset 执行前校验 DB 状态：终态直接 discarded 不进业务逻辑；mark_processing 返回 False（非法转换）安全中止 - 孤儿/超时清理标 failed 时同时 revoke + 清队列消息 - pending 超时阈值 15→45 分钟，与 running 孤儿(20min)区分 测试：新增 22 个单测（路由表/真实 Redis 消息清理/终态守卫/ 非法转换中止/标 failed 后消息不重投/入队持久化），全量 14301 passed；067 迁移隔离 DDL 验证 upgrade/downgrade 通过。
52 lines
2.1 KiB
Python
Executable File
52 lines
2.1 KiB
Python
Executable File
from celery import Celery
|
||
from worker_app.core.config import get_settings
|
||
|
||
settings = get_settings()
|
||
celery_app = Celery(settings.worker_name)
|
||
celery_app.conf.broker_url = settings.broker_url
|
||
celery_app.conf.result_backend = settings.result_backend
|
||
celery_app.conf.broker_connection_retry_on_startup = True
|
||
|
||
# #1714 队列隔离：generation（高优，独占 worker）/ transcode（素材转码）/ celery（默认）
|
||
from packages.shared.celery_queues import ( # noqa: E402
|
||
GENERATION_WORKER_PREFETCH_MULTIPLIER,
|
||
apply_queue_settings,
|
||
)
|
||
|
||
apply_queue_settings(celery_app)
|
||
# 长渲染任务预取 1，避免任务被预取占住导致调度不均
|
||
celery_app.conf.worker_prefetch_multiplier = GENERATION_WORKER_PREFETCH_MULTIPLIER
|
||
celery_app.conf.task_acks_late = True # worker 崩溃时未完成任务重回队列，由执行前守卫丢弃作废消息
|
||
|
||
celery_app.conf.imports = (
|
||
"worker_app.tasks.health",
|
||
"worker_app.tasks.ingest",
|
||
"worker_app.tasks.classification",
|
||
"worker_app.tasks.generation",
|
||
"worker_app.tasks.voice_extraction",
|
||
"worker_app.tasks.voice_clone",
|
||
"worker_app.tasks.tts_synthesis",
|
||
"worker_app.tasks.batch_download",
|
||
"worker_app.tasks.duplication_check",
|
||
"worker_app.tasks._startup",
|
||
"apps.worker.video_processing.dedup",
|
||
"worker_app.tasks.cleanup",
|
||
)
|
||
|
||
# Celery Beat 定时任务调度
|
||
# 注：worker 单实例内嵌 beat（entrypoint-worker.sh -B），定时任务不会重复执行
|
||
celery_app.conf.beat_schedule = {
|
||
# pending 任务超时清理：worker 停止消费后，卡 pending 的任务 15 分钟内释放限流名额
|
||
"cleanup-stale-pending-tasks": {
|
||
"task": "worker.cleanup_stale_pending_tasks",
|
||
"schedule": 300.0, # 每 5 分钟（秒）
|
||
"options": {"expires": 240}, # 4 分钟过期，避免堆积
|
||
},
|
||
# running 孤儿任务巡检：容器重启/进程被杀后卡 running 的任务，20 分钟无更新则判失败
|
||
"cleanup-stale-running-tasks": {
|
||
"task": "worker.cleanup_stale_running_tasks",
|
||
"schedule": 300.0, # 每 5 分钟（秒）
|
||
"options": {"expires": 240},
|
||
},
|
||
}
|