fix(lipsync): 修复Celery事务竞态导致job永远卡在tts_processing(P0)
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 1s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 2s
CI/CD Pipeline / PR Build Web Image (pull_request) Successful in 30s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 33s
CI/CD Pipeline / Frontend Lint (pull_request) Successful in 1m39s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 1m39s
CI/CD Pipeline / Frontend Unit Tests (pull_request) Successful in 1m47s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 2m44s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 2m55s
AI Code Review / AI Code Review (pull_request) Successful in 6m31s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 8m15s
CI/CD Pipeline / Unit Tests (pull_request) Successful in 8m17s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 10m28s
CI/CD Pipeline / Validate - Style (pull_request) Has been cancelled
CI/CD Pipeline / Validate - Security (pull_request) Has been cancelled
CI/CD Pipeline / Build Production API Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Web Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been cancelled
CI/CD Pipeline / Deploy Production (pull_request) Has been cancelled
CI/CD Pipeline / Production Browser E2E (pull_request) Has been cancelled
CI/CD Pipeline / Canary Release to Production (pull_request) Has been cancelled
CI/CD Pipeline / CI Gate (pull_request) Has been cancelled
PR Automation / Auto Merge on CI Green + Approved (pull_request) Has been cancelled
CI/CD Pipeline / Staging API Integration Tests (pull_request) Failing after 89h23m49s
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Failing after 89h23m53s
CI/CD Pipeline / Build Staging Web Image (pull_request) Failing after 89h23m55s
CI/CD Pipeline / Build Staging API Image (pull_request) Failing after 89h23m33s
CI/CD Pipeline / ACR Image Cleanup (pull_request) Failing after 89h23m27s
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Failing after 89h23m31s
CI/CD Pipeline / Build Staging Worker Image (pull_request) Failing after 89h23m33s
CI/CD Pipeline / Staging E2E Tests (pull_request) Failing after 89h23m27s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Failing after 89h23m28s
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Failing after 89h23m31s
CI/CD Pipeline / Check push changed paths (pull_request) Failing after 89h24m9s
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 1s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 2s
CI/CD Pipeline / PR Build Web Image (pull_request) Successful in 30s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 33s
CI/CD Pipeline / Frontend Lint (pull_request) Successful in 1m39s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 1m39s
CI/CD Pipeline / Frontend Unit Tests (pull_request) Successful in 1m47s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 2m44s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 2m55s
AI Code Review / AI Code Review (pull_request) Successful in 6m31s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 8m15s
CI/CD Pipeline / Unit Tests (pull_request) Successful in 8m17s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 10m28s
CI/CD Pipeline / Validate - Style (pull_request) Has been cancelled
CI/CD Pipeline / Validate - Security (pull_request) Has been cancelled
CI/CD Pipeline / Build Production API Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Web Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been cancelled
CI/CD Pipeline / Deploy Production (pull_request) Has been cancelled
CI/CD Pipeline / Production Browser E2E (pull_request) Has been cancelled
CI/CD Pipeline / Canary Release to Production (pull_request) Has been cancelled
CI/CD Pipeline / CI Gate (pull_request) Has been cancelled
PR Automation / Auto Merge on CI Green + Approved (pull_request) Has been cancelled
CI/CD Pipeline / Staging API Integration Tests (pull_request) Failing after 89h23m49s
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Failing after 89h23m53s
CI/CD Pipeline / Build Staging Web Image (pull_request) Failing after 89h23m55s
CI/CD Pipeline / Build Staging API Image (pull_request) Failing after 89h23m33s
CI/CD Pipeline / ACR Image Cleanup (pull_request) Failing after 89h23m27s
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Failing after 89h23m31s
CI/CD Pipeline / Build Staging Worker Image (pull_request) Failing after 89h23m33s
CI/CD Pipeline / Staging E2E Tests (pull_request) Failing after 89h23m27s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Failing after 89h23m28s
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Failing after 89h23m31s
CI/CD Pipeline / Check push changed paths (pull_request) Failing after 89h24m9s
根因:create_job() 先 apply_async() 发送 Celery 任务,再 db.commit() 提交事务。 worker 是独立进程+独立DB连接,任务<4ms就被消费,但此时 API 事务还未 commit, worker 查询 job 返回 None → 静默 return 不重试 → job 永远卡在 tts_processing。 修复: 1. API侧(lipsync_service.py):先 db.commit()+db.refresh(job),再 apply_async() 发任务; 投递失败/MediaKit提交失败分支也各自 commit,确保状态及时落库。 2. Worker侧(lipsync_tts.py):job not found 时改用 self.retry() 递增重试3次 (1s/3s/7s退避),作为竞态场景的第二道防线; 增加 autoretry_for=(OSError, ConnectionError) 自动重试网络抖动; max_retries 从2调整到5。 子agent在staging实锤:受影响job共5个,手工重投递后全部3秒内完成TTS+提交MediaKit, 证实TTS本身只需要2秒,卡顿完全是因为竞态。 单测:新增 TestCreateJobCommitOrder 验证 commit 在 apply_async 之前。
This commit is contained in:
@@ -198,6 +198,12 @@ class LipsyncService:
|
||||
self.db.add(job)
|
||||
self.db.flush()
|
||||
|
||||
# ⚠️ 必须先 commit 再发 Celery 任务,避免事务竞态:
|
||||
# worker 是独立进程+独立DB连接,任务被消费(<4ms)时若本事务还未提交,
|
||||
# worker 查询 job 会返回 None → 静默 return 不重试,job 永远卡在 tts_processing。
|
||||
self.db.commit()
|
||||
self.db.refresh(job)
|
||||
|
||||
if is_tts_mode:
|
||||
# 2a. TTS 模式:dispatch Celery 异步任务处理 TTS 合成 + MediaKit 提交
|
||||
try:
|
||||
@@ -223,6 +229,7 @@ class LipsyncService:
|
||||
job.error_message = f"Celery 任务投递失败: {exc}"
|
||||
job.error_code = "AsyncDispatchFailed"
|
||||
job.updated_at = datetime.now(timezone.utc)
|
||||
self.db.commit() # 投递失败也要落库失败状态
|
||||
else:
|
||||
# 2b. 直接音频模式:同步签名并提交 MediaKit
|
||||
video_url = self._sign_media_url(video_url)
|
||||
@@ -240,15 +247,15 @@ class LipsyncService:
|
||||
job.mediakit_task_id = result["task_id"]
|
||||
job.status = "submitted"
|
||||
job.submitted_at = datetime.now(timezone.utc)
|
||||
self.db.commit() # submitted 状态落库
|
||||
except MediaKitError as exc:
|
||||
job.status = "failed"
|
||||
job.error_message = str(exc)
|
||||
job.error_code = exc.code
|
||||
logger.error("提交对口型任务失败: %s", exc)
|
||||
self.db.commit()
|
||||
raise
|
||||
|
||||
self.db.commit()
|
||||
self.db.refresh(job)
|
||||
return job
|
||||
|
||||
# ── 查询任务 ──────────────────────────────────────────────────────────
|
||||
|
||||
@@ -211,8 +211,11 @@ def _estimate_sentence_timings_by_chars(sentences: list[str], total_duration: fl
|
||||
@shared_task(
|
||||
bind=True,
|
||||
name="lipsync_tts.synthesize_and_submit",
|
||||
max_retries=2,
|
||||
max_retries=5, # 事务竞态重试3次(job not found)+ TTS偶发错误2次
|
||||
default_retry_delay=30,
|
||||
autoretry_for=(OSError, ConnectionError), # 网络/连接错误自动重试
|
||||
retry_backoff=True,
|
||||
retry_backoff_max=30,
|
||||
soft_time_limit=180,
|
||||
time_limit=200,
|
||||
)
|
||||
@@ -259,7 +262,28 @@ def tts_synthesize_and_submit(
|
||||
)
|
||||
|
||||
if job is None:
|
||||
logger.error("[lipsync_tts] Job not found: job_id=%s", job_id)
|
||||
# 事务竞态防御:API 在 commit 前投递了任务,worker 消费时事务尚未提交。
|
||||
# Celery 内置 autoretry_for 不支持"业务条件重试",这里手动 retry 3 次,
|
||||
# 间隔递增(1s/3s/7s),让 API 事务有时间提交。
|
||||
# max_retries 由 self.request(retries) 维护;默认 self.max_retries=3 由装饰器 soft_time_limit 下方指定。
|
||||
retries = getattr(self.request, "retries", 0)
|
||||
max_retries = 3
|
||||
if retries < max_retries:
|
||||
backoff = (2 ** retries) + (retries * 1) # 1s, 3s, 7s
|
||||
logger.warning(
|
||||
"[lipsync_tts] Job not found yet (retry %d/%d, backoff %ds): job_id=%s",
|
||||
retries + 1,
|
||||
max_retries,
|
||||
backoff,
|
||||
job_id,
|
||||
)
|
||||
self.db.close()
|
||||
raise self.retry(countdown=backoff, max_retries=max_retries)
|
||||
logger.error(
|
||||
"[lipsync_tts] Job not found after %d retries, giving up: job_id=%s",
|
||||
max_retries,
|
||||
job_id,
|
||||
)
|
||||
return
|
||||
|
||||
# 已取消的任务不再处理
|
||||
|
||||
@@ -288,3 +288,44 @@ class TestCancelJobTtsProcessing:
|
||||
|
||||
result = svc.cancel_job("job-1", "user-1")
|
||||
assert result.status == "cancelled"
|
||||
|
||||
|
||||
class TestCreateJobCommitOrder:
|
||||
"""验证事务顺序修复:create_job 必须先 commit 再发 Celery 任务,避免 worker 消费时 job 不可见。"""
|
||||
|
||||
def test_commit_called_before_apply_async_in_tts_mode(self):
|
||||
"""TTS 模式:db.commit() 必须在 apply_async() 之前调用,防止 worker 查不到 job 永远卡在 tts_processing。"""
|
||||
svc, client, cosy = _make_service_with_mocks()
|
||||
call_order: list[str] = []
|
||||
|
||||
def track_commit():
|
||||
call_order.append("commit")
|
||||
def track_apply_async(*args, **kwargs):
|
||||
call_order.append("apply_async")
|
||||
|
||||
svc.db.commit.side_effect = track_commit
|
||||
|
||||
with patch("app.services.lipsync_service.tts_synthesize_and_submit") as mock_task:
|
||||
mock_task.apply_async = MagicMock(side_effect=track_apply_async)
|
||||
svc.create_job(
|
||||
user_id="user-1",
|
||||
video_url="https://example.com/video.mp4",
|
||||
voice_id="v-1",
|
||||
script_text="测试",
|
||||
)
|
||||
|
||||
# 至少有一次 commit 在 apply_async 之前
|
||||
assert "commit" in call_order, "db.commit 必须被调用"
|
||||
assert "apply_async" in call_order, "apply_async 必须被调用"
|
||||
assert call_order.index("commit") < call_order.index("apply_async"), (
|
||||
f"事务顺序错误:commit 必须在 apply_async 之前,实际顺序 {call_order}"
|
||||
)
|
||||
|
||||
def test_job_not_found_retry_mechanism_exists(self):
|
||||
"""worker 侧 job not found 必须有重试机制(self.retry),而不是静默 return。"""
|
||||
import inspect
|
||||
from app.tasks.lipsync_tts import tts_synthesize_and_submit
|
||||
source = inspect.getsource(tts_synthesize_and_submit.run)
|
||||
assert "self.retry" in source or "retry" in source, (
|
||||
"tts_synthesize_and_submit 在 job not found 时必须重试,防止静默失败"
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user