fix(gpu): #1970 MuseTalk worker 推理期心跳/超时 900/重试收敛/短视频前置失败
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 0s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 1s
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 1m23s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 2m23s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 3m3s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 3m0s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 3m1s
CI/CD Pipeline / Validate - Style (pull_request) Successful in 3m5s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 4m19s
AI Code Review / AI Code Review (pull_request) Successful in 6m27s
CI/CD Pipeline / Validate - Security (pull_request) Successful in 6m44s
CI/CD Pipeline / Unit Tests (pull_request) Failing after 8m42s
CI/CD Pipeline / Build Production API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / Canary Release to Production (pull_request) Has been skipped
CI/CD Pipeline / CI Gate (pull_request) Failing after 1s
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 5m54s

- GPU_TASK_TIMEOUT_SECONDS 默认 300→900(base.py + env 模板),worker
  REQUEST_TIMEOUT 默认同步 300→900,RTX2060 6G 处理 720p 长视频不再超时
- worker 新增 TaskHeartbeat daemon 线程:任务处理期间每 30s POST
  /gpu/register(task_id=...) 续任务心跳,服务端只在任务心跳真正停滞
  超过 900s(崩溃/断网)或 worker 明确上报 failed 时才回退 pending,
  长推理阻塞主循环不再导致误回退
- register schema/service 支持 task_id:_touch_task_heartbeat 只刷新
  属于该 worker 且仍 processing 的任务,已完成/已被回收重派的过期心跳忽略
- worker 本地 TASK_MAX_RETRY 2→1,且仅对瞬时错误(连接失败/超时/5xx)重试;
  4xx、结果过小等确定性失败不本地重试,服务端 MAX_ATTEMPTS=3 不变,
  消除 3×3=9 次推理放大
- <3s 输入视频(MuseTalk division by zero)下载后 ffprobe 前置校验,
  直接上报 failed"视频过短",不调用推理;ffprobe 不可用时不拦截
- 新增 11 个单测(worker 独立脚本按路径加载),全量 15839 passed
This commit is contained in:
xiaoxia
2026-09-19 14:29:22 +08:00
parent a1f25a4426
commit a8f1069cd2
12 changed files with 514 additions and 59 deletions
+67
View File
@@ -181,6 +181,73 @@ def test_register_worker_creates_then_updates(svc):
assert w2.created_at == w.created_at # 没新建
def test_register_with_task_id_refreshes_task_heartbeat(svc):
"""#1970 推理期心跳:register(task_id=...) 只刷新本 worker 的 processing 任务."""
from packages.adapters.sqlalchemy_impl.models import GpuWorkerModel
t = svc.create_task(video_url="v", audio_url="a")
svc.poll_task("w-1")
svc.db.refresh(t)
old_hb = t.last_heartbeat_at
assert t.status == "processing"
# 模拟时间流逝后心跳到达
svc.db.query(GpuWorkerModel).filter_by(worker_id="w-1").update(
{"last_heartbeat_at": old_hb - timedelta(seconds=300)}
)
svc.db.commit()
svc.register_worker("w-1", task_id=t.id)
svc.db.refresh(t)
assert t.last_heartbeat_at > old_hb
assert t.status == "processing" # 心跳不改变状态
# worker 表心跳也被刷新
w = svc.db.query(GpuWorkerModel).filter_by(worker_id="w-1").one()
assert w.last_heartbeat_at > old_hb
def test_register_task_heartbeat_ignores_finished_or_foreign_task(svc):
"""任务已 done,或已被超时回收重新派发给别的 worker 时,旧心跳必须忽略."""
from packages.adapters.sqlalchemy_impl.models import GpuLipsyncTaskModel, GpuWorkerModel
# 场景 1:任务已完成 → register 带 task_id 不得改写任务心跳
t = svc.create_task(video_url="v", audio_url="a")
svc.poll_task("w-1")
done = svc.report_result(t.id, "w-1", success=True, duration_seconds=10.0)
hb_when_done = done.last_heartbeat_at
svc.register_worker("w-1", task_id=t.id)
svc.db.refresh(t)
assert t.status == "done"
assert t.last_heartbeat_at == hb_when_done # 没被改写
# 场景 2:任务超时回收后被 w-2 重新认领,旧 worker w-1 的迟到心跳无效
t2 = svc.create_task(video_url="v2", audio_url="a2")
svc.poll_task("w-1")
svc.db.refresh(t2)
t2.last_heartbeat_at = datetime.now(UTC) - timedelta(days=1)
svc.db.commit()
claimed = svc.poll_task("w-2") # 触发回收并由 w-2 重新认领
assert claimed is not None and claimed.id == t2.id
owner_hb = claimed.last_heartbeat_at
# 把 w-2 的 worker 心跳拨早,确认旧心跳不会影响任务归属
svc.db.query(GpuWorkerModel).filter_by(worker_id="w-2").update(
{"last_heartbeat_at": owner_hb - timedelta(seconds=600)}
)
svc.db.commit()
svc.register_worker("w-1", task_id=t2.id) # 旧 worker 迟到心跳
svc.db.refresh(t2)
assert t2.worker_id == "w-2"
assert t2.status == "processing"
assert t2.last_heartbeat_at == owner_hb
# 场景 3:不存在的 task_id 不报错
svc.register_worker("w-1", task_id="nonexistent-id")
assert svc.db.get(GpuLipsyncTaskModel, "nonexistent-id") is None
def test_default_gpu_task_timeout_is_900(svc):
"""#1970 默认超时 300→900,覆盖 RTX2060 长视频推理."""
assert svc.settings.gpu_task_timeout_seconds == 900
# ── get_by_lipsync_job ─────────────────────────────────────────────