fix(gpu): #1970 MuseTalk worker 推理期心跳/超时900/重试收敛/短视频前置失败 #1990
Merged
auto-approve-bot
merged 3 commits from 2026-09-19 15:06:53 +08:00
fix/1970-gpu-worker-heartbeat-timeout into develop
3 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 xiaoxia
|
19ace80123 |
style(gpu): black/ruff 整理（顺带移除 worker 未使用的 import json）
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 2s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 2s
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 29s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 30s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 2m14s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 2m24s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 2m48s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 2m55s
CI/CD Pipeline / Validate - Style (pull_request) Successful in 3m0s
AI Code Review / AI Code Review (pull_request) Successful in 6m25s
CI/CD Pipeline / Validate - Security (pull_request) Successful in 7m54s
CI/CD Pipeline / Unit Tests (pull_request) Successful in 8m43s
CI/CD Pipeline / Build Production API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI/CD Pipeline / CI Gate (pull_request) Successful in 1s
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / Canary Release to Production (pull_request) Has been skipped
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 6m32s
Preview Cleanup / Cleanup Preview Environment (pull_request) Successful in 30s
ACR Cleanup / ACR Image Cleanup (pull_request_target) Successful in 3m16s
|
||
|
xiaoxia
|
10294e9eb9 |
fix(gpu): TaskHeartbeat 的 _stop 属性改名 _stop_event，避免覆盖 Thread._stop 方法
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 1s
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 2s
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 38s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 59s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 2m21s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 2m34s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 2m54s
CI/CD Pipeline / Validate - Style (pull_request) Successful in 2m53s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 2m59s
CI/CD Pipeline / Build Production API Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Web Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been cancelled
CI/CD Pipeline / Deploy Production (pull_request) Has been cancelled
CI/CD Pipeline / Production Browser E2E (pull_request) Has been cancelled
CI/CD Pipeline / Canary Release to Production (pull_request) Has been cancelled
CI/CD Pipeline / CI Gate (pull_request) Has been cancelled
CI/CD Pipeline / Validate - Security (pull_request) Has been cancelled
CI/CD Pipeline / Unit Tests (pull_request) Has been cancelled
AI Code Review / AI Code Review (pull_request) Has been cancelled
PR Automation / Auto Merge on CI Green + Approved (pull_request) Has been cancelled
CI 单测报错 'Event' object is not callable：threading.Thread 内部存在 _stop() 方法（join/GC 时调用），子类用 self._stop = Event() 覆盖了它， 线程回收时 TypeError。本地因时序差异未复现。 |
||
|
xiaoxia
|
a8f1069cd2 |
fix(gpu): #1970 MuseTalk worker 推理期心跳/超时 900/重试收敛/短视频前置失败
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 0s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 1s
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 1m23s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 2m23s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 3m3s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 3m0s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 3m1s
CI/CD Pipeline / Validate - Style (pull_request) Successful in 3m5s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 4m19s
AI Code Review / AI Code Review (pull_request) Successful in 6m27s
CI/CD Pipeline / Validate - Security (pull_request) Successful in 6m44s
CI/CD Pipeline / Unit Tests (pull_request) Failing after 8m42s
CI/CD Pipeline / Build Production API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / Canary Release to Production (pull_request) Has been skipped
CI/CD Pipeline / CI Gate (pull_request) Failing after 1s
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 5m54s
- GPU_TASK_TIMEOUT_SECONDS 默认 300→900（base.py + env 模板），worker REQUEST_TIMEOUT 默认同步 300→900，RTX2060 6G 处理 720p 长视频不再超时 - worker 新增 TaskHeartbeat daemon 线程：任务处理期间每 30s POST /gpu/register(task_id=...) 续任务心跳，服务端只在任务心跳真正停滞 超过 900s（崩溃/断网）或 worker 明确上报 failed 时才回退 pending， 长推理阻塞主循环不再导致误回退 - register schema/service 支持 task_id：_touch_task_heartbeat 只刷新 属于该 worker 且仍 processing 的任务，已完成/已被回收重派的过期心跳忽略 - worker 本地 TASK_MAX_RETRY 2→1，且仅对瞬时错误（连接失败/超时/5xx）重试； 4xx、结果过小等确定性失败不本地重试，服务端 MAX_ATTEMPTS=3 不变， 消除 3×3=9 次推理放大 - <3s 输入视频（MuseTalk division by zero）下载后 ffprobe 前置校验， 直接上报 failed"视频过短"，不调用推理；ffprobe 不可用时不拦截 - 新增 11 个单测（worker 独立脚本按路径加载），全量 15839 passed |