feat(gpu): #1978 MuseTalk GPU Worker 反向轮询对接 - 后端API+Worker脚本
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 3s
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 2s
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 52s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 55s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 2m12s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 3m10s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 5m26s
CI/CD Pipeline / Validate - Style (pull_request) Successful in 6m46s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 7m42s
AI Code Review / AI Code Review (pull_request) Successful in 7m59s
CI/CD Pipeline / Unit Tests (pull_request) Successful in 16m43s
CI/CD Pipeline / Validate - Security (pull_request) Successful in 18m23s
CI/CD Pipeline / Build Production API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / Canary Release to Production (pull_request) Has been skipped
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / CI Gate (pull_request) Successful in 2s
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 11m6s
ACR Cleanup / ACR Image Cleanup (pull_request_target) Successful in 25s
Preview Cleanup / Cleanup Preview Environment (pull_request) Successful in 1m4s

新增 MuseTalk 本地 GPU Worker 反向轮询能力,解决阿里云CPU服务器无法跑GPU推理的问题。

后端API层(新增):
- 表 gpu_lipsync_tasks(id/status/video_url/audio_url/result_url/worker_id/
  attempt/error_msg/created_at/started_at/finished_at/last_heartbeat_at)
- 表 gpu_workers(worker_id/hostname/gpu_name/free_vram_mb/capabilities/last_heartbeat_at)
- alembic 迁移 081_add_gpu_lipsync_tasks
- 4个接口(/api/v1/gpu 前缀,长期 GPU_WORKER_TOKEN Bearer 鉴权):
  * POST /gpu/register — Worker 注册/心跳
  * GET  /gpu/lipsync/poll?worker_id=xxx — 拉取 pending 任务(带预签名下载+PUT上传URL),
    原子 UPDATE WHERE status=pending 防并发;无任务返回204
  * POST /gpu/lipsync/result — multipart 上报结果(支持Worker代传文件到OSS,
    或先自PUT到预签名URL再无文件上报)
  * GET  /gpu/lipsync/status/{task_id} — 任务状态查询
- 超时回退:processing 任务超过 gpu_task_timeout_seconds(默认300s)无心跳
  自动回退 pending 重试,最多 MAX_ATTEMPTS(3) 次
- 配置项:GPU_WORKER_TOKEN / GPU_TASK_TIMEOUT_SECONDS(SharedSettings)
- OSS 存储:SharedStorageService 新增 get_upload_url 预签名 PUT URL
- CI/环境:.env/.env.staging/.env.production + render_env.sh + ci-pipeline.yml
  均注入 GPU_WORKER_TOKEN

Worker脚本(deploy/gpu_worker/,零项目代码依赖,仅依赖requests):
- gpu_worker.py:启动register→5s轮询→下载视频/音频→POST本地MuseTalk /inference
  →multipart回传结果→失败本地重试+上报failed→单任务串行
- requirements.txt:仅 requests>=2.31
- xiaoxia-gpu-worker.service:systemd 开机自启单元
- .env.example:环境变量样例
- README.md:Linux/Windows部署+联调+故障排查

单元测试:10个用例覆盖创建/轮询/并发认领/成功/失败重试/
最大重试失败/超时回退/注册心跳/按lipsync_job查询,全绿。
This commit is contained in:
xiaoxia
2026-09-18 19:37:37 +08:00
parent 34ffe14aae
commit 5ab30f7706
19 changed files with 1538 additions and 1 deletions
@@ -854,3 +854,61 @@ class DailyUsageRecordModel(Base):
usage_type = Column(String(50), nullable=False, default="free_clip")
count = Column(Integer, nullable=False, default=0)
updated_at = Column(DateTime, nullable=False, default=lambda: datetime.now(UTC))
class GpuLipsyncTaskModel(Base):
"""GPU 口型同步任务 ORM 模型 — MuseTalk 反向轮询模式.
业务侧(AI 数字人生成/lipsync 流程)提交任务后,GPU Worker 主动 poll 拉取、
调用本地 MuseTalk 推理、再通过 result 接口回传结果视频。
"""
__tablename__ = "gpu_lipsync_tasks"
id = Column(String(36), primary_key=True)
# 业务关联(原 lipsync_job_id,方便双向查询)
lipsync_job_id = Column(String(36), nullable=False, default="", index=True)
user_id = Column(String(36), nullable=False, default="", index=True)
project_id = Column(String(36), nullable=False, default="", index=True)
# 输入(预签名下载 URL,由 API 侧生成)
video_url = Column(Text, nullable=False)
audio_url = Column(Text, nullable=False)
# 结果
result_url = Column(Text, nullable=False, default="")
result_duration = Column(Float, nullable=False, default=0.0)
# 任务状态
status = Column(
String(20),
nullable=False,
default="pending",
index=True,
) # pending → processing → done / failed / timeout
worker_id = Column(String(100), nullable=False, default="", index=True)
attempt = Column(Integer, nullable=False, default=0)
error_msg = Column(Text, nullable=False, default="")
# 时间戳
created_at = Column(DateTime, nullable=False, default=lambda: datetime.now(UTC))
started_at = Column(DateTime, nullable=True)
finished_at = Column(DateTime, nullable=True)
updated_at = Column(DateTime, nullable=False, default=lambda: datetime.now(UTC))
# 心跳:worker 最近一次 poll/result 的时间,用于判定 worker 失联
last_heartbeat_at = Column(DateTime, nullable=True)
class GpuWorkerModel(Base):
"""GPU Worker 注册表 — 反向轮询模式下用于心跳与监控."""
__tablename__ = "gpu_workers"
worker_id = Column(String(100), primary_key=True)
hostname = Column(String(200), nullable=False, default="")
gpu_name = Column(String(200), nullable=False, default="")
free_vram_mb = Column(Integer, nullable=False, default=0)
capabilities = Column(String(500), nullable=False, default="") # 逗号分隔,如 "musetalk"
last_heartbeat_at = Column(DateTime, nullable=True, index=True)
created_at = Column(DateTime, nullable=False, default=lambda: datetime.now(UTC))