Files
xiaoxia-saas/deploy/gpu_worker
xiaoxia a8f1069cd2
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 0s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 1s
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 1m23s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 2m23s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 3m3s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 3m0s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 3m1s
CI/CD Pipeline / Validate - Style (pull_request) Successful in 3m5s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 4m19s
AI Code Review / AI Code Review (pull_request) Successful in 6m27s
CI/CD Pipeline / Validate - Security (pull_request) Successful in 6m44s
CI/CD Pipeline / Unit Tests (pull_request) Failing after 8m42s
CI/CD Pipeline / Build Production API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / Canary Release to Production (pull_request) Has been skipped
CI/CD Pipeline / CI Gate (pull_request) Failing after 1s
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 5m54s
fix(gpu): #1970 MuseTalk worker 推理期心跳/超时 900/重试收敛/短视频前置失败
- GPU_TASK_TIMEOUT_SECONDS 默认 300→900(base.py + env 模板),worker
  REQUEST_TIMEOUT 默认同步 300→900,RTX2060 6G 处理 720p 长视频不再超时
- worker 新增 TaskHeartbeat daemon 线程:任务处理期间每 30s POST
  /gpu/register(task_id=...) 续任务心跳,服务端只在任务心跳真正停滞
  超过 900s(崩溃/断网)或 worker 明确上报 failed 时才回退 pending,
  长推理阻塞主循环不再导致误回退
- register schema/service 支持 task_id:_touch_task_heartbeat 只刷新
  属于该 worker 且仍 processing 的任务,已完成/已被回收重派的过期心跳忽略
- worker 本地 TASK_MAX_RETRY 2→1,且仅对瞬时错误(连接失败/超时/5xx)重试;
  4xx、结果过小等确定性失败不本地重试,服务端 MAX_ATTEMPTS=3 不变,
  消除 3×3=9 次推理放大
- <3s 输入视频(MuseTalk division by zero)下载后 ffprobe 前置校验,
  直接上报 failed"视频过短",不调用推理;ffprobe 不可用时不拦截
- 新增 11 个单测(worker 独立脚本按路径加载),全量 15839 passed
2026-09-19 14:29:22 +08:00
..

MuseTalk GPU Worker — 部署指南

本目录包含 RTX2060 本地电脑上运行的 GPU Worker 脚本。 Worker 采用 反向轮询模式:主动向 SaaS API 拉取待处理的口型同步任务 → 调用本地 MuseTalk 推理 → 把结果视频回传到 SaaS。不需要内网穿透。

目录文件

文件 作用
gpu_worker.py Worker 主程序(单文件,零项目代码依赖,仅依赖 requests)
requirements.txt Python 依赖(只有 requests)
xiaoxia-gpu-worker.service systemd 服务单元(开机自启、异常自动重启)
.env.example 环境变量样例,复制为 .env 后填入真实值

一、环境准备

  1. Python 3.10+(Windows 建议从 python.org 安装;Linux 自带)
  2. 本地 MuseTalk 服务 已启动在 http://127.0.0.1:7861,health 接口返回 {"status":"ok","free_vram_mb":...}
  3. ffmpeg(可选,用于读取输出视频时长;未装则 duration 报 0,不影响功能)
  4. 网络能访问 staging / 生产 API(curl https://staging-api.xiaoxiajianji.com/health 应返回 {"status":"healthy"})

二、部署步骤(Linux,推荐 systemd)

# 1. 创建部署目录
sudo mkdir -p /opt/xiaoxia-gpu-worker
sudo chown $USER:$USER /opt/xiaoxia-gpu-worker
cd /opt/xiaoxia-gpu-worker

# 2. 拷贝脚本和依赖
cp /path/to/deploy/gpu_worker/{gpu_worker.py,requirements.txt,xiaoxia-gpu-worker.service,.env.example} .
cp .env.example .env
#    编辑 .env,填入 API_BASE_URL 和 GPU_WORKER_TOKEN

# 3. 创建虚拟环境并安装依赖
python3 -m venv venv
./venv/bin/pip install -r requirements.txt

# 4. 前台先跑一次,确认日志正常
./venv/bin/python gpu_worker.py
#    看到 "MuseTalk 健康检查通过" 和 "注册/心跳" 成功即可 Ctrl+C 退出

# 5. 安装 systemd 服务
sudo cp xiaoxia-gpu-worker.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now xiaoxia-gpu-worker

# 6. 查看日志
sudo journalctl -u xiaoxia-gpu-worker -f

三、部署步骤(Windows,快速测试)

:: 创建虚拟环境
python -m venv venv
venv\Scripts\pip install -r requirements.txt

:: 复制并编辑 .env
copy .env.example .env
notepad .env

:: 运行
venv\Scripts\python gpu_worker.py

可在任务计划程序中添加开机启动项:程序选 venv\Scripts\python.exe,参数填 gpu_worker.py,起始目录填脚本所在目录。

四、SaaS 侧配套配置

SaaS 后端部署完成后需配置:

  1. 服务端环境变量 GPU_WORKER_TOKEN 设为一个随机强 Token(和 Worker .env 中一致)
  2. 数据库已跑迁移 081_add_gpu_lipsync_tasks(自动随 API 启动的 alembic upgrade head 完成)
  3. OSS bucket 中 gpu-lipsync/results/ 路径可写(默认 bucket 已配)

五、验证联调

  1. Worker 启动后日志看到 注册/心跳 成功
  2. 后端调用 GpuLipsyncService.create_task(video_url=..., audio_url=...) 放入一条测试任务
  3. Worker 在 5 秒内拉到任务,下载 → 推理 → 上传 → 上报
  4. 后端 GET /api/v1/gpu/lipsync/status/{task_id} 返回 status=done,result_url 非空

六、故障排查

现象 可能原因 / 排查
日志 401 Invalid GPU worker token .env 的 GPU_WORKER_TOKEN 与服务端不一致
日志 MuseTalk 健康检查未通过 本地 MuseTalk 没启动,或端口不是 7861;curl http://127.0.0.1:7861/health 验证
任务长时间不被拉取 Worker 和服务端连不上;检查 API_BASE_URL 是否可达、Token 是否正确
推理后上传 OSS 失败 本地出口网络被防火墙拦截 OSS 域名(oss-cn-hangzhou.aliyuncs.com)
服务端看到任务回退到 pending 重试 任务心跳真正超时(默认 900s):Worker 进程崩溃/断网,或推理彻底卡死;正常长推理期间心跳线程每 30s 续期,不会回退
日志 MuseTalk 推理超时或连接失败 视频太长或显存不足;可临时调大 REQUEST_TIMEOUT(服务端 GPU_TASK_TIMEOUT_SECONDS 需同步调大),或限制输入视频时长
日志 视频过短(x.xxs < 3s) 输入视频不足 3s,MuseTalk 对短视频会 division by zero,已在本地直接上报失败;可用 MIN_VIDEO_DURATION_SECONDS 调整阈值

七、安全注意事项

  • .env 包含长期 Token,文件权限设为 600(chmod 600 .env)
  • Token 泄露要立即在服务端更换 GPU_WORKER_TOKEN 并重启 Worker
  • Worker 只需要出站访问 SaaS API 和 OSS,不需要开放任何入站端口