Files
xiaoxia-saas/deploy/gpu_worker/README.md
T
xiaoxia 5ab30f7706
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 3s
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 2s
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 52s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 55s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 2m12s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 3m10s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 5m26s
CI/CD Pipeline / Validate - Style (pull_request) Successful in 6m46s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 7m42s
AI Code Review / AI Code Review (pull_request) Successful in 7m59s
CI/CD Pipeline / Unit Tests (pull_request) Successful in 16m43s
CI/CD Pipeline / Validate - Security (pull_request) Successful in 18m23s
CI/CD Pipeline / Build Production API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / Canary Release to Production (pull_request) Has been skipped
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / CI Gate (pull_request) Successful in 2s
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 11m6s
ACR Cleanup / ACR Image Cleanup (pull_request_target) Successful in 25s
Preview Cleanup / Cleanup Preview Environment (pull_request) Successful in 1m4s
feat(gpu): #1978 MuseTalk GPU Worker 反向轮询对接 - 后端API+Worker脚本
新增 MuseTalk 本地 GPU Worker 反向轮询能力,解决阿里云CPU服务器无法跑GPU推理的问题。

后端API层(新增):
- 表 gpu_lipsync_tasks(id/status/video_url/audio_url/result_url/worker_id/
  attempt/error_msg/created_at/started_at/finished_at/last_heartbeat_at)
- 表 gpu_workers(worker_id/hostname/gpu_name/free_vram_mb/capabilities/last_heartbeat_at)
- alembic 迁移 081_add_gpu_lipsync_tasks
- 4个接口(/api/v1/gpu 前缀,长期 GPU_WORKER_TOKEN Bearer 鉴权):
  * POST /gpu/register — Worker 注册/心跳
  * GET  /gpu/lipsync/poll?worker_id=xxx — 拉取 pending 任务(带预签名下载+PUT上传URL),
    原子 UPDATE WHERE status=pending 防并发;无任务返回204
  * POST /gpu/lipsync/result — multipart 上报结果(支持Worker代传文件到OSS,
    或先自PUT到预签名URL再无文件上报)
  * GET  /gpu/lipsync/status/{task_id} — 任务状态查询
- 超时回退:processing 任务超过 gpu_task_timeout_seconds(默认300s)无心跳
  自动回退 pending 重试,最多 MAX_ATTEMPTS(3) 次
- 配置项:GPU_WORKER_TOKEN / GPU_TASK_TIMEOUT_SECONDS(SharedSettings)
- OSS 存储:SharedStorageService 新增 get_upload_url 预签名 PUT URL
- CI/环境:.env/.env.staging/.env.production + render_env.sh + ci-pipeline.yml
  均注入 GPU_WORKER_TOKEN

Worker脚本(deploy/gpu_worker/,零项目代码依赖,仅依赖requests):
- gpu_worker.py:启动register→5s轮询→下载视频/音频→POST本地MuseTalk /inference
  →multipart回传结果→失败本地重试+上报failed→单任务串行
- requirements.txt:仅 requests>=2.31
- xiaoxia-gpu-worker.service:systemd 开机自启单元
- .env.example:环境变量样例
- README.md:Linux/Windows部署+联调+故障排查

单元测试:10个用例覆盖创建/轮询/并发认领/成功/失败重试/
最大重试失败/超时回退/注册心跳/按lipsync_job查询,全绿。
2026-09-18 19:37:37 +08:00

100 lines
4.2 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# MuseTalk GPU Worker — 部署指南
本目录包含 RTX2060 本地电脑上运行的 GPU Worker 脚本。
Worker 采用 **反向轮询模式**:主动向 SaaS API 拉取待处理的口型同步任务 → 调用本地 MuseTalk 推理 → 把结果视频回传到 SaaS。不需要内网穿透。
## 目录文件
| 文件 | 作用 |
|---|---|
| `gpu_worker.py` | Worker 主程序(单文件,零项目代码依赖,仅依赖 `requests`) |
| `requirements.txt` | Python 依赖(只有 `requests`) |
| `xiaoxia-gpu-worker.service` | systemd 服务单元(开机自启、异常自动重启) |
| `.env.example` | 环境变量样例,复制为 `.env` 后填入真实值 |
## 一、环境准备
1. **Python 3.10+**(Windows 建议从 python.org 安装;Linux 自带)
2. **本地 MuseTalk 服务** 已启动在 `http://127.0.0.1:7861`,health 接口返回 `{"status":"ok","free_vram_mb":...}`
3. **ffmpeg**(可选,用于读取输出视频时长;未装则 duration 报 0,不影响功能)
4. 网络能访问 staging / 生产 API(`curl https://staging-api.xiaoxiajianji.com/health` 应返回 `{"status":"healthy"}`)
## 二、部署步骤(Linux,推荐 systemd)
```bash
# 1. 创建部署目录
sudo mkdir -p /opt/xiaoxia-gpu-worker
sudo chown $USER:$USER /opt/xiaoxia-gpu-worker
cd /opt/xiaoxia-gpu-worker
# 2. 拷贝脚本和依赖
cp /path/to/deploy/gpu_worker/{gpu_worker.py,requirements.txt,xiaoxia-gpu-worker.service,.env.example} .
cp .env.example .env
# 编辑 .env,填入 API_BASE_URL 和 GPU_WORKER_TOKEN
# 3. 创建虚拟环境并安装依赖
python3 -m venv venv
./venv/bin/pip install -r requirements.txt
# 4. 前台先跑一次,确认日志正常
./venv/bin/python gpu_worker.py
# 看到 "MuseTalk 健康检查通过" 和 "注册/心跳" 成功即可 Ctrl+C 退出
# 5. 安装 systemd 服务
sudo cp xiaoxia-gpu-worker.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now xiaoxia-gpu-worker
# 6. 查看日志
sudo journalctl -u xiaoxia-gpu-worker -f
```
## 三、部署步骤(Windows,快速测试)
```bat
:: 创建虚拟环境
python -m venv venv
venv\Scripts\pip install -r requirements.txt
:: 复制并编辑 .env
copy .env.example .env
notepad .env
:: 运行
venv\Scripts\python gpu_worker.py
```
可在任务计划程序中添加开机启动项:程序选 `venv\Scripts\python.exe`,参数填 `gpu_worker.py`,起始目录填脚本所在目录。
## 四、SaaS 侧配套配置
SaaS 后端部署完成后需配置:
1. 服务端环境变量 `GPU_WORKER_TOKEN` 设为一个随机强 Token(和 Worker `.env` 中一致)
2. 数据库已跑迁移 `081_add_gpu_lipsync_tasks`(自动随 API 启动的 alembic upgrade head 完成)
3. OSS bucket 中 `gpu-lipsync/results/` 路径可写(默认 bucket 已配)
## 五、验证联调
1. Worker 启动后日志看到 `注册/心跳` 成功
2. 后端调用 `GpuLipsyncService.create_task(video_url=..., audio_url=...)` 放入一条测试任务
3. Worker 在 5 秒内拉到任务,下载 → 推理 → 上传 → 上报
4. 后端 `GET /api/v1/gpu/lipsync/status/{task_id}` 返回 `status=done`,`result_url` 非空
## 六、故障排查
| 现象 | 可能原因 / 排查 |
|---|---|
| 日志 401 `Invalid GPU worker token` | `.env` 的 `GPU_WORKER_TOKEN` 与服务端不一致 |
| 日志 `MuseTalk 健康检查未通过` | 本地 MuseTalk 没启动,或端口不是 7861;`curl http://127.0.0.1:7861/health` 验证 |
| 任务长时间不被拉取 | Worker 和服务端连不上;检查 API_BASE_URL 是否可达、Token 是否正确 |
| 推理后上传 OSS 失败 | 本地出口网络被防火墙拦截 OSS 域名(oss-cn-hangzhou.aliyuncs.com) |
| 服务端看到任务回退到 pending 重试 | Worker 心跳超时(默认 5 分钟);Worker 进程崩溃或推理卡死超过 5 分钟 |
| 日志 `MuseTalk 推理超时` | 视频太长或显存不足;可临时调大 REQUEST_TIMEOUT,或限制输入视频时长 |
## 七、安全注意事项
- `.env` 包含长期 Token,文件权限设为 600(`chmod 600 .env`)
- Token 泄露要立即在服务端更换 `GPU_WORKER_TOKEN` 并重启 Worker
- Worker 只需要出站访问 SaaS API 和 OSS,不需要开放任何入站端口