5ab30f7706
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 3s
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 2s
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 52s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 55s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 2m12s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 3m10s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 5m26s
CI/CD Pipeline / Validate - Style (pull_request) Successful in 6m46s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 7m42s
AI Code Review / AI Code Review (pull_request) Successful in 7m59s
CI/CD Pipeline / Unit Tests (pull_request) Successful in 16m43s
CI/CD Pipeline / Validate - Security (pull_request) Successful in 18m23s
CI/CD Pipeline / Build Production API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / Canary Release to Production (pull_request) Has been skipped
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / CI Gate (pull_request) Successful in 2s
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 11m6s
ACR Cleanup / ACR Image Cleanup (pull_request_target) Successful in 25s
Preview Cleanup / Cleanup Preview Environment (pull_request) Successful in 1m4s
新增 MuseTalk 本地 GPU Worker 反向轮询能力,解决阿里云CPU服务器无法跑GPU推理的问题。
后端API层(新增):
- 表 gpu_lipsync_tasks(id/status/video_url/audio_url/result_url/worker_id/
attempt/error_msg/created_at/started_at/finished_at/last_heartbeat_at)
- 表 gpu_workers(worker_id/hostname/gpu_name/free_vram_mb/capabilities/last_heartbeat_at)
- alembic 迁移 081_add_gpu_lipsync_tasks
- 4个接口(/api/v1/gpu 前缀,长期 GPU_WORKER_TOKEN Bearer 鉴权):
* POST /gpu/register — Worker 注册/心跳
* GET /gpu/lipsync/poll?worker_id=xxx — 拉取 pending 任务(带预签名下载+PUT上传URL),
原子 UPDATE WHERE status=pending 防并发;无任务返回204
* POST /gpu/lipsync/result — multipart 上报结果(支持Worker代传文件到OSS,
或先自PUT到预签名URL再无文件上报)
* GET /gpu/lipsync/status/{task_id} — 任务状态查询
- 超时回退:processing 任务超过 gpu_task_timeout_seconds(默认300s)无心跳
自动回退 pending 重试,最多 MAX_ATTEMPTS(3) 次
- 配置项:GPU_WORKER_TOKEN / GPU_TASK_TIMEOUT_SECONDS(SharedSettings)
- OSS 存储:SharedStorageService 新增 get_upload_url 预签名 PUT URL
- CI/环境:.env/.env.staging/.env.production + render_env.sh + ci-pipeline.yml
均注入 GPU_WORKER_TOKEN
Worker脚本(deploy/gpu_worker/,零项目代码依赖,仅依赖requests):
- gpu_worker.py:启动register→5s轮询→下载视频/音频→POST本地MuseTalk /inference
→multipart回传结果→失败本地重试+上报failed→单任务串行
- requirements.txt:仅 requests>=2.31
- xiaoxia-gpu-worker.service:systemd 开机自启单元
- .env.example:环境变量样例
- README.md:Linux/Windows部署+联调+故障排查
单元测试:10个用例覆盖创建/轮询/并发认领/成功/失败重试/
最大重试失败/超时回退/注册心跳/按lipsync_job查询,全绿。
100 lines
4.2 KiB
Markdown
100 lines
4.2 KiB
Markdown
# MuseTalk GPU Worker — 部署指南
|
||
|
||
本目录包含 RTX2060 本地电脑上运行的 GPU Worker 脚本。
|
||
Worker 采用 **反向轮询模式**:主动向 SaaS API 拉取待处理的口型同步任务 → 调用本地 MuseTalk 推理 → 把结果视频回传到 SaaS。不需要内网穿透。
|
||
|
||
## 目录文件
|
||
|
||
| 文件 | 作用 |
|
||
|---|---|
|
||
| `gpu_worker.py` | Worker 主程序(单文件,零项目代码依赖,仅依赖 `requests`) |
|
||
| `requirements.txt` | Python 依赖(只有 `requests`) |
|
||
| `xiaoxia-gpu-worker.service` | systemd 服务单元(开机自启、异常自动重启) |
|
||
| `.env.example` | 环境变量样例,复制为 `.env` 后填入真实值 |
|
||
|
||
## 一、环境准备
|
||
|
||
1. **Python 3.10+**(Windows 建议从 python.org 安装;Linux 自带)
|
||
2. **本地 MuseTalk 服务** 已启动在 `http://127.0.0.1:7861`,health 接口返回 `{"status":"ok","free_vram_mb":...}`
|
||
3. **ffmpeg**(可选,用于读取输出视频时长;未装则 duration 报 0,不影响功能)
|
||
4. 网络能访问 staging / 生产 API(`curl https://staging-api.xiaoxiajianji.com/health` 应返回 `{"status":"healthy"}`)
|
||
|
||
## 二、部署步骤(Linux,推荐 systemd)
|
||
|
||
```bash
|
||
# 1. 创建部署目录
|
||
sudo mkdir -p /opt/xiaoxia-gpu-worker
|
||
sudo chown $USER:$USER /opt/xiaoxia-gpu-worker
|
||
cd /opt/xiaoxia-gpu-worker
|
||
|
||
# 2. 拷贝脚本和依赖
|
||
cp /path/to/deploy/gpu_worker/{gpu_worker.py,requirements.txt,xiaoxia-gpu-worker.service,.env.example} .
|
||
cp .env.example .env
|
||
# 编辑 .env,填入 API_BASE_URL 和 GPU_WORKER_TOKEN
|
||
|
||
# 3. 创建虚拟环境并安装依赖
|
||
python3 -m venv venv
|
||
./venv/bin/pip install -r requirements.txt
|
||
|
||
# 4. 前台先跑一次,确认日志正常
|
||
./venv/bin/python gpu_worker.py
|
||
# 看到 "MuseTalk 健康检查通过" 和 "注册/心跳" 成功即可 Ctrl+C 退出
|
||
|
||
# 5. 安装 systemd 服务
|
||
sudo cp xiaoxia-gpu-worker.service /etc/systemd/system/
|
||
sudo systemctl daemon-reload
|
||
sudo systemctl enable --now xiaoxia-gpu-worker
|
||
|
||
# 6. 查看日志
|
||
sudo journalctl -u xiaoxia-gpu-worker -f
|
||
```
|
||
|
||
## 三、部署步骤(Windows,快速测试)
|
||
|
||
```bat
|
||
:: 创建虚拟环境
|
||
python -m venv venv
|
||
venv\Scripts\pip install -r requirements.txt
|
||
|
||
:: 复制并编辑 .env
|
||
copy .env.example .env
|
||
notepad .env
|
||
|
||
:: 运行
|
||
venv\Scripts\python gpu_worker.py
|
||
```
|
||
|
||
可在任务计划程序中添加开机启动项:程序选 `venv\Scripts\python.exe`,参数填 `gpu_worker.py`,起始目录填脚本所在目录。
|
||
|
||
## 四、SaaS 侧配套配置
|
||
|
||
SaaS 后端部署完成后需配置:
|
||
|
||
1. 服务端环境变量 `GPU_WORKER_TOKEN` 设为一个随机强 Token(和 Worker `.env` 中一致)
|
||
2. 数据库已跑迁移 `081_add_gpu_lipsync_tasks`(自动随 API 启动的 alembic upgrade head 完成)
|
||
3. OSS bucket 中 `gpu-lipsync/results/` 路径可写(默认 bucket 已配)
|
||
|
||
## 五、验证联调
|
||
|
||
1. Worker 启动后日志看到 `注册/心跳` 成功
|
||
2. 后端调用 `GpuLipsyncService.create_task(video_url=..., audio_url=...)` 放入一条测试任务
|
||
3. Worker 在 5 秒内拉到任务,下载 → 推理 → 上传 → 上报
|
||
4. 后端 `GET /api/v1/gpu/lipsync/status/{task_id}` 返回 `status=done`,`result_url` 非空
|
||
|
||
## 六、故障排查
|
||
|
||
| 现象 | 可能原因 / 排查 |
|
||
|---|---|
|
||
| 日志 401 `Invalid GPU worker token` | `.env` 的 `GPU_WORKER_TOKEN` 与服务端不一致 |
|
||
| 日志 `MuseTalk 健康检查未通过` | 本地 MuseTalk 没启动,或端口不是 7861;`curl http://127.0.0.1:7861/health` 验证 |
|
||
| 任务长时间不被拉取 | Worker 和服务端连不上;检查 API_BASE_URL 是否可达、Token 是否正确 |
|
||
| 推理后上传 OSS 失败 | 本地出口网络被防火墙拦截 OSS 域名(oss-cn-hangzhou.aliyuncs.com) |
|
||
| 服务端看到任务回退到 pending 重试 | Worker 心跳超时(默认 5 分钟);Worker 进程崩溃或推理卡死超过 5 分钟 |
|
||
| 日志 `MuseTalk 推理超时` | 视频太长或显存不足;可临时调大 REQUEST_TIMEOUT,或限制输入视频时长 |
|
||
|
||
## 七、安全注意事项
|
||
|
||
- `.env` 包含长期 Token,文件权限设为 600(`chmod 600 .env`)
|
||
- Token 泄露要立即在服务端更换 `GPU_WORKER_TOKEN` 并重启 Worker
|
||
- Worker 只需要出站访问 SaaS API 和 OSS,不需要开放任何入站端口
|