diff --git a/deploy/gpu_worker/README.md b/deploy/gpu_worker/README.md index bb1f8b198..7ecd0cbf1 100644 --- a/deploy/gpu_worker/README.md +++ b/deploy/gpu_worker/README.md @@ -223,3 +223,131 @@ MuseTalk 健康检查通过: {...} - **循环仅兜底**:仅当 MuseTalk 输出画面短于音频时(极端情况),才 `-stream_loop` + NVENC 兜底补齐 - **业务侧异步化**:POST /lipsync/jobs 创建 GPU 任务后立即返回 `job.status="processing"`,Celery 异步等待结果回写。前端 GET /jobs/{id} 轮询。避免同步阻塞 HTTP 请求 >200s - **删除 `MUSE_ENABLE_VIDEO_LOOP`**:不再需要此开关 + +--- + +## 七、自动部署 + +从 2026-09-20 起,GPU 节点配置文件和脚本全部入库到 `deploy/gpu_worker/`,支持一键初始化新节点 + develop 分支 push 后 30 秒内自动拉取更新。 + +### 7.1 服务架构 + +每个 GPU 渲染节点运行三个 systemd 单元: + +| 单元 | 类型 | 作用 | +|---|---|---| +| `musetalk-worker.service` | simple(常驻) | MuseTalk Flask 推理 API(监听 127.0.0.1:7861) | +| `xiaoxia-gpu-worker.service` | simple(常驻) | 反向轮询 SaaS API 拉口型任务的 Worker 客户端 | +| `gpu-poll.timer` + `gpu-poll.service` | timer(每 30s 触发 oneshot) | 轮询 Gitea `deploy/gpu_worker/` 最新 commit,有变更自动执行 update 脚本 | + +脚本目录(节点本地): + +| 路径 | 来源 | 作用 | +|---|---|---| +| `~/projects/update-gpu-worker.sh` | `scripts/update-gpu-worker.sh` | 备份 → 拉代码 → 重启两个服务 → 健康检查 → 失败回滚 | +| `~/projects/gpu-webhook/poll_and_update.sh` | `scripts/poll_and_update.sh` | 轮询 Gitea API 比对 SHA,有新 commit 时触发 update | + +### 7.2 新节点部署步骤 + +**前置准备**(手动,首次部署必做): + +1. 安装 NVIDIA 驱动 + CUDA 11.8+,`nvidia-smi` 能看到 GPU +2. 克隆 MuseTalk 代码到 `~/projects/MuseTalk/`,下载模型权重到 `~/projects/MuseTalk/models/musetalk/`(权重约几 GB,不适合自动下载) +3. 创建 Python 虚拟环境 `~/projects/MuseTalk/venv/` 并安装 MuseTalk 依赖(PyTorch CUDA 版等) +4. 创建 Worker 虚拟环境 `/opt/xiaoxia-gpu-worker/venv/` 并 `pip install -r requirements.txt` +5. 准备 `.env` 文件(Worker 端):`/opt/xiaoxia-gpu-worker/.env`,填好 `API_BASE_URL`、`GPU_WORKER_TOKEN`、`MUSE_TALK_URL` 等(参考 `.env.example`) + +> ⚠️ 模型权重和 Python 虚拟环境(含 CUDA 版 PyTorch)体积大、安装慢,首次部署必须手动准备;后续脚本只更新 `.py` 文件和配置,不碰权重和 venv。 + +**一键初始化**: + +```bash +# 从仓库拉取 setup 脚本并执行(在全新 GPU 机器上以 ying 用户执行) +wget -q -O /tmp/setup-gpu-node.sh \ + "https://git.xiaoxiajianji.com/xiaoxia/xiaoxia-saas/raw/branch/develop/deploy/gpu_worker/scripts/setup-gpu-node.sh" +bash /tmp/setup-gpu-node.sh +``` + +脚本自动完成: + +1. apt 安装系统依赖(python3、ffmpeg、wget、curl、git) +2. 创建必要目录(`~/projects/MuseTalk`、`~/projects/gpu-webhook`、`/opt/xiaoxia-gpu-worker`) +3. 从仓库拉取三个 systemd 单元文件 + update/poll 脚本到本地 +4. 安装 systemd 服务到 `/etc/systemd/system/` +5. 配置 sudo 免密(仅允许 `ying` 用户免密 restart 两个服务、status、journalctl、cp、chmod、tee) +6. 首次执行 update 脚本拉取最新 `musetalk_server.py` 和 `gpu_worker.py` +7. `systemctl daemon-reload` + enable + start 三个单元 + +**初始化后检查**: + +```bash +sudo systemctl status musetalk-worker # 应 active (running) +sudo systemctl status xiaoxia-gpu-worker # 应 active (running) +sudo systemctl status gpu-poll.timer # 应 active (waiting) +curl http://127.0.0.1:7861/health # 应返回 healthy + GPU 显存信息 +``` + +### 7.3 自动更新机制 + +push 到 `develop` 分支且修改了 `deploy/gpu_worker/` 下任何文件后: + +1. `gpu-poll.timer` 每 30 秒触发 `gpu-poll.service` +2. `poll_and_update.sh` 调用 Gitea API 取 `deploy/gpu_worker/` 路径最新 commit SHA +3. 与本地 `~/projects/gpu-webhook/.last_commit` 比对,无变更直接退出 +4. 有变更:写入新 SHA → 执行 `update-gpu-worker.sh` +5. `update-gpu-worker.sh` 执行流程: + - 备份当前 `musetalk_server.py` / `gpu_worker.py`(带时间戳后缀) + - wget 拉取最新 `musetalk_server.py`、`gpu_worker.py` + - 比对 `requirements.txt`,有变化则 pip install + - `sudo systemctl restart musetalk-worker`,等 5 秒 + - `sudo systemctl restart xiaoxia-gpu-worker`,等 8 秒 + - `curl http://127.0.0.1:7861/health` 健康检查 + - 健康 → 写日志退出 0 + - 不健康 → 回滚到最新备份 → 重启 → 退出 1(日志记录 rolled back) + +端到端延迟:从 push 到节点拉到新代码并重启,约 30~60 秒。 + +### 7.4 手动更新命令 + +```bash +# 立即手动触发一次更新(不依赖 timer) +bash ~/projects/update-gpu-worker.sh + +# 查看更新日志 +tail -f /tmp/gpu-worker-update.log + +# 查看轮询日志 +tail -f /tmp/gpu-poll.log + +# 查看服务运行日志 +journalctl -u musetalk-worker -f # MuseTalk 推理服务日志 +journalctl -u xiaoxia-gpu-worker -f # GPU Worker 客户端日志 +journalctl -u gpu-poll.service -f # 轮询/更新触发日志 +``` + +### 7.5 仓库文件清单(自动部署相关) + +``` +deploy/gpu_worker/ +├── musetalk-worker.service # MuseTalk 推理 API 的 systemd 服务 +├── gpu-poll.service # 自动更新轮询 oneshot service +├── gpu-poll.timer # 每 30 秒触发轮询的 timer +├── xiaoxia-gpu-worker.service # GPU Worker 客户端 systemd 服务(已有) +├── gpu_worker.py # GPU Worker 客户端脚本(已有,自动更新) +├── musetalk_server.py # MuseTalk Flask 服务端(已有,自动更新) +├── requirements.txt # Worker Python 依赖(已有) +├── .env.example # Worker 环境变量模板(已有) +├── README.md # 本文档 +└── scripts/ + ├── update-gpu-worker.sh # 更新脚本:备份→拉取→重启→健康检查→回滚 + ├── poll_and_update.sh # 轮询脚本:SHA 比对→触发更新 + └── setup-gpu-node.sh # 新节点一键初始化脚本 +``` + +### 7.6 注意事项 + +- **首次部署必须手动准备**:MuseTalk 代码仓库、模型权重(`models/musetalk/`,几 GB)、MuseTalk 的 Python 虚拟环境(`venv/`,含 CUDA 版 PyTorch)。这些体积大、安装耗时长,不在自动更新范围内。 +- **脚本路径写死**:当前脚本路径固定为 `/home/ying/projects/` 和 `/opt/xiaoxia-gpu-worker/`,用户名固定 `ying`。后续如有多节点/多用户需求再做参数化。 +- **sudo 免密范围最小化**:setup 脚本写入 `/etc/sudoers.d/ying-gpu-update`,仅放行 restart/status 两个 GPU 相关服务、daemon-reload、journalctl、cp、chmod、tee,不开放全量 root。 +- **回滚只回滚 .py 文件**:健康检查失败只回滚 `musetalk_server.py` 和 `gpu_worker.py`,不回滚 pip 依赖(requirements.txt 变化概率低,且 pip 操作本身可能失败)。如需完全回滚,手动 `pip install -r requirements.txt` 指定旧版本。 +- **poll 脚本容错**:Gitea API 请求失败直接跳过,不触发更新,不会因为网络抖动误重启服务。 diff --git a/deploy/gpu_worker/gpu-poll.service b/deploy/gpu_worker/gpu-poll.service new file mode 100644 index 000000000..5387a045e --- /dev/null +++ b/deploy/gpu_worker/gpu-poll.service @@ -0,0 +1,9 @@ +[Unit] +Description=GPU Worker Auto-Update Poller + +[Service] +Type=oneshot +User=ying +ExecStart=/bin/bash /home/ying/projects/gpu-webhook/poll_and_update.sh +StandardOutput=journal +StandardError=journal diff --git a/deploy/gpu_worker/gpu-poll.timer b/deploy/gpu_worker/gpu-poll.timer new file mode 100644 index 000000000..448e22b72 --- /dev/null +++ b/deploy/gpu_worker/gpu-poll.timer @@ -0,0 +1,10 @@ +[Unit] +Description=Poll Gitea for GPU worker updates every 30 seconds + +[Timer] +OnBootSec=30 +OnUnitActiveSec=30 +AccuracySec=5 + +[Install] +WantedBy=timers.target diff --git a/deploy/gpu_worker/musetalk-worker.service b/deploy/gpu_worker/musetalk-worker.service new file mode 100644 index 000000000..7553cdf38 --- /dev/null +++ b/deploy/gpu_worker/musetalk-worker.service @@ -0,0 +1,19 @@ +[Unit] +Description=MuseTalk Inference API Server +After=network.target nvidia-persistenced.service + +[Service] +Type=simple +User=ying +WorkingDirectory=/home/ying/projects/MuseTalk +Environment=PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:128 +Environment=PATH=/home/ying/projects/MuseTalk/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin +ExecStart=/home/ying/projects/MuseTalk/venv/bin/python /home/ying/projects/MuseTalk/musetalk_server.py +Restart=always +RestartSec=10 +StandardOutput=journal +StandardError=journal +SyslogIdentifier=musetalk-server + +[Install] +WantedBy=multi-user.target diff --git a/deploy/gpu_worker/scripts/poll_and_update.sh b/deploy/gpu_worker/scripts/poll_and_update.sh new file mode 100644 index 000000000..fa7ecb55b --- /dev/null +++ b/deploy/gpu_worker/scripts/poll_and_update.sh @@ -0,0 +1,47 @@ +#!/bin/bash + +REPO_API="https://git.xiaoxiajianji.com/api/v1/repos/xiaoxia/xiaoxia-saas/commits?sha=develop&path=deploy/gpu_worker&limit=1" +STATE_FILE="/home/ying/projects/gpu-webhook/.last_commit" +UPDATE_SCRIPT="/home/ying/projects/update-gpu-worker.sh" +LOG_FILE="/tmp/gpu-poll.log" + +log() { + echo "[$(date +"%Y-%m-%d %H:%M:%S")] $*" >> "$LOG_FILE" +} + +LATEST_SHA=$(curl -sk --max-time 10 "$REPO_API" | python3 -c " +import sys, json +try: + data = json.load(sys.stdin) + if isinstance(data, list) and len(data) > 0: + print(data[0].get('sha', '')) + else: + print('') +except: + print('') +" 2>/dev/null) + +if [ -z "$LATEST_SHA" ]; then + log "get latest commit failed, skip" + exit 0 +fi + +LAST_SHA="" +if [ -f "$STATE_FILE" ]; then + LAST_SHA=$(cat "$STATE_FILE") +fi + +if [ "$LATEST_SHA" = "$LAST_SHA" ]; then + exit 0 +fi + +if [ -z "$LAST_SHA" ]; then + echo "$LATEST_SHA" > "$STATE_FILE" + log "first run, recording SHA: $LATEST_SHA" + exit 0 +fi + +log "new commit detected: $LAST_SHA -> $LATEST_SHA, triggering update" +echo "$LATEST_SHA" > "$STATE_FILE" +bash "$UPDATE_SCRIPT" >> "$LOG_FILE" 2>&1 +log "update completed" diff --git a/deploy/gpu_worker/scripts/setup-gpu-node.sh b/deploy/gpu_worker/scripts/setup-gpu-node.sh new file mode 100644 index 000000000..57a0085c5 --- /dev/null +++ b/deploy/gpu_worker/scripts/setup-gpu-node.sh @@ -0,0 +1,62 @@ +#!/bin/bash +# GPU节点一键初始化脚本 - 在全新GPU机器上执行 + +set -e + +echo "=== 1. 安装系统依赖 ===" +sudo apt-get update -qq +sudo apt-get install -y -qq python3 python3-pip python3-venv ffmpeg wget curl git + +echo "=== 2. 创建目录 ===" +mkdir -p ~/projects/MuseTalk ~/projects/gpu-webhook /opt/xiaoxia-gpu-worker + +echo "=== 3. 安装nvidia-container-toolkit(如需要Docker)===" +# 可选,当前不使用Docker,跳过 +# distribution=$(. /etc/os-release;echo $ID$VERSION_ID) +# curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - +# curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list +# sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit +# sudo nvidia-ctk runtime configure --runtime=docker +# sudo systemctl restart docker + +echo "=== 4. 拉取服务配置和脚本 ===" +REPO_URL="https://git.xiaoxiajianji.com/xiaoxia/xiaoxia-saas/raw/branch/develop/deploy/gpu_worker" +wget -q -O /tmp/musetalk-worker.service "$REPO_URL/musetalk-worker.service" +wget -q -O /tmp/gpu-poll.service "$REPO_URL/gpu-poll.service" +wget -q -O /tmp/gpu-poll.timer "$REPO_URL/gpu-poll.timer" +wget -q -O ~/projects/update-gpu-worker.sh "$REPO_URL/scripts/update-gpu-worker.sh" +wget -q -O ~/projects/gpu-webhook/poll_and_update.sh "$REPO_URL/scripts/poll_and_update.sh" +chmod +x ~/projects/update-gpu-worker.sh ~/projects/gpu-webhook/poll_and_update.sh + +echo "=== 5. 安装systemd服务 ===" +sudo cp /tmp/musetalk-worker.service /etc/systemd/system/ +sudo cp /tmp/gpu-poll.service /etc/systemd/system/ +sudo cp /tmp/gpu-poll.timer /etc/systemd/system/ + +echo "=== 6. 配置sudo免密 ===" +sudo bash -c 'cat > /etc/sudoers.d/ying-gpu-update << EOF +ying ALL=(ALL) NOPASSWD: /bin/systemctl restart musetalk-worker +ying ALL=(ALL) NOPASSWD: /bin/systemctl restart xiaoxia-gpu-worker +ying ALL=(ALL) NOPASSWD: /bin/systemctl status musetalk-worker +ying ALL=(ALL) NOPASSWD: /bin/systemctl status xiaoxia-gpu-worker +ying ALL=(ALL) NOPASSWD: /bin/systemctl daemon-reload +ying ALL=(ALL) NOPASSWD: /usr/bin/journalctl +ying ALL=(ALL) NOPASSWD: /bin/cp +ying ALL=(ALL) NOPASSWD: /bin/chmod +ying ALL=(ALL) NOPASSWD: /usr/bin/tee +EOF' +sudo chmod 440 /etc/sudoers.d/ying-gpu-update + +echo "=== 7. 首次拉取代码并启动服务 ===" +bash ~/projects/update-gpu-worker.sh +sudo systemctl daemon-reload +sudo systemctl enable musetalk-worker xiaoxia-gpu-worker gpu-poll.timer +sudo systemctl start musetalk-worker xiaoxia-gpu-worker gpu-poll.timer + +echo "=== 完成! ===" +echo "检查服务状态:" +echo " sudo systemctl status musetalk-worker" +echo " sudo systemctl status xiaoxia-gpu-worker" +echo " sudo systemctl status gpu-poll.timer" +echo "健康检查:curl http://127.0.0.1:7861/health" +echo "更新日志:tail -f /tmp/gpu-worker-update.log" diff --git a/deploy/gpu_worker/scripts/update-gpu-worker.sh b/deploy/gpu_worker/scripts/update-gpu-worker.sh new file mode 100644 index 000000000..949e00778 --- /dev/null +++ b/deploy/gpu_worker/scripts/update-gpu-worker.sh @@ -0,0 +1,62 @@ +#!/bin/bash +set -e + +REPO_URL="https://git.xiaoxiajianji.com/xiaoxia/xiaoxia-saas/raw/branch/develop/deploy/gpu_worker" +MUSE_DIR="/home/ying/projects/MuseTalk" +WORKER_DIR="/opt/xiaoxia-gpu-worker" +LOG_FILE="/tmp/gpu-worker-update.log" + +log() { + local NOW + NOW=$(date +"%Y-%m-%d %H:%M:%S") + echo "[$NOW] $*" | tee -a "$LOG_FILE" +} + +log "========== start update ==========" + +BAK_SUFFIX=$(date +"%Y%m%d%H%M%S") +cp "$MUSE_DIR/musetalk_server.py" "$MUSE_DIR/musetalk_server.py.bak.$BAK_SUFFIX" +cp "$WORKER_DIR/gpu_worker.py" "$WORKER_DIR/gpu_worker.py.bak.$BAK_SUFFIX" +log "backup done ($BAK_SUFFIX)" + +wget -q -O "$MUSE_DIR/musetalk_server.py" "$REPO_URL/musetalk_server.py" +log "musetalk_server.py updated" + +wget -q -O "$WORKER_DIR/gpu_worker.py" "$REPO_URL/gpu_worker.py" +log "gpu_worker.py updated" + +wget -q -O /tmp/gpu-requirements.txt "$REPO_URL/requirements.txt" +if [ -f "$WORKER_DIR/requirements.txt" ] && ! diff -q "$WORKER_DIR/requirements.txt" /tmp/gpu-requirements.txt > /dev/null 2>&1; then + log "requirements changed, updating..." + cp /tmp/gpu-requirements.txt "$WORKER_DIR/requirements.txt" + "$WORKER_DIR/venv/bin/pip" install -r "$WORKER_DIR/requirements.txt" -q + log "pip install done" +else + log "requirements no change, skip pip" +fi + +sudo systemctl restart musetalk-worker +log "musetalk restarted" +sleep 5 + +sudo systemctl restart xiaoxia-gpu-worker +log "gpu-worker restarted" +sleep 8 + +HEALTH=$(curl -s http://127.0.0.1:7861/health 2>/dev/null) +if echo "$HEALTH" | grep -q "healthy\|ok"; then + log "health check OK" + log "========== update done ==========" + exit 0 +else + log "health check FAILED, rolling back..." + LATEST_MUSE_BAK=$(ls -t "$MUSE_DIR/musetalk_server.py.bak."* 2>/dev/null | head -1) + LATEST_WORKER_BAK=$(ls -t "$WORKER_DIR/gpu_worker.py.bak."* 2>/dev/null | head -1) + [ -n "$LATEST_MUSE_BAK" ] && cp "$LATEST_MUSE_BAK" "$MUSE_DIR/musetalk_server.py" + [ -n "$LATEST_WORKER_BAK" ] && cp "$LATEST_WORKER_BAK" "$WORKER_DIR/gpu_worker.py" + sudo systemctl restart musetalk-worker + sleep 5 + sudo systemctl restart xiaoxia-gpu-worker + log "rolled back" + exit 1 +fi