Files
xiaoxia-saas/infra/docker/deploy-staging-registry.sh
T
saas-backend-bot df99305dd6
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 1s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 2s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 29s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 29s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 49s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 1m39s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m44s
CI/CD Pipeline / Validate - Style (pull_request) Successful in 2m19s
AI Code Review / AI Code Review (pull_request) Failing after 2m52s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 2m54s
CI/CD Pipeline / Validate - Security (pull_request) Successful in 4m11s
CI/CD Pipeline / Unit Tests (pull_request) Failing after 6m23s
CI/CD Pipeline / CI Gate (pull_request) Failing after 1s
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 3m39s
CI/CD Pipeline / Canary Release to Production (pull_request) Failing after 260h6m22s
CI/CD Pipeline / Build Production Worker Image (pull_request) Failing after 260h6m25s
CI/CD Pipeline / Build Production Web Image (pull_request) Failing after 260h6m25s
CI/CD Pipeline / ACR Image Cleanup (pull_request) Failing after 260h12m32s
CI/CD Pipeline / Staging API Integration Tests (pull_request) Failing after 260h12m33s
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Failing after 260h12m43s
CI/CD Pipeline / Staging E2E Tests (pull_request) Failing after 260h12m33s
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Failing after 260h12m43s
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Failing after 260h12m43s
CI/CD Pipeline / Build Staging Web Image (pull_request) Failing after 260h12m48s
CI/CD Pipeline / Build Staging API Image (pull_request) Failing after 260h12m49s
CI/CD Pipeline / Frontend Lint (pull_request) Failing after 260h12m51s
CI/CD Pipeline / PR Build Web Image (pull_request) Failing after 260h12m49s
CI/CD Pipeline / Check push changed paths (pull_request) Failing after 260h12m57s
CI/CD Pipeline / Deploy Production (pull_request) Failing after 260h40m56s
CI/CD Pipeline / Build Production API Image (pull_request) Failing after 260h40m59s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Failing after 260h47m12s
CI/CD Pipeline / Build Staging Worker Image (pull_request) Failing after 260h47m22s
CI/CD Pipeline / Frontend Unit Tests (pull_request) Failing after 260h47m24s
feat(worker): celery 队列隔离 + 孤儿任务消息作废 (#1714)
问题:素材转码与视频生成共用 celery 默认队列、worker 单进程消费,
20+ 转码积压会把用户生成任务堵 40 分钟以上;孤儿清理把任务标 failed
后 Redis 队列消息未作废,消息被重投导致 failed→running 非法转换,
worker 打印 ERROR 后继续产出半成品。

队列隔离:
- 新增 packages/shared/celery_queues.py:generation/transcode/celery
  三队列与 task_routes(generate_video→generation;ingest_asset/
  classify_asset/duplication→transcode),apply_queue_settings()
- worker 入口改双进程:generation worker 独占队列并内嵌 beat
  (prefetch=1, GENERATION_CONCURRENCY 默认 2),transcode worker
  消费 transcode,celery(并发=总-2,最小 1),任一退出则整体终止
- compose/部署脚本/ps1 同步新增 GENERATION_CONCURRENCY 与健康检查

消息作废:
- 新增 packages/shared/celery_orphan_guard.py:终态守卫
  ensure_task_claimable、Redis 队列消息物理清理(JSON 信封解析,
  按业务 id + celery headers.id 双匹配,未命中 rpush 保序)、
  revoke_and_purge(control.revoke + 物理清队列双保险)
- 入队点(生成/上传/分片/重试)send_task 后持久化 celery_task_id
  到 generation_tasks/ingest_jobs(新列,067 迁移,失败仅 warning)
- generate_video/ingest_asset 执行前校验 DB 状态:终态直接 discarded
  不进业务逻辑;mark_processing 返回 False(非法转换)安全中止
- 孤儿/超时清理标 failed 时同时 revoke + 清队列消息
- pending 超时阈值 15→45 分钟,与 running 孤儿(20min)区分

测试:新增 22 个单测(路由表/真实 Redis 消息清理/终态守卫/
非法转换中止/标 failed 后消息不重投/入队持久化),全量
14301 passed;067 迁移隔离 DDL 验证 upgrade/downgrade 通过。
2026-09-05 19:07:47 +08:00

180 lines
5.7 KiB
Bash
Executable File
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
#!/bin/sh
set -eu
# ============================================
# Staging 部署脚本 - Registry 方式
# 用法:IMAGE_TAG=<sha|version> REGISTRY_TOKEN=<token> sh deploy-staging.sh
# ============================================
IMAGE_TAG="${IMAGE_TAG:-}"
REGISTRY="${REGISTRY:-git.xiaoxiajianji.com/xiaoxia/xiaoxia-saas}"
REGISTRY_USER="${REGISTRY_USER:-xiaoxia}"
REGISTRY_TOKEN="${REGISTRY_TOKEN:-}"
ENV_FILE="${ENV_FILE:-/var/lib/xiaoxia-saas-staging/.env}"
COMPOSE_DIR="${COMPOSE_DIR:-/var/lib/xiaoxia-saas-staging/repo/infra/docker}"
GENERATED_DIR="${GENERATED_DIR:-/var/lib/xiaoxia-saas-staging/generated}"
if [ -z "$IMAGE_TAG" ]; then
echo "ERROR: IMAGE_TAG is required"
exit 1
fi
test -f "$ENV_FILE"
mkdir -p "$GENERATED_DIR"
# ---- 登录 Registry ----
if [ -n "$REGISTRY_TOKEN" ]; then
echo "Logging in to registry: $REGISTRY"
printf %s "$REGISTRY_TOKEN" | docker login "$(echo $REGISTRY | cut -d/ -f1)" -u "$REGISTRY_USER" --password-stdin 2>/dev/null || {
echo "WARN: docker login failed, will try to pull anyway"
}
fi
# ---- Pull 三镜像 ----
REGISTRY_API="${REGISTRY}/xiaoxia-saas-api:${IMAGE_TAG}"
REGISTRY_WORKER="${REGISTRY}/xiaoxia-saas-worker:${IMAGE_TAG}"
REGISTRY_WEB="${REGISTRY}/xiaoxia-saas-web:${IMAGE_TAG}"
LOCAL_API="${REGISTRY}/xiaoxia-saas-api:staging"
LOCAL_WORKER="${REGISTRY}/xiaoxia-saas-worker:staging"
LOCAL_WEB="${REGISTRY}/xiaoxia-saas-web:staging"
echo "Pulling API image..."
docker pull "$REGISTRY_API"
echo "Pulling Worker image..."
docker pull "$REGISTRY_WORKER"
echo "Pulling Web image..."
docker pull "$REGISTRY_WEB"
# ---- Re-tag 成本地名 ----
docker tag "$REGISTRY_API" "$LOCAL_API"
docker tag "$REGISTRY_WORKER" "$LOCAL_WORKER"
docker tag "$REGISTRY_WEB" "$LOCAL_WEB"
echo "All images pulled and tagged."
# ---- 确保基础设施容器在运行 ----
for c in xiaoxia-postgres-staging xiaoxia-redis-staging; do
if ! docker inspect "$c" >/dev/null 2>&1; then
echo "ERROR: Required container not found: $c"
exit 1
fi
state=$(docker inspect -f {{.State.Status}} "$c")
if [ "$state" != "running" ]; then
echo "ERROR: Container not running: $c ($state)"
exit 1
fi
done
# ---- 确保 staging 网络存在 ----
docker network create xiaoxia-net-staging 2>/dev/null || true
# ---- 执行数据库 Migration ----
echo "Running database migrations..."
docker run --rm --env-file "$ENV_FILE" --network xiaoxia-net-staging "$LOCAL_API" sh -c "cd /app && alembic upgrade head"
echo "Migrations completed."
# ---- 停止旧容器 ----
docker rm -f xiaoxia-api-staging 2>/dev/null || true
docker rm -f xiaoxia-worker-staging 2>/dev/null || true
docker rm -f xiaoxia-web-staging 2>/dev/null || true
# ---- 启动 API ----
echo "Starting API container..."
docker run -d \
--name xiaoxia-api-staging \
--env-file "$ENV_FILE" \
--network xiaoxia-net-staging \
-p 127.0.0.1:8000:8000 \
-e APP_ENV=staging \
-e APP_VERSION="$IMAGE_TAG" \
-e GENERATED_FILES_DIR=/app/generated \
-e GENERATED_FILES_URL_PREFIX=/generated-files \
-v "$GENERATED_DIR:/app/generated" \
--restart unless-stopped \
--label com.centurylinklabs.watchtower.enable=true \
--health-cmd "python -c \"import urllib.request; urllib.request.urlopen('http://localhost:8000/health', timeout=5)\"" \
--health-interval 30s \
--health-timeout 10s \
--health-retries 3 \
--health-start-period 40s \
"$LOCAL_API"
# ---- 启动 Worker ----
echo "Starting Worker container..."
docker run -d \
--name xiaoxia-worker-staging \
--env-file "$ENV_FILE" \
--network xiaoxia-net-staging \
-e APP_ENV=staging \
-e APP_VERSION="$IMAGE_TAG" \
-e WORKER_CONCURRENCY="${WORKER_CONCURRENCY:-4}" \
-e GENERATION_CONCURRENCY="${GENERATION_CONCURRENCY:-2}" \
-e WORKER_MAX_TASKS_PER_CHILD=100 \
-e GENERATED_FILES_DIR=/app/generated \
-e GENERATED_FILES_URL_PREFIX=/generated-files \
-v "$GENERATED_DIR:/app/generated" \
--restart unless-stopped \
--label com.centurylinklabs.watchtower.enable=true \
--health-cmd "sh -c \"pgrep -f 'celery.*worker' >/dev/null 2>&1 || grep -q celery /proc/1/cmdline || exit 1\"" \
--health-interval 30s \
--health-timeout 10s \
--health-retries 3 \
--health-start-period 30s \
"$LOCAL_WORKER"
# ---- 启动 Web ----
# Web 镜像默认打包 production nginx.conf,staging 需要挂载 staging 配置
NGINX_CONF="${NGINX_CONF:-${COMPOSE_DIR}/nginx-staging.conf}"
if [ ! -f "$NGINX_CONF" ]; then
echo "WARN: nginx config not found at $NGINX_CONF, using image default"
NGINX_VOLUME=""
else
NGINX_VOLUME="-v ${NGINX_CONF}:/etc/nginx/conf.d/default.conf:ro"
fi
echo "Starting Web container..."
docker run -d \
--name xiaoxia-web-staging \
--network xiaoxia-net-staging \
-p 127.0.0.1:3001:80 \
--restart unless-stopped \
--label com.centurylinklabs.watchtower.enable=true \
$NGINX_VOLUME \
--health-cmd "wget --spider -q http://127.0.0.1:80" \
--health-interval 30s \
--health-timeout 5s \
--health-retries 3 \
"$LOCAL_WEB"
# ---- 等待 API 健康 ----
echo "Waiting for API to become healthy..."
i=0
while [ "$i" -lt 30 ]; do
if curl -sf --max-time 5 http://127.0.0.1:8000/health >/dev/null 2>&1; then
echo "API is healthy!"
break
fi
i=$((i + 1))
echo " Waiting... ($i/30)"
sleep 2
done
if [ "$i" -ge 30 ]; then
echo "ERROR: API did not become healthy within 60s"
docker logs --tail 30 xiaoxia-api-staging
exit 1
fi
# ---- 清理旧镜像 ----
docker image prune -af --filter "until=72h" 2>/dev/null || true
echo ""
echo "=== Staging deployment complete ==="
echo "API: http://127.0.0.1:8000"
echo "Web: http://127.0.0.1:3001"
echo "Version: $IMAGE_TAG"
docker ps --format "table {{.Names}}\t{{.Status}}" | grep staging
# Watchtower auto-update: 容器加com.centurylinklabs.watchtower.enable=true标签,用:staging tag启动