df99305dd6
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 1s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 2s
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 29s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 29s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 49s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 1m39s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m44s
CI/CD Pipeline / Validate - Style (pull_request) Successful in 2m19s
AI Code Review / AI Code Review (pull_request) Failing after 2m52s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 2m54s
CI/CD Pipeline / Validate - Security (pull_request) Successful in 4m11s
CI/CD Pipeline / Unit Tests (pull_request) Failing after 6m23s
CI/CD Pipeline / Build Production API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI/CD Pipeline / CI Gate (pull_request) Failing after 1s
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / Canary Release to Production (pull_request) Has been skipped
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 3m39s
问题:素材转码与视频生成共用 celery 默认队列、worker 单进程消费, 20+ 转码积压会把用户生成任务堵 40 分钟以上;孤儿清理把任务标 failed 后 Redis 队列消息未作废,消息被重投导致 failed→running 非法转换, worker 打印 ERROR 后继续产出半成品。 队列隔离: - 新增 packages/shared/celery_queues.py:generation/transcode/celery 三队列与 task_routes(generate_video→generation;ingest_asset/ classify_asset/duplication→transcode),apply_queue_settings() - worker 入口改双进程:generation worker 独占队列并内嵌 beat (prefetch=1, GENERATION_CONCURRENCY 默认 2),transcode worker 消费 transcode,celery(并发=总-2,最小 1),任一退出则整体终止 - compose/部署脚本/ps1 同步新增 GENERATION_CONCURRENCY 与健康检查 消息作废: - 新增 packages/shared/celery_orphan_guard.py:终态守卫 ensure_task_claimable、Redis 队列消息物理清理(JSON 信封解析, 按业务 id + celery headers.id 双匹配,未命中 rpush 保序)、 revoke_and_purge(control.revoke + 物理清队列双保险) - 入队点(生成/上传/分片/重试)send_task 后持久化 celery_task_id 到 generation_tasks/ingest_jobs(新列,067 迁移,失败仅 warning) - generate_video/ingest_asset 执行前校验 DB 状态:终态直接 discarded 不进业务逻辑;mark_processing 返回 False(非法转换)安全中止 - 孤儿/超时清理标 failed 时同时 revoke + 清队列消息 - pending 超时阈值 15→45 分钟,与 running 孤儿(20min)区分 测试:新增 22 个单测(路由表/真实 Redis 消息清理/终态守卫/ 非法转换中止/标 failed 后消息不重投/入队持久化),全量 14301 passed;067 迁移隔离 DDL 验证 upgrade/downgrade 通过。
180 lines
5.7 KiB
Bash
Executable File
180 lines
5.7 KiB
Bash
Executable File
#!/bin/sh
|
||
set -eu
|
||
|
||
# ============================================
|
||
# Staging 部署脚本 - Registry 方式
|
||
# 用法:IMAGE_TAG=<sha|version> REGISTRY_TOKEN=<token> sh deploy-staging.sh
|
||
# ============================================
|
||
|
||
IMAGE_TAG="${IMAGE_TAG:-}"
|
||
REGISTRY="${REGISTRY:-git.xiaoxiajianji.com/xiaoxia/xiaoxia-saas}"
|
||
REGISTRY_USER="${REGISTRY_USER:-xiaoxia}"
|
||
REGISTRY_TOKEN="${REGISTRY_TOKEN:-}"
|
||
|
||
ENV_FILE="${ENV_FILE:-/var/lib/xiaoxia-saas-staging/.env}"
|
||
COMPOSE_DIR="${COMPOSE_DIR:-/var/lib/xiaoxia-saas-staging/repo/infra/docker}"
|
||
GENERATED_DIR="${GENERATED_DIR:-/var/lib/xiaoxia-saas-staging/generated}"
|
||
|
||
if [ -z "$IMAGE_TAG" ]; then
|
||
echo "ERROR: IMAGE_TAG is required"
|
||
exit 1
|
||
fi
|
||
|
||
test -f "$ENV_FILE"
|
||
mkdir -p "$GENERATED_DIR"
|
||
|
||
# ---- 登录 Registry ----
|
||
if [ -n "$REGISTRY_TOKEN" ]; then
|
||
echo "Logging in to registry: $REGISTRY"
|
||
printf %s "$REGISTRY_TOKEN" | docker login "$(echo $REGISTRY | cut -d/ -f1)" -u "$REGISTRY_USER" --password-stdin 2>/dev/null || {
|
||
echo "WARN: docker login failed, will try to pull anyway"
|
||
}
|
||
fi
|
||
|
||
# ---- Pull 三镜像 ----
|
||
REGISTRY_API="${REGISTRY}/xiaoxia-saas-api:${IMAGE_TAG}"
|
||
REGISTRY_WORKER="${REGISTRY}/xiaoxia-saas-worker:${IMAGE_TAG}"
|
||
REGISTRY_WEB="${REGISTRY}/xiaoxia-saas-web:${IMAGE_TAG}"
|
||
|
||
LOCAL_API="${REGISTRY}/xiaoxia-saas-api:staging"
|
||
LOCAL_WORKER="${REGISTRY}/xiaoxia-saas-worker:staging"
|
||
LOCAL_WEB="${REGISTRY}/xiaoxia-saas-web:staging"
|
||
|
||
echo "Pulling API image..."
|
||
docker pull "$REGISTRY_API"
|
||
echo "Pulling Worker image..."
|
||
docker pull "$REGISTRY_WORKER"
|
||
echo "Pulling Web image..."
|
||
docker pull "$REGISTRY_WEB"
|
||
|
||
# ---- Re-tag 成本地名 ----
|
||
docker tag "$REGISTRY_API" "$LOCAL_API"
|
||
docker tag "$REGISTRY_WORKER" "$LOCAL_WORKER"
|
||
docker tag "$REGISTRY_WEB" "$LOCAL_WEB"
|
||
echo "All images pulled and tagged."
|
||
|
||
# ---- 确保基础设施容器在运行 ----
|
||
for c in xiaoxia-postgres-staging xiaoxia-redis-staging; do
|
||
if ! docker inspect "$c" >/dev/null 2>&1; then
|
||
echo "ERROR: Required container not found: $c"
|
||
exit 1
|
||
fi
|
||
state=$(docker inspect -f {{.State.Status}} "$c")
|
||
if [ "$state" != "running" ]; then
|
||
echo "ERROR: Container not running: $c ($state)"
|
||
exit 1
|
||
fi
|
||
done
|
||
|
||
# ---- 确保 staging 网络存在 ----
|
||
docker network create xiaoxia-net-staging 2>/dev/null || true
|
||
|
||
# ---- 执行数据库 Migration ----
|
||
echo "Running database migrations..."
|
||
docker run --rm --env-file "$ENV_FILE" --network xiaoxia-net-staging "$LOCAL_API" sh -c "cd /app && alembic upgrade head"
|
||
echo "Migrations completed."
|
||
|
||
# ---- 停止旧容器 ----
|
||
docker rm -f xiaoxia-api-staging 2>/dev/null || true
|
||
docker rm -f xiaoxia-worker-staging 2>/dev/null || true
|
||
docker rm -f xiaoxia-web-staging 2>/dev/null || true
|
||
|
||
# ---- 启动 API ----
|
||
echo "Starting API container..."
|
||
docker run -d \
|
||
--name xiaoxia-api-staging \
|
||
--env-file "$ENV_FILE" \
|
||
--network xiaoxia-net-staging \
|
||
-p 127.0.0.1:8000:8000 \
|
||
-e APP_ENV=staging \
|
||
-e APP_VERSION="$IMAGE_TAG" \
|
||
-e GENERATED_FILES_DIR=/app/generated \
|
||
-e GENERATED_FILES_URL_PREFIX=/generated-files \
|
||
-v "$GENERATED_DIR:/app/generated" \
|
||
--restart unless-stopped \
|
||
--label com.centurylinklabs.watchtower.enable=true \
|
||
--health-cmd "python -c \"import urllib.request; urllib.request.urlopen('http://localhost:8000/health', timeout=5)\"" \
|
||
--health-interval 30s \
|
||
--health-timeout 10s \
|
||
--health-retries 3 \
|
||
--health-start-period 40s \
|
||
"$LOCAL_API"
|
||
|
||
# ---- 启动 Worker ----
|
||
echo "Starting Worker container..."
|
||
docker run -d \
|
||
--name xiaoxia-worker-staging \
|
||
--env-file "$ENV_FILE" \
|
||
--network xiaoxia-net-staging \
|
||
-e APP_ENV=staging \
|
||
-e APP_VERSION="$IMAGE_TAG" \
|
||
-e WORKER_CONCURRENCY="${WORKER_CONCURRENCY:-4}" \
|
||
-e GENERATION_CONCURRENCY="${GENERATION_CONCURRENCY:-2}" \
|
||
-e WORKER_MAX_TASKS_PER_CHILD=100 \
|
||
-e GENERATED_FILES_DIR=/app/generated \
|
||
-e GENERATED_FILES_URL_PREFIX=/generated-files \
|
||
-v "$GENERATED_DIR:/app/generated" \
|
||
--restart unless-stopped \
|
||
--label com.centurylinklabs.watchtower.enable=true \
|
||
--health-cmd "sh -c \"pgrep -f 'celery.*worker' >/dev/null 2>&1 || grep -q celery /proc/1/cmdline || exit 1\"" \
|
||
--health-interval 30s \
|
||
--health-timeout 10s \
|
||
--health-retries 3 \
|
||
--health-start-period 30s \
|
||
"$LOCAL_WORKER"
|
||
|
||
# ---- 启动 Web ----
|
||
# Web 镜像默认打包 production nginx.conf,staging 需要挂载 staging 配置
|
||
NGINX_CONF="${NGINX_CONF:-${COMPOSE_DIR}/nginx-staging.conf}"
|
||
if [ ! -f "$NGINX_CONF" ]; then
|
||
echo "WARN: nginx config not found at $NGINX_CONF, using image default"
|
||
NGINX_VOLUME=""
|
||
else
|
||
NGINX_VOLUME="-v ${NGINX_CONF}:/etc/nginx/conf.d/default.conf:ro"
|
||
fi
|
||
|
||
echo "Starting Web container..."
|
||
docker run -d \
|
||
--name xiaoxia-web-staging \
|
||
--network xiaoxia-net-staging \
|
||
-p 127.0.0.1:3001:80 \
|
||
--restart unless-stopped \
|
||
--label com.centurylinklabs.watchtower.enable=true \
|
||
$NGINX_VOLUME \
|
||
--health-cmd "wget --spider -q http://127.0.0.1:80" \
|
||
--health-interval 30s \
|
||
--health-timeout 5s \
|
||
--health-retries 3 \
|
||
"$LOCAL_WEB"
|
||
|
||
# ---- 等待 API 健康 ----
|
||
echo "Waiting for API to become healthy..."
|
||
i=0
|
||
while [ "$i" -lt 30 ]; do
|
||
if curl -sf --max-time 5 http://127.0.0.1:8000/health >/dev/null 2>&1; then
|
||
echo "API is healthy!"
|
||
break
|
||
fi
|
||
i=$((i + 1))
|
||
echo " Waiting... ($i/30)"
|
||
sleep 2
|
||
done
|
||
|
||
if [ "$i" -ge 30 ]; then
|
||
echo "ERROR: API did not become healthy within 60s"
|
||
docker logs --tail 30 xiaoxia-api-staging
|
||
exit 1
|
||
fi
|
||
|
||
# ---- 清理旧镜像 ----
|
||
docker image prune -af --filter "until=72h" 2>/dev/null || true
|
||
|
||
echo ""
|
||
echo "=== Staging deployment complete ==="
|
||
echo "API: http://127.0.0.1:8000"
|
||
echo "Web: http://127.0.0.1:3001"
|
||
echo "Version: $IMAGE_TAG"
|
||
docker ps --format "table {{.Names}}\t{{.Status}}" | grep staging
|
||
|
||
# Watchtower auto-update: 容器加com.centurylinklabs.watchtower.enable=true标签,用:staging tag启动
|