df99305dd6
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 1s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 2s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 29s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 29s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 49s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 1m39s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m44s
CI/CD Pipeline / Validate - Style (pull_request) Successful in 2m19s
AI Code Review / AI Code Review (pull_request) Failing after 2m52s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 2m54s
CI/CD Pipeline / Validate - Security (pull_request) Successful in 4m11s
CI/CD Pipeline / Unit Tests (pull_request) Failing after 6m23s
CI/CD Pipeline / CI Gate (pull_request) Failing after 1s
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 3m39s
CI/CD Pipeline / Canary Release to Production (pull_request) Failing after 260h6m22s
CI/CD Pipeline / Build Production Worker Image (pull_request) Failing after 260h6m25s
CI/CD Pipeline / Build Production Web Image (pull_request) Failing after 260h6m25s
CI/CD Pipeline / ACR Image Cleanup (pull_request) Failing after 260h12m32s
CI/CD Pipeline / Staging API Integration Tests (pull_request) Failing after 260h12m33s
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Failing after 260h12m43s
CI/CD Pipeline / Staging E2E Tests (pull_request) Failing after 260h12m33s
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Failing after 260h12m43s
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Failing after 260h12m43s
CI/CD Pipeline / Build Staging Web Image (pull_request) Failing after 260h12m48s
CI/CD Pipeline / Build Staging API Image (pull_request) Failing after 260h12m49s
CI/CD Pipeline / Frontend Lint (pull_request) Failing after 260h12m51s
CI/CD Pipeline / PR Build Web Image (pull_request) Failing after 260h12m49s
CI/CD Pipeline / Check push changed paths (pull_request) Failing after 260h12m57s
CI/CD Pipeline / Deploy Production (pull_request) Failing after 260h40m56s
CI/CD Pipeline / Build Production API Image (pull_request) Failing after 260h40m59s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Failing after 260h47m12s
CI/CD Pipeline / Build Staging Worker Image (pull_request) Failing after 260h47m22s
CI/CD Pipeline / Frontend Unit Tests (pull_request) Failing after 260h47m24s
问题:素材转码与视频生成共用 celery 默认队列、worker 单进程消费, 20+ 转码积压会把用户生成任务堵 40 分钟以上;孤儿清理把任务标 failed 后 Redis 队列消息未作废,消息被重投导致 failed→running 非法转换, worker 打印 ERROR 后继续产出半成品。 队列隔离: - 新增 packages/shared/celery_queues.py:generation/transcode/celery 三队列与 task_routes(generate_video→generation;ingest_asset/ classify_asset/duplication→transcode),apply_queue_settings() - worker 入口改双进程:generation worker 独占队列并内嵌 beat (prefetch=1, GENERATION_CONCURRENCY 默认 2),transcode worker 消费 transcode,celery(并发=总-2,最小 1),任一退出则整体终止 - compose/部署脚本/ps1 同步新增 GENERATION_CONCURRENCY 与健康检查 消息作废: - 新增 packages/shared/celery_orphan_guard.py:终态守卫 ensure_task_claimable、Redis 队列消息物理清理(JSON 信封解析, 按业务 id + celery headers.id 双匹配,未命中 rpush 保序)、 revoke_and_purge(control.revoke + 物理清队列双保险) - 入队点(生成/上传/分片/重试)send_task 后持久化 celery_task_id 到 generation_tasks/ingest_jobs(新列,067 迁移,失败仅 warning) - generate_video/ingest_asset 执行前校验 DB 状态:终态直接 discarded 不进业务逻辑;mark_processing 返回 False(非法转换)安全中止 - 孤儿/超时清理标 failed 时同时 revoke + 清队列消息 - pending 超时阈值 15→45 分钟,与 running 孤儿(20min)区分 测试:新增 22 个单测(路由表/真实 Redis 消息清理/终态守卫/ 非法转换中止/标 failed 后消息不重投/入队持久化),全量 14301 passed;067 迁移隔离 DDL 验证 upgrade/downgrade 通过。
242 lines
7.6 KiB
Bash
Executable File
242 lines
7.6 KiB
Bash
Executable File
#!/bin/sh
|
||
set -eu
|
||
|
||
# ============================================
|
||
# Production 部署脚本 - Registry 方式
|
||
# 用法:IMAGE_TAG=<version> REGISTRY_TOKEN=<token> sh deploy-production-registry.sh
|
||
# ============================================
|
||
|
||
IMAGE_TAG="${IMAGE_TAG:-}"
|
||
REGISTRY="${REGISTRY:-git.xiaoxiajianji.com/xiaoxia/xiaoxia-saas}"
|
||
REGISTRY_USER="${REGISTRY_USER:-xiaoxia}"
|
||
REGISTRY_TOKEN="${REGISTRY_TOKEN:-}"
|
||
|
||
ENV_FILE="${ENV_FILE:-/var/lib/xiaoxia-saas-production/.env}"
|
||
GENERATED_DIR="${GENERATED_DIR:-/var/lib/xiaoxia-saas-production/generated}"
|
||
LEGACY_ASSETS_DIR="${LEGACY_ASSETS_DIR:-/var/lib/xiaoxia-saas-production/legacy-assets}"
|
||
|
||
if [ -z "$IMAGE_TAG" ]; then
|
||
echo "ERROR: IMAGE_TAG is required"
|
||
exit 1
|
||
fi
|
||
|
||
test -f "$ENV_FILE"
|
||
mkdir -p "$GENERATED_DIR"
|
||
mkdir -p "$LEGACY_ASSETS_DIR"
|
||
|
||
# ---- 登录 Registry ----
|
||
if [ -n "$REGISTRY_TOKEN" ]; then
|
||
echo "Logging in to registry: $REGISTRY"
|
||
REGISTRY_HOST=$(echo "$REGISTRY" | cut -d/ -f1)
|
||
printf %s "$REGISTRY_TOKEN" | docker login "$REGISTRY_HOST" -u "$REGISTRY_USER" --password-stdin 2>/dev/null || {
|
||
echo "WARN: docker login failed, will try to pull anyway"
|
||
}
|
||
fi
|
||
|
||
# ---- Pull 三镜像 ----
|
||
REGISTRY_API="${REGISTRY}/xiaoxia-saas-api:${IMAGE_TAG}"
|
||
REGISTRY_WORKER="${REGISTRY}/xiaoxia-saas-worker:${IMAGE_TAG}"
|
||
REGISTRY_WEB="${REGISTRY}/xiaoxia-saas-web:${IMAGE_TAG}"
|
||
|
||
LOCAL_API="xiaoxia-saas-api:${IMAGE_TAG}"
|
||
LOCAL_WORKER="xiaoxia-saas-worker:${IMAGE_TAG}"
|
||
LOCAL_WEB="xiaoxia-saas-web:${IMAGE_TAG}"
|
||
|
||
echo "Pulling API image..."
|
||
docker pull "$REGISTRY_API"
|
||
echo "Pulling Worker image..."
|
||
docker pull "$REGISTRY_WORKER"
|
||
echo "Pulling Web image..."
|
||
docker pull "$REGISTRY_WEB"
|
||
|
||
# ---- Re-tag 成本地名 ----
|
||
docker tag "$REGISTRY_API" "$LOCAL_API"
|
||
docker tag "$REGISTRY_WORKER" "$LOCAL_WORKER"
|
||
docker tag "$REGISTRY_WEB" "$LOCAL_WEB"
|
||
echo "All images pulled and tagged."
|
||
|
||
# ---- 备份旧版 assets(部署期间缓存用户不 404) ----
|
||
echo "Backing up legacy assets from current web container..."
|
||
if docker inspect xiaoxia-web-production >/dev/null 2>&1; then
|
||
_tmpdir="/tmp/legacy-assets-$$"
|
||
rm -rf "$_tmpdir"
|
||
mkdir -p "$_tmpdir"
|
||
docker cp xiaoxia-web-production:/usr/share/nginx/html/assets/. "$_tmpdir/" 2>/dev/null || true
|
||
# 合并到 LEGACY_ASSETS_DIR(保留所有历史版本的 assets)
|
||
if [ -d "$_tmpdir" ] && [ "$(ls -A "$_tmpdir" 2>/dev/null)" ]; then
|
||
cp -an "$_tmpdir"/. "$LEGACY_ASSETS_DIR"/ 2>/dev/null || true
|
||
echo "Legacy assets backed up: $(ls "$_tmpdir" | wc -l) files"
|
||
fi
|
||
rm -rf "$_tmpdir"
|
||
else
|
||
echo "No existing web container, skipping legacy assets backup"
|
||
fi
|
||
|
||
# 清理超过 7 天的旧 assets 文件(避免无限增长)
|
||
if [ -d "$LEGACY_ASSETS_DIR" ]; then
|
||
find "$LEGACY_ASSETS_DIR" -type f -mtime +7 -delete 2>/dev/null || true
|
||
echo "Legacy assets cleanup done (retain 7 days)"
|
||
fi
|
||
|
||
# ---- 确保基础设施容器在运行 ----
|
||
echo "Checking infrastructure containers..."
|
||
for c in xiaoxia-postgres-production xiaoxia-redis-production; do
|
||
if ! docker inspect "$c" >/dev/null 2>&1; then
|
||
echo "ERROR: Required container not found: $c"
|
||
exit 1
|
||
fi
|
||
state=$(docker inspect -f '{{.State.Status}}' "$c")
|
||
if [ "$state" != "running" ]; then
|
||
echo "ERROR: Container not running: $c ($state)"
|
||
exit 1
|
||
fi
|
||
done
|
||
|
||
# ---- 确保生产网络存在 ----
|
||
docker network create xiaoxia-net-production 2>/dev/null || true
|
||
|
||
# ---- 执行数据库 Migration ----
|
||
echo "Running database migrations..."
|
||
docker run --rm \
|
||
--env-file "$ENV_FILE" \
|
||
--network xiaoxia-net-production \
|
||
-e APP_ENV=production \
|
||
"$LOCAL_API" sh -c "cd /app && alembic upgrade head"
|
||
echo "Migrations completed."
|
||
|
||
# ---- 停止旧容器 ----
|
||
echo "Stopping old containers..."
|
||
docker rm -f xiaoxia-api-production 2>/dev/null || true
|
||
docker rm -f xiaoxia-worker-production 2>/dev/null || true
|
||
docker rm -f xiaoxia-web-production 2>/dev/null || true
|
||
|
||
# ---- 日志配置(所有容器共用) ----
|
||
LOG_OPTS="--log-driver json-file --log-opt max-size=50m --log-opt max-file=3"
|
||
|
||
# ---- 启动 API ----
|
||
echo "Starting API container..."
|
||
docker run -d \
|
||
--name xiaoxia-api-production \
|
||
--env-file "$ENV_FILE" \
|
||
--network xiaoxia-net-production \
|
||
-p 127.0.0.1:8001:8000 \
|
||
-e APP_ENV=production \
|
||
-e APP_VERSION="$IMAGE_TAG" \
|
||
-e GENERATED_FILES_DIR=/app/generated \
|
||
-e GENERATED_FILES_URL_PREFIX=/generated-files \
|
||
-e PUBLIC_API_BASE_URL=https://api.xiaoxiajianji.com \
|
||
-v "$GENERATED_DIR:/app/generated" \
|
||
--restart unless-stopped \
|
||
--cpus 2 \
|
||
--memory 2g \
|
||
--health-cmd "python -c \"import urllib.request; urllib.request.urlopen('http://localhost:8000/health', timeout=5)\"" \
|
||
--health-interval 30s \
|
||
--health-timeout 10s \
|
||
--health-retries 3 \
|
||
--health-start-period 40s \
|
||
$LOG_OPTS \
|
||
"$LOCAL_API"
|
||
|
||
# ---- 启动 Worker ----
|
||
echo "Starting Worker container..."
|
||
docker run -d \
|
||
--name xiaoxia-worker-production \
|
||
--env-file "$ENV_FILE" \
|
||
--network xiaoxia-net-production \
|
||
-e APP_ENV=production \
|
||
-e APP_VERSION="$IMAGE_TAG" \
|
||
-e WORKER_CONCURRENCY="${WORKER_CONCURRENCY:-4}" \
|
||
-e GENERATION_CONCURRENCY="${GENERATION_CONCURRENCY:-2}" \
|
||
-e WORKER_MAX_TASKS_PER_CHILD=100 \
|
||
-e GENERATED_FILES_DIR=/app/generated \
|
||
-e GENERATED_FILES_URL_PREFIX=/generated-files \
|
||
-e PUBLIC_API_BASE_URL=https://api.xiaoxiajianji.com \
|
||
-v "$GENERATED_DIR:/app/generated" \
|
||
--restart unless-stopped \
|
||
--cpus 2 \
|
||
--memory 2g \
|
||
--health-cmd "sh -c \"pgrep -f 'celery.*worker' >/dev/null 2>&1 || grep -q celery /proc/1/cmdline || exit 1\"" \
|
||
--health-interval 30s \
|
||
--health-timeout 10s \
|
||
--health-retries 3 \
|
||
--health-start-period 30s \
|
||
$LOG_OPTS \
|
||
"$LOCAL_WORKER"
|
||
|
||
# ---- 启动 Web ----
|
||
# Legacy assets 挂载到 /usr/share/nginx/html/assets-legacy/assets/
|
||
# nginx 配置中 assets location 有 fallback 逻辑
|
||
LEGACY_VOLUME=""
|
||
if [ -d "$LEGACY_ASSETS_DIR" ] && [ "$(ls -A "$LEGACY_ASSETS_DIR" 2>/dev/null)" ]; then
|
||
LEGACY_VOLUME="-v ${LEGACY_ASSETS_DIR}:/usr/share/nginx/html/assets-legacy/assets:ro"
|
||
echo "Web container: legacy assets mounted (fallback)"
|
||
else
|
||
echo "Web container: no legacy assets to mount"
|
||
fi
|
||
|
||
echo "Starting Web container..."
|
||
docker run -d \
|
||
--name xiaoxia-web-production \
|
||
--network xiaoxia-net-production \
|
||
-p 127.0.0.1:3002:80 \
|
||
--restart unless-stopped \
|
||
--cpus 0.5 \
|
||
--memory 512m \
|
||
$LEGACY_VOLUME \
|
||
--health-cmd "wget --spider -q http://127.0.0.1:80" \
|
||
--health-interval 30s \
|
||
--health-timeout 5s \
|
||
--health-retries 3 \
|
||
$LOG_OPTS \
|
||
"$LOCAL_WEB"
|
||
|
||
# ---- 等待 API 健康 ----
|
||
echo "Waiting for API to become healthy..."
|
||
i=0
|
||
while [ "$i" -lt 40 ]; do
|
||
if curl -sf --max-time 5 http://127.0.0.1:8001/health >/dev/null 2>&1; then
|
||
echo "API is healthy!"
|
||
break
|
||
fi
|
||
i=$((i + 1))
|
||
echo " Waiting... ($i/40)"
|
||
sleep 3
|
||
done
|
||
|
||
if [ "$i" -ge 40 ]; then
|
||
echo "ERROR: API did not become healthy within 120s"
|
||
docker logs --tail 50 xiaoxia-api-production
|
||
exit 1
|
||
fi
|
||
|
||
# ---- 等待 Web 健康 ----
|
||
echo "Waiting for Web to become healthy..."
|
||
i=0
|
||
while [ "$i" -lt 15 ]; do
|
||
if curl -sf --max-time 5 http://127.0.0.1:3002/ >/dev/null 2>&1; then
|
||
echo "Web is healthy!"
|
||
break
|
||
fi
|
||
i=$((i + 1))
|
||
echo " Waiting... ($i/15)"
|
||
sleep 2
|
||
done
|
||
|
||
if [ "$i" -ge 15 ]; then
|
||
echo "ERROR: Web did not become healthy within 30s"
|
||
docker logs --tail 30 xiaoxia-web-production
|
||
exit 1
|
||
fi
|
||
|
||
# ---- 清理旧镜像 ----
|
||
echo "Cleaning up old images..."
|
||
docker image prune -af --filter "until=168h" 2>/dev/null || true
|
||
docker builder prune -af --filter "until=168h" 2>/dev/null || true
|
||
|
||
echo ""
|
||
echo "=== Production deployment complete ==="
|
||
echo "API: http://127.0.0.1:8001"
|
||
echo "Web: http://127.0.0.1:3002"
|
||
echo "Version: $IMAGE_TAG"
|
||
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Image}}" | grep production
|