Files
xiaoxia-saas/infra/docker/deploy-staging-registry.sh
T
xiaoxia 54defa4429
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 1m1s
AI Code Review / AI Code Review (pull_request) Successful in 6m36s
PR Automation / Auto Approve on CI Green (pull_request) Failing after 13m19s
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 1s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 1s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 15s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 16s
ACR Cleanup / ACR Image Cleanup (pull_request_target) Successful in 11s
Preview Cleanup / Cleanup Preview Environment (pull_request) Successful in 30s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m16s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 1m58s
CI/CD Pipeline / Validate - Style (pull_request) Successful in 2m53s
CI/CD Pipeline / Validate - Security (pull_request) Successful in 6m33s
CI/CD Pipeline / Unit Tests (pull_request) Successful in 12m20s
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / CI Gate (pull_request) Successful in 6s
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Failing after 164h14m54s
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Failing after 164h14m54s
CI/CD Pipeline / Frontend Lint (pull_request) Failing after 164h14m55s
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Failing after 164h14m54s
CI/CD Pipeline / Build Staging Worker Image (pull_request) Failing after 164h14m57s
CI/CD Pipeline / Check push changed paths (pull_request) Failing after 164h15m0s
CI/CD Pipeline / Canary Release to Production (pull_request) Failing after 164h2m30s
CI/CD Pipeline / Build Staging Web Image (pull_request) Failing after 164h14m57s
CI/CD Pipeline / Build Production Worker Image (pull_request) Failing after 164h2m35s
CI/CD Pipeline / Staging E2E Tests (pull_request) Failing after 164h14m48s
CI/CD Pipeline / Build Production API Image (pull_request) Failing after 164h2m36s
PR Automation / Auto Merge on CI Green + Approved (pull_request) Failing after 164h27m44s
CI/CD Pipeline / ACR Image Cleanup (pull_request) Failing after 164h14m48s
CI/CD Pipeline / Build Production Web Image (pull_request) Failing after 164h37m21s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Failing after 164h49m35s
CI/CD Pipeline / Frontend Unit Tests (pull_request) Failing after 164h49m41s
CI/CD Pipeline / Build Staging API Image (pull_request) Failing after 164h49m43s
CI/CD Pipeline / Deploy Production (pull_request) Failing after 164h37m16s
CI/CD Pipeline / Staging API Integration Tests (pull_request) Failing after 164h49m34s
CI/CD Pipeline / PR Build Web Image (pull_request) Failing after 164h49m41s
fix: Worker health check 扫描全部 /proc 进程修复 unhealthy 误判
问题:Worker 容器内无 pgrep/ps 工具,旧 health check 使用
'grep -q celery /proc/1/cmdline' 检查 PID 1 的 cmdline,
但 PID 1 是 entrypoint-worker.sh(bash 脚本),不含 celery 关键字,
导致 health check 永远返回 unhealthy。

同时部署脚本中 sh -c 嵌套引号导致 $pid 变量被外层 shell 提前展开为空。

修复:改用 grep -lq celery /proc/[0-9]*/cmdline 扫描所有进程 cmdline,
找到任意 celery 进程即判定 healthy。无需变量、无嵌套 shell、无 pgrep 依赖。

影响文件:
- infra/docker/compose.yml
- infra/docker/deploy-staging-registry.sh
- infra/docker/deploy-production-registry.sh
2026-09-09 18:42:17 +08:00

180 lines
5.7 KiB
Bash
Executable File
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
#!/bin/sh
set -eu
# ============================================
# Staging 部署脚本 - Registry 方式
# 用法:IMAGE_TAG=<sha|version> REGISTRY_TOKEN=<token> sh deploy-staging.sh
# ============================================
IMAGE_TAG="${IMAGE_TAG:-}"
REGISTRY="${REGISTRY:-git.xiaoxiajianji.com/xiaoxia/xiaoxia-saas}"
REGISTRY_USER="${REGISTRY_USER:-xiaoxia}"
REGISTRY_TOKEN="${REGISTRY_TOKEN:-}"
ENV_FILE="${ENV_FILE:-/var/lib/xiaoxia-saas-staging/.env}"
COMPOSE_DIR="${COMPOSE_DIR:-/var/lib/xiaoxia-saas-staging/repo/infra/docker}"
GENERATED_DIR="${GENERATED_DIR:-/var/lib/xiaoxia-saas-staging/generated}"
if [ -z "$IMAGE_TAG" ]; then
echo "ERROR: IMAGE_TAG is required"
exit 1
fi
test -f "$ENV_FILE"
mkdir -p "$GENERATED_DIR"
# ---- 登录 Registry ----
if [ -n "$REGISTRY_TOKEN" ]; then
echo "Logging in to registry: $REGISTRY"
printf %s "$REGISTRY_TOKEN" | docker login "$(echo $REGISTRY | cut -d/ -f1)" -u "$REGISTRY_USER" --password-stdin 2>/dev/null || {
echo "WARN: docker login failed, will try to pull anyway"
}
fi
# ---- Pull 三镜像 ----
REGISTRY_API="${REGISTRY}/xiaoxia-saas-api:${IMAGE_TAG}"
REGISTRY_WORKER="${REGISTRY}/xiaoxia-saas-worker:${IMAGE_TAG}"
REGISTRY_WEB="${REGISTRY}/xiaoxia-saas-web:${IMAGE_TAG}"
LOCAL_API="${REGISTRY}/xiaoxia-saas-api:staging"
LOCAL_WORKER="${REGISTRY}/xiaoxia-saas-worker:staging"
LOCAL_WEB="${REGISTRY}/xiaoxia-saas-web:staging"
echo "Pulling API image..."
docker pull "$REGISTRY_API"
echo "Pulling Worker image..."
docker pull "$REGISTRY_WORKER"
echo "Pulling Web image..."
docker pull "$REGISTRY_WEB"
# ---- Re-tag 成本地名 ----
docker tag "$REGISTRY_API" "$LOCAL_API"
docker tag "$REGISTRY_WORKER" "$LOCAL_WORKER"
docker tag "$REGISTRY_WEB" "$LOCAL_WEB"
echo "All images pulled and tagged."
# ---- 确保基础设施容器在运行 ----
for c in xiaoxia-postgres-staging xiaoxia-redis-staging; do
if ! docker inspect "$c" >/dev/null 2>&1; then
echo "ERROR: Required container not found: $c"
exit 1
fi
state=$(docker inspect -f {{.State.Status}} "$c")
if [ "$state" != "running" ]; then
echo "ERROR: Container not running: $c ($state)"
exit 1
fi
done
# ---- 确保 staging 网络存在 ----
docker network create xiaoxia-net-staging 2>/dev/null || true
# ---- 执行数据库 Migration ----
echo "Running database migrations..."
docker run --rm --env-file "$ENV_FILE" --network xiaoxia-net-staging "$LOCAL_API" sh -c "cd /app && alembic upgrade head"
echo "Migrations completed."
# ---- 停止旧容器 ----
docker rm -f xiaoxia-api-staging 2>/dev/null || true
docker rm -f xiaoxia-worker-staging 2>/dev/null || true
docker rm -f xiaoxia-web-staging 2>/dev/null || true
# ---- 启动 API ----
echo "Starting API container..."
docker run -d \
--name xiaoxia-api-staging \
--env-file "$ENV_FILE" \
--network xiaoxia-net-staging \
-p 127.0.0.1:8000:8000 \
-e APP_ENV=staging \
-e APP_VERSION="$IMAGE_TAG" \
-e GENERATED_FILES_DIR=/app/generated \
-e GENERATED_FILES_URL_PREFIX=/generated-files \
-v "$GENERATED_DIR:/app/generated" \
--restart unless-stopped \
--label com.centurylinklabs.watchtower.enable=true \
--health-cmd "python -c \"import urllib.request; urllib.request.urlopen('http://localhost:8000/health', timeout=5)\"" \
--health-interval 30s \
--health-timeout 10s \
--health-retries 3 \
--health-start-period 40s \
"$LOCAL_API"
# ---- 启动 Worker ----
echo "Starting Worker container..."
docker run -d \
--name xiaoxia-worker-staging \
--env-file "$ENV_FILE" \
--network xiaoxia-net-staging \
-e APP_ENV=staging \
-e APP_VERSION="$IMAGE_TAG" \
-e WORKER_CONCURRENCY="${WORKER_CONCURRENCY:-4}" \
-e GENERATION_CONCURRENCY="${GENERATION_CONCURRENCY:-2}" \
-e WORKER_MAX_TASKS_PER_CHILD=100 \
-e GENERATED_FILES_DIR=/app/generated \
-e GENERATED_FILES_URL_PREFIX=/generated-files \
-v "$GENERATED_DIR:/app/generated" \
--restart unless-stopped \
--label com.centurylinklabs.watchtower.enable=true \
--health-cmd "sh -c \"grep -lq celery /proc/[0-9]*/cmdline 2>/dev/null || exit 1\"" \
--health-interval 30s \
--health-timeout 10s \
--health-retries 3 \
--health-start-period 30s \
"$LOCAL_WORKER"
# ---- 启动 Web ----
# Web 镜像默认打包 production nginx.conf,staging 需要挂载 staging 配置
NGINX_CONF="${NGINX_CONF:-${COMPOSE_DIR}/nginx-staging.conf}"
if [ ! -f "$NGINX_CONF" ]; then
echo "WARN: nginx config not found at $NGINX_CONF, using image default"
NGINX_VOLUME=""
else
NGINX_VOLUME="-v ${NGINX_CONF}:/etc/nginx/conf.d/default.conf:ro"
fi
echo "Starting Web container..."
docker run -d \
--name xiaoxia-web-staging \
--network xiaoxia-net-staging \
-p 127.0.0.1:3001:80 \
--restart unless-stopped \
--label com.centurylinklabs.watchtower.enable=true \
$NGINX_VOLUME \
--health-cmd "wget --spider -q http://127.0.0.1:80" \
--health-interval 30s \
--health-timeout 5s \
--health-retries 3 \
"$LOCAL_WEB"
# ---- 等待 API 健康 ----
echo "Waiting for API to become healthy..."
i=0
while [ "$i" -lt 30 ]; do
if curl -sf --max-time 5 http://127.0.0.1:8000/health >/dev/null 2>&1; then
echo "API is healthy!"
break
fi
i=$((i + 1))
echo " Waiting... ($i/30)"
sleep 2
done
if [ "$i" -ge 30 ]; then
echo "ERROR: API did not become healthy within 60s"
docker logs --tail 30 xiaoxia-api-staging
exit 1
fi
# ---- 清理旧镜像 ----
docker image prune -af --filter "until=72h" 2>/dev/null || true
echo ""
echo "=== Staging deployment complete ==="
echo "API: http://127.0.0.1:8000"
echo "Web: http://127.0.0.1:3001"
echo "Version: $IMAGE_TAG"
docker ps --format "table {{.Names}}\t{{.Status}}" | grep staging
# Watchtower auto-update: 容器加com.centurylinklabs.watchtower.enable=true标签,用:staging tag启动