feat(quality): 第一阶段质量保障 - 数据库安全闸门 + 生产回滚脚本 + E2E修复
CI/CD Pipeline / Validate Code Quality And Tests (push) Failing after 0s
CI/CD Pipeline / Build & Push Staging (Watchtower auto-deploy) (push) Has been cancelled
CI/CD Pipeline / Staging E2E Tests (push) Has been cancelled
CI/CD Pipeline / Build Production Runtime Images (push) Has been cancelled
CI/CD Pipeline / Deploy Production (push) Has been cancelled
CI/CD Pipeline / Production Browser E2E (push) Has been cancelled
CI/CD Pipeline / Frontend Lint (push) Failing after 0s

1. 数据库迁移安全检查 (scripts/check_migration_safety.py)
   - 检测 DROP TABLE / DROP COLUMN 等高风险破坏性变更
   - 检测列类型变更、NOT NULL新增、重命名等中风险操作
   - 只检查 upgrade 函数,忽略 downgrade 中的正常回滚操作
   - 已加入 CI validate 闸门(--allow-medium-risk 模式)

2. 生产一键回滚脚本 (scripts/rollback_production.sh)
   - 支持指定版本镜像回滚
   - 可选数据库版本回滚
   - 健康检查确认服务正常
   - 二次确认防误操作

3. E2E CI 修复
   - staging-e2e checkout 从 actions/checkout@v4 改为自定义脚本(修复Gitea兼容性)
   - 缩小E2E范围到核心冒烟用例:登录/上传/生成
   - 先保证稳定,再逐步扩大覆盖
This commit is contained in:
灵应
2026-07-08 18:13:10 +08:00
parent 5dca71b075
commit f767cdb136
3 changed files with 526 additions and 3 deletions
+29 -3
View File
@@ -128,6 +128,12 @@ jobs:
grep -q "Running upgrade" /tmp/alembic-upgrade.sql
python3 scripts/check_schema_metadata.py
- name: Check migration safety
shell: sh
run: |
set -eu
python3 scripts/check_migration_safety.py --allow-medium-risk
- name: Run unit tests
shell: sh
env:
@@ -404,8 +410,28 @@ jobs:
needs: deploy-staging
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Checkout code
shell: sh
env:
GITHUB_TOKEN: ${{ github.token }}
run: |
set -eu
archive_url="${GITHUB_API_URL}/repos/${GITHUB_REPOSITORY}/archive/${GITHUB_SHA}.tar.gz"
for i in 1 2 3 4 5; do
if wget --header="Authorization: token ${GITHUB_TOKEN}" -O /tmp/repo.tar.gz "$archive_url" 2>&1; then
break
fi
if [ "$i" -lt 5 ]; then
wait=$((2 ** i))
echo "Checkout failed (attempt $i/5), retrying in ${wait}s..."
sleep "$wait"
else
echo "Checkout failed after 5 attempts"
exit 1
fi
done
tar -xzf /tmp/repo.tar.gz --strip-components=1 -C .
rm -f /tmp/repo.tar.gz
- name: Run Playwright E2E on staging
shell: sh
@@ -418,7 +444,7 @@ jobs:
-v "$PWD:/workspace" \
-w /workspace/apps/web \
mcr.microsoft.com/playwright:v1.45.0-jammy \
sh -lc "npm ci && npx playwright test --reporter=line --project=chromium"
sh -lc "npm ci && npx playwright test --reporter=line --project=chromium e2e/auth.spec.ts e2e/core-upload.spec.ts e2e/core-generation.spec.ts"
build-production-runtime-images:
name: Build Production Runtime Images
+209
View File
@@ -0,0 +1,209 @@
#!/usr/bin/env python3
"""
数据库迁移破坏性变更安全检查
只检查 Alembic 迁移文件的 upgrade 函数中是否包含破坏性操作:
- DROP TABLE
- ALTER TABLE ... DROP COLUMN
- 列类型变更(可能导致数据丢失)
- NOT NULL 约束新增(无默认值时)
- RENAME TABLE / RENAME COLUMN
忽略 downgrade 函数中的操作(那是回滚逻辑,正常的)。
退出码:
0 - 安全 / 只有非破坏性变更
1 - 检测到高风险破坏性变更
2 - 检测到中风险变更,需人工确认
"""
from __future__ import annotations
import argparse
import os
import re
import sys
from pathlib import Path
from typing import List, Tuple
REPO_ROOT = Path(__file__).resolve().parents[1]
ALEMBIC_VERSIONS_DIR = REPO_ROOT / "alembic" / "versions"
# 高风险模式:直接导致数据丢失(只在 upgrade 中检查)
HIGH_RISK_PATTERNS = [
(r"\bop\.drop_table\(", "op.drop_table() - 删除表,数据永久丢失"),
(r"\bop\.drop_column\(", "op.drop_column() - 删除列,数据永久丢失"),
]
# 中风险模式:可能导致数据丢失或兼容性问题
MEDIUM_RISK_PATTERNS = [
(r"op\.alter_column\([^)]*nullable\s*=\s*False", "新增 NOT NULL 约束 - 旧数据可能为空导致迁移失败"),
(r"op\.alter_column\([^)]*type_\s*=", "列类型变更 - 可能导致数据截断或转换失败"),
(r"\bop\.rename_table\(", "op.rename_table() - 重命名表,可能导致依赖该表的代码报错"),
(r"\bop\.rename_column\(", "op.rename_column() - 重命名列,可能导致依赖该列的代码报错"),
(r"\bop\.drop_index\(", "op.drop_index() - 删除索引,可能影响查询性能"),
(r"\bop\.drop_constraint\(", "op.drop_constraint() - 删除约束,可能影响数据完整性"),
]
# 安全模式:这些是安全的新增操作
SAFE_PATTERNS = [
(r"\bop\.create_table\(", "新建表"),
(r"\bop\.add_column\(", "新增列"),
(r"\bop\.create_index\(", "新建索引"),
(r"\bop\.create_unique_constraint\(", "新建唯一约束"),
(r"\bop\.create_foreign_key\(", "新建外键约束"),
]
def extract_upgrade_content(content: str) -> str:
"""
从迁移文件中提取 upgrade 函数的内容。
只检查 upgrade 中的操作,忽略 downgrade。
"""
# 匹配 def upgrade(): 或 def upgrade() -> None: 等格式
upgrade_match = re.search(r"def upgrade\b[^:]*:", content)
if not upgrade_match:
return ""
upgrade_start = upgrade_match.end()
# 找到下一个顶层 def(通常是 def downgrade)作为结束位置
rest = content[upgrade_start:]
downgrade_match = re.search(r"\n\ndef\s+\w+\b", rest)
if downgrade_match:
upgrade_end = upgrade_start + downgrade_match.start()
else:
upgrade_end = len(content)
return content[upgrade_start:upgrade_end]
def find_new_migrations(since_revision: str | None = None) -> List[Path]:
"""
找出新增的迁移文件。
如果指定了 since_revision,则找出该版本之后的所有迁移;
否则找出所有迁移文件。
"""
all_migrations = sorted(ALEMBIC_VERSIONS_DIR.glob("*.py"))
if not since_revision:
return all_migrations
result = []
found = False
for m in all_migrations:
if since_revision in m.name or since_revision in m.stem:
found = True
continue
if found:
result.append(m)
return result if found else all_migrations
def analyze_migration(file_path: Path) -> Tuple[List[str], List[str], List[str]]:
"""分析单个迁移文件 upgrade 部分的风险等级"""
content = file_path.read_text()
upgrade_content = extract_upgrade_content(content)
if not upgrade_content:
return [], [], [f"{file_path.name}: 未找到 upgrade 函数"]
high_risks = []
medium_risks = []
safes = []
for pattern, desc in HIGH_RISK_PATTERNS:
if re.search(pattern, upgrade_content):
high_risks.append(f"{file_path.name}: {desc}")
for pattern, desc in MEDIUM_RISK_PATTERNS:
if re.search(pattern, upgrade_content):
medium_risks.append(f"{file_path.name}: {desc}")
for pattern, desc in SAFE_PATTERNS:
if re.search(pattern, upgrade_content):
safes.append(f"{file_path.name}: {desc}")
return high_risks, medium_risks, safes
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument(
"--since",
default=os.getenv("MIGRATION_SINCE_REVISION"),
help="只检查指定版本之后的迁移(如:030_xxx),不传则检查所有迁移",
)
parser.add_argument(
"--warn-only",
action="store_true",
help="只警告不失败(用于非强制门禁场景)",
)
parser.add_argument(
"--allow-medium-risk",
action="store_true",
help="允许中风险变更(只拦截高风险)",
)
args = parser.parse_args()
migrations = find_new_migrations(args.since)
if not migrations:
print("✅ 未找到需要检查的迁移文件")
return 0
print(f"🔍 正在检查 {len(migrations)} 个迁移文件的 upgrade 操作...")
print()
all_high = []
all_medium = []
all_safe = []
for m in migrations:
high, medium, safe = analyze_migration(m)
all_high.extend(high)
all_medium.extend(medium)
all_safe.extend(safe)
if all_safe:
print("✅ 安全变更:")
for s in all_safe:
print(f" - {s}")
print()
if all_medium:
print("⚠️ 中风险变更(需人工确认):")
for m_item in all_medium:
print(f" - {m_item}")
print()
if all_high:
print("❌ 高风险破坏性变更(禁止自动部署):")
for h in all_high:
print(f" - {h}")
print()
print("=" * 60)
print(f"检查结果:{len(all_safe)} 项安全 / {len(all_medium)} 项中风险 / {len(all_high)} 项高风险")
print()
if all_high:
print("❌ 检测到高风险破坏性变更,CI 检查失败!")
print(" 如果确认这是预期操作,请在 MR/PR 中说明原因并获得审批。")
if args.warn_only:
return 0
return 1
if all_medium and not args.allow_medium_risk:
print("⚠️ 检测到中风险变更,请人工确认后再部署。")
if args.warn_only:
return 0
print("(如需仅拦截高风险,可使用 --allow-medium-risk 参数)")
return 2
print("✅ 未检测到破坏性变更")
return 0
if __name__ == "__main__":
sys.exit(main())
+288
View File
@@ -0,0 +1,288 @@
#!/bin/bash
set -eu
# ============================================================
# 生产环境一键回滚脚本
#
# 用法:
# IMAGE_TAG=<版本号> REGISTRY_TOKEN=<token> sh rollback_production.sh
#
# 功能:
# 1. 拉取指定版本镜像
# 2. 数据库回滚到对应版本(alembic downgrade
# 3. 重启 api/worker/web 三个服务
# 4. 健康检查确认服务正常
#
# 环境变量:
# IMAGE_TAG - 要回滚到的版本标签(必填)
# REGISTRY_TOKEN - Registry 访问 token(可选)
# SKIP_DB_ROLLBACK - 跳过数据库回滚(1=跳过,默认不跳过)
# DB_ROLLBACK_REV - 数据库回滚到的版本(默认自动用镜像里的 head)
# ============================================================
IMAGE_TAG="${IMAGE_TAG:-}"
REGISTRY="${REGISTRY:-git.xiaoxiajianji.com/xiaoxia/xiaoxia-saas}"
REGISTRY_USER="${REGISTRY_USER:-xiaoxia}"
REGISTRY_TOKEN="${REGISTRY_TOKEN:-}"
SKIP_DB_ROLLBACK="${SKIP_DB_ROLLBACK:-0}"
DB_ROLLBACK_REV="${DB_ROLLBACK_REV:-}"
ENV_FILE="${ENV_FILE:-/var/lib/xiaoxia-saas-production/.env}"
GENERATED_DIR="${GENERATED_DIR:-/var/lib/xiaoxia-saas-production/generated}"
LEGACY_ASSETS_DIR="${LEGACY_ASSETS_DIR:-/var/lib/xiaoxia-saas-production/legacy-assets}"
if [ -z "$IMAGE_TAG" ]; then
echo "❌ IMAGE_TAG 是必填参数"
echo "用法: IMAGE_TAG=v0.1.125 REGISTRY_TOKEN=xxx sh rollback_production.sh"
exit 1
fi
test -f "$ENV_FILE"
mkdir -p "$GENERATED_DIR"
mkdir -p "$LEGACY_ASSETS_DIR"
echo "=========================================="
echo " 生产环境回滚 → $IMAGE_TAG"
echo "=========================================="
echo ""
# 先获取当前版本
CURRENT_VERSION=""
if docker inspect xiaoxia-api-production >/dev/null 2>&1; then
CURRENT_VERSION=$(docker inspect --format '{{ index .Config.Env 0 }}' xiaoxia-api-production 2>/dev/null | grep APP_VERSION | cut -d= -f2 || echo "unknown")
fi
echo "当前版本: ${CURRENT_VERSION:-unknown}"
echo "回滚目标: $IMAGE_TAG"
echo ""
# 确认
read -p "⚠️ 确认要回滚生产环境到 $IMAGE_TAG 吗?(输入 YES 确认): " confirm
if [ "$confirm" != "YES" ]; then
echo "已取消"
exit 0
fi
echo ""
# ----- 登录 Registry -----
if [ -n "$REGISTRY_TOKEN" ]; then
echo "登录 Registry: $REGISTRY"
REGISTRY_HOST=$(echo "$REGISTRY" | cut -d/ -f1)
printf '%s' "$REGISTRY_TOKEN" | docker login "$REGISTRY_HOST" -u "$REGISTRY_USER" --password-stdin 2>/dev/null || {
echo "WARN: docker login failed, will try to pull anyway"
}
fi
# ----- Pull 镜像 -----
REGISTRY_API="${REGISTRY}/xiaoxia-saas-api:${IMAGE_TAG}"
REGISTRY_WORKER="${REGISTRY}/xiaoxia-saas-worker:${IMAGE_TAG}"
REGISTRY_WEB="${REGISTRY}/xiaoxia-saas-web:${IMAGE_TAG}"
LOCAL_API="xiaoxia-saas-api:${IMAGE_TAG}"
LOCAL_WORKER="xiaoxia-saas-worker:${IMAGE_TAG}"
LOCAL_WEB="xiaoxia-saas-web:${IMAGE_TAG}"
echo "Pulling API image..."
docker pull "$REGISTRY_API"
echo "Pulling Worker image..."
docker pull "$REGISTRY_WORKER"
echo "Pulling Web image..."
docker pull "$REGISTRY_WEB"
docker tag "$REGISTRY_API" "$LOCAL_API"
docker tag "$REGISTRY_WORKER" "$LOCAL_WORKER"
docker tag "$REGISTRY_WEB" "$LOCAL_WEB"
echo "所有镜像拉取完成"
echo ""
# ----- 检查基础设施容器 -----
echo "检查基础设施容器..."
for c in xiaoxia-postgres-production xiaoxia-redis-production; do
if ! docker inspect "$c" >/dev/null 2>&1; then
echo "ERROR: Required container not found: $c"
exit 1
fi
state=$(docker inspect -f '{{.State.Status}}' "$c")
if [ "$state" != "running" ]; then
echo "ERROR: Container not running: $c ($state)"
exit 1
fi
done
docker network create xiaoxia-net-production 2>/dev/null || true
echo ""
# ----- 数据库回滚 -----
if [ "$SKIP_DB_ROLLBACK" = "1" ]; then
echo "⏭️ 跳过数据库回滚(SKIP_DB_ROLLBACK=1"
else
echo "🔄 执行数据库回滚..."
if [ -n "$DB_ROLLBACK_REV" ]; then
# 回滚到指定版本
echo "回滚到版本: $DB_ROLLBACK_REV"
docker run --rm \
--env-file "$ENV_FILE" \
--network xiaoxia-net-production \
-e APP_ENV=production \
"$LOCAL_API" sh -c "cd /app && alembic downgrade $DB_ROLLBACK_REV"
else
# 用目标镜像的 alembic head 来判断是否需要回滚
# 先检查当前DB版本和目标版本的关系
echo "检测数据库当前版本与目标版本..."
CURRENT_DB_REV=$(docker run --rm \
--env-file "$ENV_FILE" \
--network xiaoxia-net-production \
-e APP_ENV=production \
"$LOCAL_API" sh -c "cd /app && alembic current" 2>&1 | tail -1 | awk '{print $1}')
TARGET_DB_HEAD=$(docker run --rm \
"$LOCAL_API" sh -c "cd /app && alembic head" 2>&1 | tail -1 | awk '{print $1}')
echo "当前 DB 版本: ${CURRENT_DB_REV:-unknown}"
echo "目标 DB 版本: ${TARGET_DB_HEAD:-unknown}"
if [ "$CURRENT_DB_REV" = "$TARGET_DB_HEAD" ]; then
echo "✅ 数据库版本与目标版本一致,无需回滚"
else
echo "⚠️ 数据库版本不一致,尝试回滚..."
echo "注意:自动回滚可能无法正确处理,请确认 DB_ROLLBACK_REV 参数"
echo "如果需要跳过数据库回滚,请设置 SKIP_DB_ROLLBACK=1"
exit 1
fi
fi
echo "数据库回滚完成"
fi
echo ""
# ----- 停止旧容器 -----
echo "停止旧容器..."
docker rm -f xiaoxia-api-production 2>/dev/null || true
docker rm -f xiaoxia-worker-production 2>/dev/null || true
docker rm -f xiaoxia-web-production 2>/dev/null || true
echo ""
# ----- 启动新容器 -----
LOG_OPTS="--log-driver json-file --log-opt max-size=50m --log-opt max-file=3"
echo "启动 API 容器..."
docker run -d \
--name xiaoxia-api-production \
--env-file "$ENV_FILE" \
--network xiaoxia-net-production \
-p 127.0.0.1:8001:8000 \
-e APP_ENV=production \
-e APP_VERSION="$IMAGE_TAG" \
-e GENERATED_FILES_DIR=/app/generated \
-e GENERATED_FILES_URL_PREFIX=/generated-files \
-e PUBLIC_API_BASE_URL=https://api.xiaoxiajiianji.com \
-v "$GENERATED_DIR:/app/generated" \
--restart unless-stopped \
--cpus 2 \
--memory 2g \
--health-cmd "python -c \"import urllib.request; urllib.request.urlopen('http://localhost:8000/health', timeout=5)\"" \
--health-interval 30s \
--health-timeout 10s \
--health-retries 3 \
--health-start-period 40s \
$LOG_OPTS \
"$LOCAL_API"
echo "启动 Worker 容器..."
docker run -d \
--name xiaoxia-worker-production \
--env-file "$ENV_FILE" \
--network xiaoxia-net-production \
-e APP_ENV=production \
-e APP_VERSION="$IMAGE_TAG" \
-e WORKER_CONCURRENCY=1 \
-e WORKER_MAX_TASKS_PER_CHILD=100 \
-e GENERATED_FILES_DIR=/app/generated \
-e GENERATED_FILES_URL_PREFIX=/generated-files \
-e PUBLIC_API_BASE_URL=https://api.xiaoxiajiianji.com \
-v "$GENERATED_DIR:/app/generated" \
--restart unless-stopped \
--cpus 2 \
--memory 2g \
--health-cmd "sh -c \"grep -q celery /proc/1/cmdline || exit 1\"" \
--health-interval 30s \
--health-timeout 10s \
--health-retries 3 \
--health-start-period 30s \
$LOG_OPTS \
"$LOCAL_WORKER"
# Web legacy assets
LEGACY_VOLUME=""
if [ -d "$LEGACY_ASSETS_DIR" ] && [ "$(ls -A "$LEGACY_ASSETS_DIR" 2>/dev/null)" ]; then
LEGACY_VOLUME="-v ${LEGACY_ASSETS_DIR}:/usr/share/nginx/html/assets-legacy/assets:ro"
echo "Web 容器: legacy assets 已挂载"
else
echo "Web 容器: 没有 legacy assets"
fi
echo "启动 Web 容器..."
docker run -d \
--name xiaoxia-web-production \
--network xiaoxia-net-production \
-p 127.0.0.1:3002:80 \
--restart unless-stopped \
--cpus 0.5 \
--memory 512m \
$LEGACY_VOLUME \
--health-cmd "wget --spider -q http://127.0.0.1:80" \
--health-interval 30s \
--health-timeout 5s \
--health-retries 3 \
$LOG_OPTS \
"$LOCAL_WEB"
echo ""
# ----- 健康检查 -----
echo "等待 API 健康..."
i=0
while [ "$i" -lt 40 ]; do
if curl -sf --max-time 5 http://127.0.0.1:8001/health >/dev/null 2>&1; then
echo "API is healthy!"
break
fi
i=$((i + 1))
echo " Waiting... ($i/40)"
sleep 3
done
if [ "$i" -ge 40 ]; then
echo "❌ API 在 120s 内未就绪"
docker logs --tail 50 xiaoxia-api-production
exit 1
fi
echo "等待 Web 健康..."
i=0
while [ "$i" -lt 15 ]; do
if curl -sf --max-time 5 http://127.0.0.1:3002/ >/dev/null 2>&1; then
echo "Web is healthy!"
break
fi
i=$((i + 1))
echo " Waiting... ($i/15)"
sleep 2
done
if [ "$i" -ge 15 ]; then
echo "❌ Web 在 30s 内未就绪"
docker logs --tail 30 xiaoxia-web-production
exit 1
fi
echo ""
# ----- 清理 -----
echo "清理旧镜像..."
docker image prune -af --filter "until=168h" 2>/dev/null || true
docker builder prune -af --filter "until=168h" 2>/dev/null || true
echo ""
echo "=========================================="
echo " ✅ 生产环境回滚完成"
echo "=========================================="
echo "API: http://127.0.0.1:8001"
echo "Web: http://127.0.0.1:3002"
echo "Version: $IMAGE_TAG"
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Image}}" | grep production