fix(ci): 集成测试xdist worker数按内存动态计算,防止OOM
CI/CD Pipeline / Validate - Type Check (mypy) (pull_request) Successful in 40s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 46s
CI/CD Pipeline / Validate - Migration (alembic) (pull_request) Successful in 45s
CI/CD Pipeline / PR Build Web Image (pull_request) Successful in 1m6s
CI/CD Pipeline / Frontend Lint (pull_request) Successful in 1m16s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 49s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 2m3s
CI/CD Pipeline / Validate - Code Quality (pull_request) Successful in 3m15s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 3m29s
CI/CD Pipeline / Unit Tests (pull_request) Successful in 3m55s
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 5m8s
CI/CD Pipeline / PR Build API Image (pull_request) Has been cancelled
CI/CD Pipeline / PR Build Worker Image (pull_request) Has been cancelled
AI Code Review / AI Code Review (pull_request) Has been cancelled
CI/CD Pipeline / Frontend Unit Tests (pull_request) Failing after 1338h11m10s
CI/CD Pipeline / ACR Image Cleanup (pull_request) Failing after 1338h11m18s
CI/CD Pipeline / Production Browser E2E (pull_request) Failing after 1338h11m14s
CI/CD Pipeline / Staging API Integration Tests (pull_request) Failing after 1338h11m20s
CI/CD Pipeline / Staging E2E Tests (pull_request) Failing after 1338h11m22s
CI/CD Pipeline / Build Production Web Image (pull_request) Failing after 1338h11m39s
CI/CD Pipeline / Build Production API Image (pull_request) Failing after 1338h11m40s
CI/CD Pipeline / Build Staging Web Image (pull_request) Failing after 1338h12m26s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Failing after 1338h11m41s
CI/CD Pipeline / Build Staging API Image (pull_request) Failing after 1338h12m27s
CI/CD Pipeline / Build Staging Worker Image (pull_request) Failing after 1338h44m43s
CI/CD Pipeline / Deploy Production (pull_request) Failing after 1338h43m33s
CI/CD Pipeline / Build Production Worker Image (pull_request) Failing after 1338h43m55s

之前 -n auto 用满所有CPU核,CI runner内存不足导致worker crash
和ffmpeg子进程被SIGKILL。改为根据可用内存动态计算worker数,
预留1GB给系统和ffmpeg等子进程,限制2-8个worker。
同时 maxfail 从1调到3,容忍偶发不稳定。
This commit is contained in:
CI Bot
2026-07-22 21:16:16 +08:00
parent a75f749b39
commit 8ea3263a12
+19 -4
View File
@@ -252,14 +252,29 @@ echo ""
echo "=== 运行集成测试(pytest-xdist 并行模式) ==="
echo "CPU 核数: $(nproc 2>/dev/null || echo 'unknown')"
# 根据可用内存动态计算 worker 数,防止 OOM
# 每个 worker 约占 200-300MB(含 DB 连接 + FastAPI test client)
# 预留 1GB 给系统 + ffmpeg 等子进程
AVAIL_MEM_MB=$(($(grep MemAvailable /proc/meminfo 2>/dev/null | awk '{print $2}' || echo 2097152) / 1024))
RESERVED_MB=1024
PER_WORKER_MB=256
MAX_WORKERS=$(( (AVAIL_MEM_MB - RESERVED_MB) / PER_WORKER_MB ))
# 下限 2,上限 8,CPU 核数也作为上限
CPU_CORES=$(nproc 2>/dev/null || echo 4)
XDIST_WORKERS=$MAX_WORKERS
[ $XDIST_WORKERS -lt 2 ] && XDIST_WORKERS=2
[ $XDIST_WORKERS -gt 8 ] && XDIST_WORKERS=8
[ $XDIST_WORKERS -gt $CPU_CORES ] && XDIST_WORKERS=$CPU_CORES
echo "可用内存: ${AVAIL_MEM_MB}MB, CPU核数: ${CPU_CORES}, xdist workers: ${XDIST_WORKERS}"
# 集成测试使用 pytest-xdist 并行加速(coverage 由单元测试负责,并行模式下 coverage 不稳定)
# -n auto: 自动使用 CPU 核数
# -n N: 并行 worker 数
# --dist loadfile: 同一测试文件分配到同一 worker(共享 fixture 更高效)
# --maxfail=1: 遇到失败停止调度新测试(并行模式下等价于 -x)
# --maxfail=3: 容忍少量失败(避免偶发 OOM 导致全挂)
PYTHONPATH="$PWD/apps/api:$PWD" python3 -m pytest tests/integration \
-q --timeout=60 --maxfail=1 --reruns 2 --reruns-delay 1 \
-q --timeout=60 --maxfail=3 --reruns 2 --reruns-delay 1 \
-m "not performance" \
-n auto --dist loadfile \
-n $XDIST_WORKERS --dist loadfile \
-p no:cacheprovider
echo "✅ 集成测试通过"