fix(ci): 集成测试xdist worker数按内存动态计算,防止OOM
CI/CD Pipeline / Validate - Type Check (mypy) (pull_request) Successful in 40s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 46s
CI/CD Pipeline / Validate - Migration (alembic) (pull_request) Successful in 45s
CI/CD Pipeline / PR Build Web Image (pull_request) Successful in 1m6s
CI/CD Pipeline / Frontend Lint (pull_request) Successful in 1m16s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 49s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 2m3s
CI/CD Pipeline / Validate - Code Quality (pull_request) Successful in 3m15s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 3m29s
CI/CD Pipeline / Unit Tests (pull_request) Successful in 3m55s
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 5m8s
CI/CD Pipeline / PR Build API Image (pull_request) Has been cancelled
CI/CD Pipeline / PR Build Worker Image (pull_request) Has been cancelled
AI Code Review / AI Code Review (pull_request) Has been cancelled
CI/CD Pipeline / Frontend Unit Tests (pull_request) Failing after 1338h11m10s
CI/CD Pipeline / ACR Image Cleanup (pull_request) Failing after 1338h11m18s
CI/CD Pipeline / Production Browser E2E (pull_request) Failing after 1338h11m14s
CI/CD Pipeline / Staging API Integration Tests (pull_request) Failing after 1338h11m20s
CI/CD Pipeline / Staging E2E Tests (pull_request) Failing after 1338h11m22s
CI/CD Pipeline / Build Production Web Image (pull_request) Failing after 1338h11m39s
CI/CD Pipeline / Build Production API Image (pull_request) Failing after 1338h11m40s
CI/CD Pipeline / Build Staging Web Image (pull_request) Failing after 1338h12m26s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Failing after 1338h11m41s
CI/CD Pipeline / Build Staging API Image (pull_request) Failing after 1338h12m27s
CI/CD Pipeline / Build Staging Worker Image (pull_request) Failing after 1338h44m43s
CI/CD Pipeline / Deploy Production (pull_request) Failing after 1338h43m33s
CI/CD Pipeline / Build Production Worker Image (pull_request) Failing after 1338h43m55s
CI/CD Pipeline / Validate - Type Check (mypy) (pull_request) Successful in 40s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 46s
CI/CD Pipeline / Validate - Migration (alembic) (pull_request) Successful in 45s
CI/CD Pipeline / PR Build Web Image (pull_request) Successful in 1m6s
CI/CD Pipeline / Frontend Lint (pull_request) Successful in 1m16s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 49s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 2m3s
CI/CD Pipeline / Validate - Code Quality (pull_request) Successful in 3m15s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 3m29s
CI/CD Pipeline / Unit Tests (pull_request) Successful in 3m55s
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 5m8s
CI/CD Pipeline / PR Build API Image (pull_request) Has been cancelled
CI/CD Pipeline / PR Build Worker Image (pull_request) Has been cancelled
AI Code Review / AI Code Review (pull_request) Has been cancelled
CI/CD Pipeline / Frontend Unit Tests (pull_request) Failing after 1338h11m10s
CI/CD Pipeline / ACR Image Cleanup (pull_request) Failing after 1338h11m18s
CI/CD Pipeline / Production Browser E2E (pull_request) Failing after 1338h11m14s
CI/CD Pipeline / Staging API Integration Tests (pull_request) Failing after 1338h11m20s
CI/CD Pipeline / Staging E2E Tests (pull_request) Failing after 1338h11m22s
CI/CD Pipeline / Build Production Web Image (pull_request) Failing after 1338h11m39s
CI/CD Pipeline / Build Production API Image (pull_request) Failing after 1338h11m40s
CI/CD Pipeline / Build Staging Web Image (pull_request) Failing after 1338h12m26s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Failing after 1338h11m41s
CI/CD Pipeline / Build Staging API Image (pull_request) Failing after 1338h12m27s
CI/CD Pipeline / Build Staging Worker Image (pull_request) Failing after 1338h44m43s
CI/CD Pipeline / Deploy Production (pull_request) Failing after 1338h43m33s
CI/CD Pipeline / Build Production Worker Image (pull_request) Failing after 1338h43m55s
之前 -n auto 用满所有CPU核,CI runner内存不足导致worker crash 和ffmpeg子进程被SIGKILL。改为根据可用内存动态计算worker数, 预留1GB给系统和ffmpeg等子进程,限制2-8个worker。 同时 maxfail 从1调到3,容忍偶发不稳定。
This commit is contained in:
@@ -252,14 +252,29 @@ echo ""
|
||||
echo "=== 运行集成测试(pytest-xdist 并行模式) ==="
|
||||
echo "CPU 核数: $(nproc 2>/dev/null || echo 'unknown')"
|
||||
|
||||
# 根据可用内存动态计算 worker 数,防止 OOM
|
||||
# 每个 worker 约占 200-300MB(含 DB 连接 + FastAPI test client)
|
||||
# 预留 1GB 给系统 + ffmpeg 等子进程
|
||||
AVAIL_MEM_MB=$(($(grep MemAvailable /proc/meminfo 2>/dev/null | awk '{print $2}' || echo 2097152) / 1024))
|
||||
RESERVED_MB=1024
|
||||
PER_WORKER_MB=256
|
||||
MAX_WORKERS=$(( (AVAIL_MEM_MB - RESERVED_MB) / PER_WORKER_MB ))
|
||||
# 下限 2,上限 8,CPU 核数也作为上限
|
||||
CPU_CORES=$(nproc 2>/dev/null || echo 4)
|
||||
XDIST_WORKERS=$MAX_WORKERS
|
||||
[ $XDIST_WORKERS -lt 2 ] && XDIST_WORKERS=2
|
||||
[ $XDIST_WORKERS -gt 8 ] && XDIST_WORKERS=8
|
||||
[ $XDIST_WORKERS -gt $CPU_CORES ] && XDIST_WORKERS=$CPU_CORES
|
||||
echo "可用内存: ${AVAIL_MEM_MB}MB, CPU核数: ${CPU_CORES}, xdist workers: ${XDIST_WORKERS}"
|
||||
|
||||
# 集成测试使用 pytest-xdist 并行加速(coverage 由单元测试负责,并行模式下 coverage 不稳定)
|
||||
# -n auto: 自动使用 CPU 核数
|
||||
# -n N: 并行 worker 数
|
||||
# --dist loadfile: 同一测试文件分配到同一 worker(共享 fixture 更高效)
|
||||
# --maxfail=1: 遇到失败停止调度新测试(并行模式下等价于 -x)
|
||||
# --maxfail=3: 容忍少量失败(避免偶发 OOM 导致全挂)
|
||||
PYTHONPATH="$PWD/apps/api:$PWD" python3 -m pytest tests/integration \
|
||||
-q --timeout=60 --maxfail=1 --reruns 2 --reruns-delay 1 \
|
||||
-q --timeout=60 --maxfail=3 --reruns 2 --reruns-delay 1 \
|
||||
-m "not performance" \
|
||||
-n auto --dist loadfile \
|
||||
-n $XDIST_WORKERS --dist loadfile \
|
||||
-p no:cacheprovider
|
||||
|
||||
echo "✅ 集成测试通过"
|
||||
|
||||
Reference in New Issue
Block a user