ci: 瞬态故障自动重试——runner 故障时自动 re-run，无需人工干预 #1572
Merged
xiaoxia
merged 1 commits from 2026-08-31 15:23:41 +08:00
ci/transient-fault-auto-retry into develop
1 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 xiaoxia
|
115dc462d7 |
ci: CI 瞬态故障自动重试机制
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 1s
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 3m28s
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 4m19s
CI/CD Pipeline / Validate - Migration (alembic) (pull_request) Successful in 4m22s
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 4m26s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 1m20s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 1m21s
CI/CD Pipeline / Validate - Type Check (mypy) (pull_request) Successful in 5m11s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 5m56s
CI/CD Pipeline / Validate - Code Quality (pull_request) Successful in 8m16s
AI Code Review / AI Code Review (pull_request) Successful in 9m3s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 6m32s
CI/CD Pipeline / Unit Tests (pull_request) Successful in 15m44s
CI/CD Pipeline / Build Production API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Canary Release to Production (pull_request) Has been cancelled
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / CI Gate (pull_request) Successful in 3m16s
ACR Cleanup / ACR Image Cleanup (pull_request_target) Successful in 4m6s
Preview Cleanup / Cleanup Preview Environment (pull_request) Successful in 4m35s
问题：
当 runner 出现瞬态故障（磁盘满/docker挂/网络抖动），分配到该 runner
的 job 会在 checkout/setup 阶段失败。这种失败与代码无关，但没有自动
重试机制，需要人工干预。
方案：
ci_transient_retry.sh - 每 5 分钟扫描最近 30 分钟的 runs：
- 检测特征：失败 job 的 checkout/setup 阶段失败 + 同 run 其他 job 成功
- 自动 re-run 整个 workflow（Gitea API /rerun）
- 每个 run 最多重试 1 次（状态文件跟踪，24h 自动清理）
- 支持 --dry-run 模式
- 日志记录到 /var/log/ci-auto-retry.log
判定逻辑：
- 失败 job 第一个 step 是 checkout/setup 且 conclusion=failure → 瞬态
- 同一 run 至少 2 个 job 成功（排除代码全挂）
- 纯代码失败（lint/test/compile）→ 不重试
部署：
- systemd timer: ci_transient_retry.{service,timer}
- 或 crontab: */5 * * * *
- Token 通过环境变量 / EnvironmentFile 注入，不硬编码
|