xiaoxia
115dc462d7
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 1s
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 3m28s
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 4m19s
CI/CD Pipeline / Validate - Migration (alembic) (pull_request) Successful in 4m22s
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 4m26s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 1m20s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 1m21s
CI/CD Pipeline / Validate - Type Check (mypy) (pull_request) Successful in 5m11s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 5m56s
CI/CD Pipeline / Validate - Code Quality (pull_request) Successful in 8m16s
AI Code Review / AI Code Review (pull_request) Successful in 9m3s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 6m32s
CI/CD Pipeline / Unit Tests (pull_request) Successful in 15m44s
CI/CD Pipeline / Build Production API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Canary Release to Production (pull_request) Has been cancelled
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / CI Gate (pull_request) Successful in 3m16s
ACR Cleanup / ACR Image Cleanup (pull_request_target) Successful in 4m6s
Preview Cleanup / Cleanup Preview Environment (pull_request) Successful in 4m35s
问题：
当 runner 出现瞬态故障（磁盘满/docker挂/网络抖动），分配到该 runner
的 job 会在 checkout/setup 阶段失败。这种失败与代码无关，但没有自动
重试机制，需要人工干预。
方案：
ci_transient_retry.sh - 每 5 分钟扫描最近 30 分钟的 runs：
- 检测特征：失败 job 的 checkout/setup 阶段失败 + 同 run 其他 job 成功
- 自动 re-run 整个 workflow（Gitea API /rerun）
- 每个 run 最多重试 1 次（状态文件跟踪，24h 自动清理）
- 支持 --dry-run 模式
- 日志记录到 /var/log/ci-auto-retry.log
判定逻辑：
- 失败 job 第一个 step 是 checkout/setup 且 conclusion=failure → 瞬态
- 同一 run 至少 2 个 job 成功（排除代码全挂）
- 纯代码失败（lint/test/compile）→ 不重试
部署：
- systemd timer: ci_transient_retry.{service,timer}
- 或 crontab: */5 * * * *
- Token 通过环境变量 / EnvironmentFile 注入，不硬编码
11 lines
164 B
SYSTEMD
11 lines
164 B
SYSTEMD
[Unit]
|
|
Description=Run CI Transient Fault Auto-Retry every 5 minutes
|
|
|
|
[Timer]
|
|
OnBootSec=2min
|
|
OnUnitActiveSec=5min
|
|
AccuracySec=30s
|
|
|
|
[Install]
|
|
WantedBy=timers.target
|