main
1 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 xiaoxia
|
243b0e7c78 |
ci: CI 瞬态故障自动重试机制
CI/CD Pipeline / Check if frontend-only change (push) Has been skipped
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (push) Successful in 2s
CI/CD Pipeline / PR Build API Image (push) Has been skipped
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / PR Build Web Image (push) Has been skipped
CI/CD Pipeline / PR Build Worker Image (push) Has been skipped
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 4s
PR Automation / Auto Merge on CI Green + Approved (pull_request) Has been skipped
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 3m48s
CI/CD Pipeline / Frontend Lint (push) Has been skipped
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Validate - Code Quality (pull_request) Has been skipped
CI/CD Pipeline / Validate - Type Check (mypy) (pull_request) Has been skipped
CI/CD Pipeline / Validate - Migration (alembic) (pull_request) Has been skipped
CI/CD Pipeline / Check push changed paths (push) Successful in 4m41s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 5m35s
CI/CD Pipeline / Unit Tests (pull_request) Has been skipped
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
CI/CD Pipeline / Validate - Type Check (mypy) (push) Successful in 5m51s
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
CI/CD Pipeline / Validate - Migration (alembic) (push) Successful in 5m27s
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Web Image (push) Has been skipped
AI Code Review / AI Code Review (pull_request) Successful in 8m20s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 2m23s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 2m44s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 9m20s
CI/CD Pipeline / Frontend Unit Tests (push) Successful in 5m37s
CI/CD Pipeline / Build Staging API Image (push) Successful in 2m50s
CI/CD Pipeline / Validate - Code Quality (push) Successful in 10m14s
CI/CD Pipeline / Build Production API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (push) Successful in 3m25s
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (push) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (push) Has been skipped
CI/CD Pipeline / CI Gate (pull_request) Successful in 3m7s
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / Canary Release to Production (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (push) Successful in 2m6s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (push) Successful in 1m22s
CI/CD Pipeline / Unit Tests (push) Successful in 17m19s
CI/CD Pipeline / Build Production API Image (push) Has been skipped
CI/CD Pipeline / Build Production Web Image (push) Has been skipped
CI/CD Pipeline / Build Production Worker Image (push) Has been skipped
CI/CD Pipeline / Deploy Production (push) Has been skipped
CI/CD Pipeline / Production Browser E2E (push) Has been skipped
CI/CD Pipeline / Integration Tests (push) Successful in 6m12s
CI/CD Pipeline / CI Gate (push) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (push) Successful in 2m21s
CI/CD Pipeline / Staging API Integration Tests (push) Successful in 3m56s
CI/CD Pipeline / Canary Release to Production (push) Has been skipped
CI/CD Pipeline / Staging E2E Tests (push) Successful in 4m39s
问题：
当 runner 出现瞬态故障（磁盘满/docker挂/网络抖动），分配到该 runner
的 job 会在 checkout/setup 阶段失败。这种失败与代码无关，但没有自动
重试机制，需要人工干预。
方案：
ci_transient_retry.sh - 每 5 分钟扫描最近 30 分钟的 runs：
- 检测特征：失败 job 的 checkout/setup 阶段失败 + 同 run 其他 job 成功
- 自动 re-run 整个 workflow（Gitea API /rerun）
- 每个 run 最多重试 1 次（状态文件跟踪，24h 自动清理）
- 支持 --dry-run 模式
- 日志记录到 /var/log/ci-auto-retry.log
判定逻辑：
- 失败 job 第一个 step 是 checkout/setup 且 conclusion=failure → 瞬态
- 同一 run 至少 2 个 job 成功（排除代码全挂）
- 纯代码失败（lint/test/compile）→ 不重试
部署：
- systemd timer: ci_transient_retry.{service,timer}
- 或 crontab: */5 * * * *
- Token 通过环境变量 / EnvironmentFile 注入，不硬编码
|