feat(ci): P2-3 Runner监控告警 - 离线/磁盘/内存/队列积压主动告警 #537
Merged
auto-approve-bot
merged 2 commits from 2026-07-18 20:55:43 +08:00
ci/runner-monitor-alerting into develop
2 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 CI Bot
|
34abf855b0 |
style: isort修复import排序
CI Build & Deploy Pipeline / Build Staging API Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Production API Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Production Web Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI Build & Deploy Pipeline / Deploy Production (pull_request) Has been skipped
CI Build & Deploy Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 37s
CI/CD Pipeline / Frontend Lint (pull_request) Successful in 56s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 1m9s
CI/CD Pipeline / Validate Code Quality And Tests (pull_request) Successful in 1m56s
CI/CD Pipeline / Unit Tests (pull_request) Successful in 3m3s
Auto Approve CI PRs / Auto Approve on CI Green (pull_request) Successful in 3m54s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m29s
AI Code Review / AI Code Review (pull_request) Successful in 4m19s
Auto Merge CI PRs / Auto Merge on CI Green + Approved (pull_request) Successful in 4m24s
Preview Cleanup / Cleanup Preview Environment (pull_request) Successful in 26s
|
||
|
CI Bot
|
df2effbd62 |
feat(ci): P2-3 Runner监控告警 - 离线/磁盘/内存/队列积压主动告警
CI Build & Deploy Pipeline / Build Staging API Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Production API Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Production Web Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI Build & Deploy Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI Build & Deploy Pipeline / Deploy Production (pull_request) Has been skipped
CI Build & Deploy Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI Build & Deploy Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 24s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 57s
CI/CD Pipeline / Frontend Lint (pull_request) Successful in 58s
CI/CD Pipeline / Validate Code Quality And Tests (pull_request) Failing after 1m39s
Auto Merge CI PRs / Auto Merge on CI Green + Approved (pull_request) Successful in 2m11s
AI Code Review / AI Code Review (pull_request) Successful in 3m34s
CI/CD Pipeline / Unit Tests (pull_request) Successful in 3m22s
Auto Approve CI PRs / Auto Approve on CI Green (pull_request) Successful in 3m51s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m46s
新增 scripts/ci/runner_monitor/ 目录，纯CI运维工具，不动业务代码。 工单: #449 里程碑: Phase 4 3个核心能力: - runner_status.py - Runner在线状态巡检（Gitea API查runner列表+状态+队列） - runner_metrics.py - 系统指标采集骨架（CPU/内存/磁盘，SSH实装后补） - alert_manager.py - 告警调度（阈值判断+去重+飞书通知+快照输出） 告警规则: P1（严重）: - Runner离线 → 立即告警（Gitea status != online） - 磁盘使用率 > 90% P2（警告）: - 磁盘使用率 > 85% - 内存使用率 > 90% 持续5分钟 - CI队列积压 > 10个pending超过10分钟 设计要点: - 配置化：所有阈值、检测间隔、webhook地址全走环境变量 - 告警去重：同一问题30分钟内只报一次，避免刷屏 - 复用chatops：飞书通知走同一个webhook，GiteaClient复用 - 状态快照：每次检测生成JSON快照，供CI看板消费 - 无额外强依赖：urllib实现，和现有ci脚本风格一致 - SSH指标采集留好接口，后续补充不影响现有逻辑 black+ruff全绿，冒烟测试通过。 |