xiaoxia/xiaoxia-saas
1
0
Fork 0
Code Issues 9 Pull Requests 2 Actions Packages Projects Releases 7 Wiki Activity

feat(ci): P2-3 Runner监控告警 - 离线/磁盘/内存/队列积压主动告警 #537

Merged
auto-approve-bot merged 2 commits from ci/runner-monitor-alerting into develop 2026-07-18 20:55:43 +08:00
Conversation 5 Commits 2 Files Changed 5
+998
xiaoxia commented 2026-07-18 19:28:45 +08:00
Owner
Copy Link
Copy Source

概述

P2-3 Runner 监控告警 - Phase 4 最后一块，纯 CI 运维工具，不动业务代码。

工单: #449

新增内容

目录结构

scripts/ci/runner_monitor/
├── __init__.py          # 包定义
├── config.py            # 配置管理（全走环境变量）
├── runner_status.py     # Runner在线状态巡检 + 队列积压检测
├── runner_metrics.py    # 系统指标采集骨架（SSH实装后补）
└── alert_manager.py     # 告警调度器（阈值+去重+飞书+快照）

3个核心能力

  • runner_status.py - 调 Gitea API 查 runner 列表、在线状态、忙闲、队列积压
  • runner_metrics.py - 系统指标采集骨架（CPU/内存/磁盘，SSH 实装后补）
  • alert_manager.py - 告警调度：阈值判断 + 去重 + 飞书通知 + 快照输出

告警规则

等级 规则 阈值
🔥 P1 Runner 离线 status != online（立即告警）
🔥 P1 磁盘使用率过高 > 90%
⚠️ P2 磁盘使用率偏高 > 85%
⚠️ P2 内存使用率持续过高 > 90% 持续 5 分钟
⚠️ P2 CI 队列积压 >10 个 pending 超过 10 分钟

设计要点

  • ✅ 配置化：阈值、检测间隔、webhook 全走环境变量
  • ✅ 告警去重：同一问题 30 分钟内只报一次，避免刷屏
  • ✅ 复用 chatops：飞书通知走同一个 webhook，GiteaClient 复用不重复造轮子
  • ✅ 状态快照：每次检测生成 JSON 快照，供 CI 看板消费
  • ✅ SSH 预留接口：指标采集骨架已搭好，后续 SSH 实装不影响现有逻辑
  • ✅ 无额外强依赖：urllib 实现，和现有 ci 脚本风格一致

用法

# 列出所有 runner
python3 scripts/ci/runner_monitor/runner_status.py --list

# 执行一次完整检查（检测+告警+快照）
python3 scripts/ci/runner_monitor/alert_manager.py --check

# 只检测不发通知（dry-run）
python3 scripts/ci/runner_monitor/alert_manager.py --dry-run

# 持续运行（daemon模式）
python3 scripts/ci/runner_monitor/alert_manager.py --daemon

# 生成快照JSON
python3 scripts/ci/runner_monitor/alert_manager.py --snapshot

验证

  • ✅ black 格式化通过
  • ✅ ruff 检查通过
  • ✅ Gitea API 实际调用：12 个 runner 状态正确返回
  • ✅ 告警检测逻辑：当前无离线/无积压 → 0 告警，正确
  • ✅ 快照输出：JSON 结构完整（runner摘要+离线列表+队列+指标+活跃告警）

后续迭代

  • SSH 实装：连接 runner 服务器采集 CPU/内存/磁盘实际指标
  • 部署到构建服务器 + crontab 定时执行
  • 集成到 CI 看板展示 runner 状态
## 概述 P2-3 Runner 监控告警 - Phase 4 最后一块，纯 CI 运维工具，不动业务代码。 工单: #449 ## 新增内容 ### 目录结构 ``` scripts/ci/runner_monitor/ ├── __init__.py # 包定义 ├── config.py # 配置管理（全走环境变量） ├── runner_status.py # Runner在线状态巡检 + 队列积压检测 ├── runner_metrics.py # 系统指标采集骨架（SSH实装后补） └── alert_manager.py # 告警调度器（阈值+去重+飞书+快照） ``` ### 3个核心能力 - **runner_status.py** - 调 Gitea API 查 runner 列表、在线状态、忙闲、队列积压 - **runner_metrics.py** - 系统指标采集骨架（CPU/内存/磁盘，SSH 实装后补） - **alert_manager.py** - 告警调度：阈值判断 + 去重 + 飞书通知 + 快照输出 ### 告警规则 | 等级 | 规则 | 阈值 | |------|------|------| | 🔥 P1 | Runner 离线 | status != online（立即告警） | | 🔥 P1 | 磁盘使用率过高 | > 90% | | ⚠️ P2 | 磁盘使用率偏高 | > 85% | | ⚠️ P2 | 内存使用率持续过高 | > 90% 持续 5 分钟 | | ⚠️ P2 | CI 队列积压 | >10 个 pending 超过 10 分钟 | ### 设计要点 - ✅ **配置化**：阈值、检测间隔、webhook 全走环境变量 - ✅ **告警去重**：同一问题 30 分钟内只报一次，避免刷屏 - ✅ **复用 chatops**：飞书通知走同一个 webhook，GiteaClient 复用不重复造轮子 - ✅ **状态快照**：每次检测生成 JSON 快照，供 CI 看板消费 - ✅ **SSH 预留接口**：指标采集骨架已搭好，后续 SSH 实装不影响现有逻辑 - ✅ **无额外强依赖**：urllib 实现，和现有 ci 脚本风格一致 ### 用法 ```bash # 列出所有 runner python3 scripts/ci/runner_monitor/runner_status.py --list # 执行一次完整检查（检测+告警+快照） python3 scripts/ci/runner_monitor/alert_manager.py --check # 只检测不发通知（dry-run） python3 scripts/ci/runner_monitor/alert_manager.py --dry-run # 持续运行（daemon模式） python3 scripts/ci/runner_monitor/alert_manager.py --daemon # 生成快照JSON python3 scripts/ci/runner_monitor/alert_manager.py --snapshot ``` ## 验证 - ✅ black 格式化通过 - ✅ ruff 检查通过 - ✅ Gitea API 实际调用：12 个 runner 状态正确返回 - ✅ 告警检测逻辑：当前无离线/无积压 → 0 告警，正确 - ✅ 快照输出：JSON 结构完整（runner摘要+离线列表+队列+指标+活跃告警） ## 后续迭代 - SSH 实装：连接 runner 服务器采集 CPU/内存/磁盘实际指标 - 部署到构建服务器 + crontab 定时执行 - 集成到 CI 看板展示 runner 状态
xiaoxia added 1 commit 2026-07-18 19:28:45 +08:00
feat(ci): P2-3 Runner监控告警 - 离线/磁盘/内存/队列积压主动告警
CI Build & Deploy Pipeline / Build Staging API Image (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Build Staging Web Image (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Build Staging Worker Image (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Build Production API Image (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Build Production Web Image (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Build Production Worker Image (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Staging E2E Tests (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Deploy Production (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Staging API Integration Tests (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Production Browser E2E (pull_request) Has been skipped
Details
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 24s
Details
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 57s
Details
CI/CD Pipeline / Frontend Lint (pull_request) Successful in 58s
Details
CI/CD Pipeline / Validate Code Quality And Tests (pull_request) Failing after 1m39s
Details
Auto Merge CI PRs / Auto Merge on CI Green + Approved (pull_request) Successful in 2m11s
Details
AI Code Review / AI Code Review (pull_request) Successful in 3m34s
Details
CI/CD Pipeline / Unit Tests (pull_request) Successful in 3m22s
Details
Auto Approve CI PRs / Auto Approve on CI Green (pull_request) Successful in 3m51s
Details
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m46s
Details
df2effbd62 
新增 scripts/ci/runner_monitor/ 目录，纯CI运维工具，不动业务代码。

工单: #449
里程碑: Phase 4

3个核心能力:
- runner_status.py  - Runner在线状态巡检（Gitea API查runner列表+状态+队列）
- runner_metrics.py - 系统指标采集骨架（CPU/内存/磁盘，SSH实装后补）
- alert_manager.py  - 告警调度（阈值判断+去重+飞书通知+快照输出）

告警规则:
  P1（严重）:
    - Runner离线 → 立即告警（Gitea status != online）
    - 磁盘使用率 > 90%
  P2（警告）:
    - 磁盘使用率 > 85%
    - 内存使用率 > 90% 持续5分钟
    - CI队列积压 > 10个pending超过10分钟

设计要点:
- 配置化：所有阈值、检测间隔、webhook地址全走环境变量
- 告警去重：同一问题30分钟内只报一次，避免刷屏
- 复用chatops：飞书通知走同一个webhook，GiteaClient复用
- 状态快照：每次检测生成JSON快照，供CI看板消费
- 无额外强依赖：urllib实现，和现有ci脚本风格一致
- SSH指标采集留好接口，后续补充不影响现有逻辑

black+ruff全绿，冒烟测试通过。
gitea-actions bot commented 2026-07-18 19:29:44 +08:00
Copy Link
Copy Source

🚀 预览环境已部署

项目 详情
PR号 #537
预览链接 https://pr-537.preview.xiaoxiajianji.com
API环境 staging

💡 预览环境使用 staging API 数据，请勿在预览环境中操作重要数据。

🔄 每次提交新代码后预览环境会自动更新。

🗑️ PR 关闭或合并后，预览环境会自动清理。

🚀 **预览环境已部署** | 项目 | 详情 | |------|------| | PR号 | #537 | | 预览链接 | [https://pr-537.preview.xiaoxiajianji.com](https://pr-537.preview.xiaoxiajianji.com) | | API环境 | staging | > 💡 预览环境使用 staging API 数据，请勿在预览环境中操作重要数据。 > > 🔄 每次提交新代码后预览环境会自动更新。 > > 🗑️ PR 关闭或合并后，预览环境会自动清理。
xiaoxia added 1 commit 2026-07-18 20:51:18 +08:00
style: isort修复import排序
CI Build & Deploy Pipeline / Build Staging API Image (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Build Staging Web Image (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Build Staging Worker Image (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Build Production API Image (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Build Production Web Image (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Staging E2E Tests (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Build Production Worker Image (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Staging API Integration Tests (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Deploy Production (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Production Browser E2E (pull_request) Has been skipped
Details
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 37s
Details
CI/CD Pipeline / Frontend Lint (pull_request) Successful in 56s
Details
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 1m9s
Details
CI/CD Pipeline / Validate Code Quality And Tests (pull_request) Successful in 1m56s
Details
CI/CD Pipeline / Unit Tests (pull_request) Successful in 3m3s
Details
Auto Approve CI PRs / Auto Approve on CI Green (pull_request) Successful in 3m54s
Details
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m29s
Details
AI Code Review / AI Code Review (pull_request) Successful in 4m19s
Details
Auto Merge CI PRs / Auto Merge on CI Green + Approved (pull_request) Successful in 4m24s
Details
Preview Cleanup / Cleanup Preview Environment (pull_request) Successful in 26s
Details
34abf855b0
auto-approve-bot approved these changes 2026-07-18 20:55:12 +08:00
auto-approve-bot left a comment
Collaborator
Copy Link
Copy Source

CI全绿，自动审批通过。

CI全绿，自动审批通过。
auto-approve-bot approved these changes 2026-07-18 20:55:12 +08:00
auto-approve-bot left a comment
Collaborator
Copy Link
Copy Source

CI全绿，自动审批通过。

CI全绿，自动审批通过。
auto-approve-bot commented 2026-07-18 20:55:38 +08:00
Collaborator
Copy Link
Copy Source

代码审查结果 - PR #537

⚠️ 问题（4个需要修改）

  1. scripts/ci/runner_monitor/__init__.py 第13行：__all__ 中包含了 "snapshot"，但在修改文件列表中未提供 snapshot.py 文件。这会导致外部执行 from runner_monitor import snapshot 时抛出 ImportError 或 AttributeError。
  2. scripts/ci/runner_monitor/config.py 第36行：使用 int(os.environ.get(..., "default")) 进行类型转换时缺乏异常处理。如果环境变量配置了非数字字符串（如 "abc"），模块加载时会直接抛出 ValueError 导致程序崩溃。
  3. scripts/ci/runner_monitor/alert_manager.py 第338行：使用 config.__dict__.get("FEISHU_WEBHOOK_URL", "") 访问配置极不规范且不安全。该方式绕过了属性访问机制，且根据当前 diff，config.py 中并未定义 FEISHU_WEBHOOK_URL，导致此行逻辑永远取不到值，只能依赖后续的 fallback 逻辑，属于无效代码。
  4. scripts/ci/runner_monitor/alert_manager.py 第451行：在 save_snapshot 方法内部通过 from runner_monitor.runner_status import RunnerStatusChecker as RSC 导入类仅为了调用静态方法 _cleanup_old_snapshots。这种设计造成了不必要的模块耦合和重复导入开销，且静态方法放在实体类中也不符合单一职责原则。

💡 建议（2个可选）

  1. scripts/ci/runner_monitor/alert_manager.py 第338行：建议直接从 chatops 导入配置，或者在 config.py 中显式定义并转发 FEISHU_WEBHOOK_URL，避免使用 __dict__ 这种黑魔法。
  2. scripts/ci/runner_monitor/alert_manager.py 第149行：在生成 label 列表时使用了 label.get('name')，假设 label 一定是字典。如果 API 返回的数据结构发生变化（如 label 为字符串或 None），此处会崩溃。建议增加类型检查或更安全的防御性代码。

✅ 格式检查通过 | ❌ 逻辑审查需修改 | ✅ 建议关注性能

🤖 由 AI 代码审查机器人自动生成 | 2026-07-18 20:55:38 | 模型:

## 代码审查结果 - PR #537 ### ⚠️ 问题（4个需要修改） 1. **scripts/ci/runner_monitor/__init__.py 第13行**：`__all__` 中包含了 `"snapshot"`，但在修改文件列表中未提供 `snapshot.py` 文件。这会导致外部执行 `from runner_monitor import snapshot` 时抛出 `ImportError` 或 `AttributeError`。 2. **scripts/ci/runner_monitor/config.py 第36行**：使用 `int(os.environ.get(..., "default"))` 进行类型转换时缺乏异常处理。如果环境变量配置了非数字字符串（如 "abc"），模块加载时会直接抛出 `ValueError` 导致程序崩溃。 3. **scripts/ci/runner_monitor/alert_manager.py 第338行**：使用 `config.__dict__.get("FEISHU_WEBHOOK_URL", "")` 访问配置极不规范且不安全。该方式绕过了属性访问机制，且根据当前 diff，`config.py` 中并未定义 `FEISHU_WEBHOOK_URL`，导致此行逻辑永远取不到值，只能依赖后续的 fallback 逻辑，属于无效代码。 4. **scripts/ci/runner_monitor/alert_manager.py 第451行**：在 `save_snapshot` 方法内部通过 `from runner_monitor.runner_status import RunnerStatusChecker as RSC` 导入类仅为了调用静态方法 `_cleanup_old_snapshots`。这种设计造成了不必要的模块耦合和重复导入开销，且静态方法放在实体类中也不符合单一职责原则。 ### 💡 建议（2个可选） 1. **scripts/ci/runner_monitor/alert_manager.py 第338行**：建议直接从 `chatops` 导入配置，或者在 `config.py` 中显式定义并转发 `FEISHU_WEBHOOK_URL`，避免使用 `__dict__` 这种黑魔法。 2. **scripts/ci/runner_monitor/alert_manager.py 第149行**：在生成 label 列表时使用了 `label.get('name')`，假设 `label` 一定是字典。如果 API 返回的数据结构发生变化（如 label 为字符串或 None），此处会崩溃。建议增加类型检查或更安全的防御性代码。 --- ✅ 格式检查通过 | ❌ 逻辑审查需修改 | ✅ 建议关注性能 --- <sub>🤖 由 AI 代码审查机器人自动生成 | 2026-07-18 20:55:38 | 模型: </sub> <!-- AI_CODE_REVIEW_AUTO_COMMENT -->
auto-approve-bot merged commit efc79b4457 into develop 2026-07-18 20:55:43 +08:00
auto-approve-bot deleted branch ci/runner-monitor-alerting 2026-07-18 20:55:43 +08:00
gitea-actions bot commented 2026-07-18 20:56:08 +08:00
Copy Link
Copy Source

🗑️ 预览环境已清理

PR #537 已关闭或合并，对应的预览环境已被清理。

如有需要，可以重新打开 PR 来重新生成预览环境。

🗑️ **预览环境已清理** PR #537 已关闭或合并，对应的预览环境已被清理。 > 如有需要，可以重新打开 PR 来重新生成预览环境。
Sign in to join this conversation.
Reviewers
No Reviewers
auto-approve-bot
Labels
Clear labels
P0
阻塞性，必须立即处理
 P1
高优先级
 P2
中优先级
 P3
低优先级/后考虑
 architecture
架构设计
 backend
后端相关
 blocked
阻塞中
 bug
Bug修复
 ci-bug
CI本身的bug
 ci-optimization
CI优化类任务
 ci-pipeline
CI流水线/workflow相关
 cleanup
代码清理
 devops
运维/部署/CI
 documentation
文档相关
 feature
新功能
 frontend
前端相关
 question
需要讨论确认
 ready
PR CI通过，可合并
 refactor
代码重构
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
auto-approve-bot xiaoxia
No Assignees
3 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: xiaoxia/xiaoxia-saas#537
Delete Branch "ci/runner-monitor-alerting"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?