Files
xiaoxia-saas/scripts/ci/runner_monitor/runner_metrics.py
T
CI Bot df2effbd62
CI Build & Deploy Pipeline / Build Staging API Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Production API Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Production Web Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI Build & Deploy Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI Build & Deploy Pipeline / Deploy Production (pull_request) Has been skipped
CI Build & Deploy Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI Build & Deploy Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 24s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 57s
CI/CD Pipeline / Frontend Lint (pull_request) Successful in 58s
CI/CD Pipeline / Validate Code Quality And Tests (pull_request) Failing after 1m39s
Auto Merge CI PRs / Auto Merge on CI Green + Approved (pull_request) Successful in 2m11s
AI Code Review / AI Code Review (pull_request) Successful in 3m34s
CI/CD Pipeline / Unit Tests (pull_request) Successful in 3m22s
Auto Approve CI PRs / Auto Approve on CI Green (pull_request) Successful in 3m51s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m46s
feat(ci): P2-3 Runner监控告警 - 离线/磁盘/内存/队列积压主动告警
新增 scripts/ci/runner_monitor/ 目录,纯CI运维工具,不动业务代码。

工单: #449
里程碑: Phase 4

3个核心能力:
- runner_status.py  - Runner在线状态巡检(Gitea API查runner列表+状态+队列)
- runner_metrics.py - 系统指标采集骨架(CPU/内存/磁盘,SSH实装后补)
- alert_manager.py  - 告警调度(阈值判断+去重+飞书通知+快照输出)

告警规则:
  P1(严重):
    - Runner离线 → 立即告警(Gitea status != online)
    - 磁盘使用率 > 90%
  P2(警告):
    - 磁盘使用率 > 85%
    - 内存使用率 > 90% 持续5分钟
    - CI队列积压 > 10个pending超过10分钟

设计要点:
- 配置化:所有阈值、检测间隔、webhook地址全走环境变量
- 告警去重:同一问题30分钟内只报一次,避免刷屏
- 复用chatops:飞书通知走同一个webhook,GiteaClient复用
- 状态快照:每次检测生成JSON快照,供CI看板消费
- 无额外强依赖:urllib实现,和现有ci脚本风格一致
- SSH指标采集留好接口,后续补充不影响现有逻辑

black+ruff全绿,冒烟测试通过。
2026-07-18 19:28:09 +08:00

83 lines
2.7 KiB
Python
Executable File

#!/usr/bin/env python3
"""
Runner 系统指标采集 - CPU/内存/磁盘(通过 SSH 连接构建服务器)
⚠️ 第一版:骨架 + 接口定义,SSH 实装后续补充
原因:跨机器 SSH 需要密钥管理和网络权限,先把监控框架搭好。
接口约定(与 alert_manager 对接):
metrics = RunnerMetricsCollector().collect_all()
# 返回: [{"host": "...", "cpu_percent": 75.2, "mem_percent": 80.1, "disk_percent": 65.0,
# "disk_total_gb": 500, "disk_used_gb": 325, "status": "ok"}, ...]
当 SSH 不可用时,返回空列表,不影响其他监控功能。
"""
from runner_monitor import config
class RunnerMetricsCollector:
"""Runner 系统指标采集器
第一版:返回空数据(SSH 实装待后续迭代)
接口已定义好,alert_manager 直接消费。
"""
def __init__(self, ssh_hosts=None, ssh_key_path=None, ssh_user=None):
self.ssh_hosts = ssh_hosts or config.SSH_HOSTS
self.ssh_key_path = ssh_key_path or config.SSH_KEY_PATH
self.ssh_user = ssh_user or config.SSH_USER
def collect_all(self):
"""采集所有 runner 的系统指标
Returns:
list[dict]: 每台机器的指标数据
"""
if not self.ssh_hosts:
# 没有配置 SSH 主机,返回空列表
return []
results = []
for host in self.ssh_hosts:
try:
metrics = self._collect_one(host)
results.append(metrics)
except Exception as e:
results.append(
{
"host": host,
"status": "error",
"error": str(e),
}
)
return results
def _collect_one(self, host):
"""采集单台机器的指标(SSH 实装待后续)
当前直接返回 not_available 状态。
后续实现方案:用 paramiko 或 subprocess + ssh 命令执行:
- top / mpstat 取 CPU
- free 取内存
- df -h 取磁盘
"""
return {
"host": host,
"status": "not_available",
"cpu_percent": None,
"mem_percent": None,
"disk_percent": None,
"disk_total_gb": None,
"disk_used_gb": None,
"note": "SSH metrics collection not implemented yet",
}
# ── 便捷方法 ──────────────────────────────────────
@staticmethod
def is_available():
"""是否有可用的指标采集(SSH 已配置)"""
return bool(config.SSH_HOSTS and config.SSH_KEY_PATH)