Files
xiaoxia-saas/scripts/ci/runner_monitor/config.py
T
CI Bot df2effbd62
CI Build & Deploy Pipeline / Build Staging API Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Production API Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Production Web Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI Build & Deploy Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI Build & Deploy Pipeline / Deploy Production (pull_request) Has been skipped
CI Build & Deploy Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI Build & Deploy Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 24s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 57s
CI/CD Pipeline / Frontend Lint (pull_request) Successful in 58s
CI/CD Pipeline / Validate Code Quality And Tests (pull_request) Failing after 1m39s
Auto Merge CI PRs / Auto Merge on CI Green + Approved (pull_request) Successful in 2m11s
AI Code Review / AI Code Review (pull_request) Successful in 3m34s
CI/CD Pipeline / Unit Tests (pull_request) Successful in 3m22s
Auto Approve CI PRs / Auto Approve on CI Green (pull_request) Successful in 3m51s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m46s
feat(ci): P2-3 Runner监控告警 - 离线/磁盘/内存/队列积压主动告警
新增 scripts/ci/runner_monitor/ 目录,纯CI运维工具,不动业务代码。

工单: #449
里程碑: Phase 4

3个核心能力:
- runner_status.py  - Runner在线状态巡检(Gitea API查runner列表+状态+队列)
- runner_metrics.py - 系统指标采集骨架(CPU/内存/磁盘,SSH实装后补)
- alert_manager.py  - 告警调度(阈值判断+去重+飞书通知+快照输出)

告警规则:
  P1(严重):
    - Runner离线 → 立即告警(Gitea status != online)
    - 磁盘使用率 > 90%
  P2(警告):
    - 磁盘使用率 > 85%
    - 内存使用率 > 90% 持续5分钟
    - CI队列积压 > 10个pending超过10分钟

设计要点:
- 配置化:所有阈值、检测间隔、webhook地址全走环境变量
- 告警去重:同一问题30分钟内只报一次,避免刷屏
- 复用chatops:飞书通知走同一个webhook,GiteaClient复用
- 状态快照:每次检测生成JSON快照,供CI看板消费
- 无额外强依赖:urllib实现,和现有ci脚本风格一致
- SSH指标采集留好接口,后续补充不影响现有逻辑

black+ruff全绿,冒烟测试通过。
2026-07-18 19:28:09 +08:00

88 lines
3.8 KiB
Python
Executable File
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
#!/usr/bin/env python3
"""
Runner 监控告警配置 - 全部走环境变量,不硬编码
环境变量:
GITEA_URL / GITEA_REPO / GITEA_TOKEN / GITEA_USERNAME / GITEA_PASSWORD
(复用 chatops 的 Gitea 配置)
FEISHU_WEBHOOK_URL
飞书 webhook 地址(复用 chatops
ALERT_RUNNER_OFFLINE_MINUTES
Runner 离线超过多少分钟触发告警,默认 5 分钟(P1)
ALERT_DISK_WARN_PERCENT 磁盘告警阈值 P2,默认 85
ALERT_DISK_CRIT_PERCENT 磁盘告警阈值 P1,默认 90
ALERT_MEM_WARN_PERCENT 内存告警阈值 P2,默认 90
ALERT_MEM_DURATION_MINUTES 内存持续超阈值多久告警,默认 5 分钟
ALERT_QUEUE_PENDING_COUNT CI 队列积压数量阈值,默认 10
ALERT_QUEUE_PENDING_MINUTES CI 队列积压持续时间阈值(分钟),默认 10
ALERT_CHECK_INTERVAL 检测间隔(秒),默认 60
ALERT_DEDUPE_WINDOW 同一告警去重窗口(秒),默认 1800(30分钟)
ALERT_P1_ENABLED P1 告警开关,默认 true
ALERT_P2_ENABLED P2 告警开关,默认 true
RUNNER_MONITOR_OUTPUT_DIR 状态快照输出目录,默认 scripts/ci/runner_monitor/snapshots
"""
import os
# ── Runner 离线告警 ───────────────────────────────────
RUNNER_OFFLINE_MINUTES = int(os.environ.get("ALERT_RUNNER_OFFLINE_MINUTES", "5"))
# ── 磁盘告警 ──────────────────────────────────────────
DISK_WARN_PERCENT = int(os.environ.get("ALERT_DISK_WARN_PERCENT", "85"))
DISK_CRIT_PERCENT = int(os.environ.get("ALERT_DISK_CRIT_PERCENT", "90"))
# ── 内存告警 ──────────────────────────────────────────
MEM_WARN_PERCENT = int(os.environ.get("ALERT_MEM_WARN_PERCENT", "90"))
MEM_DURATION_MINUTES = int(os.environ.get("ALERT_MEM_DURATION_MINUTES", "5"))
# ── 队列积压告警 ──────────────────────────────────────
QUEUE_PENDING_COUNT = int(os.environ.get("ALERT_QUEUE_PENDING_COUNT", "10"))
QUEUE_PENDING_MINUTES = int(os.environ.get("ALERT_QUEUE_PENDING_MINUTES", "10"))
# ── 检测与去重 ────────────────────────────────────────
CHECK_INTERVAL = int(os.environ.get("ALERT_CHECK_INTERVAL", "60"))
DEDUPE_WINDOW = int(os.environ.get("ALERT_DEDUPE_WINDOW", "1800"))
# ── 告警等级开关 ──────────────────────────────────────
P1_ENABLED = os.environ.get("ALERT_P1_ENABLED", "true").lower() == "true"
P2_ENABLED = os.environ.get("ALERT_P2_ENABLED", "true").lower() == "true"
# ── 输出目录 ──────────────────────────────────────────
OUTPUT_DIR = os.environ.get(
"RUNNER_MONITOR_OUTPUT_DIR",
"scripts/ci/runner_monitor/snapshots",
)
# ── SSH 配置(后补) ─────────────────────────────────
# SSH 主机列表,格式: user@host:port,user@host2:port
SSH_HOSTS = [h.strip() for h in os.environ.get("RUNNER_SSH_HOSTS", "").split(",") if h.strip()]
SSH_KEY_PATH = os.environ.get("RUNNER_SSH_KEY_PATH", "")
SSH_USER = os.environ.get("RUNNER_SSH_USER", "root")
# ── 告警等级常量 ──────────────────────────────────────
P1 = "P1"
P2 = "P2"
INFO = "INFO"
# 等级对应飞书卡片颜色
LEVEL_COLOR = {
P1: "red",
P2: "orange",
INFO: "blue",
}
LEVEL_EMOJI = {
P1: "🔥",
P2: "⚠️",
INFO: "️",
}