df2effbd62
CI Build & Deploy Pipeline / Build Staging API Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Production API Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Production Web Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI Build & Deploy Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI Build & Deploy Pipeline / Deploy Production (pull_request) Has been skipped
CI Build & Deploy Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI Build & Deploy Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 24s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 57s
CI/CD Pipeline / Frontend Lint (pull_request) Successful in 58s
CI/CD Pipeline / Validate Code Quality And Tests (pull_request) Failing after 1m39s
Auto Merge CI PRs / Auto Merge on CI Green + Approved (pull_request) Successful in 2m11s
AI Code Review / AI Code Review (pull_request) Successful in 3m34s
CI/CD Pipeline / Unit Tests (pull_request) Successful in 3m22s
Auto Approve CI PRs / Auto Approve on CI Green (pull_request) Successful in 3m51s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m46s
新增 scripts/ci/runner_monitor/ 目录,纯CI运维工具,不动业务代码。 工单: #449 里程碑: Phase 4 3个核心能力: - runner_status.py - Runner在线状态巡检(Gitea API查runner列表+状态+队列) - runner_metrics.py - 系统指标采集骨架(CPU/内存/磁盘,SSH实装后补) - alert_manager.py - 告警调度(阈值判断+去重+飞书通知+快照输出) 告警规则: P1(严重): - Runner离线 → 立即告警(Gitea status != online) - 磁盘使用率 > 90% P2(警告): - 磁盘使用率 > 85% - 内存使用率 > 90% 持续5分钟 - CI队列积压 > 10个pending超过10分钟 设计要点: - 配置化:所有阈值、检测间隔、webhook地址全走环境变量 - 告警去重:同一问题30分钟内只报一次,避免刷屏 - 复用chatops:飞书通知走同一个webhook,GiteaClient复用 - 状态快照:每次检测生成JSON快照,供CI看板消费 - 无额外强依赖:urllib实现,和现有ci脚本风格一致 - SSH指标采集留好接口,后续补充不影响现有逻辑 black+ruff全绿,冒烟测试通过。
88 lines
3.8 KiB
Python
Executable File
88 lines
3.8 KiB
Python
Executable File
#!/usr/bin/env python3
|
||
"""
|
||
Runner 监控告警配置 - 全部走环境变量,不硬编码
|
||
|
||
环境变量:
|
||
GITEA_URL / GITEA_REPO / GITEA_TOKEN / GITEA_USERNAME / GITEA_PASSWORD
|
||
(复用 chatops 的 Gitea 配置)
|
||
|
||
FEISHU_WEBHOOK_URL
|
||
飞书 webhook 地址(复用 chatops)
|
||
|
||
ALERT_RUNNER_OFFLINE_MINUTES
|
||
Runner 离线超过多少分钟触发告警,默认 5 分钟(P1)
|
||
|
||
ALERT_DISK_WARN_PERCENT 磁盘告警阈值 P2,默认 85
|
||
ALERT_DISK_CRIT_PERCENT 磁盘告警阈值 P1,默认 90
|
||
|
||
ALERT_MEM_WARN_PERCENT 内存告警阈值 P2,默认 90
|
||
ALERT_MEM_DURATION_MINUTES 内存持续超阈值多久告警,默认 5 分钟
|
||
|
||
ALERT_QUEUE_PENDING_COUNT CI 队列积压数量阈值,默认 10
|
||
ALERT_QUEUE_PENDING_MINUTES CI 队列积压持续时间阈值(分钟),默认 10
|
||
|
||
ALERT_CHECK_INTERVAL 检测间隔(秒),默认 60
|
||
ALERT_DEDUPE_WINDOW 同一告警去重窗口(秒),默认 1800(30分钟)
|
||
|
||
ALERT_P1_ENABLED P1 告警开关,默认 true
|
||
ALERT_P2_ENABLED P2 告警开关,默认 true
|
||
|
||
RUNNER_MONITOR_OUTPUT_DIR 状态快照输出目录,默认 scripts/ci/runner_monitor/snapshots
|
||
"""
|
||
|
||
import os
|
||
|
||
# ── Runner 离线告警 ───────────────────────────────────
|
||
RUNNER_OFFLINE_MINUTES = int(os.environ.get("ALERT_RUNNER_OFFLINE_MINUTES", "5"))
|
||
|
||
# ── 磁盘告警 ──────────────────────────────────────────
|
||
DISK_WARN_PERCENT = int(os.environ.get("ALERT_DISK_WARN_PERCENT", "85"))
|
||
DISK_CRIT_PERCENT = int(os.environ.get("ALERT_DISK_CRIT_PERCENT", "90"))
|
||
|
||
# ── 内存告警 ──────────────────────────────────────────
|
||
MEM_WARN_PERCENT = int(os.environ.get("ALERT_MEM_WARN_PERCENT", "90"))
|
||
MEM_DURATION_MINUTES = int(os.environ.get("ALERT_MEM_DURATION_MINUTES", "5"))
|
||
|
||
# ── 队列积压告警 ──────────────────────────────────────
|
||
QUEUE_PENDING_COUNT = int(os.environ.get("ALERT_QUEUE_PENDING_COUNT", "10"))
|
||
QUEUE_PENDING_MINUTES = int(os.environ.get("ALERT_QUEUE_PENDING_MINUTES", "10"))
|
||
|
||
# ── 检测与去重 ────────────────────────────────────────
|
||
CHECK_INTERVAL = int(os.environ.get("ALERT_CHECK_INTERVAL", "60"))
|
||
DEDUPE_WINDOW = int(os.environ.get("ALERT_DEDUPE_WINDOW", "1800"))
|
||
|
||
# ── 告警等级开关 ──────────────────────────────────────
|
||
P1_ENABLED = os.environ.get("ALERT_P1_ENABLED", "true").lower() == "true"
|
||
P2_ENABLED = os.environ.get("ALERT_P2_ENABLED", "true").lower() == "true"
|
||
|
||
# ── 输出目录 ──────────────────────────────────────────
|
||
OUTPUT_DIR = os.environ.get(
|
||
"RUNNER_MONITOR_OUTPUT_DIR",
|
||
"scripts/ci/runner_monitor/snapshots",
|
||
)
|
||
|
||
# ── SSH 配置(后补) ─────────────────────────────────
|
||
# SSH 主机列表,格式: user@host:port,user@host2:port
|
||
SSH_HOSTS = [h.strip() for h in os.environ.get("RUNNER_SSH_HOSTS", "").split(",") if h.strip()]
|
||
SSH_KEY_PATH = os.environ.get("RUNNER_SSH_KEY_PATH", "")
|
||
SSH_USER = os.environ.get("RUNNER_SSH_USER", "root")
|
||
|
||
|
||
# ── 告警等级常量 ──────────────────────────────────────
|
||
P1 = "P1"
|
||
P2 = "P2"
|
||
INFO = "INFO"
|
||
|
||
# 等级对应飞书卡片颜色
|
||
LEVEL_COLOR = {
|
||
P1: "red",
|
||
P2: "orange",
|
||
INFO: "blue",
|
||
}
|
||
|
||
LEVEL_EMOJI = {
|
||
P1: "🔥",
|
||
P2: "⚠️",
|
||
INFO: "ℹ️",
|
||
}
|