Files
xiaoxia-saas/scripts/ci/runner_monitor/config.py
T
xiaoxia efc79b4457
CI/CD Pipeline / Validate Code Quality And Tests (push) Has been cancelled
CI/CD Pipeline / Unit Tests (push) Has been cancelled
CI/CD Pipeline / Integration Tests (push) Has been cancelled
CI Build & Deploy Pipeline / Production Browser E2E (push) Failing after 1434h33m15s
CI Build & Deploy Pipeline / Build Production API Image (push) Failing after 1434h33m17s
CI Build & Deploy Pipeline / Deploy Production (push) Failing after 1434h33m15s
CI/CD Pipeline / Check if frontend-only change (push) Failing after 1434h33m51s
CI Build & Deploy Pipeline / Build Production Worker Image (push) Failing after 1434h33m17s
CI Build & Deploy Pipeline / Build Staging API Image (push) Has been skipped
CI Build & Deploy Pipeline / Build Staging Web Image (push) Has been skipped
CI Build & Deploy Pipeline / Build Staging Worker Image (push) Has been skipped
CI Build & Deploy Pipeline / Deploy Staging (Watchtower auto-deploy) (push) Has been skipped
CI Build & Deploy Pipeline / Staging E2E Tests (push) Failing after 1m55s
CI Build & Deploy Pipeline / Staging API Integration Tests (push) Has been skipped
CI Build & Deploy Pipeline / Build Production Web Image (push) Failing after 1435h5m20s
CI/CD Pipeline / Frontend Lint (push) Has been skipped
feat(ci): P2-3 Runner监控告警 - 离线/磁盘/内存/队列积压主动告警 (#537)
Co-authored-by: xiaoxia <dev@xiaoxiajianji.com>
Co-committed-by: xiaoxia <dev@xiaoxiajianji.com>
2026-07-18 20:55:43 +08:00

88 lines
3.8 KiB
Python
Executable File
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
#!/usr/bin/env python3
"""
Runner 监控告警配置 - 全部走环境变量,不硬编码
环境变量:
GITEA_URL / GITEA_REPO / GITEA_TOKEN / GITEA_USERNAME / GITEA_PASSWORD
(复用 chatops 的 Gitea 配置)
FEISHU_WEBHOOK_URL
飞书 webhook 地址(复用 chatops)
ALERT_RUNNER_OFFLINE_MINUTES
Runner 离线超过多少分钟触发告警,默认 5 分钟(P1)
ALERT_DISK_WARN_PERCENT 磁盘告警阈值 P2,默认 85
ALERT_DISK_CRIT_PERCENT 磁盘告警阈值 P1,默认 90
ALERT_MEM_WARN_PERCENT 内存告警阈值 P2,默认 90
ALERT_MEM_DURATION_MINUTES 内存持续超阈值多久告警,默认 5 分钟
ALERT_QUEUE_PENDING_COUNT CI 队列积压数量阈值,默认 10
ALERT_QUEUE_PENDING_MINUTES CI 队列积压持续时间阈值(分钟),默认 10
ALERT_CHECK_INTERVAL 检测间隔(秒),默认 60
ALERT_DEDUPE_WINDOW 同一告警去重窗口(秒),默认 1800(30分钟)
ALERT_P1_ENABLED P1 告警开关,默认 true
ALERT_P2_ENABLED P2 告警开关,默认 true
RUNNER_MONITOR_OUTPUT_DIR 状态快照输出目录,默认 scripts/ci/runner_monitor/snapshots
"""
import os
# ── Runner 离线告警 ───────────────────────────────────
RUNNER_OFFLINE_MINUTES = int(os.environ.get("ALERT_RUNNER_OFFLINE_MINUTES", "5"))
# ── 磁盘告警 ──────────────────────────────────────────
DISK_WARN_PERCENT = int(os.environ.get("ALERT_DISK_WARN_PERCENT", "85"))
DISK_CRIT_PERCENT = int(os.environ.get("ALERT_DISK_CRIT_PERCENT", "90"))
# ── 内存告警 ──────────────────────────────────────────
MEM_WARN_PERCENT = int(os.environ.get("ALERT_MEM_WARN_PERCENT", "90"))
MEM_DURATION_MINUTES = int(os.environ.get("ALERT_MEM_DURATION_MINUTES", "5"))
# ── 队列积压告警 ──────────────────────────────────────
QUEUE_PENDING_COUNT = int(os.environ.get("ALERT_QUEUE_PENDING_COUNT", "10"))
QUEUE_PENDING_MINUTES = int(os.environ.get("ALERT_QUEUE_PENDING_MINUTES", "10"))
# ── 检测与去重 ────────────────────────────────────────
CHECK_INTERVAL = int(os.environ.get("ALERT_CHECK_INTERVAL", "60"))
DEDUPE_WINDOW = int(os.environ.get("ALERT_DEDUPE_WINDOW", "1800"))
# ── 告警等级开关 ──────────────────────────────────────
P1_ENABLED = os.environ.get("ALERT_P1_ENABLED", "true").lower() == "true"
P2_ENABLED = os.environ.get("ALERT_P2_ENABLED", "true").lower() == "true"
# ── 输出目录 ──────────────────────────────────────────
OUTPUT_DIR = os.environ.get(
"RUNNER_MONITOR_OUTPUT_DIR",
"scripts/ci/runner_monitor/snapshots",
)
# ── SSH 配置(后补) ─────────────────────────────────
# SSH 主机列表,格式: user@host:port,user@host2:port
SSH_HOSTS = [h.strip() for h in os.environ.get("RUNNER_SSH_HOSTS", "").split(",") if h.strip()]
SSH_KEY_PATH = os.environ.get("RUNNER_SSH_KEY_PATH", "")
SSH_USER = os.environ.get("RUNNER_SSH_USER", "root")
# ── 告警等级常量 ──────────────────────────────────────
P1 = "P1"
P2 = "P2"
INFO = "INFO"
# 等级对应飞书卡片颜色
LEVEL_COLOR = {
P1: "red",
P2: "orange",
INFO: "blue",
}
LEVEL_EMOJI = {
P1: "🔥",
P2: "⚠️",
INFO: "ℹ️",
}