fix(ci): 僵尸run复活防护——重负载job自检+dedupe竞态增强

P1修复(2026-08-30 CI巡检实证):
Gitea 1.26.4 调度器bug:被 concurrency cancel 的 run,其 job 会在数小时后
被复活重跑(attempt=2)。实测 42597 号 run 18:46 被取消,21:46 复活,
与正常流水线并行烧了1.5小时 runner(Validate+UnitTests+staging构建全套),
合并高峰双流水线CPU超卖导致单测 14分→30分。

改动:
1. 新增 scripts/ci/ci_run_selfcheck.sh:job 启动第一步实时查本 run 状态,
   若 conclusion 为 cancelled/skipped(被复活的僵尸job)立即退出(exit 78),
   API 故障时不阻断(unknown 照常执行)
2. ci-pipeline.yml 12 个重量级 job(validate×3、unit/integration/frontend×2、
   build-pr/build-staging/staging-e2e/api-tests/build-production)首步插入自检
3. dedupe-check 情形2增强:同一 head_sha 的 push 流水线不只是在跑时去重,
   已 success 完成时 PR 侧测试同样跳过(消除竞态窗口双跑)
This commit is contained in:
xiaoxia
2026-08-30 23:04:40 +08:00
parent 301b591037
commit 3fcb66e40d
2 changed files with 109 additions and 1 deletions
+75 -1
View File
@@ -59,9 +59,11 @@ jobs:
exit 0
fi
# 情形2:同一 head_sha 已有在跑/排队的 push 流水线(rebase/ff 合并竞态)
# 情形3(增强):同一 head_sha 的 push 流水线已成功完成(竞态窗口内两边都通过了dedup,
# 此刻 push 已跑完时PR侧测试纯属重复,2026-08-30 双跑实测CPU超卖14分→30分)
DUP=$(curl -sfH "Authorization: token $GITHUB_TOKEN" \
"${GITHUB_API_URL}/repos/${GITHUB_REPOSITORY}/actions/runs?head_sha=${HEAD_SHA}&per_page=30" \
| python3 -c "import json,sys; d=json.load(sys.stdin); runs=d if isinstance(d,list) else d.get('workflow_runs',d.get('runs',[])); hit=[r for r in runs if r.get('event')=='push' and r.get('status') in ('in_progress','queued','waiting','pending')]; print('true' if hit else 'false')" || echo false)
| python3 -c "import json,sys; d=json.load(sys.stdin); runs=d if isinstance(d,list) else d.get('workflow_runs',d.get('runs',[])); hit=[r for r in runs if r.get('event')=='push' and (r.get('status') in ('in_progress','queued','waiting','pending') or r.get('conclusion')=='success')]; print('true' if hit else 'false')" || echo false)
if [ "$DUP" = "true" ]; then
echo "skip_tests=true" >> $GITHUB_OUTPUT
echo "reason=duplicate-push-run-active" >> $GITHUB_OUTPUT
@@ -138,6 +140,12 @@ jobs:
permissions:
contents: write
steps:
- name: Zombie run selfcheck
shell: sh
env:
GITHUB_TOKEN: ${{ github.token }}
GITHUB_RUN_ID: ${{ github.run_id }}
run: bash scripts/ci/ci_run_selfcheck.sh
- name: Checkout code
shell: sh
env:
@@ -238,6 +246,12 @@ jobs:
permissions:
contents: read
steps:
- name: Zombie run selfcheck
shell: sh
env:
GITHUB_TOKEN: ${{ github.token }}
GITHUB_RUN_ID: ${{ github.run_id }}
run: bash scripts/ci/ci_run_selfcheck.sh
- name: Checkout code
shell: sh
env:
@@ -319,6 +333,12 @@ jobs:
USE_IN_MEMORY_DB: 'false'
CI_USE_SHARED_PG: 'true'
steps:
- name: Zombie run selfcheck
shell: sh
env:
GITHUB_TOKEN: ${{ github.token }}
GITHUB_RUN_ID: ${{ github.run_id }}
run: bash scripts/ci/ci_run_selfcheck.sh
- name: Checkout code
shell: sh
env:
@@ -402,6 +422,12 @@ jobs:
OSS_BUCKET_NAME: xiaoxia-autocut
OSS_ENDPOINT: oss-cn-hangzhou.aliyuncs.com
steps:
- name: Zombie run selfcheck
shell: sh
env:
GITHUB_TOKEN: ${{ github.token }}
GITHUB_RUN_ID: ${{ github.run_id }}
run: bash scripts/ci/ci_run_selfcheck.sh
- name: Checkout code
shell: sh
env:
@@ -482,6 +508,12 @@ jobs:
OSS_BUCKET_NAME: xiaoxia-autocut
OSS_ENDPOINT: oss-cn-hangzhou.aliyuncs.com
steps:
- name: Zombie run selfcheck
shell: sh
env:
GITHUB_TOKEN: ${{ github.token }}
GITHUB_RUN_ID: ${{ github.run_id }}
run: bash scripts/ci/ci_run_selfcheck.sh
- name: Checkout code
shell: sh
env:
@@ -536,6 +568,12 @@ jobs:
needs: [check-frontend-only, dedupe-check]
if: needs.dedupe-check.outputs.skip_tests != 'true' && needs.check-frontend-only.outputs.skip_frontend != 'true'
steps:
- name: Zombie run selfcheck
shell: sh
env:
GITHUB_TOKEN: ${{ github.token }}
GITHUB_RUN_ID: ${{ github.run_id }}
run: bash scripts/ci/ci_run_selfcheck.sh
- name: Checkout code
shell: sh
env:
@@ -598,6 +636,12 @@ jobs:
needs: [check-frontend-only, dedupe-check]
if: always() && needs.dedupe-check.outputs.skip_tests != 'true' && needs.check-frontend-only.outputs.skip_frontend != 'true'
steps:
- name: Zombie run selfcheck
shell: sh
env:
GITHUB_TOKEN: ${{ github.token }}
GITHUB_RUN_ID: ${{ github.run_id }}
run: bash scripts/ci/ci_run_selfcheck.sh
- name: Checkout code
shell: sh
env:
@@ -689,6 +733,12 @@ jobs:
cache_name: web-cache
timeout: 30
steps:
- name: Zombie run selfcheck
shell: sh
env:
GITHUB_TOKEN: ${{ github.token }}
GITHUB_RUN_ID: ${{ github.run_id }}
run: bash scripts/ci/ci_run_selfcheck.sh
- name: Checkout code
shell: sh
env:
@@ -867,6 +917,12 @@ jobs:
cache_name: web-cache
timeout: 30
steps:
- name: Zombie run selfcheck
shell: sh
env:
GITHUB_TOKEN: ${{ github.token }}
GITHUB_RUN_ID: ${{ github.run_id }}
run: bash scripts/ci/ci_run_selfcheck.sh
- name: Checkout code
shell: sh
env:
@@ -1240,6 +1296,12 @@ jobs:
if: success() && (github.ref_name == 'develop' || github.ref_name == 'main')
needs: deploy-staging
steps:
- name: Zombie run selfcheck
shell: sh
env:
GITHUB_TOKEN: ${{ github.token }}
GITHUB_RUN_ID: ${{ github.run_id }}
run: bash scripts/ci/ci_run_selfcheck.sh
- name: Checkout code
shell: sh
env:
@@ -1287,6 +1349,12 @@ jobs:
if: success() && (github.ref_name == 'develop' || github.ref_name == 'main')
needs: deploy-staging
steps:
- name: Zombie run selfcheck
shell: sh
env:
GITHUB_TOKEN: ${{ github.token }}
GITHUB_RUN_ID: ${{ github.run_id }}
run: bash scripts/ci/ci_run_selfcheck.sh
- name: Checkout code
shell: sh
env:
@@ -1361,6 +1429,12 @@ jobs:
cache_name: web-cache
timeout: 30
steps:
- name: Zombie run selfcheck
shell: sh
env:
GITHUB_TOKEN: ${{ github.token }}
GITHUB_RUN_ID: ${{ github.run_id }}
run: bash scripts/ci/ci_run_selfcheck.sh
- name: Checkout code
shell: sh
env:
+34
View File
@@ -0,0 +1,34 @@
#!/bin/bash
# CI job 运行前自检 —— 防 Gitea 1.26 调度 bug
# 被 concurrency cancel 的 run,其已派发的 job 可能在数小时后被调度器复活重跑
# 2026-08-30 实测:18:46 取消的 run21:46 复活 attempt=2,与正常流水线抢 runner 1.5 小时)。
# job 启动第一步先实时查本 run 状态,若已终结(cancelled/skipped),容器立即退出不烧资源。
#
# 用法:在重量级 jobValidate/Unit Tests/Integration/Frontend/Build/E2E 类)的
# 首个 step 中加入:bash scripts/ci/ci_run_selfcheck.sh
set -u
echo "▶ CI 运行前自检(僵尸 run 复活防护)..."
if [ -z "${GITHUB_API_URL:-}" ] || [ -z "${GITHUB_REPOSITORY:-}" ] || [ -z "${GITHUB_TOKEN:-}" ] || [ -z "${GITHUB_RUN_ID:-}" ]; then
echo "⚠️ 缺少 GITHUB_* 环境变量(GITHUB_RUN_ID/GITHUB_TOKEN),跳过自检(不阻断)"
exit 0
fi
CONCLUSION=$(curl -sf -H "Authorization: token ${GITHUB_TOKEN}" --max-time 10 \
"${GITHUB_API_URL}/repos/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}" 2>/dev/null \
| python3 -c "import json,sys; print(json.load(sys.stdin).get('conclusion') or 'running')" 2>/dev/null || echo "unknown")
case "${CONCLUSION}" in
cancelled|skipped)
echo "::error::本 run(${GITHUB_RUN_ID}) 状态已为 ${CONCLUSION},却仍被调度器派发执行——判定为 Gitea 复活的僵尸 job,立即退出,不再占用 runner"
exit 78 # EX_CONFIG:非零退出让调度器知晓该 job 终结(避免被当成功空跑)
;;
running|unknown|"")
# unknown 时不阻断(API 临时故障),正常执行
echo "✅ run ${GITHUB_RUN_ID} 状态正常(${CONCLUSION:-running}),继续执行"
;;
*)
echo "✅ run ${GITHUB_RUN_ID} 结论=${CONCLUSION}(已终结但非取消),继续执行"
;;
esac