fix(ci): 僵尸run复活防护——重负载job自检+dedupe竞态增强
P1修复(2026-08-30 CI巡检实证): Gitea 1.26.4 调度器bug:被 concurrency cancel 的 run,其 job 会在数小时后 被复活重跑(attempt=2)。实测 42597 号 run 18:46 被取消,21:46 复活, 与正常流水线并行烧了1.5小时 runner(Validate+UnitTests+staging构建全套), 合并高峰双流水线CPU超卖导致单测 14分→30分。 改动: 1. 新增 scripts/ci/ci_run_selfcheck.sh:job 启动第一步实时查本 run 状态, 若 conclusion 为 cancelled/skipped(被复活的僵尸job)立即退出(exit 78), API 故障时不阻断(unknown 照常执行) 2. ci-pipeline.yml 12 个重量级 job(validate×3、unit/integration/frontend×2、 build-pr/build-staging/staging-e2e/api-tests/build-production)首步插入自检 3. dedupe-check 情形2增强:同一 head_sha 的 push 流水线不只是在跑时去重, 已 success 完成时 PR 侧测试同样跳过(消除竞态窗口双跑)
This commit is contained in:
@@ -59,9 +59,11 @@ jobs:
|
||||
exit 0
|
||||
fi
|
||||
# 情形2:同一 head_sha 已有在跑/排队的 push 流水线(rebase/ff 合并竞态)
|
||||
# 情形3(增强):同一 head_sha 的 push 流水线已成功完成(竞态窗口内两边都通过了dedup,
|
||||
# 此刻 push 已跑完时PR侧测试纯属重复,2026-08-30 双跑实测CPU超卖14分→30分)
|
||||
DUP=$(curl -sfH "Authorization: token $GITHUB_TOKEN" \
|
||||
"${GITHUB_API_URL}/repos/${GITHUB_REPOSITORY}/actions/runs?head_sha=${HEAD_SHA}&per_page=30" \
|
||||
| python3 -c "import json,sys; d=json.load(sys.stdin); runs=d if isinstance(d,list) else d.get('workflow_runs',d.get('runs',[])); hit=[r for r in runs if r.get('event')=='push' and r.get('status') in ('in_progress','queued','waiting','pending')]; print('true' if hit else 'false')" || echo false)
|
||||
| python3 -c "import json,sys; d=json.load(sys.stdin); runs=d if isinstance(d,list) else d.get('workflow_runs',d.get('runs',[])); hit=[r for r in runs if r.get('event')=='push' and (r.get('status') in ('in_progress','queued','waiting','pending') or r.get('conclusion')=='success')]; print('true' if hit else 'false')" || echo false)
|
||||
if [ "$DUP" = "true" ]; then
|
||||
echo "skip_tests=true" >> $GITHUB_OUTPUT
|
||||
echo "reason=duplicate-push-run-active" >> $GITHUB_OUTPUT
|
||||
@@ -138,6 +140,12 @@ jobs:
|
||||
permissions:
|
||||
contents: write
|
||||
steps:
|
||||
- name: Zombie run selfcheck
|
||||
shell: sh
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ github.token }}
|
||||
GITHUB_RUN_ID: ${{ github.run_id }}
|
||||
run: bash scripts/ci/ci_run_selfcheck.sh
|
||||
- name: Checkout code
|
||||
shell: sh
|
||||
env:
|
||||
@@ -238,6 +246,12 @@ jobs:
|
||||
permissions:
|
||||
contents: read
|
||||
steps:
|
||||
- name: Zombie run selfcheck
|
||||
shell: sh
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ github.token }}
|
||||
GITHUB_RUN_ID: ${{ github.run_id }}
|
||||
run: bash scripts/ci/ci_run_selfcheck.sh
|
||||
- name: Checkout code
|
||||
shell: sh
|
||||
env:
|
||||
@@ -319,6 +333,12 @@ jobs:
|
||||
USE_IN_MEMORY_DB: 'false'
|
||||
CI_USE_SHARED_PG: 'true'
|
||||
steps:
|
||||
- name: Zombie run selfcheck
|
||||
shell: sh
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ github.token }}
|
||||
GITHUB_RUN_ID: ${{ github.run_id }}
|
||||
run: bash scripts/ci/ci_run_selfcheck.sh
|
||||
- name: Checkout code
|
||||
shell: sh
|
||||
env:
|
||||
@@ -402,6 +422,12 @@ jobs:
|
||||
OSS_BUCKET_NAME: xiaoxia-autocut
|
||||
OSS_ENDPOINT: oss-cn-hangzhou.aliyuncs.com
|
||||
steps:
|
||||
- name: Zombie run selfcheck
|
||||
shell: sh
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ github.token }}
|
||||
GITHUB_RUN_ID: ${{ github.run_id }}
|
||||
run: bash scripts/ci/ci_run_selfcheck.sh
|
||||
- name: Checkout code
|
||||
shell: sh
|
||||
env:
|
||||
@@ -482,6 +508,12 @@ jobs:
|
||||
OSS_BUCKET_NAME: xiaoxia-autocut
|
||||
OSS_ENDPOINT: oss-cn-hangzhou.aliyuncs.com
|
||||
steps:
|
||||
- name: Zombie run selfcheck
|
||||
shell: sh
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ github.token }}
|
||||
GITHUB_RUN_ID: ${{ github.run_id }}
|
||||
run: bash scripts/ci/ci_run_selfcheck.sh
|
||||
- name: Checkout code
|
||||
shell: sh
|
||||
env:
|
||||
@@ -536,6 +568,12 @@ jobs:
|
||||
needs: [check-frontend-only, dedupe-check]
|
||||
if: needs.dedupe-check.outputs.skip_tests != 'true' && needs.check-frontend-only.outputs.skip_frontend != 'true'
|
||||
steps:
|
||||
- name: Zombie run selfcheck
|
||||
shell: sh
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ github.token }}
|
||||
GITHUB_RUN_ID: ${{ github.run_id }}
|
||||
run: bash scripts/ci/ci_run_selfcheck.sh
|
||||
- name: Checkout code
|
||||
shell: sh
|
||||
env:
|
||||
@@ -598,6 +636,12 @@ jobs:
|
||||
needs: [check-frontend-only, dedupe-check]
|
||||
if: always() && needs.dedupe-check.outputs.skip_tests != 'true' && needs.check-frontend-only.outputs.skip_frontend != 'true'
|
||||
steps:
|
||||
- name: Zombie run selfcheck
|
||||
shell: sh
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ github.token }}
|
||||
GITHUB_RUN_ID: ${{ github.run_id }}
|
||||
run: bash scripts/ci/ci_run_selfcheck.sh
|
||||
- name: Checkout code
|
||||
shell: sh
|
||||
env:
|
||||
@@ -689,6 +733,12 @@ jobs:
|
||||
cache_name: web-cache
|
||||
timeout: 30
|
||||
steps:
|
||||
- name: Zombie run selfcheck
|
||||
shell: sh
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ github.token }}
|
||||
GITHUB_RUN_ID: ${{ github.run_id }}
|
||||
run: bash scripts/ci/ci_run_selfcheck.sh
|
||||
- name: Checkout code
|
||||
shell: sh
|
||||
env:
|
||||
@@ -867,6 +917,12 @@ jobs:
|
||||
cache_name: web-cache
|
||||
timeout: 30
|
||||
steps:
|
||||
- name: Zombie run selfcheck
|
||||
shell: sh
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ github.token }}
|
||||
GITHUB_RUN_ID: ${{ github.run_id }}
|
||||
run: bash scripts/ci/ci_run_selfcheck.sh
|
||||
- name: Checkout code
|
||||
shell: sh
|
||||
env:
|
||||
@@ -1240,6 +1296,12 @@ jobs:
|
||||
if: success() && (github.ref_name == 'develop' || github.ref_name == 'main')
|
||||
needs: deploy-staging
|
||||
steps:
|
||||
- name: Zombie run selfcheck
|
||||
shell: sh
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ github.token }}
|
||||
GITHUB_RUN_ID: ${{ github.run_id }}
|
||||
run: bash scripts/ci/ci_run_selfcheck.sh
|
||||
- name: Checkout code
|
||||
shell: sh
|
||||
env:
|
||||
@@ -1287,6 +1349,12 @@ jobs:
|
||||
if: success() && (github.ref_name == 'develop' || github.ref_name == 'main')
|
||||
needs: deploy-staging
|
||||
steps:
|
||||
- name: Zombie run selfcheck
|
||||
shell: sh
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ github.token }}
|
||||
GITHUB_RUN_ID: ${{ github.run_id }}
|
||||
run: bash scripts/ci/ci_run_selfcheck.sh
|
||||
- name: Checkout code
|
||||
shell: sh
|
||||
env:
|
||||
@@ -1361,6 +1429,12 @@ jobs:
|
||||
cache_name: web-cache
|
||||
timeout: 30
|
||||
steps:
|
||||
- name: Zombie run selfcheck
|
||||
shell: sh
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ github.token }}
|
||||
GITHUB_RUN_ID: ${{ github.run_id }}
|
||||
run: bash scripts/ci/ci_run_selfcheck.sh
|
||||
- name: Checkout code
|
||||
shell: sh
|
||||
env:
|
||||
|
||||
Executable
+34
@@ -0,0 +1,34 @@
|
||||
#!/bin/bash
|
||||
# CI job 运行前自检 —— 防 Gitea 1.26 调度 bug:
|
||||
# 被 concurrency cancel 的 run,其已派发的 job 可能在数小时后被调度器复活重跑
|
||||
# (2026-08-30 实测:18:46 取消的 run,21:46 复活 attempt=2,与正常流水线抢 runner 1.5 小时)。
|
||||
# job 启动第一步先实时查本 run 状态,若已终结(cancelled/skipped),容器立即退出不烧资源。
|
||||
#
|
||||
# 用法:在重量级 job(Validate/Unit Tests/Integration/Frontend/Build/E2E 类)的
|
||||
# 首个 step 中加入:bash scripts/ci/ci_run_selfcheck.sh
|
||||
set -u
|
||||
|
||||
echo "▶ CI 运行前自检(僵尸 run 复活防护)..."
|
||||
|
||||
if [ -z "${GITHUB_API_URL:-}" ] || [ -z "${GITHUB_REPOSITORY:-}" ] || [ -z "${GITHUB_TOKEN:-}" ] || [ -z "${GITHUB_RUN_ID:-}" ]; then
|
||||
echo "⚠️ 缺少 GITHUB_* 环境变量(GITHUB_RUN_ID/GITHUB_TOKEN),跳过自检(不阻断)"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
CONCLUSION=$(curl -sf -H "Authorization: token ${GITHUB_TOKEN}" --max-time 10 \
|
||||
"${GITHUB_API_URL}/repos/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}" 2>/dev/null \
|
||||
| python3 -c "import json,sys; print(json.load(sys.stdin).get('conclusion') or 'running')" 2>/dev/null || echo "unknown")
|
||||
|
||||
case "${CONCLUSION}" in
|
||||
cancelled|skipped)
|
||||
echo "::error::本 run(${GITHUB_RUN_ID}) 状态已为 ${CONCLUSION},却仍被调度器派发执行——判定为 Gitea 复活的僵尸 job,立即退出,不再占用 runner"
|
||||
exit 78 # EX_CONFIG:非零退出让调度器知晓该 job 终结(避免被当成功空跑)
|
||||
;;
|
||||
running|unknown|"")
|
||||
# unknown 时不阻断(API 临时故障),正常执行
|
||||
echo "✅ run ${GITHUB_RUN_ID} 状态正常(${CONCLUSION:-running}),继续执行"
|
||||
;;
|
||||
*)
|
||||
echo "✅ run ${GITHUB_RUN_ID} 结论=${CONCLUSION}(已终结但非取消),继续执行"
|
||||
;;
|
||||
esac
|
||||
Reference in New Issue
Block a user