fix: Worker health check 扫描全部 /proc 进程修复 unhealthy 误判
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 1m1s
AI Code Review / AI Code Review (pull_request) Successful in 6m36s
PR Automation / Auto Approve on CI Green (pull_request) Failing after 13m19s
PR Automation / Auto Merge on CI Green + Approved (pull_request) Has been skipped
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 1s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 1s
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 15s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 16s
ACR Cleanup / ACR Image Cleanup (pull_request_target) Successful in 11s
Preview Cleanup / Cleanup Preview Environment (pull_request) Successful in 30s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m16s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 1m58s
CI/CD Pipeline / Validate - Style (pull_request) Successful in 2m53s
CI/CD Pipeline / Validate - Security (pull_request) Successful in 6m33s
CI/CD Pipeline / Unit Tests (pull_request) Successful in 12m20s
CI/CD Pipeline / Build Production API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Web Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / Canary Release to Production (pull_request) Has been skipped
CI/CD Pipeline / CI Gate (pull_request) Successful in 6s

问题:Worker 容器内无 pgrep/ps 工具,旧 health check 使用
'grep -q celery /proc/1/cmdline' 检查 PID 1 的 cmdline,
但 PID 1 是 entrypoint-worker.sh(bash 脚本),不含 celery 关键字,
导致 health check 永远返回 unhealthy。

同时部署脚本中 sh -c 嵌套引号导致 $pid 变量被外层 shell 提前展开为空。

修复:改用 grep -lq celery /proc/[0-9]*/cmdline 扫描所有进程 cmdline,
找到任意 celery 进程即判定 healthy。无需变量、无嵌套 shell、无 pgrep 依赖。

影响文件:
- infra/docker/compose.yml
- infra/docker/deploy-staging-registry.sh
- infra/docker/deploy-production-registry.sh
This commit is contained in:
xiaoxia
2026-09-09 18:42:17 +08:00
parent da2302b7ad
commit 54defa4429
3 changed files with 4 additions and 4 deletions
+2 -2
View File
@@ -128,9 +128,9 @@ services:
- xiaoxia-net
# 健康检查配置
# 注:celery inspect ping 依赖 broker 连接,在容器内不可靠,改用进程检查
# 注:容器内无 pgrep/ps,扫描 /proc 所有进程的 cmdline 查找 celery 进程
healthcheck:
test: ["CMD-SHELL", "for pid in /proc/[0-9]*/cmdline; do if grep -ql celery \"$pid\" 2>/dev/null; then exit 0; fi; done; exit 1"]
test: ["CMD-SHELL", "grep -lq celery /proc/[0-9]*/cmdline 2>/dev/null || exit 1"]
interval: 30s
timeout: 10s
retries: 3
+1 -1
View File
@@ -155,7 +155,7 @@ docker run -d \
--restart unless-stopped \
--cpus 2 \
--memory 2g \
--health-cmd "sh -c \"for pid in /proc/[0-9]*/cmdline; do if grep -ql celery \"$pid\" 2>/dev/null; then exit 0; fi; done; exit 1\"" \
--health-cmd "sh -c \"grep -lq celery /proc/[0-9]*/cmdline 2>/dev/null || exit 1\"" \
--health-interval 30s \
--health-timeout 10s \
--health-retries 3 \
+1 -1
View File
@@ -116,7 +116,7 @@ docker run -d \
-v "$GENERATED_DIR:/app/generated" \
--restart unless-stopped \
--label com.centurylinklabs.watchtower.enable=true \
--health-cmd "sh -c \"for pid in /proc/[0-9]*/cmdline; do if grep -ql celery \"$pid\" 2>/dev/null; then exit 0; fi; done; exit 1\"" \
--health-cmd "sh -c \"grep -lq celery /proc/[0-9]*/cmdline 2>/dev/null || exit 1\"" \
--health-interval 30s \
--health-timeout 10s \
--health-retries 3 \