fix: Worker health check 扫描全部 /proc 进程修复 unhealthy 误判
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 2s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 1s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 22s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m15s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 1m34s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 2m6s
CI/CD Pipeline / Validate - Style (pull_request) Successful in 2m19s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 2m30s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 2m42s
CI/CD Pipeline / Unit Tests (pull_request) Failing after 5m45s
AI Code Review / AI Code Review (pull_request) Successful in 6m14s
CI/CD Pipeline / Validate - Security (pull_request) Successful in 7m33s
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / CI Gate (pull_request) Failing after 1s
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 5m9s
CI/CD Pipeline / Canary Release to Production (pull_request) Failing after 165h8m36s
CI/CD Pipeline / Build Production API Image (pull_request) Failing after 165h8m40s
CI/CD Pipeline / Deploy Production (pull_request) Failing after 165h8m36s
CI/CD Pipeline / ACR Image Cleanup (pull_request) Failing after 165h16m6s
CI/CD Pipeline / Build Production Worker Image (pull_request) Failing after 165h8m39s
CI/CD Pipeline / Staging E2E Tests (pull_request) Failing after 165h16m6s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Failing after 165h16m8s
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Failing after 165h16m13s
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Failing after 165h16m13s
CI/CD Pipeline / Build Staging API Image (pull_request) Failing after 165h16m13s
CI/CD Pipeline / Build Staging Worker Image (pull_request) Failing after 165h16m13s
CI/CD Pipeline / PR Build Web Image (pull_request) Failing after 165h16m14s
CI/CD Pipeline / Frontend Lint (pull_request) Failing after 165h16m14s
CI/CD Pipeline / Check push changed paths (pull_request) Failing after 165h16m18s
CI/CD Pipeline / Build Production Web Image (pull_request) Failing after 165h43m25s
CI/CD Pipeline / Staging API Integration Tests (pull_request) Failing after 165h50m52s
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Failing after 165h50m59s
CI/CD Pipeline / Build Staging Web Image (pull_request) Failing after 165h50m59s
CI/CD Pipeline / Frontend Unit Tests (pull_request) Failing after 165h51m0s
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 2s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 1s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 22s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m15s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 1m34s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 2m6s
CI/CD Pipeline / Validate - Style (pull_request) Successful in 2m19s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 2m30s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 2m42s
CI/CD Pipeline / Unit Tests (pull_request) Failing after 5m45s
AI Code Review / AI Code Review (pull_request) Successful in 6m14s
CI/CD Pipeline / Validate - Security (pull_request) Successful in 7m33s
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / CI Gate (pull_request) Failing after 1s
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 5m9s
CI/CD Pipeline / Canary Release to Production (pull_request) Failing after 165h8m36s
CI/CD Pipeline / Build Production API Image (pull_request) Failing after 165h8m40s
CI/CD Pipeline / Deploy Production (pull_request) Failing after 165h8m36s
CI/CD Pipeline / ACR Image Cleanup (pull_request) Failing after 165h16m6s
CI/CD Pipeline / Build Production Worker Image (pull_request) Failing after 165h8m39s
CI/CD Pipeline / Staging E2E Tests (pull_request) Failing after 165h16m6s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Failing after 165h16m8s
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Failing after 165h16m13s
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Failing after 165h16m13s
CI/CD Pipeline / Build Staging API Image (pull_request) Failing after 165h16m13s
CI/CD Pipeline / Build Staging Worker Image (pull_request) Failing after 165h16m13s
CI/CD Pipeline / PR Build Web Image (pull_request) Failing after 165h16m14s
CI/CD Pipeline / Frontend Lint (pull_request) Failing after 165h16m14s
CI/CD Pipeline / Check push changed paths (pull_request) Failing after 165h16m18s
CI/CD Pipeline / Build Production Web Image (pull_request) Failing after 165h43m25s
CI/CD Pipeline / Staging API Integration Tests (pull_request) Failing after 165h50m52s
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Failing after 165h50m59s
CI/CD Pipeline / Build Staging Web Image (pull_request) Failing after 165h50m59s
CI/CD Pipeline / Frontend Unit Tests (pull_request) Failing after 165h51m0s
问题:Worker 容器内无 pgrep/ps 工具,旧 health check 使用 'grep -q celery /proc/1/cmdline' 检查 PID 1 的 cmdline, 但 PID 1 是 entrypoint-worker.sh(bash 脚本),不含 celery 关键字, 导致 health check 永远返回 unhealthy。 修复:改用 'grep -lq celery /proc/[0-9]*/cmdline' 扫描容器内 所有进程的 cmdline,找到任意 celery 进程即判定 healthy。 同时修复 deploy 脚本中 sh -c 嵌套导致的 $pid 变量提前展开问题。 影响文件: - infra/docker/compose.yml - infra/docker/deploy-staging-registry.sh - infra/docker/deploy-production-registry.sh
This commit is contained in:
@@ -128,9 +128,9 @@ services:
|
||||
- xiaoxia-net
|
||||
|
||||
# 健康检查配置
|
||||
# 注:celery inspect ping 依赖 broker 连接,在容器内不可靠,改用进程检查
|
||||
# 注:容器内无 pgrep/ps,扫描 /proc 所有进程的 cmdline 查找 celery 进程
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pgrep -f 'celery.*worker' | head -n1 >/dev/null 2>&1 || grep -q celery /proc/1/cmdline || exit 1"]
|
||||
test: ["CMD-SHELL", "grep -lq celery /proc/[0-9]*/cmdline 2>/dev/null || exit 1"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
|
||||
@@ -155,7 +155,7 @@ docker run -d \
|
||||
--restart unless-stopped \
|
||||
--cpus 2 \
|
||||
--memory 2g \
|
||||
--health-cmd "sh -c \"pgrep -f 'celery.*worker' >/dev/null 2>&1 || grep -q celery /proc/1/cmdline || exit 1\"" \
|
||||
--health-cmd "grep -lq celery /proc/[0-9]*/cmdline 2>/dev/null || exit 1" \
|
||||
--health-interval 30s \
|
||||
--health-timeout 10s \
|
||||
--health-retries 3 \
|
||||
|
||||
@@ -116,7 +116,7 @@ docker run -d \
|
||||
-v "$GENERATED_DIR:/app/generated" \
|
||||
--restart unless-stopped \
|
||||
--label com.centurylinklabs.watchtower.enable=true \
|
||||
--health-cmd "sh -c \"pgrep -f 'celery.*worker' >/dev/null 2>&1 || grep -q celery /proc/1/cmdline || exit 1\"" \
|
||||
--health-cmd "grep -lq celery /proc/[0-9]*/cmdline 2>/dev/null || exit 1" \
|
||||
--health-interval 30s \
|
||||
--health-timeout 10s \
|
||||
--health-retries 3 \
|
||||
|
||||
Reference in New Issue
Block a user