Commit Graph

9 Commits

Author SHA1 Message Date
xiaoxia 29ca51da9c fix(web): 烤入所有 nginx 配置,按 APP_ENV 运行时切换,彻底解决 staging/prod 配置错乱 (#1853)
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (push) Successful in 1s
CI/CD Pipeline / Check push changed paths (push) Successful in 18s
CI/CD Pipeline / Build Staging Worker Image (push) Successful in 14s
CI/CD Pipeline / Build Staging API Image (push) Successful in 1m0s
CI/CD Pipeline / Integration Tests (push) Successful in 1m32s
CI/CD Pipeline / Validate - Python (mypy + alembic) (push) Successful in 1m40s
CI/CD Pipeline / Validate - Style (push) Successful in 2m15s
CI/CD Pipeline / Build Staging Web Image (push) Successful in 2m16s
CI/CD Pipeline / Frontend Unit Tests (push) Successful in 4m44s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (push) Failing after 2m12s
CI/CD Pipeline / Validate - Security (push) Successful in 5m30s
CI/CD Pipeline / Unit Tests (push) Successful in 7m44s
CI/CD Pipeline / Production Browser E2E (push) Has been skipped
CI/CD Pipeline / Build Production Worker Image (push) Failing after 126h13m58s
CI/CD Pipeline / Retag skipped Staging Worker Image (push) Failing after 126h19m7s
CI/CD Pipeline / PR Build Worker Image (push) Failing after 126h21m46s
CI/CD Pipeline / PR Build Web Image (push) Failing after 126h21m33s
CI/CD Pipeline / PR Build API Image (push) Failing after 126h21m34s
CI/CD Pipeline / Retag skipped Staging Web Image (push) Failing after 126h18m54s
CI/CD Pipeline / Retag skipped Staging API Image (push) Failing after 126h18m54s
CI/CD Pipeline / CI Gate (push) Failing after 126h13m45s
CI/CD Pipeline / Build Production API Image (push) Failing after 126h13m45s
CI/CD Pipeline / Check if frontend-only change (push) Failing after 126h21m35s
CI/CD Pipeline / Deploy Production (push) Failing after 126h13m42s
CI/CD Pipeline / Canary Release to Production (push) Failing after 126h13m42s
CI/CD Pipeline / Build Production Web Image (push) Failing after 126h13m45s
CI/CD Pipeline / Staging API Integration Tests (push) Failing after 126h16m37s
CI/CD Pipeline / Staging E2E Tests (push) Failing after 126h16m37s
CI/CD Pipeline / Frontend Lint (push) Failing after 126h21m29s
CI/CD Pipeline / ACR Image Cleanup (push) Failing after 126h51m43s
Co-authored-by: xiaoxia <dev@xiaoxiajianji.com>
Co-committed-by: xiaoxia <dev@xiaoxiajianji.com>
2026-09-11 09:02:01 +08:00
xiaoxia 54defa4429 fix: Worker health check 扫描全部 /proc 进程修复 unhealthy 误判
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 1m1s
AI Code Review / AI Code Review (pull_request) Successful in 6m36s
PR Automation / Auto Approve on CI Green (pull_request) Failing after 13m19s
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 1s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 1s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 15s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 16s
ACR Cleanup / ACR Image Cleanup (pull_request_target) Successful in 11s
Preview Cleanup / Cleanup Preview Environment (pull_request) Successful in 30s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m16s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 1m58s
CI/CD Pipeline / Validate - Style (pull_request) Successful in 2m53s
CI/CD Pipeline / Validate - Security (pull_request) Successful in 6m33s
CI/CD Pipeline / Unit Tests (pull_request) Successful in 12m20s
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / CI Gate (pull_request) Successful in 6s
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Failing after 164h14m54s
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Failing after 164h14m54s
CI/CD Pipeline / Frontend Lint (pull_request) Failing after 164h14m55s
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Failing after 164h14m54s
CI/CD Pipeline / Build Staging Worker Image (pull_request) Failing after 164h14m57s
CI/CD Pipeline / Check push changed paths (pull_request) Failing after 164h15m0s
CI/CD Pipeline / Canary Release to Production (pull_request) Failing after 164h2m30s
CI/CD Pipeline / Build Staging Web Image (pull_request) Failing after 164h14m57s
CI/CD Pipeline / Build Production Worker Image (pull_request) Failing after 164h2m35s
CI/CD Pipeline / Staging E2E Tests (pull_request) Failing after 164h14m48s
CI/CD Pipeline / Build Production API Image (pull_request) Failing after 164h2m36s
PR Automation / Auto Merge on CI Green + Approved (pull_request) Failing after 164h27m44s
CI/CD Pipeline / ACR Image Cleanup (pull_request) Failing after 164h14m48s
CI/CD Pipeline / Build Production Web Image (pull_request) Failing after 164h37m21s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Failing after 164h49m35s
CI/CD Pipeline / Frontend Unit Tests (pull_request) Failing after 164h49m41s
CI/CD Pipeline / Build Staging API Image (pull_request) Failing after 164h49m43s
CI/CD Pipeline / Deploy Production (pull_request) Failing after 164h37m16s
CI/CD Pipeline / Staging API Integration Tests (pull_request) Failing after 164h49m34s
CI/CD Pipeline / PR Build Web Image (pull_request) Failing after 164h49m41s
问题:Worker 容器内无 pgrep/ps 工具,旧 health check 使用
'grep -q celery /proc/1/cmdline' 检查 PID 1 的 cmdline,
但 PID 1 是 entrypoint-worker.sh(bash 脚本),不含 celery 关键字,
导致 health check 永远返回 unhealthy。

同时部署脚本中 sh -c 嵌套引号导致 $pid 变量被外层 shell 提前展开为空。

修复:改用 grep -lq celery /proc/[0-9]*/cmdline 扫描所有进程 cmdline,
找到任意 celery 进程即判定 healthy。无需变量、无嵌套 shell、无 pgrep 依赖。

影响文件:
- infra/docker/compose.yml
- infra/docker/deploy-staging-registry.sh
- infra/docker/deploy-production-registry.sh
2026-09-09 18:42:17 +08:00
CI Bot e4ac398397 fix: Worker 健康检查彻底修复 - 移除不可用的 pgrep 和 PID1 检查
根因:#1714 队列隔离后 entrypoint 启动两个 celery worker 子进程,
PID 1 变为 bash。旧健康检查依赖 pgrep(容器内不存在)或
/proc/1/cmdline(PID 1 是 bash 不是 celery),永远返回失败。
此前靠 cron 脚本 fix-worker-healthcheck.sh 每 30 分钟擦屁股。

修复:统一改为扫描 /proc/[0-9]*/cmdline 查找 celery 进程,
覆盖所有部署/回滚/CI 脚本(10 处替换)。已移除 cron 补丁。
2026-09-08 19:07:37 +08:00
saas-backend-bot df99305dd6 feat(worker): celery 队列隔离 + 孤儿任务消息作废 (#1714)
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 1s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 2s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 29s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 29s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 49s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 1m39s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m44s
CI/CD Pipeline / Validate - Style (pull_request) Successful in 2m19s
AI Code Review / AI Code Review (pull_request) Failing after 2m52s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 2m54s
CI/CD Pipeline / Validate - Security (pull_request) Successful in 4m11s
CI/CD Pipeline / Unit Tests (pull_request) Failing after 6m23s
CI/CD Pipeline / CI Gate (pull_request) Failing after 1s
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 3m39s
CI/CD Pipeline / Canary Release to Production (pull_request) Failing after 260h6m22s
CI/CD Pipeline / Build Production Worker Image (pull_request) Failing after 260h6m25s
CI/CD Pipeline / Build Production Web Image (pull_request) Failing after 260h6m25s
CI/CD Pipeline / ACR Image Cleanup (pull_request) Failing after 260h12m32s
CI/CD Pipeline / Staging API Integration Tests (pull_request) Failing after 260h12m33s
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Failing after 260h12m43s
CI/CD Pipeline / Staging E2E Tests (pull_request) Failing after 260h12m33s
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Failing after 260h12m43s
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Failing after 260h12m43s
CI/CD Pipeline / Build Staging Web Image (pull_request) Failing after 260h12m48s
CI/CD Pipeline / Build Staging API Image (pull_request) Failing after 260h12m49s
CI/CD Pipeline / Frontend Lint (pull_request) Failing after 260h12m51s
CI/CD Pipeline / PR Build Web Image (pull_request) Failing after 260h12m49s
CI/CD Pipeline / Check push changed paths (pull_request) Failing after 260h12m57s
CI/CD Pipeline / Deploy Production (pull_request) Failing after 260h40m56s
CI/CD Pipeline / Build Production API Image (pull_request) Failing after 260h40m59s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Failing after 260h47m12s
CI/CD Pipeline / Build Staging Worker Image (pull_request) Failing after 260h47m22s
CI/CD Pipeline / Frontend Unit Tests (pull_request) Failing after 260h47m24s
问题:素材转码与视频生成共用 celery 默认队列、worker 单进程消费,
20+ 转码积压会把用户生成任务堵 40 分钟以上;孤儿清理把任务标 failed
后 Redis 队列消息未作废,消息被重投导致 failed→running 非法转换,
worker 打印 ERROR 后继续产出半成品。

队列隔离:
- 新增 packages/shared/celery_queues.py:generation/transcode/celery
  三队列与 task_routes(generate_video→generation;ingest_asset/
  classify_asset/duplication→transcode),apply_queue_settings()
- worker 入口改双进程:generation worker 独占队列并内嵌 beat
  (prefetch=1, GENERATION_CONCURRENCY 默认 2),transcode worker
  消费 transcode,celery(并发=总-2,最小 1),任一退出则整体终止
- compose/部署脚本/ps1 同步新增 GENERATION_CONCURRENCY 与健康检查

消息作废:
- 新增 packages/shared/celery_orphan_guard.py:终态守卫
  ensure_task_claimable、Redis 队列消息物理清理(JSON 信封解析,
  按业务 id + celery headers.id 双匹配,未命中 rpush 保序)、
  revoke_and_purge(control.revoke + 物理清队列双保险)
- 入队点(生成/上传/分片/重试)send_task 后持久化 celery_task_id
  到 generation_tasks/ingest_jobs(新列,067 迁移,失败仅 warning)
- generate_video/ingest_asset 执行前校验 DB 状态:终态直接 discarded
  不进业务逻辑;mark_processing 返回 False(非法转换)安全中止
- 孤儿/超时清理标 failed 时同时 revoke + 清队列消息
- pending 超时阈值 15→45 分钟,与 running 孤儿(20min)区分

测试:新增 22 个单测(路由表/真实 Redis 消息清理/终态守卫/
非法转换中止/标 failed 后消息不重投/入队持久化),全量
14301 passed;067 迁移隔离 DDL 验证 upgrade/downgrade 通过。
2026-09-05 19:07:47 +08:00
xiaoxia 563131917e fix: Worker 并发默认值从 1 改为 4
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 34s
CI/CD Pipeline / Validate - Type Check (mypy) (pull_request) Successful in 50s
CI/CD Pipeline / Validate - Migration (alembic) (pull_request) Successful in 56s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 1m30s
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 1m48s
AI Code Review / AI Code Review (pull_request) Successful in 1m55s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 2m1s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 2m3s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 3m43s
CI/CD Pipeline / Validate - Code Quality (pull_request) Successful in 4m26s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m17s
CI/CD Pipeline / Unit Tests (pull_request) Successful in 7m2s
CI/CD Pipeline / CI Gate (pull_request) Successful in 55s
ACR Cleanup / ACR Image Cleanup (pull_request_target) Successful in 43s
Preview Cleanup / Cleanup Preview Environment (pull_request) Successful in 1m3s
CI/CD Pipeline / Production Browser E2E (pull_request) Failing after 936h55m54s
CI/CD Pipeline / Canary Release to Production (pull_request) Failing after 936h56m4s
CI/CD Pipeline / Build Production Worker Image (pull_request) Failing after 936h56m6s
CI/CD Pipeline / Deploy Production (pull_request) Failing after 936h56m4s
CI/CD Pipeline / Build Production Web Image (pull_request) Failing after 936h56m6s
CI/CD Pipeline / Build Production API Image (pull_request) Failing after 936h56m7s
CI/CD Pipeline / PR Build Web Image (pull_request) Failing after 937h3m7s
CI/CD Pipeline / Frontend Unit Tests (pull_request) Failing after 937h3m8s
CI/CD Pipeline / ACR Image Cleanup (pull_request) Failing after 937h3m41s
CI/CD Pipeline / Build Staging Worker Image (pull_request) Failing after 937h3m43s
CI/CD Pipeline / Staging API Integration Tests (pull_request) Failing after 937h3m41s
CI/CD Pipeline / Build Staging Web Image (pull_request) Failing after 937h3m43s
CI/CD Pipeline / Staging E2E Tests (pull_request) Failing after 937h3m41s
CI/CD Pipeline / Build Staging API Image (pull_request) Failing after 937h3m43s
CI/CD Pipeline / Frontend Lint (pull_request) Failing after 937h36m32s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Failing after 937h37m6s
deploy-staging-registry.sh / deploy-production.sh / deploy-production-registry.sh
三处 WORKER_CONCURRENCY 统一改为从环境变量读取,默认值 4。

此前硬编码为 1,导致 staging 和 production 各只有 1 个 Worker 并发,
与 PR #1292 的预期(4 并发)不符。
2026-08-08 14:23:12 +08:00
CI Test 7dc92191e0 test: watchtower自动更新验证(仅加注释)
CI/CD Pipeline / Staging E2E Tests (push) Has been cancelled
CI/CD Pipeline / Deploy Staging (push) Failing after 38h42m26s
CI/CD Pipeline / Frontend Lint (push) Failing after 38h56m36s
CI/CD Pipeline / Validate Code Quality And Tests (push) Failing after 38h58m34s
CI/CD Pipeline / Production Browser E2E (push) Failing after 1697h18m26s
CI/CD Pipeline / Build Production Runtime Images (push) Failing after 1697h18m31s
CI/CD Pipeline / Deploy Production (push) Failing after 1697h49m56s
2026-07-07 21:55:55 +08:00
CI Test 5028956cea ci: staging部署接入Watchtower自动更新
CI/CD Pipeline / Staging E2E Tests (push) Has been cancelled
CI/CD Pipeline / Deploy Staging (push) Failing after 39h4m5s
CI/CD Pipeline / Frontend Lint (push) Failing after 39h5m1s
CI/CD Pipeline / Validate Code Quality And Tests (push) Failing after 39h5m1s
CI/CD Pipeline / Production Browser E2E (push) Failing after 1697h40m7s
CI/CD Pipeline / Build Production Runtime Images (push) Failing after 1697h40m11s
CI/CD Pipeline / Deploy Production (push) Failing after 1698h11m37s
- staging构建额外打:staging tag并push到Registry
- staging容器加com.centurylinklabs.watchtower.enable=true标签
- 容器用完整Registry路径+:staging tag启动,供Watchtower监控
- 1分钟轮询,自动检测新镜像并重启
2026-07-07 21:49:28 +08:00
CI Test 1ed0d5aa75 fix: staging deploy script health-check url missing quotes
CI/CD Pipeline / Staging E2E Tests (push) Failing after 39h40m42s
CI/CD Pipeline / Deploy Staging (push) Failing after 39h42m13s
CI/CD Pipeline / Frontend Lint (push) Failing after 39h44m9s
CI/CD Pipeline / Validate Code Quality And Tests (push) Failing after 39h44m58s
CI/CD Pipeline / Production Browser E2E (push) Failing after 1698h18m7s
CI/CD Pipeline / Build Production Runtime Images (push) Failing after 1698h18m14s
CI/CD Pipeline / Deploy Production (push) Failing after 1698h49m40s
2026-07-07 21:09:28 +08:00
CI Test d99ee6fc84 ci: staging deploy改为Registry方式,去掉tar/scp/docker load
CI/CD Pipeline / Deploy Staging (push) Failing after 40h1m36s
CI/CD Pipeline / Frontend Lint (push) Failing after 40h2m31s
CI/CD Pipeline / Validate Code Quality And Tests (push) Failing after 40h2m31s
CI/CD Pipeline / Staging E2E Tests (push) Failing after 1698h36m55s
CI/CD Pipeline / Deploy Production (push) Failing after 1698h37m39s
CI/CD Pipeline / Build Production Runtime Images (push) Failing after 1698h37m42s
CI/CD Pipeline / Production Browser E2E (push) Failing after 1699h9m6s
2026-07-07 20:52:01 +08:00