bba8beffef6beee8d35fc1a8cd806b29261e01ac
9 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
29ca51da9c |
fix(web): 烤入所有 nginx 配置,按 APP_ENV 运行时切换,彻底解决 staging/prod 配置错乱 (#1853)
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (push) Successful in 1s
CI/CD Pipeline / Check push changed paths (push) Successful in 18s
CI/CD Pipeline / Build Staging Worker Image (push) Successful in 14s
CI/CD Pipeline / Build Staging API Image (push) Successful in 1m0s
CI/CD Pipeline / Integration Tests (push) Successful in 1m32s
CI/CD Pipeline / Validate - Python (mypy + alembic) (push) Successful in 1m40s
CI/CD Pipeline / Validate - Style (push) Successful in 2m15s
CI/CD Pipeline / Build Staging Web Image (push) Successful in 2m16s
CI/CD Pipeline / Frontend Unit Tests (push) Successful in 4m44s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (push) Failing after 2m12s
CI/CD Pipeline / Validate - Security (push) Successful in 5m30s
CI/CD Pipeline / Unit Tests (push) Successful in 7m44s
CI/CD Pipeline / Production Browser E2E (push) Has been skipped
CI/CD Pipeline / Build Production Worker Image (push) Failing after 126h13m58s
CI/CD Pipeline / Retag skipped Staging Worker Image (push) Failing after 126h19m7s
CI/CD Pipeline / PR Build Worker Image (push) Failing after 126h21m46s
CI/CD Pipeline / PR Build Web Image (push) Failing after 126h21m33s
CI/CD Pipeline / PR Build API Image (push) Failing after 126h21m34s
CI/CD Pipeline / Retag skipped Staging Web Image (push) Failing after 126h18m54s
CI/CD Pipeline / Retag skipped Staging API Image (push) Failing after 126h18m54s
CI/CD Pipeline / CI Gate (push) Failing after 126h13m45s
CI/CD Pipeline / Build Production API Image (push) Failing after 126h13m45s
CI/CD Pipeline / Check if frontend-only change (push) Failing after 126h21m35s
CI/CD Pipeline / Deploy Production (push) Failing after 126h13m42s
CI/CD Pipeline / Canary Release to Production (push) Failing after 126h13m42s
CI/CD Pipeline / Build Production Web Image (push) Failing after 126h13m45s
CI/CD Pipeline / Staging API Integration Tests (push) Failing after 126h16m37s
CI/CD Pipeline / Staging E2E Tests (push) Failing after 126h16m37s
CI/CD Pipeline / Frontend Lint (push) Failing after 126h21m29s
CI/CD Pipeline / ACR Image Cleanup (push) Failing after 126h51m43s
Co-authored-by: xiaoxia <dev@xiaoxiajianji.com> Co-committed-by: xiaoxia <dev@xiaoxiajianji.com> |
||
|
|
54defa4429 |
fix: Worker health check 扫描全部 /proc 进程修复 unhealthy 误判
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 1m1s
AI Code Review / AI Code Review (pull_request) Successful in 6m36s
PR Automation / Auto Approve on CI Green (pull_request) Failing after 13m19s
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 1s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 1s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 15s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 16s
ACR Cleanup / ACR Image Cleanup (pull_request_target) Successful in 11s
Preview Cleanup / Cleanup Preview Environment (pull_request) Successful in 30s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m16s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 1m58s
CI/CD Pipeline / Validate - Style (pull_request) Successful in 2m53s
CI/CD Pipeline / Validate - Security (pull_request) Successful in 6m33s
CI/CD Pipeline / Unit Tests (pull_request) Successful in 12m20s
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / CI Gate (pull_request) Successful in 6s
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Failing after 164h14m54s
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Failing after 164h14m54s
CI/CD Pipeline / Frontend Lint (pull_request) Failing after 164h14m55s
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Failing after 164h14m54s
CI/CD Pipeline / Build Staging Worker Image (pull_request) Failing after 164h14m57s
CI/CD Pipeline / Check push changed paths (pull_request) Failing after 164h15m0s
CI/CD Pipeline / Canary Release to Production (pull_request) Failing after 164h2m30s
CI/CD Pipeline / Build Staging Web Image (pull_request) Failing after 164h14m57s
CI/CD Pipeline / Build Production Worker Image (pull_request) Failing after 164h2m35s
CI/CD Pipeline / Staging E2E Tests (pull_request) Failing after 164h14m48s
CI/CD Pipeline / Build Production API Image (pull_request) Failing after 164h2m36s
PR Automation / Auto Merge on CI Green + Approved (pull_request) Failing after 164h27m44s
CI/CD Pipeline / ACR Image Cleanup (pull_request) Failing after 164h14m48s
CI/CD Pipeline / Build Production Web Image (pull_request) Failing after 164h37m21s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Failing after 164h49m35s
CI/CD Pipeline / Frontend Unit Tests (pull_request) Failing after 164h49m41s
CI/CD Pipeline / Build Staging API Image (pull_request) Failing after 164h49m43s
CI/CD Pipeline / Deploy Production (pull_request) Failing after 164h37m16s
CI/CD Pipeline / Staging API Integration Tests (pull_request) Failing after 164h49m34s
CI/CD Pipeline / PR Build Web Image (pull_request) Failing after 164h49m41s
问题:Worker 容器内无 pgrep/ps 工具,旧 health check 使用 'grep -q celery /proc/1/cmdline' 检查 PID 1 的 cmdline, 但 PID 1 是 entrypoint-worker.sh(bash 脚本),不含 celery 关键字, 导致 health check 永远返回 unhealthy。 同时部署脚本中 sh -c 嵌套引号导致 $pid 变量被外层 shell 提前展开为空。 修复:改用 grep -lq celery /proc/[0-9]*/cmdline 扫描所有进程 cmdline, 找到任意 celery 进程即判定 healthy。无需变量、无嵌套 shell、无 pgrep 依赖。 影响文件: - infra/docker/compose.yml - infra/docker/deploy-staging-registry.sh - infra/docker/deploy-production-registry.sh |
||
|
|
e4ac398397 |
fix: Worker 健康检查彻底修复 - 移除不可用的 pgrep 和 PID1 检查
根因:#1714 队列隔离后 entrypoint 启动两个 celery worker 子进程, PID 1 变为 bash。旧健康检查依赖 pgrep(容器内不存在)或 /proc/1/cmdline(PID 1 是 bash 不是 celery),永远返回失败。 此前靠 cron 脚本 fix-worker-healthcheck.sh 每 30 分钟擦屁股。 修复:统一改为扫描 /proc/[0-9]*/cmdline 查找 celery 进程, 覆盖所有部署/回滚/CI 脚本(10 处替换)。已移除 cron 补丁。 |
||
|
|
df99305dd6 |
feat(worker): celery 队列隔离 + 孤儿任务消息作废 (#1714)
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 1s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 2s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 29s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 29s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 49s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 1m39s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m44s
CI/CD Pipeline / Validate - Style (pull_request) Successful in 2m19s
AI Code Review / AI Code Review (pull_request) Failing after 2m52s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 2m54s
CI/CD Pipeline / Validate - Security (pull_request) Successful in 4m11s
CI/CD Pipeline / Unit Tests (pull_request) Failing after 6m23s
CI/CD Pipeline / CI Gate (pull_request) Failing after 1s
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 3m39s
CI/CD Pipeline / Canary Release to Production (pull_request) Failing after 260h6m22s
CI/CD Pipeline / Build Production Worker Image (pull_request) Failing after 260h6m25s
CI/CD Pipeline / Build Production Web Image (pull_request) Failing after 260h6m25s
CI/CD Pipeline / ACR Image Cleanup (pull_request) Failing after 260h12m32s
CI/CD Pipeline / Staging API Integration Tests (pull_request) Failing after 260h12m33s
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Failing after 260h12m43s
CI/CD Pipeline / Staging E2E Tests (pull_request) Failing after 260h12m33s
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Failing after 260h12m43s
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Failing after 260h12m43s
CI/CD Pipeline / Build Staging Web Image (pull_request) Failing after 260h12m48s
CI/CD Pipeline / Build Staging API Image (pull_request) Failing after 260h12m49s
CI/CD Pipeline / Frontend Lint (pull_request) Failing after 260h12m51s
CI/CD Pipeline / PR Build Web Image (pull_request) Failing after 260h12m49s
CI/CD Pipeline / Check push changed paths (pull_request) Failing after 260h12m57s
CI/CD Pipeline / Deploy Production (pull_request) Failing after 260h40m56s
CI/CD Pipeline / Build Production API Image (pull_request) Failing after 260h40m59s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Failing after 260h47m12s
CI/CD Pipeline / Build Staging Worker Image (pull_request) Failing after 260h47m22s
CI/CD Pipeline / Frontend Unit Tests (pull_request) Failing after 260h47m24s
问题:素材转码与视频生成共用 celery 默认队列、worker 单进程消费, 20+ 转码积压会把用户生成任务堵 40 分钟以上;孤儿清理把任务标 failed 后 Redis 队列消息未作废,消息被重投导致 failed→running 非法转换, worker 打印 ERROR 后继续产出半成品。 队列隔离: - 新增 packages/shared/celery_queues.py:generation/transcode/celery 三队列与 task_routes(generate_video→generation;ingest_asset/ classify_asset/duplication→transcode),apply_queue_settings() - worker 入口改双进程:generation worker 独占队列并内嵌 beat (prefetch=1, GENERATION_CONCURRENCY 默认 2),transcode worker 消费 transcode,celery(并发=总-2,最小 1),任一退出则整体终止 - compose/部署脚本/ps1 同步新增 GENERATION_CONCURRENCY 与健康检查 消息作废: - 新增 packages/shared/celery_orphan_guard.py:终态守卫 ensure_task_claimable、Redis 队列消息物理清理(JSON 信封解析, 按业务 id + celery headers.id 双匹配,未命中 rpush 保序)、 revoke_and_purge(control.revoke + 物理清队列双保险) - 入队点(生成/上传/分片/重试)send_task 后持久化 celery_task_id 到 generation_tasks/ingest_jobs(新列,067 迁移,失败仅 warning) - generate_video/ingest_asset 执行前校验 DB 状态:终态直接 discarded 不进业务逻辑;mark_processing 返回 False(非法转换)安全中止 - 孤儿/超时清理标 failed 时同时 revoke + 清队列消息 - pending 超时阈值 15→45 分钟,与 running 孤儿(20min)区分 测试:新增 22 个单测(路由表/真实 Redis 消息清理/终态守卫/ 非法转换中止/标 failed 后消息不重投/入队持久化),全量 14301 passed;067 迁移隔离 DDL 验证 upgrade/downgrade 通过。 |
||
|
|
563131917e |
fix: Worker 并发默认值从 1 改为 4
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 34s
CI/CD Pipeline / Validate - Type Check (mypy) (pull_request) Successful in 50s
CI/CD Pipeline / Validate - Migration (alembic) (pull_request) Successful in 56s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 1m30s
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 1m48s
AI Code Review / AI Code Review (pull_request) Successful in 1m55s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 2m1s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 2m3s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 3m43s
CI/CD Pipeline / Validate - Code Quality (pull_request) Successful in 4m26s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m17s
CI/CD Pipeline / Unit Tests (pull_request) Successful in 7m2s
CI/CD Pipeline / CI Gate (pull_request) Successful in 55s
ACR Cleanup / ACR Image Cleanup (pull_request_target) Successful in 43s
Preview Cleanup / Cleanup Preview Environment (pull_request) Successful in 1m3s
CI/CD Pipeline / Production Browser E2E (pull_request) Failing after 936h55m54s
CI/CD Pipeline / Canary Release to Production (pull_request) Failing after 936h56m4s
CI/CD Pipeline / Build Production Worker Image (pull_request) Failing after 936h56m6s
CI/CD Pipeline / Deploy Production (pull_request) Failing after 936h56m4s
CI/CD Pipeline / Build Production Web Image (pull_request) Failing after 936h56m6s
CI/CD Pipeline / Build Production API Image (pull_request) Failing after 936h56m7s
CI/CD Pipeline / PR Build Web Image (pull_request) Failing after 937h3m7s
CI/CD Pipeline / Frontend Unit Tests (pull_request) Failing after 937h3m8s
CI/CD Pipeline / ACR Image Cleanup (pull_request) Failing after 937h3m41s
CI/CD Pipeline / Build Staging Worker Image (pull_request) Failing after 937h3m43s
CI/CD Pipeline / Staging API Integration Tests (pull_request) Failing after 937h3m41s
CI/CD Pipeline / Build Staging Web Image (pull_request) Failing after 937h3m43s
CI/CD Pipeline / Staging E2E Tests (pull_request) Failing after 937h3m41s
CI/CD Pipeline / Build Staging API Image (pull_request) Failing after 937h3m43s
CI/CD Pipeline / Frontend Lint (pull_request) Failing after 937h36m32s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Failing after 937h37m6s
deploy-staging-registry.sh / deploy-production.sh / deploy-production-registry.sh 三处 WORKER_CONCURRENCY 统一改为从环境变量读取,默认值 4。 此前硬编码为 1,导致 staging 和 production 各只有 1 个 Worker 并发, 与 PR #1292 的预期(4 并发)不符。 |
||
|
|
7dc92191e0 |
test: watchtower自动更新验证(仅加注释)
CI/CD Pipeline / Staging E2E Tests (push) Has been cancelled
CI/CD Pipeline / Deploy Staging (push) Failing after 38h42m26s
CI/CD Pipeline / Frontend Lint (push) Failing after 38h56m36s
CI/CD Pipeline / Validate Code Quality And Tests (push) Failing after 38h58m34s
CI/CD Pipeline / Production Browser E2E (push) Failing after 1697h18m26s
CI/CD Pipeline / Build Production Runtime Images (push) Failing after 1697h18m31s
CI/CD Pipeline / Deploy Production (push) Failing after 1697h49m56s
|
||
|
|
5028956cea |
ci: staging部署接入Watchtower自动更新
CI/CD Pipeline / Staging E2E Tests (push) Has been cancelled
CI/CD Pipeline / Deploy Staging (push) Failing after 39h4m5s
CI/CD Pipeline / Frontend Lint (push) Failing after 39h5m1s
CI/CD Pipeline / Validate Code Quality And Tests (push) Failing after 39h5m1s
CI/CD Pipeline / Production Browser E2E (push) Failing after 1697h40m7s
CI/CD Pipeline / Build Production Runtime Images (push) Failing after 1697h40m11s
CI/CD Pipeline / Deploy Production (push) Failing after 1698h11m37s
- staging构建额外打:staging tag并push到Registry - staging容器加com.centurylinklabs.watchtower.enable=true标签 - 容器用完整Registry路径+:staging tag启动,供Watchtower监控 - 1分钟轮询,自动检测新镜像并重启 |
||
|
|
1ed0d5aa75 |
fix: staging deploy script health-check url missing quotes
CI/CD Pipeline / Staging E2E Tests (push) Failing after 39h40m42s
CI/CD Pipeline / Deploy Staging (push) Failing after 39h42m13s
CI/CD Pipeline / Frontend Lint (push) Failing after 39h44m9s
CI/CD Pipeline / Validate Code Quality And Tests (push) Failing after 39h44m58s
CI/CD Pipeline / Production Browser E2E (push) Failing after 1698h18m7s
CI/CD Pipeline / Build Production Runtime Images (push) Failing after 1698h18m14s
CI/CD Pipeline / Deploy Production (push) Failing after 1698h49m40s
|
||
|
|
d99ee6fc84 |
ci: staging deploy改为Registry方式,去掉tar/scp/docker load
CI/CD Pipeline / Deploy Staging (push) Failing after 40h1m36s
CI/CD Pipeline / Frontend Lint (push) Failing after 40h2m31s
CI/CD Pipeline / Validate Code Quality And Tests (push) Failing after 40h2m31s
CI/CD Pipeline / Staging E2E Tests (push) Failing after 1698h36m55s
CI/CD Pipeline / Deploy Production (push) Failing after 1698h37m39s
CI/CD Pipeline / Build Production Runtime Images (push) Failing after 1698h37m42s
CI/CD Pipeline / Production Browser E2E (push) Failing after 1699h9m6s
|