ce943faaeb4b3a6317c44cd70642ca6607602166
5 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 saas-backend-agent
|
ce943faaeb |
feat(generation): worker孤儿任务自动恢复+429限流结构化提示
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 2s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 2s
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 26s
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 36s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 1m25s
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 1m41s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m52s
CI/CD Pipeline / Validate - Style (pull_request) Successful in 2m15s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 3m9s
CI/CD Pipeline / Validate - Security (pull_request) Successful in 5m16s
AI Code Review / AI Code Review (pull_request) Successful in 6m21s
CI/CD Pipeline / Unit Tests (pull_request) Successful in 10m15s
CI/CD Pipeline / Build Production API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / Canary Release to Production (pull_request) Has been skipped
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / CI Gate (pull_request) Successful in 4s
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 7m53s
ACR Cleanup / ACR Image Cleanup (pull_request_target) Successful in 8s
Preview Cleanup / Cleanup Preview Environment (pull_request) Successful in 24s
问题：
- worker容器部署重启时，正在跑的preview/正式生成任务永久卡在pending/running
（实测3个预览任务卡80%超10小时），孤儿任务占限流名额导致新请求429误报创建失败
- 429返回纯文本，前端无法区分「限流排队」和「创建失败」
改动：
1. worker孤儿任务巡检(apps/worker/worker_app/tasks/)
- running孤儿清理从仅worker_ready启动时执行，扩展为beat每5分钟定时巡检
(cleanup_stale_running_tasks)，容器重启/进程OOM卡死持续兜底
- 阈值: running 10→20分钟(任务硬超时11分钟,20分钟绝不误杀);
pending 30→15分钟(满队列消化约10分钟,留余量)
- pending beat巡检10→5分钟,孤儿占位释放更快
- 失败原因结构化: error_type=WorkerInterrupted/PendingTimeout
2. 429/503限流结构化提示(app/core/task_enqueue.py)
- build_rate_limit_detail(): code/message/queued_count/running_count/
queue_ahead/estimated_wait_seconds/limit
- code: USER_QUEUE_FULL(429排队等待) / SYSTEM_QUEUE_FULL(503系统繁忙)
- 等待预估: ceil(排队数/worker并发4) × 最近20条完成任务平均耗时
- 预览/正式生成/确认生成/重试共7处限流响应全部结构化
- 新仓储方法鸭子类型防御,旧mock/调用方零破坏
3. 仓储新增: count_running_by_user/count_running_total/estimate_avg_duration_seconds
测试: 新增15个单测(中断running重置、stale pending重置释放名额、3孤儿批量恢复、
限流计数含预览任务、结构化detail字段、等待预估并发计算、旧仓储优雅降级);
全量14241 passed; black/isort/ruff/mypy通过
|
||
|
CI Bot
|
e9d2831850 |
feat(ci): mypy upgrade to hard gate + fix type errors
CI Build & Deploy Pipeline / Build Staging API Image (push) Has been skipped
CI Build & Deploy Pipeline / Build Staging Web Image (push) Has been skipped
CI Build & Deploy Pipeline / Build Staging Worker Image (push) Has been skipped
CI Build & Deploy Pipeline / Build Staging API Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Deploy Staging (Watchtower auto-deploy) (push) Has been skipped
CI Build & Deploy Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Production API Image (push) Has been skipped
CI Build & Deploy Pipeline / Build Production Web Image (push) Has been skipped
CI Build & Deploy Pipeline / Build Production Worker Image (push) Has been skipped
CI Build & Deploy Pipeline / Staging E2E Tests (push) Has been skipped
CI Build & Deploy Pipeline / Staging API Integration Tests (push) Has been skipped
CI Build & Deploy Pipeline / Deploy Production (push) Has been skipped
CI Build & Deploy Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI Build & Deploy Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI Build & Deploy Pipeline / Production Browser E2E (push) Has been skipped
CI/CD Pipeline / Frontend Lint (push) Successful in 1m33s
CI/CD Pipeline / Frontend Lint (pull_request) Successful in 1m30s
CI Build & Deploy Pipeline / Build Production API Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Production Web Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Deploy Production (pull_request) Has been skipped
CI Build & Deploy Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / Unit Tests (push) Successful in 3m13s
CI/CD Pipeline / Unit Tests (pull_request) Successful in 3m18s
CI/CD Pipeline / Validate Code Quality And Tests (pull_request) Failing after 16m16s
CI/CD Pipeline / Validate Code Quality And Tests (push) Failing after 16m20s
CI/CD Pipeline / Integration Tests (push) Successful in 2m30s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 2m24s
|
||
 xiaoxia
|
ad671d94c5 |
fix(voice-clone): source_audio_url 不做预签名转换，原样返回用户输入
CI/CD Pipeline / Validate Code Quality And Tests (push) Successful in 1m25s
CI/CD Pipeline / Frontend Lint (push) Successful in 2m56s
CI/CD Pipeline / Build Production Runtime Images (push) Has been skipped
CI/CD Pipeline / Deploy Production (push) Has been skipped
CI/CD Pipeline / Production Browser E2E (push) Has been skipped
CI/CD Pipeline / Build & Push Staging (Watchtower auto-deploy) (push) Successful in 6m40s
CI/CD Pipeline / Staging E2E Tests (push) Successful in 4m30s
CI/CD Pipeline / Staging API Integration Tests (push) Successful in 5m35s
fix(voice-clone): source_audio_url 不做预签名转换，原样返回用户输入 |
||
|
xiaoxia
|
efe7f6b52a |
feat: 任务队列限流防护
feat: 任务队列限流防护 - 用户级 pending 限流（3 个上限，返回 429） - 全局队列保护（20 个上限，返回 503） - 4 处入口全覆盖：批量创建、generation 重试、任务中心两级重试 - 入队前预检查 + 入队后最终校验双层兜底，处理并发竞态 - 阈值统一常量管理，边界语义一致 - 350+ 行单元测试覆盖所有边界场景 |
||
|
用户CI Test
|
9aa450bb8b |
refactor: 安全入队函数抽为公共工具，消除重复定义
CI/CD Pipeline / Validate Code Quality And Tests (push) Failing after 8s
CI/CD Pipeline / Frontend Lint (push) Successful in 1m9s
CI/CD Pipeline / Build & Push Staging (Watchtower auto-deploy) (push) Has been skipped
CI/CD Pipeline / Build Production Runtime Images (push) Has been skipped
CI/CD Pipeline / Staging E2E Tests (push) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (push) Has been skipped
CI/CD Pipeline / Deploy Production (push) Has been skipped
CI/CD Pipeline / Production Browser E2E (push) Has been skipped
- 新增 app/core/task_enqueue.py，统一 safe_enqueue_generation_task 实现 - generation_tasks.py 和 task_center.py 改为 import 公共函数 - 修复 task_center.py 缺失 logging 导入的问题 - 日志前缀通过参数传入，保持各模块语义不变 |