fix(viral_video): P0图片分析4分钟超时 VLM禁用底层重试+收紧timeout (最坏70s) #2194

Merged
xiaoxia merged 1 commits from fix/vlm-timeout-tune into develop 2026-10-05 15:15:26 +08:00
Owner

P0 根因

用户在staging实测图片分析卡4分钟+。代码层分析发现单图最坏耗时路径:

  • DoubaoClient.max_retries=1(每次VLM请求实际发2次)
  • lite timeout=30s × 2次 = 60s
  • lite失败走 pro_fallback timeout=90s × 2次 = 180s
  • 单图最坏 240s,3图2并发最坏 ~240s+,恰好命中用户描述「4分多钟」

触发场景:方舟lite高峰期偶发超时(30s内无响应)→ httpx层重试1次又等30s → 仍失败走pro→pro在高负载下也可能卡90s→重试又等90s。叠加用户体验极差。

修复

  1. _call() 临时设 client.max_retries=0:单次调用不重复请求,失败直接走外层pro_fallback,避免重试叠加
  2. lite timeout 30s→25s:超时立即降级pro
  3. pro_fallback timeout 90s→45s:pro稳定响应通常<30s,45s足够;90s过长
  4. 单图最坏耗时 25+45=70s,3图2并发最坏约70s(含celery排队)
  5. 每张图VLM调用结束打印 elapsed 日志([爆款视频] 图片 #N VLM(model) 完成 elapsed=Xs),后续便于排查单图慢

单图耗时对照

场景 修复前最坏 修复后最坏
lite一次成功(常态) 30s(含重试) 25s
lite超时→pro一次成功 30×2+90=150s 25+45=70s
lite+pro都超时 30×2+90×2=240s 25+45=70s(整体超时让任务快速失败而不是卡4min)

PR合入后需运维重建staging worker验证。

## P0 根因 用户在staging实测图片分析卡4分钟+。代码层分析发现单图最坏耗时路径: - DoubaoClient.max_retries=1(每次VLM请求实际发2次) - lite timeout=30s × 2次 = 60s - lite失败走 pro_fallback timeout=90s × 2次 = 180s - **单图最坏 240s**,3图2并发最坏 ~240s+,恰好命中用户描述「4分多钟」 触发场景:方舟lite高峰期偶发超时(30s内无响应)→ httpx层重试1次又等30s → 仍失败走pro→pro在高负载下也可能卡90s→重试又等90s。叠加用户体验极差。 ## 修复 1. **_call() 临时设 client.max_retries=0**:单次调用不重复请求,失败直接走外层pro_fallback,避免重试叠加 2. **lite timeout 30s→25s**:超时立即降级pro 3. **pro_fallback timeout 90s→45s**:pro稳定响应通常<30s,45s足够;90s过长 4. **单图最坏耗时 25+45=70s**,3图2并发最坏约70s(含celery排队) 5. **每张图VLM调用结束打印 elapsed 日志**([爆款视频] 图片 #N VLM(model) 完成 elapsed=Xs),后续便于排查单图慢 ## 单图耗时对照 | 场景 | 修复前最坏 | 修复后最坏 | |---|---|---| | lite一次成功(常态)| 30s(含重试) | 25s | | lite超时→pro一次成功 | 30×2+90=150s | 25+45=70s | | lite+pro都超时 | 30×2+90×2=240s | 25+45=70s(整体超时让任务快速失败而不是卡4min)| PR合入后需运维重建staging worker验证。
xiaoxia added 1 commit 2026-10-05 15:15:00 +08:00
fix(viral_video): #2194 VLM单图最坏240s→70s,禁用httpx底层重试+收紧timeout
CI/CD Pipeline / Check if frontend-only change (push) Has been skipped
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (push) Successful in 11s
CI/CD Pipeline / PR Build API Image (push) Has been skipped
CI/CD Pipeline / PR Build Web Image (push) Has been skipped
CI/CD Pipeline / PR Build Worker Image (push) Has been skipped
CI/CD Pipeline / Check push changed paths (push) Successful in 16s
ACR Cleanup / ACR Image Cleanup (pull_request_target) Successful in 24s
Preview Cleanup / Cleanup Preview Environment (pull_request) Successful in 35s
CI/CD Pipeline / Frontend Lint (push) Has been skipped
CI/CD Pipeline / Validate - Python (mypy + alembic) (push) Successful in 3m26s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 4m37s
CI/CD Pipeline / Validate - Security (push) Has been cancelled
CI/CD Pipeline / Unit Tests (push) Has been cancelled
CI/CD Pipeline / Retag skipped Staging API Image (push) Has been cancelled
CI/CD Pipeline / Retag skipped Staging Web Image (push) Has been cancelled
CI/CD Pipeline / Retag skipped Staging Worker Image (push) Has been cancelled
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (push) Has been cancelled
CI/CD Pipeline / Staging E2E Tests (push) Has been cancelled
CI/CD Pipeline / Build Staging API Image (push) Has been cancelled
CI/CD Pipeline / Build Staging Web Image (push) Has been cancelled
CI/CD Pipeline / Build Staging Worker Image (push) Has been cancelled
CI/CD Pipeline / Staging API Integration Tests (push) Has been cancelled
CI/CD Pipeline / Build Production API Image (push) Has been cancelled
CI/CD Pipeline / Build Production Web Image (push) Has been cancelled
CI/CD Pipeline / Build Production Worker Image (push) Has been cancelled
CI/CD Pipeline / Deploy Production (push) Has been cancelled
CI/CD Pipeline / Production Browser E2E (push) Has been cancelled
CI/CD Pipeline / ACR Image Cleanup (push) Has been cancelled
CI/CD Pipeline / Canary Release to Production (push) Has been cancelled
CI/CD Pipeline / CI Gate (push) Has been cancelled
CI/CD Pipeline / Validate - Style (push) Has been cancelled
CI/CD Pipeline / Integration Tests (push) Has been cancelled
CI/CD Pipeline / Frontend Unit Tests (push) Has been cancelled
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 1m18s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 3m12s
AI Code Review / AI Code Review (pull_request) Successful in 6m54s
CI/CD Pipeline / Validate - Security (pull_request) Successful in 8m3s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 2s
CI/CD Pipeline / Unit Tests (pull_request) Failing after 3m4s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 3m42s
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 0s
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Validate - Style (pull_request) Failing after 5m30s
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 47s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 57s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 6m10s
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
CI/CD Pipeline / Build Production API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI/CD Pipeline / CI Gate (pull_request) Failing after 2s
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / Canary Release to Production (pull_request) Has been skipped
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
fbc1df36e0
P0根因:用户实测图片分析卡4分钟+。代码分析发现:
- _analyze_single_image 底层 DoubaoClient.max_retries=1(共2次请求)
- lite timeout=30s × 2次重试 = 60s
- lite失败后 pro_fallback timeout=90s × 2次重试 = 180s
- 单图最坏 60+180 = 240s,3图2并发最坏 ~240s+,恰好命中用户4分钟描述

修复:
1. _call() 临时将 client.max_retries=0 单次调用,不重复请求
2. lite timeout 30s → 25s(超时立即降级pro)
3. pro fallback timeout 90s → 45s(pro稳定响应<30s)
4. 单图最坏 25+45=70s,3图2并发最坏约70s(含排队)
5. 每张图VLM调用结束打印elapsed日志,方便后续排查
xiaoxia merged commit fbc1df36e0 into develop 2026-10-05 15:15:26 +08:00

🗑️ 预览环境已清理

PR #2194 已关闭或合并,对应的预览环境已被清理。

如有需要,可以重新打开 PR 来重新生成预览环境。

🗑️ **预览环境已清理** PR #2194 已关闭或合并,对应的预览环境已被清理。 > 如有需要,可以重新打开 PR 来重新生成预览环境。

🚀 预览环境已部署

项目 详情
PR号 #2194
预览链接 https://pr-2194.preview.xiaoxiajianji.com
API环境 staging

💡 预览环境使用 staging API 数据,请勿在预览环境中操作重要数据。

🔄 每次提交新代码后预览环境会自动更新。

🗑️ PR 关闭或合并后,预览环境会自动清理。

🚀 **预览环境已部署** | 项目 | 详情 | |------|------| | PR号 | #2194 | | 预览链接 | [https://pr-2194.preview.xiaoxiajianji.com](https://pr-2194.preview.xiaoxiajianji.com) | | API环境 | staging | > 💡 预览环境使用 staging API 数据,请勿在预览环境中操作重要数据。 > > 🔄 每次提交新代码后预览环境会自动更新。 > > 🗑️ PR 关闭或合并后,预览环境会自动清理。
Sign in to join this conversation.