fix(viral-video): P0 渲染卡死 + P1 VLM lite 快速降级(v1.6.1)
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 1s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 2s
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
PR Automation / Auto Approve on CI Green (pull_request) Successful in 3m3s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 3m2s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 3m40s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 3m52s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 5m31s
CI/CD Pipeline / Unit Tests (pull_request) Has been cancelled
CI/CD Pipeline / Build Production API Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Web Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been cancelled
CI/CD Pipeline / Deploy Production (pull_request) Has been cancelled
CI/CD Pipeline / Production Browser E2E (pull_request) Has been cancelled
CI/CD Pipeline / Canary Release to Production (pull_request) Has been cancelled
CI/CD Pipeline / CI Gate (pull_request) Has been cancelled
CI/CD Pipeline / Validate - Style (pull_request) Has been cancelled
CI/CD Pipeline / Validate - Security (pull_request) Has been cancelled
CI/CD Pipeline / Integration Tests (pull_request) Has been cancelled
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 3m27s
AI Code Review / AI Code Review (pull_request) Has been cancelled

P0 根因修复(v9 渲染卡死 20+ min):
1. Seedance 2.5 content 数组协议错误:之前把参考音频/视频放在顶层 reference_audios/reference_videos(非官方字段),
   且首帧图不带 role 导致服务端识别为 first_frame 模式(该模式强制 ratio=adaptive),传入 ratio=9:16 导致任务
   静默排队/失败轮询到 900s 超时。
2. 修复:所有参考(图/音/视)放进 content 数组并带正确 role(reference_image/reference_audio/reference_video);
   首张图根据是否有额外参考自动选择 first_frame vs reference_image 模式。
3. ratio 智能选择:纯首帧无参考→adaptive;omni_reference→按用户 ratio(默认 9:16);
   若创建任务因 ratio 返回 HTTP 400,自动回退 adaptive 重试一次。
4. 轮询 total_timeout 从 900s 收紧到 480s(8min),下载超时 120s;
   celery 任务加 soft_time_limit(render 660s/analyze 240s/generate_copy 180s/pipeline 900s)防挂死。
5. 轮询每 5 次打一次 info 日志,失败时完整打印 error.code/message。
6. _hb_stop 提前声明避免 finally UnboundLocalError。

P1 VLM lite 快速降级:
- lite timeout 从 45s 收紧到 15s,pro 降级固定 25s,总耗时 ≤40s(之前 45+60≈100s)。
- 日志:创建任务时打印 mode(first_frame/omni_ref)、ratio、参考数量。
This commit is contained in:
xiaoxia
2026-10-02 16:34:56 +08:00
parent d707b64876
commit 70aeb5e642
3 changed files with 157 additions and 122 deletions
+25 -6
View File
@@ -1185,13 +1185,19 @@ def _wait_oss_ready(url: str, timeout_sec: int = 10) -> bool:
# ── 主编排器 ────────────────────────────────────────────────────────────
@shared_task(bind=True, max_retries=2, name="worker.run_viral_video_pipeline")
@shared_task(
bind=True,
max_retries=2,
name="worker.run_viral_video_pipeline",
soft_time_limit=900, # 15min(完整流水线)
time_limit=960,
)
def run_viral_video_pipeline(self: Task, job_id: str) -> dict:
"""旧一键流水线(保留兼容):图片分析→风格分析→意图解析→WAIT_USER_CONFIRM。"""
session = None
_hb_stop = None
try:
_recover_stale_jobs() # 顺带回收僵尸任务
_hb_stop = None
session, repo, job = _get_repo_and_job(job_id)
if job is None:
return {"ok": False, "error": "job not found"}
@@ -1324,13 +1330,19 @@ def _mark_failed_and_notify(job_id: str, session, repo, job, err_msg: str, stage
# ── v1.5/v1.6 三步分步流水线 Celery 任务 ─────────────────────────────────
@shared_task(bind=True, max_retries=1, name="worker.run_viral_video_analyze")
@shared_task(
bind=True,
max_retries=1,
name="worker.run_viral_video_analyze",
soft_time_limit=240, # 4min(VLM 并行分析)
time_limit=300,
)
def run_viral_video_analyze(self: Task, job_id: str) -> dict:
"""v1.5+ 阶段1:图片 VLM 分析 + 可选视频风格分析。"""
session = None
_hb_stop = None
try:
_recover_stale_jobs() # 顺带回收僵尸任务
_hb_stop = None
session, repo, job = _get_repo_and_job(job_id)
if job is None:
return {"ok": False, "error": "job not found"}
@@ -1386,7 +1398,13 @@ def run_viral_video_analyze(self: Task, job_id: str) -> dict:
session.close()
@shared_task(bind=True, max_retries=1, name="worker.run_viral_video_generate_copy")
@shared_task(
bind=True,
max_retries=1,
name="worker.run_viral_video_generate_copy",
soft_time_limit=180, # 3min(编导脚本生成含三级重试)
time_limit=240,
)
def run_viral_video_generate_copy(self: Task, job_id: str) -> dict:
"""v1.6 阶段2(v1.6.1 提速版):意图解析 → 编导分镜脚本生成 → 直接返回,合规审核后置到出片前。
@@ -1395,11 +1413,12 @@ def run_viral_video_generate_copy(self: Task, job_id: str) -> dict:
- _SCRIPT_GENERATION_PROMPT 精简冗余描述
- 合规审核改为异步后置:不阻塞前端,在 confirm-copy(阶段3 TTS前)再做最终审核
- 每个阶段通过 _set_stage 持久化 current_stage/phase_message 到 DB(问题8)
- v1.6.1: lite VLM 15s 快速失败,pro 25s;三级编导重试
"""
session = None
_hb_stop = None
try:
_recover_stale_jobs() # 顺带回收僵尸任务
_hb_stop = None
session, repo, job = _get_repo_and_job(job_id)
if job is None:
return {"ok": False, "error": "job not found"}
+125 -108
View File
@@ -262,26 +262,13 @@ class DoubaoClient:
reference_audios: list[str] | None = None,
reference_videos: list[str] | None = None,
) -> str | None:
"""调用 Seedance 2.5 文生/图生视频(异步任务→轮询→下载),返回本地 MP4 路径;失败返回 None。
"""调用 Seedance 2.5 生视频(异步任务→轮询→下载),返回本地 MP4 路径;失败返回 None。
v1.6: 支持多参考图(产品素材)+ 参考音频(TTS口型驱动)+ 参考视频,单次生成最长 30 秒。
Args:
prompt: 文本提示词(含完整编导脚本:总览+场景光线+逐镜头时间轴+硬约束+负面词)
image_url: 首帧参考图 URL(可选,提供则走图生视频首帧模式,ratio 仍按参数传,模型会自动居中裁剪到目标比例)
duration: 视频时长 4~30 秒
ratio: 宽高比 16:9/9:16/1:1/4:3/3:4/21:9/adaptive;图生视频时模型会自动居中裁剪首帧到目标比例
resolution: 480p/720p/1080p
generate_audio: 是否让模型原生合成音效/BGM(v1.6 默认 True,配合 reference_audios 做口型驱动)
watermark: 是否加水印
output_dir: 下载目录,默认 /tmp
model: 指定模型 ID;空则用 settings.doubao_video_model
reference_images: 多参考图 URL 列表(产品素材,最多30张;注意 image_url 为首帧单独传)
reference_audios: 参考音频 URL 列表(TTS口播,驱动口型,最多10段)
reference_videos: 参考视频 URL 列表(风格参考)
Returns:
本地 MP4 文件路径,失败返回 None。
【v1.6.1 修复】严格按官方 content 数组协议构造请求:
- 所有参考(图/音/视)必须放进 content 数组并带 role 字段,不能放顶层 reference_audios/reference_videos(非官方字段,会被忽略或导致异常)。
- 首帧图(first_frame 模式)Seedance 2.5 强制 ratio=adaptive;走 omni_reference(参考生视频)模式时才能指定 9:16/1:1 等具体比例。
判定:传了参考音频/视频或 ≥1 张多参考图时,走 omni_reference(首张图 role=reference_image);纯首帧无参考时走 first_frame(ratio 强制 adaptive)。
- 创建任务若因 ratio 报错(HTTP 400),自动回退到 ratio=adaptive 重试一次。
"""
if not self.is_available:
return None
@@ -290,18 +277,44 @@ class DoubaoClient:
settings = get_shared_settings()
poll_interval = getattr(settings, "doubao_video_poll_interval", 10) or 10
total_timeout = getattr(settings, "doubao_video_timeout", 900) or 900
# 收紧总超时:轮询 8min + 下载 2min = 最长 ~10min,防止出现 20min 卡死
total_timeout = getattr(settings, "doubao_video_timeout", 480) or 480
default_video_model = getattr(settings, "doubao_video_model", None) or "doubao-seedance-2-5-260628"
video_model = model or default_video_model
ref_audios = [u for u in (reference_audios or [])[:10] if u and isinstance(u, str)]
ref_videos = [u for u in (reference_videos or [])[:3] if u and isinstance(u, str)]
ref_imgs = [u for u in (reference_images or [])[:9] if u and isinstance(u, str)]
# 判断任务模式:有参考音/视/多图 → omni_reference(支持指定 ratio);纯首帧 → first_frame(ratio=adaptive)
has_extra_refs = bool(ref_audios or ref_videos or ref_imgs)
is_first_frame_mode = bool(image_url) and not has_extra_refs
# 最终 ratio:first_frame 模式强制 adaptive,否则按用户传值(默认 9:16)
final_ratio = "adaptive" if is_first_frame_mode else (ratio or "9:16")
# 构造 content 数组:text + 图 + 音 + 视
content: list[dict[str, Any]] = [{"type": "text", "text": prompt.strip()}]
if image_url:
content.append({"type": "image_url", "image_url": {"url": image_url}})
# v1.6: 多参考图(产品素材)
if reference_images:
for url in reference_images[:30]:
if url and isinstance(url, str):
content.append({"type": "image_url", "image_url": {"url": url}})
if has_extra_refs:
# omni_reference:首张图作为 reference_image,允许指定 ratio
content.append({
"type": "image_url",
"image_url": {"url": image_url},
"role": "reference_image",
})
else:
# 纯首帧:不带 role,服务端识别为 first_frame(或显式 role=first_frame)
content.append({
"type": "image_url",
"image_url": {"url": image_url},
"role": "first_frame",
})
for u in ref_imgs:
content.append({"type": "image_url", "image_url": {"url": u}, "role": "reference_image"})
for u in ref_audios:
content.append({"type": "audio_url", "audio_url": {"url": u}, "role": "reference_audio"})
for u in ref_videos:
content.append({"type": "video_url", "video_url": {"url": u}, "role": "reference_video"})
create_payload: dict[str, Any] = {
"model": video_model,
@@ -310,19 +323,8 @@ class DoubaoClient:
"duration": int(duration),
"resolution": resolution,
"watermark": bool(watermark),
"ratio": final_ratio,
}
# v1.6: 参考音频(TTS 驱动口型)
if reference_audios:
create_payload["reference_audios"] = [
{"url": u, "role": "audio_url"} for u in reference_audios[:10] if u and isinstance(u, str)
]
# v1.6: 参考视频(风格参考)
if reference_videos:
create_payload["reference_videos"] = [{"url": u} for u in reference_videos[:5] if u and isinstance(u, str)]
# 图生视频也必须传 ratio,否则模型默认 adaptive 可能输出 1:1(首帧方形商品图会导致 960x960)
# 官方文档:图生视频会自动居中裁剪首帧到目标比例,支持 16:9/9:16/1:1/4:3/3:4/21:9/adaptive
if ratio:
create_payload["ratio"] = ratio
headers = {
"Authorization": f"Bearer {self.api_key}",
@@ -330,78 +332,76 @@ class DoubaoClient:
}
create_url = f"{self.base_url}/contents/generations/tasks"
logger.info(
"Seedance 创建任务请求: url=%s model=%s duration=%ds ratio=%s gen_audio=%s image_url=%s ref_imgs=%d ref_audios=%d ref_videos=%d",
create_url,
"Seedance 创建任务: model=%s dur=%ds ratio=%s mode=%s gen_audio=%s img=%d aud=%d vid=%d",
video_model,
duration,
ratio or "(follow-image)",
final_ratio,
"first_frame" if is_first_frame_mode else "omni_ref",
generate_audio,
bool(image_url),
len(reference_images or []),
len(reference_audios or []),
len(reference_videos or []),
(1 if image_url else 0) + len(ref_imgs),
len(ref_audios),
len(ref_videos),
)
# 1) 创建任务(带重试)
task_id: str | None = None
last_error: Exception | None = None
for attempt in range(self.max_retries + 1):
try:
resp = httpx.post(create_url, headers=headers, json=create_payload, timeout=self.timeout)
# 测试环境下 MagicMock().status_code 是 MagicMock,与 int 比较会抛 TypeError;
# 用显式 int() 转换+类型判断,避免误判。
def _do_create(payload: dict) -> tuple[str | None, Exception | None, int, str]:
"""返回 (task_id, last_err, status_code, body_text)。"""
last_err: Exception | None = None
for attempt in range(self.max_retries + 1):
try:
_status = int(resp.status_code)
except (TypeError, ValueError):
_status = 200
if _status >= 400:
# 把响应体完整打出来(通常含 error.code/message,能直接定位:模型未开通/Key 无权限/模型 ID 错误)
logger.error(
"Seedance 创建任务 HTTP %d: body=%s",
resp.status_code,
(resp.text or "")[:1000],
)
resp.raise_for_status()
data = resp.json()
task_id = data.get("id")
if task_id:
break
last_error = RuntimeError(f"create task returned no id: {str(data)[:200]}")
except Exception as e:
last_error = e
if attempt < self.max_retries:
wait = 0.5 * (2**attempt)
logger.warning(
"Seedance 创建任务失败,%.1fs 后重试 (%d/%d): %s", wait, attempt + 1, self.max_retries + 1, e
)
time.sleep(wait)
resp = httpx.post(create_url, headers=headers, json=payload, timeout=self.timeout)
sc = int(getattr(resp, "status_code", 0) or 0)
body = (getattr(resp, "text", "") or "")[:1500]
if sc >= 400:
logger.error("Seedance 创建任务 HTTP %d: body=%s", sc, body)
try:
resp.raise_for_status()
except Exception as ee:
last_err = ee
if attempt < self.max_retries:
time.sleep(0.5 * (2 ** attempt))
continue
return None, last_err, sc, body
data = resp.json()
tid = data.get("id")
if tid:
return tid, None, sc, body
last_err = RuntimeError(f"create ok but no id: {str(data)[:300]}")
except Exception as e:
last_err = e
if attempt < self.max_retries:
wait = 0.5 * (2 ** attempt)
logger.warning("Seedance 创建任务失败,%.1fs 后重试 (%d/%d): %s", wait, attempt + 1, self.max_retries + 1, e)
time.sleep(wait)
return None, last_err, 0, ""
# 第一次尝试
task_id, last_err, sc, body = _do_create(create_payload)
# ratio 兜底:HTTP 400 且 body 提到 ratio / adaptive → 回退 adaptive 再试一次
if not task_id and sc == 400 and final_ratio != "adaptive" and ("ratio" in (body or "").lower() or "aspect" in (body or "").lower() or "adaptive" in (body or "").lower()):
logger.warning("Seedance 创建因 ratio 失败,回退 ratio=adaptive 重试")
create_payload["ratio"] = "adaptive"
task_id, last_err, sc2, body2 = _do_create(create_payload)
if not task_id:
logger.error(
"Seedance 创建任务最终失败: model=%s base_url=%s err=%s 【排查建议】"
"1) 确认方舟控制台已开通 Doubao-Seedance-2.5 模型;"
"2) DOUBAO_API_KEY 对应的账号有该模型调用权限;"
"3) DOUBAO_BASE_URL 必须为 https://ark.cn-beijing.volces.com/api/v3;"
"4) 若控制台用「推理接入点」(endpoint),请把 DOUBAO_VIDEO_MODEL 改为 ep-xxx 接入点 ID。",
video_model,
self.base_url,
last_error,
"Seedance 创建任务最终失败: model=%s base_url=%s err=%s body=%s 【排查】"
"1) 方舟控制台已开通 doubao-seedance-2-5-260628;2) API Key 有该模型权限;"
"3) DOUBAO_BASE_URL=https://ark.cn-beijing.volces.com/api/v3;4) 参考素材 URL 公网可访问。",
video_model, self.base_url, last_err, (body or "")[:500],
)
return None
logger.info(
"Seedance 任务已创建: task_id=%s model=%s duration=%ds gen_audio=%s",
task_id,
video_model,
duration,
generate_audio,
)
logger.info("Seedance 任务已创建: task_id=%s ratio=%s", task_id, create_payload["ratio"])
# 2) 轮询状态
poll_url = f"{create_url}/{task_id}"
deadline = time.time() + total_timeout
video_url: str | None = None
last_status: str = "queued"
poll_count = 0
while time.time() < deadline:
poll_count += 1
try:
resp = httpx.get(poll_url, headers=headers, timeout=self.timeout)
try:
@@ -416,48 +416,65 @@ class DoubaoClient:
content_obj = data.get("content") or {}
video_url = content_obj.get("video_url")
if video_url:
logger.info("Seedance 任务成功: task_id=%s polls=%d", task_id, poll_count)
break
last_error = RuntimeError(f"task succeeded but no video_url: {str(data)[:300]}")
last_err = RuntimeError(f"task succeeded but no video_url: {str(data)[:300]}")
logger.error("Seedance succeeded 但无 video_url: %s", last_err)
break
if status == "failed":
err = data.get("error") or {}
last_error = RuntimeError(f"task failed: {err.get('code','')} {err.get('message','')}")
last_err = RuntimeError(f"task failed: code={err.get('code','')} msg={err.get('message','')}")
logger.error("Seedance 任务失败 task_id=%s: %s", task_id, last_err)
break
if status in ("expired", "cancelled"):
last_error = RuntimeError(f"task {status}")
last_err = RuntimeError(f"task {status}")
logger.error("Seedance 任务 %s: task_id=%s", status, task_id)
break
# queued / running: 继续轮询
# 每 5 次轮询打一次 info 日志,便于观察进度
if poll_count % 5 == 0:
logger.info("Seedance 轮询中: task_id=%s status=%s polls=%d", task_id, status, poll_count)
except httpx.HTTPStatusError as e:
last_error = e
logger.warning(
"Seedance 轮询 HTTP %d: body=%s",
e.response.status_code,
(e.response.text or "")[:500],
)
last_err = e
logger.warning("Seedance 轮询 HTTP %d: %s", e.response.status_code, (e.response.text or "")[:300])
except Exception as e:
last_error = e
last_err = e
logger.debug("Seedance 轮询异常: %s", e)
time.sleep(poll_interval)
if not video_url:
logger.error("Seedance 任务未成功: task_id=%s status=%s err=%s", task_id, last_status, last_error)
logger.error(
"Seedance 任务未成功: task_id=%s last_status=%s polls=%d err=%s (总等待 %.0fs)",
task_id, last_status, poll_count, last_err, total_timeout,
)
return None
# 3) 下载到本地
# 3) 下载到本地(下载超时收紧到 120s)
try:
out_dir = output_dir or "/tmp"
os.makedirs(out_dir, exist_ok=True)
local_path = f"{out_dir}/seedance_{task_id}_{uuid.uuid4().hex[:8]}.mp4"
with httpx.stream("GET", video_url, timeout=300) as r:
download_timeout = 120.0
logger.info("Seedance 开始下载: task_id=%s url=%s timeout=%.0fs", task_id, video_url[:120], download_timeout)
with httpx.stream("GET", video_url, timeout=download_timeout) as r:
r.raise_for_status()
downloaded = 0
with open(local_path, "wb") as f:
for chunk in r.iter_bytes(chunk_size=1024 * 256):
if chunk:
f.write(chunk)
logger.info("Seedance 视频下载完成: %s (%d bytes)", local_path, os.path.getsize(local_path))
downloaded += len(chunk)
size = os.path.getsize(local_path)
logger.info("Seedance 视频下载完成: %s size=%d bytes", local_path, size)
if size == 0:
logger.error("Seedance 下载文件大小为 0")
try:
os.remove(local_path)
except Exception:
pass
return None
return local_path
except Exception as e:
logger.error("Seedance 视频下载失败: %s", e)
logger.error("Seedance 视频下载失败: %s", e, exc_info=True)
return None
+7 -8
View File
@@ -618,15 +618,14 @@ def call_video_generation(
reference_audios: list[str] | None = None,
reference_videos: list[str] | None = None,
) -> str | None:
"""调用 Seedance 2.5 生成视频(v1.6 单次出片版),返回本地 MP4 路径;失败返回 None。
"""调用 Seedance 2.5 生成视频(v1.6.1 单次出片版),返回本地 MP4 路径;失败返回 None。
v1.6:
- 默认 generate_audio=True,模型原生合成环境音效/BGM;
- reference_audios 传 TTS 音频 URL 数组做口型驱动;
- reference_images 传产品素材 URL 数组做视觉参考;
- 单次最长 30 秒,不分段不拼接;
- 图生视频(image_url 存在)也强制传 ratio,避免商品方图导致默认输出 1:1。
官方文档:图生视频时 ratio 由参数决定,模型会居中裁剪首帧到目标比例。
v1.6.1 关键约束(避免 20min 卡死):
- 参考音频/视频/多图全部放进 content 数组并带 role=reference_audio/reference_video/reference_image;
- 纯首帧无参考时(first_frame 模式),Seedance 2.5 强制 ratio=adaptive;
传了参考音/视/多图时走 omni_reference 模式,ratio 可指定为 9:16(客户端内部自动判断)。
- ratio 默认 9:16(竖屏),客户端会根据是否有参考自动在 first_frame/adaptive 与 omni/9:16 间切换;
若创建任务因 ratio 报错(HTTP 400),客户端会自动回退到 adaptive 再试一次。
"""
client = get_doubao_client()
if not client.is_available: