Files
xiaoxia-saas/packages/application/generated_video_finalize.py
T
saas-backend-agent 48f6f49f7b
PR Automation / Auto Approve on CI Green (pull_request) Successful in 3m22s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 2m32s
AI Code Review / AI Code Review (pull_request) Successful in 6m51s
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 10m53s
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 0s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 2s
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 30s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 29s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 3m46s
CI/CD Pipeline / Validate - Style (pull_request) Successful in 4m59s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 5m33s
CI/CD Pipeline / Validate - Security (pull_request) Successful in 10m57s
CI/CD Pipeline / Unit Tests (pull_request) Failing after 11m4s
CI/CD Pipeline / Build Production API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Web Image (pull_request) Has been skipped
CI/CD Pipeline / CI Gate (pull_request) Failing after 1s
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / Canary Release to Production (pull_request) Has been skipped
feat(generation): #2024 defer video finalization until cover confirmed
Two-phase pipeline: worker renders and precomputes dedup metadata, but does
not create GeneratedVideo until user confirms cover in Step 5.

Domain
- Add GenerationTaskStatus.AWAITING_COVER (non-terminal)
- Add mark_awaiting_cover(): progress=100, clears error, leaves completed_at unset
- Transitions: running → {awaiting_cover, completed, failed, cancelled};
  awaiting_cover → {completed, failed, cancelled}
- _missing_ aliases: waiting_cover/video_ready/rendered/pending_cover
- Keep running→completed for backward compat / legacy paths

Worker (apps/worker/worker_app/tasks/generation.py)
- Replace _record_video_and_dedup with _precompute_render_metadata:
  calls compute_render_fingerprint_and_dedup, returns dict without DB writes
- Remove runtime batch-rerender decision (should_rerender_for_batch_dedup path);
  dedup now happens at finalize time against all finished records
- Persist precomputed metadata into task.extra_meta['rendered_output']
- Final status: mark_awaiting_cover instead of mark_completed

Dedup helpers (apps/worker/video_processing/dedup_helpers.py)
- New compute_render_fingerprint_and_dedup(video_path,...): local fingerprint
  + historical dedup + batch dedup, returns fully serializable dict
  (fingerprint_dict, fingerprint_chunks list, is_duplicate, duplicate_of, etc.)
- create_video_record_and_dedup() retained for compat/tests; now supports
  pre_dedup_result to reuse worker-precomputed data without re-reading video
- VideoFingerprint.from_dict() added to reconstruct from serialized form

Application layer (packages/application/generated_video_finalize.py)
- RenderedOutput dataclass + from_dict() for deserializing worker output
- finalize_generated_video(): creates GeneratedVideo, bulk-inserts
  VideoFingerprintChunk rows, commits; API-layer free

API service (apps/api/app/services/generation_finalize_service.py)
- GenerationFinalizeService.finalize_task(task_id, user_id, cover_url):
  * permission / existence check
  * idempotent: if GeneratedVideo already exists for this task, just ensure
    task is marked completed and return (safe for double-click)
  * status gate: only awaiting_cover (or legacy completed) accepted
  * cover resolution: explicit cover_url arg > task.cover_url
  * delegates to finalize_generated_video, marks task completed,
    clears extra_meta['rendered_output']

API endpoint (apps/api/app/api/routes/generation_tasks.py)
- POST /api/v1/generation/tasks/{task_id}/finalize
- Body: { cover_url?: string }; returns video_id/cover_url/file_url/is_duplicate
- Adjust confirm_generation fast path: mark_confirmed keeps task in
  awaiting_cover (don't auto-complete); historical 'completed' previews
  migrated back to awaiting_cover
- Preview-reuse accepts awaiting_cover tasks

Preview / task center
- GET /preview/{task_id} constructs lightweight _PreviewVideo from
  extra_meta.rendered_output when task is awaiting_cover
- Task center step map: awaiting_cover → 等待确认封面; status filter includes it

Tests
- tests/unit/test_finalize_generation.py: 13 new tests covering status
  transitions, mark_awaiting_cover, RenderedOutput parsing, finalize use case
  (success / missing metadata / cover fallback)
- test_generation_task.py updated for 6th status value
- Full suite: 15978 passed, 28 skipped (matches #2023 baseline, no regressions)
- black/isort/ruff clean

Out of scope (intentionally untouched)
- AI avatar pipeline uses separate AiAvatarRenderJob; finalize_job and
  /{job_id}/finalize are unchanged
- PR #2023 subtitle font scaling files (ass_subtitle_builder,
  video_filter_builder, subtitle_generator) not modified
- No DB migration: GenerationTaskModel.status is String(20) without CHECK
- Temp file cleanup: leave to existing periodic job
2026-09-24 11:29:36 +08:00

192 lines
6.9 KiB
Python

"""#2024: 视频生成 finalize 入库用例。
Worker 渲染+上传完成后不自动入库,只把渲染产物与查重结果保存到
GenerationTask.extra_meta["rendered_output"],并标记为 awaiting_cover。
用户点「完成」时由 API 调用本用例:创建 GeneratedVideo 记录(复用预计算查重结果)、
推进任务到 completed,返回新记录 id。
设计原则:finalize 必须快速(仅 DB 写入,不下载视频、不重算指纹)——
所有耗时操作(指纹计算、历史/批次查重)都在 worker 渲染阶段预完成。
"""
from __future__ import annotations
import logging
from dataclasses import dataclass
from datetime import UTC, datetime
from typing import Any
from uuid import uuid4
from sqlalchemy.orm import Session
logger = logging.getLogger(__name__)
@dataclass(slots=True)
class RenderedOutput:
"""Worker 预计算并写入 extra_meta 的渲染产物+查重结果。"""
file_url: str
file_size: int = 0
duration: float = 0.0
width: int = 1280
height: int = 720
fps: float = 25.0
name: str = ""
thumbnail_url: str = ""
mode: str = "narrative"
batch_id: str = ""
project_id: str = ""
user_id: str = ""
# 查重结果(worker 预计算)
fingerprint_dict: dict[str, Any] | None = None
fingerprint_chunks: list[dict[str, Any]] | None = None
is_duplicate: bool = False
duplicate_of: str | None = None
duplicate_rate: float | None = None
match_count: int | None = None
visual_similarity: float | None = None
video_fingerprint_md5: str = ""
@classmethod
def from_dict(cls, data: dict[str, Any]) -> "RenderedOutput":
if not isinstance(data, dict):
raise ValueError("rendered_output must be a dict")
return cls(
file_url=str(data.get("file_url") or ""),
file_size=int(data.get("file_size") or 0),
duration=float(data.get("duration") or 0.0),
width=int(data.get("width") or 1280),
height=int(data.get("height") or 720),
fps=float(data.get("fps") or 25.0),
name=str(data.get("name") or ""),
thumbnail_url=str(data.get("thumbnail_url") or ""),
mode=str(data.get("mode") or "narrative"),
batch_id=str(data.get("batch_id") or ""),
project_id=str(data.get("project_id") or ""),
user_id=str(data.get("user_id") or ""),
fingerprint_dict=data.get("fingerprint_dict"),
fingerprint_chunks=data.get("fingerprint_chunks"),
is_duplicate=bool(data.get("is_duplicate", False)),
duplicate_of=data.get("duplicate_of"),
duplicate_rate=_safe_float(data.get("duplicate_rate")),
match_count=_safe_int(data.get("match_count")),
visual_similarity=_safe_float(data.get("visual_similarity")),
video_fingerprint_md5=str(data.get("video_fingerprint_md5") or ""),
)
def _safe_float(v) -> float | None:
if v is None:
return None
try:
return float(v)
except (TypeError, ValueError):
return None
def _safe_int(v) -> int | None:
if v is None:
return None
try:
return int(v)
except (TypeError, ValueError):
return None
def finalize_generated_video(
*,
task,
session: Session,
effective_cover_url: str = "",
) -> dict:
"""将 awaiting_cover 的任务正式入库。
从 ``task.extra_meta["rendered_output"]`` 读取 worker 预存的渲染结果与查重数据,
创建 GeneratedVideo 记录并 commit;调用方负责将 task 推进到 completed 并 update。
Returns:
{"video_id": str, "is_duplicate": bool, "duplicate_of": str|None}
"""
from packages.adapters.sqlalchemy_impl.generated_video_repository import (
SQLAlchemyGeneratedVideoRepository,
)
from packages.adapters.sqlalchemy_impl.models import VideoFingerprintChunkModel
from packages.domain.generated_video import GeneratedVideo
meta = dict(task.extra_meta or {})
rendered_dict = meta.get("rendered_output") or {}
rendered = RenderedOutput.from_dict(rendered_dict)
if not rendered.file_url.strip():
raise ValueError(f"task {task.id} rendered_output.file_url 为空,无法 finalize")
video_id = uuid4().hex
video_name = rendered.name.strip() or f"generated-{task.id[:8]}.mp4"
generated_video = GeneratedVideo(
id=video_id,
project_id=(rendered.project_id or task.project_id or "").strip(),
user_id=(rendered.user_id or task.created_by_user_id or "").strip(),
generation_task_id=task.id,
name=video_name,
file_url=rendered.file_url.strip(),
file_size=rendered.file_size,
duration=rendered.duration,
width=rendered.width,
height=rendered.height,
fps=rendered.fps,
status="completed",
generation_params={"mode": rendered.mode},
thumbnail_url=effective_cover_url or rendered.thumbnail_url or None,
video_fingerprint=rendered.fingerprint_dict,
is_duplicate=rendered.is_duplicate,
duplicate_of=rendered.duplicate_of,
duplicate_rate=rendered.duplicate_rate,
match_count=rendered.match_count,
visual_similarity=rendered.visual_similarity,
created_at=datetime.now(UTC),
generated_at=datetime.now(UTC),
)
# 写入分片指纹(worker 预序列化的 chunk 列表)
if rendered.fingerprint_chunks:
try:
chunk_models = []
for c in rendered.fingerprint_chunks:
if not isinstance(c, dict):
continue
chunk_models.append(
VideoFingerprintChunkModel(
id=uuid4().hex,
video_id=video_id,
project_id=generated_video.project_id,
user_id=generated_video.user_id,
start_time_ms=int(c.get("start_time_ms", 0)),
end_time_ms=int(c.get("end_time_ms", 0)),
phash_binary=str(c.get("phash_binary", "")),
color_histogram=[float(v) for v in (c.get("color_histogram") or [])],
frame_count=int(c.get("frame_count", 0)),
)
)
if chunk_models:
session.bulk_save_objects(chunk_models)
except Exception as chunk_err:
logger.warning("Failed to persist fingerprint chunks for video %s: %s", video_id, chunk_err)
video_repo = SQLAlchemyGeneratedVideoRepository(session)
video_repo.create(generated_video)
session.commit()
logger.info(
"[finalize] GeneratedVideo created: %s (task=%s, dup=%s, cover=%s)",
video_id,
task.id,
rendered.is_duplicate,
bool(effective_cover_url),
)
return {
"video_id": video_id,
"is_duplicate": rendered.is_duplicate,
"duplicate_of": rendered.duplicate_of,
}