feat: 素材片段区间持久化去重 + 区间用尽自动轮回
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 2m39s
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 3m43s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 1m8s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 1m15s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 5m30s
CI/CD Pipeline / Validate - Migration (alembic) (pull_request) Successful in 8m33s
AI Code Review / AI Code Review (pull_request) Successful in 8m50s
CI/CD Pipeline / Validate - Type Check (mypy) (pull_request) Successful in 9m35s
CI/CD Pipeline / Validate - Code Quality (pull_request) Has been cancelled
CI/CD Pipeline / Unit Tests (pull_request) Has been cancelled
CI/CD Pipeline / Integration Tests (pull_request) Has been cancelled
CI/CD Pipeline / Build Production API Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Web Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been cancelled
CI/CD Pipeline / Deploy Production (pull_request) Has been cancelled
CI/CD Pipeline / Production Browser E2E (pull_request) Has been cancelled
CI/CD Pipeline / Canary Release to Production (pull_request) Has been cancelled
CI/CD Pipeline / CI Gate (pull_request) Has been cancelled
Preview Deploy / Deploy Preview Environment (pull_request) Has been cancelled
CI/CD Pipeline / PR Build Web Image (pull_request) Failing after 427h7m30s
CI/CD Pipeline / Frontend Lint (pull_request) Failing after 427h7m32s
CI/CD Pipeline / ACR Image Cleanup (pull_request) Failing after 427h9m55s
CI/CD Pipeline / Staging API Integration Tests (pull_request) Failing after 427h9m57s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Failing after 427h9m59s
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Failing after 427h10m4s
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Failing after 427h10m7s
CI/CD Pipeline / Build Staging Worker Image (pull_request) Failing after 427h10m9s
CI/CD Pipeline / Build Staging API Image (pull_request) Failing after 427h10m11s
CI/CD Pipeline / Check push changed paths (pull_request) Failing after 427h10m13s
CI/CD Pipeline / Frontend Unit Tests (pull_request) Failing after 427h41m34s
CI/CD Pipeline / Staging E2E Tests (pull_request) Failing after 427h44m1s
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Failing after 427h44m9s
CI/CD Pipeline / Build Staging Web Image (pull_request) Failing after 427h44m14s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 2m39s
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 3m43s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 1m8s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 1m15s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 5m30s
CI/CD Pipeline / Validate - Migration (alembic) (pull_request) Successful in 8m33s
AI Code Review / AI Code Review (pull_request) Successful in 8m50s
CI/CD Pipeline / Validate - Type Check (mypy) (pull_request) Successful in 9m35s
CI/CD Pipeline / Validate - Code Quality (pull_request) Has been cancelled
CI/CD Pipeline / Unit Tests (pull_request) Has been cancelled
CI/CD Pipeline / Integration Tests (pull_request) Has been cancelled
CI/CD Pipeline / Build Production API Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Web Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been cancelled
CI/CD Pipeline / Deploy Production (pull_request) Has been cancelled
CI/CD Pipeline / Production Browser E2E (pull_request) Has been cancelled
CI/CD Pipeline / Canary Release to Production (pull_request) Has been cancelled
CI/CD Pipeline / CI Gate (pull_request) Has been cancelled
Preview Deploy / Deploy Preview Environment (pull_request) Has been cancelled
CI/CD Pipeline / PR Build Web Image (pull_request) Failing after 427h7m30s
CI/CD Pipeline / Frontend Lint (pull_request) Failing after 427h7m32s
CI/CD Pipeline / ACR Image Cleanup (pull_request) Failing after 427h9m55s
CI/CD Pipeline / Staging API Integration Tests (pull_request) Failing after 427h9m57s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Failing after 427h9m59s
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Failing after 427h10m4s
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Failing after 427h10m7s
CI/CD Pipeline / Build Staging Worker Image (pull_request) Failing after 427h10m9s
CI/CD Pipeline / Build Staging API Image (pull_request) Failing after 427h10m11s
CI/CD Pipeline / Check push changed paths (pull_request) Failing after 427h10m13s
CI/CD Pipeline / Frontend Unit Tests (pull_request) Failing after 427h41m34s
CI/CD Pipeline / Staging E2E Tests (pull_request) Failing after 427h44m1s
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Failing after 427h44m9s
CI/CD Pipeline / Build Staging Web Image (pull_request) Failing after 427h44m14s
- 新增 asset_segment_tracker 服务:素材 metadata(used_time_ranges) 持久化片段级已用区间 - from-assets 创建片段时读取历史区间,新片段跨任务/跨调用自动避开 - 片段记录与 replace_all_clips_transactional 同事务,失败整体回滚 - MediaKit 异步移动片段起点后同步更新 metadata 区间记录(失败静默) - _calc_random_start_time 新增 on_exhausted 回调:100次找不到时清空该素材历史区间再重试,实现轮完一圈自动循环 详见 PR body 的 metadata schema 说明
This commit is contained in:
@@ -23,6 +23,12 @@ import re
|
||||
from app.auth import AuthenticatedUser, get_current_user
|
||||
from app.core.storage import get_storage_service
|
||||
from app.dependencies import get_asset_repository, get_db_session
|
||||
from app.services.asset_segment_tracker import (
|
||||
get_used_segments,
|
||||
make_reset_callback,
|
||||
record_used_segments,
|
||||
remove_used_segment,
|
||||
)
|
||||
from app.services.edit_plan_service import EditPlanService
|
||||
from app.services.edit_template_service import EditTemplateService
|
||||
from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException, Query, status
|
||||
@@ -600,7 +606,12 @@ def create_clips_from_assets_editor(
|
||||
asset_durations[asset_id] = float(asset.duration or 0.0)
|
||||
|
||||
# 3. 在内存中计算所有片段数据(使用随机起始时间,不调用MediaKit)
|
||||
used_segments: dict[str, list[tuple[float, float]]] = {}
|
||||
# 读取素材 metadata 中持久化的历史已用区间(跨任务/跨调用去重),
|
||||
# 格式与 _calc_random_start_time 的 used_segments 参数一致
|
||||
used_segments: dict[str, list[tuple[float, float]]] = get_used_segments(
|
||||
db, unique_asset_ids
|
||||
)
|
||||
reset_cb = make_reset_callback(db, used_segments)
|
||||
clips_data: list[dict] = []
|
||||
|
||||
for i, (_seg_order, dur_min, dur_max) in enumerate(segments):
|
||||
@@ -630,7 +641,11 @@ def create_clips_from_assets_editor(
|
||||
|
||||
# 使用随机起始时间(不调用MediaKit,保证接口快速返回)
|
||||
start_time = _calc_random_start_time(
|
||||
asset_id, clip_duration, asset_durations, used_segments
|
||||
asset_id,
|
||||
clip_duration,
|
||||
asset_durations,
|
||||
used_segments,
|
||||
on_exhausted=reset_cb,
|
||||
)
|
||||
|
||||
if start_time is None:
|
||||
@@ -639,10 +654,15 @@ def create_clips_from_assets_editor(
|
||||
detail=f"素材 {asset_id} 时长信息缺失,无法计算起始时间",
|
||||
)
|
||||
|
||||
# 记录已使用时间段
|
||||
# 记录已使用时间段(内存,供本次后续片段避开)
|
||||
used_segments.setdefault(asset_id, []).append(
|
||||
(start_time, start_time + clip_duration)
|
||||
)
|
||||
# 同步写入素材 metadata(不 commit,与下方 replace_all_clips_transactional
|
||||
# 处于同一事务,任一步失败整体回滚,不留脏数据)
|
||||
record_used_segments(
|
||||
db, asset_id, start_time, start_time + clip_duration, plan_id
|
||||
)
|
||||
|
||||
clips_data.append(
|
||||
{
|
||||
@@ -799,8 +819,35 @@ def _update_mediakit_recommendations_async( # pragma: no cover
|
||||
|
||||
# 逐个更新并捕获异常(单点失败不影响其他片段)
|
||||
try:
|
||||
old_start = clip.start_time
|
||||
old_end = old_start + clip_duration
|
||||
plan_svc.update_clip(clip.id, start_time=recommended_start)
|
||||
db.commit()
|
||||
# MediaKit 移动了片段起点 → 同步素材 metadata 的区间记录:
|
||||
# 删除旧区间记录(按 plan_id + 旧 start 匹配),写入新区间。
|
||||
# 异步任务,失败静默,不影响已更新的片段。
|
||||
try:
|
||||
if remove_used_segment(
|
||||
db, asset_id, old_start, old_end, plan_id=plan_id
|
||||
):
|
||||
record_used_segments(
|
||||
db,
|
||||
asset_id,
|
||||
recommended_start,
|
||||
recommended_start + clip_duration,
|
||||
plan_id,
|
||||
)
|
||||
db.commit()
|
||||
except Exception as me:
|
||||
logger.warning(
|
||||
"后台任务: 同步素材区间记录失败: clip_id=%s error=%s",
|
||||
clip.id,
|
||||
me,
|
||||
)
|
||||
try:
|
||||
db.rollback()
|
||||
except Exception:
|
||||
pass
|
||||
updated_count += 1
|
||||
updated_clip_ids.add(clip.id)
|
||||
except Exception as ue:
|
||||
|
||||
@@ -0,0 +1,171 @@
|
||||
"""素材片段级使用记录追踪.
|
||||
|
||||
在素材 metadata(assets.classification_result JSON)中持久化已使用的片段时间区间,
|
||||
供 from-assets 创建片段时避开历史区间,实现跨任务/跨调用的片段去重。
|
||||
|
||||
metadata 中新增字段 ``used_time_ranges``::
|
||||
|
||||
"used_time_ranges": [
|
||||
{"start": 12.5, "end": 20.3, "plan_id": "plan-xxx", "created_at": "2026-08-29T12:00:00+00:00"},
|
||||
...
|
||||
]
|
||||
|
||||
注意:本模块所有函数都不自行 commit,由调用方控制事务边界
|
||||
(from-assets 与 replace_all_clips_transactional 同事务;异步任务各自 commit)。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
from typing import Callable
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from packages.adapters.sqlalchemy_impl.models import AssetModel
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
USED_RANGES_KEY = "used_time_ranges"
|
||||
|
||||
|
||||
def _read_ranges(model: AssetModel) -> list[dict]:
|
||||
"""从 AssetModel 读取 metadata dict(classification_result 列承载的 JSON)."""
|
||||
if not model.classification_result:
|
||||
return {}
|
||||
try:
|
||||
return json.loads(model.classification_result)
|
||||
except Exception:
|
||||
return {}
|
||||
|
||||
|
||||
def get_used_segments(db: Session, asset_ids: list[str]) -> dict[str, list[tuple[float, float]]]:
|
||||
"""聚合多个素材的历史已用片段区间。
|
||||
|
||||
Args:
|
||||
db: SQLAlchemy session
|
||||
asset_ids: 素材 ID 列表
|
||||
|
||||
Returns:
|
||||
``{asset_id: [(start, end), ...]}`` 格式,与 ``_calc_random_start_time`` 的
|
||||
``used_segments`` 参数格式一致,可直接传入。
|
||||
"""
|
||||
if not asset_ids:
|
||||
return {}
|
||||
result: dict[str, list[tuple[float, float]]] = {}
|
||||
models = db.query(AssetModel).filter(AssetModel.id.in_(list(set(asset_ids)))).all()
|
||||
for model in models:
|
||||
meta = _read_ranges(model)
|
||||
ranges = meta.get(USED_RANGES_KEY) or []
|
||||
segments: list[tuple[float, float]] = []
|
||||
for r in ranges:
|
||||
try:
|
||||
segments.append((float(r["start"]), float(r["end"])))
|
||||
except (KeyError, TypeError, ValueError):
|
||||
continue
|
||||
if segments:
|
||||
result[model.id] = segments
|
||||
return result
|
||||
|
||||
|
||||
def record_used_segments(
|
||||
db: Session,
|
||||
asset_id: str,
|
||||
start: float,
|
||||
end: float,
|
||||
plan_id: str,
|
||||
) -> None:
|
||||
"""向素材 metadata 追加一条片段使用记录(不 commit)."""
|
||||
model = db.query(AssetModel).filter(AssetModel.id == asset_id).first()
|
||||
if model is None:
|
||||
logger.warning("[片段追踪] 素材不存在,跳过记录: asset_id=%s", asset_id)
|
||||
return
|
||||
meta = _read_ranges(model)
|
||||
ranges = list(meta.get(USED_RANGES_KEY) or [])
|
||||
ranges.append(
|
||||
{
|
||||
"start": round(float(start), 3),
|
||||
"end": round(float(end), 3),
|
||||
"plan_id": plan_id,
|
||||
"created_at": datetime.now(timezone.utc).isoformat(),
|
||||
}
|
||||
)
|
||||
meta[USED_RANGES_KEY] = ranges
|
||||
model.classification_result = json.dumps(meta, ensure_ascii=False)
|
||||
model.updated_at = datetime.now(timezone.utc)
|
||||
|
||||
|
||||
def remove_used_segment(
|
||||
db: Session,
|
||||
asset_id: str,
|
||||
start: float,
|
||||
end: float,
|
||||
plan_id: str | None = None,
|
||||
tolerance: float = 0.5,
|
||||
) -> bool:
|
||||
"""删除素材 metadata 中匹配的一条使用记录(不 commit).
|
||||
|
||||
匹配规则:start/end 与记录值相差不超过 tolerance 秒;plan_id 非空时还需相等。
|
||||
Returns:
|
||||
是否找到并删除了记录。
|
||||
"""
|
||||
model = db.query(AssetModel).filter(AssetModel.id == asset_id).first()
|
||||
if model is None:
|
||||
return False
|
||||
meta = _read_ranges(model)
|
||||
ranges = list(meta.get(USED_RANGES_KEY) or [])
|
||||
remaining: list[dict] = []
|
||||
removed = False
|
||||
for r in ranges:
|
||||
try:
|
||||
match = (
|
||||
abs(float(r["start"]) - float(start)) <= tolerance
|
||||
and abs(float(r["end"]) - float(end)) <= tolerance
|
||||
)
|
||||
except (KeyError, TypeError, ValueError):
|
||||
remaining.append(r)
|
||||
continue
|
||||
if plan_id is not None and r.get("plan_id") != plan_id:
|
||||
match = False
|
||||
if match and not removed:
|
||||
removed = True
|
||||
continue
|
||||
remaining.append(r)
|
||||
if removed:
|
||||
meta[USED_RANGES_KEY] = remaining
|
||||
model.classification_result = json.dumps(meta, ensure_ascii=False)
|
||||
model.updated_at = datetime.now(timezone.utc)
|
||||
return removed
|
||||
|
||||
|
||||
def reset_used_segments(db: Session, asset_id: str) -> None:
|
||||
"""清空单个素材的历史片段使用记录(不 commit).
|
||||
|
||||
单个素材的可用区间被全部占用(轮回一圈)后调用,使后续片段可重新使用整段素材。
|
||||
"""
|
||||
model = db.query(AssetModel).filter(AssetModel.id == asset_id).first()
|
||||
if model is None:
|
||||
return
|
||||
meta = _read_ranges(model)
|
||||
if meta.get(USED_RANGES_KEY):
|
||||
meta[USED_RANGES_KEY] = []
|
||||
model.classification_result = json.dumps(meta, ensure_ascii=False)
|
||||
model.updated_at = datetime.now(timezone.utc)
|
||||
logger.info("[片段追踪] 素材区间轮回重置: asset_id=%s", asset_id)
|
||||
|
||||
|
||||
def make_reset_callback(db: Session, used_segments: dict) -> Callable[[str], None]:
|
||||
"""构造给 _calc_random_start_time 用的 reset 回调.
|
||||
|
||||
回调同时清空持久化 metadata 和内存中的 used_segments,使重试随机能覆盖全素材。
|
||||
"""
|
||||
|
||||
def _reset(asset_id: str) -> None:
|
||||
try:
|
||||
reset_used_segments(db, asset_id)
|
||||
except Exception:
|
||||
logger.warning("[片段追踪] reset 持久化记录失败: asset_id=%s", asset_id, exc_info=True)
|
||||
used_segments.pop(asset_id, None)
|
||||
|
||||
return _reset
|
||||
Reference in New Issue
Block a user