fix(ai-avatar): 端到端一致性修复(标题/封面/B-roll位置/B-roll时长)
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 2s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 1s
Preview Deploy / Deploy Preview Environment (pull_request) Failing after 57s
CI/CD Pipeline / Frontend Lint (pull_request) Failing after 50s
CI/CD Pipeline / Frontend Unit Tests (pull_request) Successful in 1m41s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 2m6s
CI/CD Pipeline / PR Build Web Image (pull_request) Failing after 2m16s
CI/CD Pipeline / Validate - Style (pull_request) Has been cancelled
CI/CD Pipeline / Validate - Security (pull_request) Has been cancelled
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Has been cancelled
CI/CD Pipeline / Unit Tests (pull_request) Has been cancelled
CI/CD Pipeline / Integration Tests (pull_request) Has been cancelled
CI/CD Pipeline / PR Build Worker Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production API Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Web Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been cancelled
CI/CD Pipeline / Deploy Production (pull_request) Has been cancelled
CI/CD Pipeline / Production Browser E2E (pull_request) Has been cancelled
CI/CD Pipeline / Canary Release to Production (pull_request) Has been cancelled
CI/CD Pipeline / CI Gate (pull_request) Has been cancelled
AI Code Review / AI Code Review (pull_request) Has been cancelled
PR Automation / Auto Approve on CI Green (pull_request) Has been cancelled
PR Automation / Auto Merge on CI Green + Approved (pull_request) Has been cancelled
CI/CD Pipeline / Staging API Integration Tests (pull_request) Failing after 109h59m25s
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Failing after 109h59m42s
CI/CD Pipeline / Build Staging Worker Image (pull_request) Failing after 109h59m30s
CI/CD Pipeline / Build Staging Web Image (pull_request) Failing after 109h59m31s
CI/CD Pipeline / ACR Image Cleanup (pull_request) Failing after 109h59m8s
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Failing after 109h59m25s
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Failing after 109h59m25s
CI/CD Pipeline / Build Staging API Image (pull_request) Failing after 109h59m31s
CI/CD Pipeline / Check push changed paths (pull_request) Failing after 109h59m38s
CI/CD Pipeline / Staging E2E Tests (pull_request) Failing after 109h59m9s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Failing after 109h59m18s
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 2s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 1s
Preview Deploy / Deploy Preview Environment (pull_request) Failing after 57s
CI/CD Pipeline / Frontend Lint (pull_request) Failing after 50s
CI/CD Pipeline / Frontend Unit Tests (pull_request) Successful in 1m41s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 2m6s
CI/CD Pipeline / PR Build Web Image (pull_request) Failing after 2m16s
CI/CD Pipeline / Validate - Style (pull_request) Has been cancelled
CI/CD Pipeline / Validate - Security (pull_request) Has been cancelled
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Has been cancelled
CI/CD Pipeline / Unit Tests (pull_request) Has been cancelled
CI/CD Pipeline / Integration Tests (pull_request) Has been cancelled
CI/CD Pipeline / PR Build Worker Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production API Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Web Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been cancelled
CI/CD Pipeline / Deploy Production (pull_request) Has been cancelled
CI/CD Pipeline / Production Browser E2E (pull_request) Has been cancelled
CI/CD Pipeline / Canary Release to Production (pull_request) Has been cancelled
CI/CD Pipeline / CI Gate (pull_request) Has been cancelled
AI Code Review / AI Code Review (pull_request) Has been cancelled
PR Automation / Auto Approve on CI Green (pull_request) Has been cancelled
PR Automation / Auto Merge on CI Green + Approved (pull_request) Has been cancelled
CI/CD Pipeline / Staging API Integration Tests (pull_request) Failing after 109h59m25s
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Failing after 109h59m42s
CI/CD Pipeline / Build Staging Worker Image (pull_request) Failing after 109h59m30s
CI/CD Pipeline / Build Staging Web Image (pull_request) Failing after 109h59m31s
CI/CD Pipeline / ACR Image Cleanup (pull_request) Failing after 109h59m8s
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Failing after 109h59m25s
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Failing after 109h59m25s
CI/CD Pipeline / Build Staging API Image (pull_request) Failing after 109h59m31s
CI/CD Pipeline / Check push changed paths (pull_request) Failing after 109h59m38s
CI/CD Pipeline / Staging E2E Tests (pull_request) Failing after 109h59m9s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Failing after 109h59m18s
修复5个问题:
1. 最终视频标题一致性
- 后端 drawtext 默认 font_size 从 36 改为 28(对齐前端 DEFAULT_TITLE_CONFIG.size)
- 后端 drawtext 默认 position 从 'top' 改为 'bottom'(对齐前端默认值)
- 前端 contract.ts font_size fallback 从 36 改为 28
2. 封面与视频一致性
- 渲染完成后自动用 output_cover_url 更新前端封面配置
- 前端封面标题预览 fontSize 改用 size*0.55 缩放(与对口型预览对齐)
- 封面默认阴影改为 none(与后端 drawtext 无 shadow 行为对齐)
3. B-roll 位置一致性
- ModalBRollEditor 使用 lipsyncJob.script_text(TTS 时锁定的文案)
而非 state.scriptText(用户可能已修改的文案)
4. B-roll 时长精确化
- 后端 lipsync_tts.py 新增 _compute_sentence_timings():
TTS 音频用 ffmpeg silencedetect 检测静音点,与句子边界对齐
- 新增 sentence_timings JSON 字段到 LipsyncJobModel + Alembic migration
- 前端 sentences.ts 不再做字数比例估算,改为接收后端精确时间戳
- ModalBRollEditor 接收 sentenceTimings prop
5. 标题拖拽位置修复
- PanelLipsyncPreview 拖拽标题发送百分比坐标(0-100) + position:'custom'
- 后端 build_title_drawtext_filter 的 custom 位置改用 drawtext 表达式
x=(w-text_w)*{pct_x} y=(h-text_h)*{pct_y}(百分比,与视频尺寸解耦)
文件变更:
- apps/api/app/tasks/lipsync_tts.py: 句子时间戳计算
- apps/api/app/schemas/lipsync.py: sentence_timings 字段
- packages/adapters/sqlalchemy_impl/models.py: sentence_timings 列
- packages/domain/video_filter_builder.py: 百分比坐标 + 默认值对齐
- alembic/versions/075_add_sentence_timings.py: 数据库迁移
- apps/web/src/pages/ai-avatar/types.ts: SentenceTiming + RenderJob.output_cover_url
- apps/web/src/pages/ai-avatar/utils/sentences.ts: 移除字数估算
- apps/web/src/pages/ai-avatar/utils/contract.ts: font_size 默认值
- apps/web/src/pages/ai-avatar/components/PanelLipsyncPreview.tsx: 拖拽百分比
- apps/web/src/pages/ai-avatar/components/PanelCoverAndGenerate.tsx: 标题缩放
- apps/web/src/pages/ai-avatar/components/ModalBRollEditor.tsx: 后端时间戳
- apps/web/src/pages/ai-avatar/AiAvatarPage.tsx: 封面更新 + 传参
- tests/unit/test_video_filter_builder.py: 测试更新
This commit is contained in:
@@ -0,0 +1,27 @@
|
||||
"""add sentence_timings to lipsync_jobs
|
||||
|
||||
Revision ID: 075_add_sentence_timings
|
||||
Revises: 074_ai_avatar_render_script_id_optional
|
||||
Create Date: 2026-09-12
|
||||
"""
|
||||
|
||||
import sqlalchemy as sa
|
||||
|
||||
from alembic import op
|
||||
|
||||
revision = "075_add_sentence_timings"
|
||||
down_revision = "074_ai_avatar_render_script_id_optional"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
with op.batch_alter_table("lipsync_jobs") as batch:
|
||||
batch.add_column(
|
||||
sa.Column("sentence_timings", sa.JSON(), nullable=True),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
with op.batch_alter_table("lipsync_jobs") as batch:
|
||||
batch.drop_column("sentence_timings")
|
||||
@@ -33,6 +33,7 @@ class LipsyncJobResponse(BaseModel):
|
||||
output_duration: float
|
||||
error_message: str
|
||||
error_code: str
|
||||
sentence_timings: Optional[list] = None
|
||||
submitted_at: Optional[datetime] = None
|
||||
completed_at: Optional[datetime] = None
|
||||
created_at: datetime
|
||||
|
||||
@@ -54,6 +54,149 @@ def _sign_media_url(url: str) -> str:
|
||||
return url
|
||||
|
||||
|
||||
|
||||
|
||||
def _split_script_into_sentences(script_text: str) -> list[str]:
|
||||
"""按句号/问号/感叹号/分号/换行分句(与前端 splitScriptIntoSentences 一致)."""
|
||||
import re
|
||||
text = (script_text or "").strip()
|
||||
if not text:
|
||||
return []
|
||||
parts = re.split(r"[。!?!?;;\n\r]+", text)
|
||||
return [p.strip() for p in parts if p.strip()]
|
||||
|
||||
|
||||
def _compute_sentence_timings(audio_data: bytes, script_text: str, total_duration: float) -> list[dict]:
|
||||
"""基于 TTS 音频的静音检测,精确计算每句文案的起止时间.
|
||||
|
||||
使用 ffmpeg silencedetect 检测静音段,将静音点与句子边界对齐。
|
||||
比字数比例估算准确得多。
|
||||
|
||||
Args:
|
||||
audio_data: TTS 音频二进制数据(MP3)
|
||||
script_text: 文案全文
|
||||
total_duration: 音频总时长(秒)
|
||||
|
||||
Returns:
|
||||
list[{"index": int, "text": str, "start_time": float, "end_time": float}]
|
||||
"""
|
||||
import re
|
||||
import subprocess
|
||||
import tempfile
|
||||
|
||||
sentences = _split_script_into_sentences(script_text)
|
||||
if not sentences:
|
||||
return []
|
||||
|
||||
# 写入临时音频文件
|
||||
with tempfile.NamedTemporaryFile(suffix=".mp3", delete=False) as tmp:
|
||||
tmp.write(audio_data)
|
||||
tmp_path = tmp.name
|
||||
|
||||
try:
|
||||
# 用 ffmpeg silencedetect 检测静音段
|
||||
result = subprocess.run(
|
||||
[
|
||||
"ffmpeg", "-i", tmp_path,
|
||||
"-af", "silencedetect=noise=-25dB:d=0.3",
|
||||
"-f", "null", "-",
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=30,
|
||||
)
|
||||
stderr = result.stderr or ""
|
||||
|
||||
# 解析静音结束时间点(silence_end: X.XXX)
|
||||
silence_ends = []
|
||||
for match in re.finditer(r"silence_end:\s*([\d.]+)", stderr):
|
||||
t = float(match.group(1))
|
||||
if 0 < t < total_duration:
|
||||
silence_ends.append(t)
|
||||
|
||||
# 如果没有检测到足够的静音点,降级为字数比例估算
|
||||
if len(silence_ends) < len(sentences) - 1:
|
||||
logger.warning(
|
||||
"[sentence_timings] 静音点不足(%d < %d),降级为字数比例估算",
|
||||
len(silence_ends), len(sentences) - 1,
|
||||
)
|
||||
return _estimate_sentence_timings_by_chars(sentences, total_duration)
|
||||
|
||||
# 贪心匹配:N-1 个句子边界对应 N-1 个静音点
|
||||
# 按时间均匀分布期望值,选择最近的静音点
|
||||
n_boundaries = len(sentences) - 1
|
||||
boundaries = []
|
||||
used_indices = set()
|
||||
|
||||
for i in range(n_boundaries):
|
||||
# 期望的边界位置(按句子数量均匀分布)
|
||||
expected_pos = (i + 1) / len(sentences) * total_duration
|
||||
# 找最近的未使用静音点
|
||||
best_idx = None
|
||||
best_dist = float("inf")
|
||||
for j, t in enumerate(silence_ends):
|
||||
if j in used_indices:
|
||||
continue
|
||||
dist = abs(t - expected_pos)
|
||||
if dist < best_dist:
|
||||
best_dist = dist
|
||||
best_idx = j
|
||||
if best_idx is not None:
|
||||
used_indices.add(best_idx)
|
||||
boundaries.append(silence_ends[best_idx])
|
||||
|
||||
boundaries.sort()
|
||||
|
||||
# 构建 sentence_timings
|
||||
timings = []
|
||||
prev_end = 0.0
|
||||
for i, sent in enumerate(sentences):
|
||||
start = prev_end
|
||||
end = boundaries[i] if i < len(boundaries) else total_duration
|
||||
timings.append({
|
||||
"index": i,
|
||||
"text": sent,
|
||||
"start_time": round(start, 2),
|
||||
"end_time": round(end, 2),
|
||||
})
|
||||
prev_end = end
|
||||
|
||||
return timings
|
||||
|
||||
except Exception as exc:
|
||||
logger.warning("[sentence_timings] 静音检测异常,降级为字数比例估算: %s", exc)
|
||||
return _estimate_sentence_timings_by_chars(sentences, total_duration)
|
||||
finally:
|
||||
import os
|
||||
try:
|
||||
os.unlink(tmp_path)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def _estimate_sentence_timings_by_chars(sentences: list[str], total_duration: float) -> list[dict]:
|
||||
"""降级方案:按字数比例估算句子时间(与原前端逻辑一致)."""
|
||||
if not sentences or total_duration <= 0:
|
||||
return []
|
||||
total_chars = sum(len(s.replace("\s", "")) for s in sentences)
|
||||
if total_chars == 0:
|
||||
return []
|
||||
|
||||
timings = []
|
||||
acc = 0
|
||||
for i, sent in enumerate(sentences):
|
||||
chars = len(sent.replace("\s", ""))
|
||||
start = (acc / total_chars) * total_duration
|
||||
end = ((acc + chars) / total_chars) * total_duration
|
||||
timings.append({
|
||||
"index": i,
|
||||
"text": sent,
|
||||
"start_time": round(start, 2),
|
||||
"end_time": round(end, 2),
|
||||
})
|
||||
acc += chars
|
||||
return timings
|
||||
|
||||
@shared_task(
|
||||
bind=True,
|
||||
name="lipsync_tts.synthesize_and_submit",
|
||||
@@ -179,6 +322,50 @@ def tts_synthesize_and_submit(
|
||||
|
||||
db.commit()
|
||||
|
||||
# 2.5 计算精确句子时间戳(基于 TTS 音频静音检测)
|
||||
import os as _os
|
||||
_st_tmp_path = None
|
||||
try:
|
||||
import subprocess as _sp
|
||||
import tempfile as _tmpf
|
||||
|
||||
# 下载音频用于探测时长和静音检测
|
||||
if isinstance(job.audio_url, str) and job.audio_url:
|
||||
from packages.shared.url_security import safe_download_bytes as _sdl
|
||||
_audio_bytes = _sdl(job.audio_url, purpose="sentence_timings", timeout=30.0)
|
||||
else:
|
||||
_audio_bytes = audio_data
|
||||
|
||||
# ffprobe 获取音频时长
|
||||
with _tmpf.NamedTemporaryFile(suffix=".mp3", delete=False) as _atmp:
|
||||
_atmp.write(_audio_bytes)
|
||||
_st_tmp_path = _atmp.name
|
||||
|
||||
_probe_result = _sp.run(
|
||||
["ffprobe", "-v", "error", "-show_entries", "format=duration",
|
||||
"-of", "default=noprint_wrappers=1:nokey=1", _st_tmp_path],
|
||||
capture_output=True, text=True, timeout=10,
|
||||
)
|
||||
_audio_duration = float(_probe_result.stdout.strip()) if _probe_result.stdout.strip() else 0.0
|
||||
|
||||
if _audio_duration > 0:
|
||||
_timings = _compute_sentence_timings(_audio_bytes, script_text, _audio_duration)
|
||||
if _timings:
|
||||
job.sentence_timings = _timings
|
||||
logger.info(
|
||||
"[lipsync_tts] 句子时间戳已计算: job_id=%s sentences=%d duration=%.1f",
|
||||
job_id, len(_timings), _audio_duration,
|
||||
)
|
||||
db.commit()
|
||||
except Exception as _st_err:
|
||||
logger.warning("[lipsync_tts] 句子时间戳计算失败(不影响主流程): job_id=%s err=%s", job_id, _st_err)
|
||||
finally:
|
||||
if _st_tmp_path:
|
||||
try:
|
||||
_os.unlink(_st_tmp_path)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# 3. 签名 URL 并提交到 MediaKit(复用模块内 _sign_media_url,避免对 LipsyncService 的耦合)
|
||||
audio_url = _sign_media_url(job.audio_url)
|
||||
video_url = _sign_media_url(job.video_url)
|
||||
|
||||
@@ -241,6 +241,15 @@ const AiAvatarPage: React.FC = () => {
|
||||
if (renderTimerRef.current) clearInterval(renderTimerRef.current)
|
||||
renderTimerRef.current = null
|
||||
setRenderStatus("completed")
|
||||
// 渲染完成后,用最终视频的封面更新前端封面配置
|
||||
if (updated.output_cover_url) {
|
||||
state.setCoverConfig((prev) => ({
|
||||
...prev,
|
||||
mode: "auto_frame",
|
||||
smart_cover_url: updated.output_cover_url,
|
||||
thumbnail_url: updated.output_cover_url,
|
||||
}))
|
||||
}
|
||||
message.success("视频已生成并保存到成片库")
|
||||
} else if (updated.status === "failed") {
|
||||
if (renderTimerRef.current) clearInterval(renderTimerRef.current)
|
||||
@@ -488,8 +497,9 @@ const AiAvatarPage: React.FC = () => {
|
||||
open={state.showBRollModal}
|
||||
onClose={() => state.setShowBRollModal(false)}
|
||||
existingSegments={state.bRollSegments}
|
||||
scriptText={state.scriptText}
|
||||
scriptText={state.lipsyncJob?.script_text || state.scriptText}
|
||||
outputDuration={state.lipsyncJob?.output_duration ?? 0}
|
||||
sentenceTimings={state.lipsyncJob?.sentence_timings}
|
||||
onConfirm={state.addBRollSegment}
|
||||
onRemove={state.removeBRollSegment}
|
||||
/>
|
||||
|
||||
@@ -5,12 +5,12 @@
|
||||
* - 左侧:先选素材库(video 库)→ 再选该库视频素材(已被其他 segment 使用的素材
|
||||
* 标灰 + "已选择" 遮罩,pointer-events:none 防重复选择)
|
||||
* - 右侧:文案句子列表(点选对应段落,替代原数字索引框)/ 全屏 or 画中画 / 四角位置+大小
|
||||
* (开始/结束时间已删除,按句子字数占比 × 口播总时长自动估算)
|
||||
* (开始/结束时间来自后端精确句子时间戳,基于 TTS 音频静音检测)
|
||||
* - 底部:已配置的画面插入列表(可删除)
|
||||
*/
|
||||
import React, { useEffect, useMemo, useState } from "react"
|
||||
import { getAssets, getAssetLibraries, type AssetItem, type AssetLibraryItem } from "@/api/assets"
|
||||
import type { BRollSegment, BRollInsertMode, PipPosition } from "../types"
|
||||
import type { BRollSegment, BRollInsertMode, PipPosition, SentenceTiming } from "../types"
|
||||
import { splitScriptIntoSentences, type ScriptSentence } from "../utils/sentences"
|
||||
|
||||
interface ModalBRollEditorProps {
|
||||
@@ -18,10 +18,12 @@ interface ModalBRollEditorProps {
|
||||
onClose: () => void
|
||||
/** 当前已有的 B-roll segments(用于标灰已选素材) */
|
||||
existingSegments: BRollSegment[]
|
||||
/** 当前文案全文(用于分句) */
|
||||
/** 文案全文(优先使用对口型时锁定的 scriptText) */
|
||||
scriptText: string
|
||||
/** 对口型成片总时长(秒),用于时间自动估算 */
|
||||
/** 对口型成片总时长(秒) */
|
||||
outputDuration: number
|
||||
/** 后端精确句子时间戳(来自 lipsyncJob.sentence_timings) */
|
||||
sentenceTimings?: SentenceTiming[] | null
|
||||
onConfirm: (segment: BRollSegment) => void
|
||||
onRemove: (id: string) => void
|
||||
}
|
||||
@@ -44,6 +46,7 @@ const ModalBRollEditor: React.FC<ModalBRollEditorProps> = ({
|
||||
existingSegments,
|
||||
scriptText,
|
||||
outputDuration,
|
||||
sentenceTimings,
|
||||
onConfirm,
|
||||
onRemove,
|
||||
}) => {
|
||||
@@ -62,10 +65,10 @@ const ModalBRollEditor: React.FC<ModalBRollEditorProps> = ({
|
||||
const [pipPosition, setPipPosition] = useState<PipPosition>("top-right")
|
||||
const [pipScale, setPipScale] = useState(0.3)
|
||||
|
||||
/** 文案分句(⑤) */
|
||||
/** 文案分句(使用后端精确时间戳) */
|
||||
const sentences = useMemo(
|
||||
() => splitScriptIntoSentences(scriptText, outputDuration),
|
||||
[scriptText, outputDuration],
|
||||
() => splitScriptIntoSentences(scriptText, sentenceTimings),
|
||||
[scriptText, sentenceTimings],
|
||||
)
|
||||
|
||||
/** 已被现有 segments 占用的素材 id 集合(标灰、禁止重复选择) */
|
||||
@@ -264,7 +267,7 @@ const ModalBRollEditor: React.FC<ModalBRollEditorProps> = ({
|
||||
>
|
||||
<span className="aa-sentence-item__idx">{sent.index + 1}</span>
|
||||
<span className="aa-sentence-item__text">{sent.text}</span>
|
||||
{outputDuration > 0 && (
|
||||
{sent.endTime > 0 && (
|
||||
<span className="aa-sentence-item__time">
|
||||
{sent.startTime.toFixed(1)}-{sent.endTime.toFixed(1)}s
|
||||
</span>
|
||||
@@ -349,7 +352,7 @@ const ModalBRollEditor: React.FC<ModalBRollEditorProps> = ({
|
||||
selectedSentence.endTime,
|
||||
selectedSentence.startTime + 0.5,
|
||||
).toFixed(1)}
|
||||
s (按字数自动估算)
|
||||
s
|
||||
</div>
|
||||
</>
|
||||
) : (
|
||||
|
||||
@@ -120,7 +120,7 @@ const PanelCoverAndGenerate: React.FC<PanelCoverAndGenerateProps> = ({
|
||||
wordBreak: "break-word",
|
||||
whiteSpace: "pre-wrap",
|
||||
color: titleConfig.color || "#ffffff",
|
||||
fontSize: `${titleConfig.size}px`,
|
||||
fontSize: `${(titleConfig.size || 28) * 0.55}px`, // 预览容器缩放,与 PanelLipsyncPreview 对齐
|
||||
fontFamily: getFontFamily(titleConfig.font),
|
||||
fontWeight: titleConfig.bold ? "bold" : "normal",
|
||||
fontStyle: titleConfig.italic ? "italic" : "normal",
|
||||
@@ -153,8 +153,8 @@ const PanelCoverAndGenerate: React.FC<PanelCoverAndGenerateProps> = ({
|
||||
} else if (titleConfig.shadow) {
|
||||
style.textShadow = "0 2px 8px rgba(0,0,0,0.7), 0 0 2px rgba(0,0,0,0.5)"
|
||||
} else {
|
||||
// 默认给轻微阴影保证白字在亮背景可读
|
||||
style.textShadow = "0 2px 6px rgba(0,0,0,0.6)"
|
||||
// 无描边无阴影时,不加额外效果(与后端 drawtext 对齐:无 stroke/shadow 则不加)
|
||||
style.textShadow = "none"
|
||||
}
|
||||
|
||||
return style
|
||||
|
||||
@@ -14,8 +14,8 @@ interface PanelLipsyncPreviewProps {
|
||||
onRemoveBRoll: (id: string) => void
|
||||
/** 标题配置(实时叠加预览用) */
|
||||
titleConfig?: AiAvatarTitleConfig
|
||||
/** 标题位置变更回调(拖拽结束时调用) */
|
||||
onTitlePositionChange?: (pos: { pos_x: number; pos_y: number }) => void
|
||||
/** 标题位置变更回调(拖拽结束时调用,发送百分比坐标 + position:"custom") */
|
||||
onTitlePositionChange?: (pos: { pos_x: number; pos_y: number; position: string }) => void
|
||||
}
|
||||
|
||||
const BROLL_MODE_LABEL: Record<BRollSegment["mode"], string> = {
|
||||
@@ -68,11 +68,13 @@ export function PanelLipsyncPreview({
|
||||
padding: "4px 8px",
|
||||
textShadow: titleConfig.shadow ? "0 2px 4px rgba(0,0,0,0.8)" : undefined,
|
||||
WebkitTextStroke: titleConfig.stroke ? "1.5px #000" : undefined,
|
||||
...(titleConfig.position === "top"
|
||||
? { top: 8 }
|
||||
: titleConfig.position === "bottom"
|
||||
? { bottom: 8 }
|
||||
: { top: "50%", transform: "translateX(-50%) translateY(-50%)" }),
|
||||
...(titleConfig.position === "custom" && titleConfig.pos_x != null && titleConfig.pos_y != null
|
||||
? { left: `${titleConfig.pos_x}%`, top: `${titleConfig.pos_y}%`, transform: "translateX(-50%) translateY(-50%)" }
|
||||
: titleConfig.position === "top"
|
||||
? { top: 8 }
|
||||
: titleConfig.position === "bottom"
|
||||
? { bottom: 8 }
|
||||
: { top: "50%", transform: "translateX(-50%) translateY(-50%)" }),
|
||||
}
|
||||
: null
|
||||
|
||||
@@ -105,7 +107,10 @@ export function PanelLipsyncPreview({
|
||||
const rect = previewContainerRef.current.getBoundingClientRect()
|
||||
const relX = Math.max(0, Math.min(rect.width, e.clientX - rect.left))
|
||||
const relY = Math.max(0, Math.min(rect.height, e.clientY - rect.top))
|
||||
onTitlePositionChange({ pos_x: relX, pos_y: relY })
|
||||
// 发送百分比坐标(0-100),与后端 drawtext 百分比表达式对齐
|
||||
const xpct = Math.round((relX / rect.width) * 1000) / 10
|
||||
const ypct = Math.round((relY / rect.height) * 1000) / 10
|
||||
onTitlePositionChange({ pos_x: xpct, pos_y: ypct, position: "custom" })
|
||||
}
|
||||
;(e.currentTarget as HTMLDivElement).style.cursor = "grab"
|
||||
}
|
||||
|
||||
@@ -44,12 +44,22 @@ export interface LipsyncJob {
|
||||
status: LipsyncStatus
|
||||
progress: number
|
||||
output_video_url: string | null
|
||||
/** 对口型成片总时长(秒),后端返回;用于 B-roll 时间自动估算(#1809 ⑥) */
|
||||
/** 对口型成片总时长(秒),后端返回 */
|
||||
output_duration?: number
|
||||
/** 精确句子时间戳(后端基于 TTS 音频静音检测计算) */
|
||||
sentence_timings?: SentenceTiming[] | null
|
||||
error_message: string | null
|
||||
created_at: string
|
||||
}
|
||||
|
||||
/* ── 句子时间戳(后端精确计算) ── */
|
||||
export interface SentenceTiming {
|
||||
index: number
|
||||
text: string
|
||||
start_time: number
|
||||
end_time: number
|
||||
}
|
||||
|
||||
/* ── B-roll 画面插入 ── */
|
||||
export type BRollInsertMode = "fullscreen" | "pip"
|
||||
export type PipPosition = "top-left" | "top-right" | "bottom-left" | "bottom-right"
|
||||
@@ -77,7 +87,7 @@ export interface AiAvatarTitleConfig {
|
||||
shadow: boolean
|
||||
color: string
|
||||
auto_subtitle: boolean
|
||||
/** 自定义位置坐标(position=custom 时生效,像素) */
|
||||
/** 自定义位置坐标(position=custom 时生效,百分比 0-100) */
|
||||
pos_x?: number
|
||||
pos_y?: number
|
||||
}
|
||||
@@ -101,6 +111,7 @@ export interface RenderJob {
|
||||
status: RenderStatus
|
||||
progress: number
|
||||
output_video_url: string | null
|
||||
output_cover_url: string | null
|
||||
error_message: string | null
|
||||
created_at: string
|
||||
}
|
||||
|
||||
@@ -39,7 +39,7 @@ export function buildTitleConfigPayload(cfg: AiAvatarTitleConfig): Record<string
|
||||
text,
|
||||
enabled: true,
|
||||
font: cfg.font || "思源黑体",
|
||||
font_size: Math.round(cfg.size) || 36,
|
||||
font_size: Math.round(cfg.size) || 28,
|
||||
font_color: cfg.color || "#ffffff",
|
||||
position,
|
||||
bold: !!cfg.bold,
|
||||
|
||||
@@ -1,6 +1,10 @@
|
||||
/**
|
||||
* AI数字人 — 文案分句 & B-roll 时间自动估算(#1809 ⑤⑥)
|
||||
* AI数字人 — 文案分句工具
|
||||
*
|
||||
* 分句规则与后端 _split_script_into_sentences 保持一致。
|
||||
* 时间戳由后端基于 TTS 音频静音检测精确计算,前端不再做字数比例估算。
|
||||
*/
|
||||
import type { SentenceTiming } from "../types"
|
||||
|
||||
export interface ScriptSentence {
|
||||
/** 句子序号(从 0 开始,对应提交给后端的 script_segment_index) */
|
||||
@@ -9,21 +13,21 @@ export interface ScriptSentence {
|
||||
text: string
|
||||
/** 句子字数(按中文/字符计,去除空白) */
|
||||
charCount: number
|
||||
/** 累计起始字数(用于时间估算) */
|
||||
/** 累计起始字数 */
|
||||
startChar: number
|
||||
/** 估算的对口型视频内起始时间(秒) */
|
||||
/** 精确起始时间(秒),来自后端 sentence_timings;无数据时为 0 */
|
||||
startTime: number
|
||||
/** 估算的对口型视频内结束时间(秒) */
|
||||
/** 精确结束时间(秒),来自后端 sentence_timings;无数据时为 0 */
|
||||
endTime: number
|
||||
}
|
||||
|
||||
/**
|
||||
* 按句号/问号/感叹号/分号/换行分句(兼容中英文标点)。
|
||||
* 空文案返回空数组。时间按「该句字数 ÷ 全文总字数 × 口播总时长」线性估算。
|
||||
* 时间戳从后端 sentence_timings 获取(精确);若无则返回 0(由调用方降级处理)。
|
||||
*/
|
||||
export function splitScriptIntoSentences(
|
||||
scriptText: string,
|
||||
outputDuration: number,
|
||||
sentenceTimings?: SentenceTiming[] | null,
|
||||
): ScriptSentence[] {
|
||||
const text = (scriptText || "").trim()
|
||||
if (!text) return []
|
||||
@@ -33,30 +37,25 @@ export function splitScriptIntoSentences(
|
||||
.map((part) => part.trim())
|
||||
.filter((part) => part.length > 0)
|
||||
|
||||
const totalChars = rawParts.reduce((sum, part) => sum + part.replace(/\s/g, "").length, 0)
|
||||
const duration = outputDuration > 0 ? outputDuration : 0
|
||||
|
||||
const sentences: ScriptSentence[] = []
|
||||
let accChar = 0
|
||||
rawParts.forEach((part, i) => {
|
||||
const charCount = part.replace(/\s/g, "").length
|
||||
const startTime = duration > 0 && totalChars > 0 ? (accChar / totalChars) * duration : 0
|
||||
const endTime =
|
||||
duration > 0 && totalChars > 0 ? ((accChar + charCount) / totalChars) * duration : 0
|
||||
// 从后端精确时间戳获取;无数据时返回 0
|
||||
const timing = sentenceTimings?.[i]
|
||||
const startTime = timing?.start_time ?? 0
|
||||
const endTime = timing?.end_time ?? 0
|
||||
|
||||
sentences.push({
|
||||
index: i,
|
||||
text: part,
|
||||
charCount,
|
||||
startChar: accChar,
|
||||
startTime: round1(startTime),
|
||||
endTime: round1(endTime),
|
||||
startTime,
|
||||
endTime,
|
||||
})
|
||||
accChar += charCount
|
||||
})
|
||||
|
||||
return sentences
|
||||
}
|
||||
|
||||
function round1(n: number): number {
|
||||
return Math.round(n * 10) / 10
|
||||
}
|
||||
|
||||
@@ -703,6 +703,9 @@ class LipsyncJobModel(Base):
|
||||
error_message = Column(Text, nullable=False, default="")
|
||||
error_code = Column(String(100), nullable=False, default="")
|
||||
|
||||
# 精确句子时间戳(TTS 合成后由 silencedetect 计算,用于 B-roll 精确定位)
|
||||
sentence_timings = Column(JSON, nullable=True) # list[{index,text,start_time,end_time}]
|
||||
|
||||
# 时间戳
|
||||
submitted_at = Column(DateTime, nullable=True)
|
||||
completed_at = Column(DateTime, nullable=True)
|
||||
|
||||
@@ -481,13 +481,13 @@ def build_title_drawtext_filter(
|
||||
|
||||
# ── 样式参数 ──
|
||||
font_name = title_config.get("font") or title_config.get("font_preset") or "思源黑体"
|
||||
font_size = int(title_config.get("font_size") or title_config.get("size") or 36)
|
||||
font_size = int(title_config.get("font_size") or title_config.get("size") or 28)
|
||||
font_color = title_config.get("font_color") or title_config.get("color") or "#ffffff"
|
||||
# 去掉 # 前缀(drawtext 用纯 hex 或颜色名)
|
||||
if font_color.startswith("#"):
|
||||
font_color = font_color[1:]
|
||||
|
||||
position = title_config.get("position", "top")
|
||||
position = title_config.get("position") or "bottom"
|
||||
bold = bool(title_config.get("bold", True))
|
||||
stroke = title_config.get("stroke")
|
||||
shadow = title_config.get("shadow")
|
||||
@@ -556,8 +556,13 @@ def build_title_drawtext_filter(
|
||||
and not isinstance(pos_x, bool)
|
||||
and not isinstance(pos_y, bool)
|
||||
):
|
||||
params.append(f"x={int(pos_x)}")
|
||||
params.append(f"y={int(pos_y)}")
|
||||
# pos_x/pos_y 为百分比坐标(0-100),转换为 drawtext 表达式
|
||||
# 例如 pos_x=50 → x=(w-text_w)*0.50(水平居中偏50%)
|
||||
# pos_y=30 → y=(h-text_h)*0.30
|
||||
pct_x = max(0.0, min(100.0, float(pos_x))) / 100.0
|
||||
pct_y = max(0.0, min(100.0, float(pos_y))) / 100.0
|
||||
params.append(f"x=(w-text_w)*{pct_x:.4f}")
|
||||
params.append(f"y=(h-text_h)*{pct_y:.4f}")
|
||||
else:
|
||||
# 三档预设位置:top / center / bottom
|
||||
# x 始终水平居中:(w-text_w)/2
|
||||
|
||||
@@ -1065,12 +1065,23 @@ class TestDrawtextPositionBranches(unittest.TestCase):
|
||||
self.assertIn("y=h-text_h-50", result)
|
||||
|
||||
@patch("packages.domain.video_filter_builder._resolve_font_path")
|
||||
def test_position_custom_with_float_coords(self, mock_font):
|
||||
def test_position_custom_with_percentage_coords(self, mock_font):
|
||||
"""自定义位置:百分比坐标转换为 drawtext 表达式."""
|
||||
mock_font.return_value = ""
|
||||
# pos_x=50, pos_y=30 → x=(w-text_w)*0.5000, y=(h-text_h)*0.3000
|
||||
result = build_title_drawtext_filter({"text": "标题", "position": "custom", "pos_x": 50, "pos_y": 30})
|
||||
self.assertIsNotNone(result)
|
||||
self.assertIn("x=(w-text_w)*0.5000", result)
|
||||
self.assertIn("y=(h-text_h)*0.3000", result)
|
||||
|
||||
@patch("packages.domain.video_filter_builder._resolve_font_path")
|
||||
def test_position_custom_clamped_to_100(self, mock_font):
|
||||
"""自定义位置:超过100的坐标被截断到100%."""
|
||||
mock_font.return_value = ""
|
||||
result = build_title_drawtext_filter({"text": "标题", "position": "custom", "pos_x": 100.7, "pos_y": 200.3})
|
||||
self.assertIsNotNone(result)
|
||||
self.assertIn("x=100", result)
|
||||
self.assertIn("y=200", result)
|
||||
self.assertIn("x=(w-text_w)*1.0000", result)
|
||||
self.assertIn("y=(h-text_h)*1.0000", result)
|
||||
|
||||
@patch("packages.domain.video_filter_builder._resolve_font_path")
|
||||
def test_position_custom_bool_coords_fallback(self, mock_font):
|
||||
|
||||
Reference in New Issue
Block a user