Compare commits

..

4 Commits

Author SHA1 Message Date
CI Bot dc6c442ea1 style: auto-format with black + isort + ruff + prettier [skip ci-format-check]
CI/CD Pipeline / Build Production API Image (pull_request) Blocked by required conditions
CI/CD Pipeline / Build Production Web Image (pull_request) Blocked by required conditions
CI/CD Pipeline / Build Production Worker Image (pull_request) Blocked by required conditions
CI/CD Pipeline / Deploy Production (pull_request) Blocked by required conditions
CI/CD Pipeline / Production Browser E2E (pull_request) Blocked by required conditions
CI/CD Pipeline / Canary Release to Production (pull_request) Blocked by required conditions
CI/CD Pipeline / CI Gate (pull_request) Blocked by required conditions
PR Automation / Auto Merge on CI Green + Approved (pull_request) Blocked by required conditions
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 2s
AI Code Review / AI Code Review (pull_request) Has started running
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 1s
PR Automation / Auto Approve on CI Green (pull_request) Has started running
CI/CD Pipeline / Validate - Style (pull_request) Has started running
CI/CD Pipeline / Validate - Security (pull_request) Has started running
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Has started running
CI/CD Pipeline / Unit Tests (pull_request) Has started running
CI/CD Pipeline / Integration Tests (pull_request) Has started running
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 44s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 46s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 1m3s
2026-09-26 06:47:35 +00:00
saas-backend fa8928174a test(voice-clone): 修复 _resolve_task 跨 Celery 版本鲁棒性
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 4s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 4s
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 46s
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 2m45s
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 1m24s
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
CI/CD Pipeline / Integration Tests (pull_request) Successful in 2m55s
CI/CD Pipeline / Unit Tests (pull_request) Failing after 2m57s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 3m56s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 2m48s
AI Code Review / AI Code Review (pull_request) Successful in 7m8s
CI/CD Pipeline / Validate - Security (pull_request) Successful in 7m37s
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 10m48s
CI/CD Pipeline / Validate - Style (pull_request) Has been cancelled
CI/CD Pipeline / Build Production API Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Web Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been cancelled
CI/CD Pipeline / Deploy Production (pull_request) Has been cancelled
CI/CD Pipeline / Production Browser E2E (pull_request) Has been cancelled
CI/CD Pipeline / Canary Release to Production (pull_request) Has been cancelled
CI/CD Pipeline / CI Gate (pull_request) Has been cancelled
- _get_current_object() 加 try/except 兜底,无 active app context 时不抛 AttributeError
- _resolve_task 统一返回 (callable, mock_self, real_task) 三元组
- timeout 重试用例复用 real_task patch retry,不再二次调用无保护的 _get_current_object()
- mock_self 判断改为 is not None(语义更清晰,兼容 MagicMock 真值)
2026-09-26 14:30:39 +08:00
CI Bot 04b18e3d64 style: auto-format with black + isort + ruff + prettier [skip ci-format-check]
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 1m12s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 3m2s
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 2m29s
AI Code Review / AI Code Review (pull_request) Successful in 6m33s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 0s
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 1s
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 2m35s
CI/CD Pipeline / Unit Tests (pull_request) Failing after 2m36s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 2m59s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 3m31s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 3m40s
CI/CD Pipeline / Validate - Style (pull_request) Successful in 3m52s
CI/CD Pipeline / Validate - Security (pull_request) Successful in 7m9s
CI/CD Pipeline / Build Production API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI/CD Pipeline / CI Gate (pull_request) Failing after 0s
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / Canary Release to Production (pull_request) Has been skipped
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
2026-09-26 03:43:35 +00:00
CI Bot 146effcf07 fix(voice-clone): 处理中卡死兜底 — worker重启/Celery消息丢失后processing记录永久卡住问题
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 1s
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 2s
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 1m31s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m55s
CI/CD Pipeline / Unit Tests (pull_request) Failing after 2m8s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 3m3s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 3m8s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 3m18s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 3m17s
CI/CD Pipeline / Validate - Security (pull_request) Successful in 5m17s
AI Code Review / AI Code Review (pull_request) Successful in 6m30s
CI/CD Pipeline / Build Production API Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Web Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been cancelled
CI/CD Pipeline / Deploy Production (pull_request) Has been cancelled
CI/CD Pipeline / Production Browser E2E (pull_request) Has been cancelled
CI/CD Pipeline / Canary Release to Production (pull_request) Has been cancelled
CI/CD Pipeline / CI Gate (pull_request) Has been cancelled
CI/CD Pipeline / Validate - Style (pull_request) Has been cancelled
PR Automation / Auto Merge on CI Green + Approved (pull_request) Has been cancelled
根因:
- 声音克隆走 API提交CosyVoice→Celery worker轮询5分钟→mark ready/failed
- 当worker容器重启、进程OOM或Celeryprefetch消息丢失时,已进入processing状态的voice_clone_profile没有兜底恢复机制
- 对比ingest链路有cleanup_stale_ingest_jobs beat任务+worker_ready启动恢复,voice_clone链路缺失
- 结果: 用户看到"克隆处理中,请稍候..."永久转圈,刷新也不变

修复:
1. SQLAlchemyVoiceCloneProfileRepository新增cleanup_stale_processing方法:
   扫描updated_at超过10分钟的processing记录(正常克隆<5分钟),标记failed并提示用户重试
2. _startup.py新增recover_stale_voice_clones_on_startup+worker_ready信号:
   worker启动时一次性扫描恢复,用户重启/部署后立即解锁卡死任务
3. cleanup.py新增scheduled_cleanup_stale_voice_clones beat任务:
   每5分钟巡检兜底,防止运行期OOM/消息丢失导致的新卡死
4. celery_app.py beat_schedule注册新任务
5. voice_clone.py任务入口加received日志,便于日志排查
2026-09-26 11:34:43 +08:00
11 changed files with 229 additions and 153 deletions
@@ -562,8 +562,6 @@ const GeneratePage: React.FC = () => {
generateError={generateError}
progress={progress}
generatedVideos={generatedVideos}
currentTaskId={currentTaskId}
onRetry={handleRetryGenerate}
onRetryBatchTask={handleRetryBatchTask}
onDismissError={handleDismissError}
@@ -91,8 +91,6 @@ export interface GenerateStepContentProps {
selectedVariantIds?: number[]
selectedCoverTemplate?: string
onSelectedCoverTemplateChange?: (templateId: string) => void
/** 单视频任务 ID(兜底,awaiting_cover 状态下 results 接口未入库时用) */
currentTaskId?: string
/** Step3 右上角确认生成按钮 */
onConfirmGenerate?: () => void | Promise<void>
}
@@ -266,7 +264,6 @@ export const GenerateStepContent: React.FC<GenerateStepContentProps> = (props) =
selectedVariantIndexes={selectedVariantIds}
selectedTemplate={selectedCoverTemplate}
onTemplateChange={onSelectedCoverTemplateChange}
currentTaskId={props.currentTaskId}
/>
)
default:
@@ -30,8 +30,6 @@ interface Step6CoverSettingsProps {
onPreviewCoversChange?: (urls: string[]) => void
selectedVariantIndexes?: number[]
onTemplateChange?: (templateId: string) => void
/** 单视频任务 ID(awaiting_cover 阶段 results 接口可能返回 preview-xxx 合成对象,兜底用) */
currentTaskId?: string
}
const Step6CoverSettings: React.FC<Step6CoverSettingsProps> = (props) => {
@@ -49,32 +47,6 @@ const Step6CoverSettings: React.FC<Step6CoverSettingsProps> = (props) => {
props.generatedVideos.find((v) => v.status === "completed" || v.status === "awaiting_cover") ||
props.generatedVideos[0]
/**
* 兜底任务/视频 ID:awaiting_cover 阶段后端 /results 可能还没有入库 GeneratedVideo,
* 只返回合成的 preview-{taskId} 轻量对象;此时用 currentTaskId 兜底让后端能找到任务。
* 同时统一抽取 taskId(generation_task_id 优先)用于日志/错误提示。
*/
const effectiveTaskId =
(finalVideo as { generation_task_id?: string } | undefined)?.generation_task_id ||
props.currentTaskId ||
""
const _rawVideoId =
(finalVideo as { id?: string; video_id?: string } | undefined)?.id ||
(finalVideo as { video_id?: string } | undefined)?.video_id ||
""
// preview-{taskId} 是后端合成的临时 id,gv_repo.get 查不到 → 不传 generated_video_id,
// 让后端走 plan.config.generation_task_id / rendered_storage_key 兜底路径。
const effectiveVideoId = _rawVideoId && !_rawVideoId.startsWith("preview-") ? _rawVideoId : ""
const effectiveVideoUrl = finalVideo?.file_url || finalVideo?.download_url || ""
/** 按钮可用:非批量 且 (有 finalVideo 对象或兜底 taskId) 且 视频状态已完成/等待封面/未设置 */
const isVideoReady =
!finalVideo ||
finalVideo.status === "completed" ||
finalVideo.status === "awaiting_cover" ||
!finalVideo.status
const canGenerateCover = !isBatch && (!!finalVideo || !!effectiveTaskId) && isVideoReady
const completedVideos = useMemo(
() =>
props.generatedVideos.filter(
@@ -88,58 +60,45 @@ const Step6CoverSettings: React.FC<Step6CoverSettingsProps> = (props) => {
* 批量场景 canGenerate=false,避免 shared.generateAutoCover 被误触发
*/
const shared = useSharedCover({
canGenerate: canGenerateCover,
disabledHint: isBatch
? "批量场景请在上方操作卡片"
: !finalVideo && !effectiveTaskId
? "请先生成视频再选择封面"
: "视频尚未就绪,请稍候",
canGenerate:
!!finalVideo &&
!isBatch &&
(finalVideo.status === "completed" ||
finalVideo.status === "awaiting_cover" ||
!finalVideo.status),
disabledHint: isBatch ? "批量场景请在上方操作卡片" : "请先生成视频再选择封面",
initialTemplateId: "default", // 封面模板独立于编辑模板,默认用 default
generateFn: async (tplId) => {
if (isBatch) return null
if (!finalVideo && !effectiveTaskId) {
console.warn("[Cover] generateAutoCover: no finalVideo and no taskId")
return null
}
// 请求体:generated_video_id 仅在后端已入库(非 preview-xxx 合成id)时传;
// video_url 兜底让后端能直接下载视频抽帧;generation_task_id 后端已从 plan.config 自动读取。
const requestBody: {
generated_video_id?: string
video_url?: string
cover_type: "ai_frame"
title_config?: Record<string, unknown>
} = {
if (!finalVideo || isBatch) return null
const response = await apiGenerateCover(tplId, {
generated_video_id:
(finalVideo as { id?: string; video_id?: string }).id ||
(finalVideo as { video_id?: string }).video_id ||
"",
video_url: finalVideo.file_url || finalVideo.download_url || "",
cover_type: "ai_frame",
}
if (effectiveVideoId) {
requestBody.generated_video_id = effectiveVideoId
}
if (effectiveVideoUrl) {
requestBody.video_url = effectiveVideoUrl
}
if (props.titleSettings?.title) {
requestBody.title_config = {
text: props.titleSettings.title,
font: props.titleSettings.font,
font_size: props.titleSettings.size,
font_color: props.titleSettings.color,
position: props.titleSettings.position,
bold: props.titleSettings.bold,
stroke: props.titleSettings.stroke,
shadow: props.titleSettings.shadow,
}
}
console.log("[Cover] auto-generate request:", { tplId, ...requestBody })
const response = await apiGenerateCover(tplId, requestBody)
const url = response.cover?.image_url || response.cover?.thumbnail_url || ""
...(props.titleSettings?.title
? {
title_config: {
text: props.titleSettings.title,
font: props.titleSettings.font,
font_size: props.titleSettings.size,
font_color: props.titleSettings.color,
position: props.titleSettings.position,
bold: props.titleSettings.bold,
stroke: props.titleSettings.stroke,
shadow: props.titleSettings.shadow,
},
}
: {}),
})
const url = response.cover?.image_url || ""
if (url) {
props.onCoverSettingsChange({
...props.coverSettings,
thumbnail_url: url,
ai_suggested_time: response.cover?.frame_time ?? null,
})
} else {
console.warn("[Cover] generate returned empty url:", response)
}
return url
},
@@ -364,7 +323,7 @@ const Step6CoverSettings: React.FC<Step6CoverSettingsProps> = (props) => {
<div className="xx-form-section">
<h3>🖼️ 选择封面</h3>
{(finalVideo || effectiveTaskId) && (
{finalVideo && (
<div
style={{
padding: "10px 14px",
@@ -376,7 +335,7 @@ const Step6CoverSettings: React.FC<Step6CoverSettingsProps> = (props) => {
color: "var(--text-secondary, #666)",
}}
>
🎬 封面将从最终成片{finalVideo?.name ? `「${finalVideo.name}」` : ""}中智能选帧
🎬 封面将从最终成片「{finalVideo.name}」中智能选帧
{shared.selectedTemplateId && shared.selectedTemplateId !== "default" && (
<>
{" "}
@@ -390,9 +349,8 @@ const Step6CoverSettings: React.FC<Step6CoverSettingsProps> = (props) => {
<Button
buttonType="primary"
onClick={() => void shared.generateAutoCover()}
disabled={!canGenerateCover || shared.generating}
disabled={!finalVideo || shared.generating}
loading={shared.generating}
title={!canGenerateCover ? "请先完成视频生成" : ""}
>
✨ 自动生成封面
</Button>
@@ -16,19 +16,6 @@ import "@/components/cover/cover.css"
const mergeEditorConfig = (partial?: Partial<CoverEditorConfig> | null): CoverEditorConfig => {
const def = DEFAULT_EDITOR_CONFIG
const src = partial || {}
/** 兼容老模板:老版本 background 有 posX/posY/rotation,新版改为 offsetY(相对文字位置偏移)。
* 老模板黑底默认 posY 通常是 50(与文字对齐)或 80(副标题偏下),统一归一为 offsetY=0,
* 因为新版背景位置已自动跟随文字位置,offsetY 仅做相对微调。 */
const normalizeBg = (bg: Record<string, unknown> | undefined) => {
if (!bg) return {}
// 兼容老模板字段:posX/posY/rotation 在新版中已改为 offsetY(背景位置自动跟随文字)
const normalized = { ...bg }
delete (normalized as Record<string, unknown>).posX
delete (normalized as Record<string, unknown>).posY
delete (normalized as Record<string, unknown>).rotation
if (normalized.offsetY == null) normalized.offsetY = 0
return normalized
}
const mergeText = (
base: TextStyleConfig,
patch?: Partial<TextStyleConfig> | null,
@@ -36,10 +23,7 @@ const mergeEditorConfig = (partial?: Partial<CoverEditorConfig> | null): CoverEd
...base,
...(patch || {}),
position: { ...base.position, ...(patch?.position || {}) },
background: {
...base.background,
...normalizeBg(patch?.background as Record<string, unknown> | undefined),
},
background: { ...base.background, ...(patch?.background || {}) },
shadows: Array.isArray(patch?.shadows) ? [...patch!.shadows] : [...base.shadows],
})
return {
@@ -414,13 +398,13 @@ const TextStylePanel: React.FC<{
/>
</div>
<div className="xx-ce-row">
<label className="xx-ce-label">上下偏移: {config.background.offsetY ?? 0}%</label>
<label className="xx-ce-label">上下偏移: {config.background.posY - 50}%</label>
<Slider
min={-30}
max={30}
min={-50}
max={50}
step={1}
value={config.background.offsetY ?? 0}
onChange={(v) => upd("background", { ...config.background, offsetY: v })}
value={config.background.posY - 50}
onChange={(v) => upd("background", { ...config.background, posY: 50 + v })}
/>
</div>
</div>
@@ -611,9 +595,7 @@ const CoverEditorModal: React.FC<CoverEditorModalProps> = ({ open, onClose, temp
const renderTextBgStyle = (tc: TextStyleConfig | undefined | null): React.CSSProperties => {
if (!tc?.background?.enabled) return { display: "none" }
const bg = tc.background
// 背景位置跟随文字:left/top 对齐文字中心,用 offsetY(-50~50% 相对文字位置)做上下微调
// 这样文字拖拽时背景会自动跟随,不需要独立的位置控制
const radius = bg.shape === "polygon" ? `${Math.max(4, Math.round(bg.height / 4))}px` : "0"
const radius = bg.shape === "polygon" ? `${Math.max(4, Math.round(bg.height / 2))}px` : "0"
const alpha = Math.max(0, Math.min(1, bg.opacity / 100))
const hex = (bg.color || "#000000").replace("#", "")
let r = 0,
@@ -625,12 +607,10 @@ const CoverEditorModal: React.FC<CoverEditorModalProps> = ({ open, onClose, temp
b = parseInt(hex.substring(4, 6), 16)
}
const rgba = `rgba(${r}, ${g}, ${b}, ${alpha})`
// bg.offsetY 是相对文字位置的上下偏移(-50~50,单位%画布高度),默认 0 表示与文字中心对齐
const offsetY = typeof bg.offsetY === "number" ? bg.offsetY : 0
return {
position: "absolute",
left: `${tc.position.x}%`,
top: `calc(${tc.position.y}% + ${offsetY}%)`,
left: `${bg.posX}%`,
top: `${bg.posY}%`,
width: `${bg.width}%`,
height: `${bg.height}%`,
transform: "translate(-50%, -50%)",
+9 -4
View File
@@ -63,8 +63,9 @@ export interface TextBackground {
shape: TextBgShape
width: number
height: number
/** 相对文字的上下偏移(百分比),背景自动跟随文字位置 */
offsetY: number
posX: number
posY: number
rotation: number
}
/** 文字样式配置(主标题/副标题共用) */
@@ -153,7 +154,9 @@ export const DEFAULT_TITLE_CONFIG: TextStyleConfig = {
shape: "polygon",
width: 30,
height: 10,
offsetY: 0,
posX: 50,
posY: 50,
rotation: 0,
},
}
@@ -181,7 +184,9 @@ export const DEFAULT_SUBTITLE_CONFIG: TextStyleConfig = {
shape: "rectangle",
width: 100,
height: 20,
offsetY: 8,
posX: 50,
posY: 80,
rotation: 0,
},
}
+6
View File
@@ -76,4 +76,10 @@ celery_app.conf.beat_schedule = {
"schedule": 600.0, # 每 10 分钟(秒)
"options": {"expires": 540},
},
# 音色克隆卡死巡检:worker 重启/消息丢失后 processing 卡 10 分钟标 failed,用户可点重试
"cleanup-stale-voice-clones": {
"task": "worker.cleanup_stale_voice_clones",
"schedule": 300.0, # 每 5 分钟
"options": {"expires": 240},
},
}
+47
View File
@@ -287,3 +287,50 @@ def _recover_stuck_ingest_jobs_on_ready(sender, **kwargs): # pragma: no cover
logger.info("Worker 启动 ingest 恢复完成,共重新派单 %d 个卡死任务", recovered)
except Exception as e: # noqa: BLE001 — 启动恢复失败不能阻断 worker 起服
logger.error("启动 ingest 恢复扫描失败(beat 巡检仍会兜底标 failed): %s", e, exc_info=True)
def recover_stale_voice_clones_on_startup(timeout_minutes: int = 10) -> int:
"""Worker 启动时恢复卡死在 processing 的音色克隆任务。
容器重启/进程 OOM 时 worker 中正在轮询的克隆任务会丢失,
voice_clone_profiles 永久卡在 processing 无兜底。启动时扫描
updated_at 超过 timeout_minutes 的 processing 记录,直接标记
为 failed(错误信息指引用户重试)。选择标 failed 而非重新派单,
因为 CosyVoice 侧的 voice_id 无法在无上下文下恢复轮询,重试需
用户确认后显式触发。
Args:
timeout_minutes: 判定卡死的阈值,默认 10 分钟
Returns:
恢复的记录数
"""
from packages.adapters.sqlalchemy_impl.voice_clone_profile_repository import (
SQLAlchemyVoiceCloneProfileRepository,
)
try:
session = SessionLocal()
try:
repo = SQLAlchemyVoiceCloneProfileRepository(session)
count = repo.cleanup_stale_processing(timeout_minutes)
finally:
session.close()
if count > 0:
logger.warning("启动时恢复了 %d 个卡死在 processing 的音色克隆(超时 %d 分钟)", count, timeout_minutes)
else:
logger.info("无卡死 processing 音色克隆需要恢复")
return count
except Exception as e:
logger.error("启动时音色克隆恢复扫描失败(beat 巡检仍会兜底): %s", e, exc_info=True)
return 0
@worker_ready.connect
def _recover_stuck_voice_clones_on_ready(sender, **kwargs):
"""Worker 启动完成后恢复卡死的音色克隆任务。"""
try:
recovered = recover_stale_voice_clones_on_startup()
logger.info("Worker 启动音色克隆恢复完成,共标记 %d 个卡死任务为 failed", recovered)
except Exception as e:
logger.error("启动音色克隆恢复失败(beat 巡检仍会兜底标 failed): %s", e, exc_info=True)
+41
View File
@@ -22,6 +22,10 @@ from packages.application.ingest_orphan_cleanup import (
INGEST_PROCESSING_TIMEOUT_MINUTES,
)
# 音色克隆 processing 超时:正常克隆轮询最多 5 分钟,10 分钟无更新视为卡死
VOICE_CLONE_PROCESSING_TIMEOUT_MINUTES = 10
logger = logging.getLogger(__name__)
@@ -125,3 +129,40 @@ def scheduled_cleanup_stale_ingest_jobs(
purged,
)
return {"stale_jobs": total_jobs, "assets_to_error": total_assets, "purged_messages": purged}
@shared_task(name="worker.cleanup_stale_voice_clones")
def scheduled_cleanup_stale_voice_clones(
processing_timeout_minutes: int = VOICE_CLONE_PROCESSING_TIMEOUT_MINUTES,
) -> dict:
"""Celery Beat: 清理卡死在 processing 的音色克隆档案。
每 5 分钟执行一次。worker 重启/Celery 消息丢失/进程 OOM 时,
已 prefetch 的克隆任务消息丢失,voice_clone_profile 永久卡在 processing。
超过 processing_timeout_minutes 未更新的记录标记为 failed,
错误信息指引用户点击重试。
"""
from worker_app.db import SessionLocal
from packages.adapters.sqlalchemy_impl.voice_clone_profile_repository import (
SQLAlchemyVoiceCloneProfileRepository,
)
session = None
try:
session = SessionLocal()
repo = SQLAlchemyVoiceCloneProfileRepository(session)
count = repo.cleanup_stale_processing(processing_timeout_minutes)
if count > 0:
logger.warning(
"[Beat] 清理了 %d 个卡死 processing 的音色克隆(超时 %d 分钟)",
count,
processing_timeout_minutes,
)
return {"cleaned": count}
except Exception as e:
logger.error("[Beat] 清理卡死音色克隆失败: %s", e, exc_info=True)
return {"cleaned": 0, "error": str(e)}
finally:
if session is not None:
session.close()
@@ -44,6 +44,7 @@ def process_voice_clone(self: Task, profile_id: str) -> dict:
# P2-2 修复:session 初始化为 None,避免 SessionLocal() 抛异常时
# finally 块中 session.close() 触发 UnboundLocalError
session = None
logger.info(f"Voice clone task started: profile_id={profile_id}")
try:
session = SessionLocal()
repo = SQLAlchemyVoiceCloneProfileRepository(session)
@@ -136,6 +136,39 @@ class SQLAlchemyVoiceCloneProfileRepository:
)
return {voice_id: profile_id for voice_id, profile_id in rows}
def cleanup_stale_processing(self, timeout_minutes: int = 10) -> int:
"""清理超时卡在 processing 的克隆档案。
worker 重启、Celery 任务丢失或 OOM 被杀时,processing 档案会永久卡住。
updated_at < NOW() - timeout_minutes 的 processing 记录,标记为 failed
并附带明确错误信息,用户可在前端点击「重试」。
Args:
timeout_minutes: 超时分钟数,默认 10 分钟(正常克隆 < 5 分钟)
Returns:
清理的记录数
"""
from datetime import UTC, datetime, timedelta
cutoff = datetime.now(UTC) - timedelta(minutes=timeout_minutes)
models = (
self.session.query(VoiceCloneProfileModel)
.filter(
VoiceCloneProfileModel.status == "processing",
VoiceCloneProfileModel.updated_at < cutoff,
)
.all()
)
count = 0
for model in models:
model.status = "failed"
model.error_message = f"克隆任务执行超时(超过 {timeout_minutes} 分钟未更新,可能因服务重启中断),请重试"
count += 1
if count > 0:
self.session.commit()
return count
@staticmethod
def _model_to_entity(model: VoiceCloneProfileModel) -> VoiceCloneProfile:
return VoiceCloneProfile(
+50 -40
View File
@@ -8,13 +8,14 @@
Celery bind=True 任务的底层函数签名为 (self, profile_id),
CosyVoiceService 在 voice_clone.py 中被实例化传入 workflow,必须 mock 防止真实初始化。
跨环境兼容:
Python 3.13 + Celery 5.4.0 → import 返回 Celery Proxy
→ _get_current_object() 返回 Task 实例 → .run 是 bound method(self 已绑定)
→ 调用方式:task.run(profile_id),retry mock 在 task.run.retry
Python 3.10 + Celery 5.4.0 → import 返回原始函数(装饰器未生效)
→ 签名 (self, profile_id),需手动传 mock_self
→ 调用方式:func(mock_self, profile_id),retry mock 在 mock_self.retry
跨环境兼容(_resolve_task):
不同 Celery 版本 / Python 版本 / 是否有 active Celery app,task 对象形态不同:
1) Celery Proxy(LocalProxy/LazyProxy):import 结果是代理对象,调用
_get_current_object() 可能抛 RuntimeError(无 active context),必须 try 保护。
成功取到真实 Task 实例后,使用 bound method .run。
2) Celery Task 实例(bind=True 时 @task 返回的典型形态):直接有 .run/.retry。
3) 原始函数(某些环境装饰器未生效或 patch 时序问题):需手动传 mock_self。
统一返回 (callable, mock_self, real_task),调用方不需要重复解析。
"""
from __future__ import annotations
@@ -58,24 +59,33 @@ def _make_mock_profile(
def _resolve_task(task_obj):
"""解析 Celery 任务对象,返回 (callable, mock_self_or_none)。
"""解析 Celery 任务对象,兼容 Proxy / Task 实例 / 原始函数三种形态。
跨环境兼容 Celery Proxy / Task 实例 / 原始函数三种情况。
所有分支均做异常保护,避免因 Celery Proxy 在无 app context 时抛错导致测试挂掉。
Returns:
tuple: (callable, mock_self)
- Proxy/Task: callable 是 bound method task.run,mock_self=None
- 原始函数: callable 是原始函数,mock_self 需由调用方提供
tuple: (callable, mock_self, real_task)
- callable: 最终执行用的可调用对象
- mock_self: 仅原始函数分支需要手动传入 mock self;其他分支为 None
- real_task: 真实 Task 实例(Proxy 分支为 _get_current_object() 结果;
Task 分支为 task_obj 本身;原始函数分支为 None)。用于 patch .retry。
"""
# Case 1: Celery Proxy → 提取 Task 实例的 .run(bound method)
# Case 1: Celery Proxy → 安全尝试 _get_current_object()
if hasattr(task_obj, "_get_current_object"):
real_task = task_obj._get_current_object()
return real_task.run, None
try:
real_task = task_obj._get_current_object()
if real_task is not None and hasattr(real_task, "run"):
return real_task.run, None, real_task
except Exception:
# 无 active app context 或 Proxy 未绑定,退化为其他分支处理
pass
# Case 2: Celery Task 实例(非 Proxy)
if hasattr(task_obj, "run") and hasattr(task_obj, "retry"):
return task_obj.run, None
# Case 3: 原始函数(CI 环境中装饰器未生效)
return task_obj, MagicMock()
return task_obj.run, None, task_obj
# Case 3: 原始函数(装饰器未生效)
return task_obj, MagicMock(), None
# ── 成功场景 ──────────────────────────────────────────────
@@ -110,8 +120,8 @@ class TestProcessVoiceCloneSuccess:
from worker_app.tasks.voice_clone import process_voice_clone
func, mock_self = _resolve_task(process_voice_clone)
args = (mock_self, "profile-123") if mock_self else ("profile-123",)
func, mock_self, _ = _resolve_task(process_voice_clone)
args = (mock_self, "profile-123") if mock_self is not None else ("profile-123",)
result = func(*args)
assert result["ok"] is True
@@ -148,8 +158,8 @@ class TestProcessVoiceCloneSuccess:
from worker_app.tasks.voice_clone import process_voice_clone
func, mock_self = _resolve_task(process_voice_clone)
args = (mock_self, "nonexistent") if mock_self else ("nonexistent",)
func, mock_self, _ = _resolve_task(process_voice_clone)
args = (mock_self, "nonexistent") if mock_self is not None else ("nonexistent",)
result = func(*args)
assert result["ok"] is False
@@ -189,24 +199,24 @@ class TestProcessVoiceCloneTimeout:
from worker_app.tasks.voice_clone import process_voice_clone
func, mock_self = _resolve_task(process_voice_clone)
func, mock_self, real_task = _resolve_task(process_voice_clone)
# 设置 retry mock:根据环境不同,retry 在不同对象上
if mock_self is None:
# Proxy/Task 环境:retry 在 Task 实例上(func 是 bound method task.run)
real_task = process_voice_clone._get_current_object()
mock_retry = MagicMock()
mock_retry.side_effect = Retry("retrying")
with patch.object(real_task, "retry", mock_retry):
with pytest.raises(Retry):
func("profile-123")
mock_retry.assert_called_once()
else:
if mock_self is not None:
# 原始函数环境:retry 在 mock_self 上
mock_self.retry.side_effect = Retry("retrying")
with pytest.raises(Retry):
func(mock_self, "profile-123")
mock_self.retry.assert_called_once()
else:
# Proxy/Task 环境:retry 在 Task 实例上。用 _resolve_task 返回的 real_task,
# 避免再次 _get_current_object() 在无 context 时抛 AttributeError。
retry_target = real_task if real_task is not None else process_voice_clone
mock_retry = MagicMock()
mock_retry.side_effect = Retry("retrying")
with patch.object(retry_target, "retry", mock_retry):
with pytest.raises(Retry):
func("profile-123")
mock_retry.assert_called_once()
mock_session.rollback.assert_called_once()
mock_session.close.assert_called_once()
@@ -243,8 +253,8 @@ class TestProcessVoiceCloneFailure:
from worker_app.tasks.voice_clone import process_voice_clone
func, mock_self = _resolve_task(process_voice_clone)
args = (mock_self, "profile-123") if mock_self else ("profile-123",)
func, mock_self, _ = _resolve_task(process_voice_clone)
args = (mock_self, "profile-123") if mock_self is not None else ("profile-123",)
result = func(*args)
assert result["ok"] is False
@@ -277,8 +287,8 @@ class TestProcessVoiceCloneFailure:
from worker_app.tasks.voice_clone import process_voice_clone
func, mock_self = _resolve_task(process_voice_clone)
args = (mock_self, "profile-123") if mock_self else ("profile-123",)
func, mock_self, _ = _resolve_task(process_voice_clone)
args = (mock_self, "profile-123") if mock_self is not None else ("profile-123",)
result = func(*args)
assert result["ok"] is False
@@ -311,8 +321,8 @@ class TestProcessVoiceCloneFailure:
from worker_app.tasks.voice_clone import process_voice_clone
func, mock_self = _resolve_task(process_voice_clone)
args = (mock_self, "profile-123") if mock_self else ("profile-123",)
func, mock_self, _ = _resolve_task(process_voice_clone)
args = (mock_self, "profile-123") if mock_self is not None else ("profile-123",)
result = func(*args)
assert result["ok"] is False