7 Commits

Author SHA1 Message Date
xiaoxia 28b3010668 fix(dedup): 修复查重率恒为0%——指纹绕开降重裁剪+局部片段复用+阈值校准+3个单位bug (#1702) (#1703)
CI/CD Pipeline / Check if frontend-only change (push) Has been skipped
CI/CD Pipeline / PR Build API Image (push) Has been skipped
CI/CD Pipeline / PR Build Web Image (push) Has been skipped
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (push) Successful in 2s
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / PR Build Worker Image (push) Has been skipped
CI/CD Pipeline / Check push changed paths (push) Successful in 4s
CI/CD Pipeline / Frontend Lint (push) Has been skipped
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 5s
CI/CD Pipeline / Build Staging Web Image (push) Has been skipped
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 7s
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging API Image (push) Successful in 38s
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 18s
CI/CD Pipeline / Validate - Style (pull_request) Has been skipped
CI/CD Pipeline / Validate - Security (pull_request) Has been skipped
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Has been skipped
CI/CD Pipeline / Unit Tests (pull_request) Has been skipped
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
CI/CD Pipeline / Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
CI/CD Pipeline / Build Production API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Web Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (push) Successful in 43s
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (push) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (push) Has been skipped
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / Canary Release to Production (pull_request) Has been skipped
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 22s
CI/CD Pipeline / CI Gate (pull_request) Successful in 8s
CI/CD Pipeline / Retag skipped Staging Web Image (push) Successful in 28s
CI/CD Pipeline / Validate - Python (mypy + alembic) (push) Successful in 2m1s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 2m21s
CI/CD Pipeline / Integration Tests (push) Successful in 2m30s
AI Code Review / AI Code Review (pull_request) Failing after 2m33s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (push) Successful in 1m10s
CI/CD Pipeline / Validate - Style (push) Successful in 3m13s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 3m12s
PR Automation / Auto Merge on CI Green + Approved (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (push) Successful in 1m42s
CI/CD Pipeline / Validate - Security (push) Successful in 5m58s
CI/CD Pipeline / Frontend Unit Tests (push) Successful in 6m6s
CI/CD Pipeline / Staging API Integration Tests (push) Successful in 4m38s
CI/CD Pipeline / Unit Tests (push) Successful in 8m43s
CI/CD Pipeline / Build Production API Image (push) Has been skipped
CI/CD Pipeline / Build Production Web Image (push) Has been skipped
CI/CD Pipeline / Build Production Worker Image (push) Has been skipped
CI/CD Pipeline / CI Gate (push) Has been skipped
CI/CD Pipeline / Deploy Production (push) Has been skipped
CI/CD Pipeline / Production Browser E2E (push) Has been skipped
CI/CD Pipeline / Canary Release to Production (push) Has been skipped
CI/CD Pipeline / Staging E2E Tests (push) Successful in 6m35s
Co-authored-by: xiaoxia <dev@xiaoxiajianji.com>
Co-committed-by: xiaoxia <dev@xiaoxiajianji.com>
2026-09-05 07:57:23 +08:00
xiaoxia fac80b1f77 feat(dedup): 动态抽帧 + 滑动窗口时序匹配 (#1659)
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 1s
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 2s
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 22s
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 9s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
CI/CD Pipeline / Unit Tests (pull_request) Successful in 1m22s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m25s
AI Code Review / AI Code Review (pull_request) Failing after 1m50s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 2m39s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 2m44s
CI/CD Pipeline / Validate - Style (pull_request) Has been cancelled
CI/CD Pipeline / Validate - Security (pull_request) Has been cancelled
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Has been cancelled
CI/CD Pipeline / Build Production API Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Web Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been cancelled
CI/CD Pipeline / Deploy Production (pull_request) Has been cancelled
CI/CD Pipeline / Production Browser E2E (pull_request) Has been cancelled
CI/CD Pipeline / Canary Release to Production (pull_request) Has been cancelled
CI/CD Pipeline / CI Gate (pull_request) Has been cancelled
PR Automation / Auto Merge on CI Green + Approved (pull_request) Has been cancelled
1. 动态抽帧策略 — detect_keyframe_timestamps()
   - 降采样到 320x240 逐帧灰度差异检测场景切换
   - 最小间隔过滤(保留差异最大的候选帧)
   - 数量裁剪到 [MIN_KEYFRAMES=5, MAX_KEYFRAMES=30]
   - 长视频(>3分钟)每 30 秒分段保底

2. 滑动窗口时序匹配 — find_duplicate_segments()
   - 逐帧最佳匹配 → 连续 run 检测(允许 MAX_GAP=2 间隙)
   - 最少 MIN_CONSECUTIVE_MATCHES=5 帧才报告
   - 返回 DuplicateSegment(query/target 时间范围 + 平均距离)

3. 查重算法升级
   - 均值距离 → 中位数距离(抵抗异常值)
   - 新增帧匹配比例条件(match_ratio >= 0.7)
   - Bhattacharyya 系数替代余弦相似度
   - pHash + 直方图加权融合(0.7/0.3)
   - 判定重复后附加 duplicate_segments 字段

4. 删除旧代码
   - 移除 SHORT_VIDEO_CHUNK_SEC/LONG_VIDEO_CHUNK_SEC 固定间隔
   - 移除 compute_chunk_interval()
   - 移除 _average_histogram_similarity()

5. 测试
   - 新增 test_dedup_v2.py: 34 个测试
   - 更新 test_dedup_engine.py/test_duplicate_rate.py/test_dedup_pure.py
   - 清理 test_fingerprint_chunks.py 中旧常量测试
2026-09-03 23:00:21 +08:00
xiaoxia 30457629da fix(ci): fix preview deploy frontend build - remove DooD, use direct runner (#919)
CI/CD Pipeline / Production Browser E2E (push) Has been skipped
CI/CD Pipeline / Check if frontend-only change (push) Has been skipped
CI/CD Pipeline / Validate - Code Quality (push) Failing after 1m20s
CI/CD Pipeline / Validate - Type Check (mypy) (push) Successful in 48s
CI/CD Pipeline / Validate - Migration (alembic) (push) Successful in 51s
CI/CD Pipeline / Unit Tests (push) Successful in 4m54s
CI/CD Pipeline / Integration Tests (push) Successful in 2m6s
CI/CD Pipeline / Frontend Lint (push) Successful in 28s
CI/CD Pipeline / Frontend Unit Tests (push) Successful in 55s
CI/CD Pipeline / PR Build API Image (push) Has been skipped
CI/CD Pipeline / PR Build Web Image (push) Has been skipped
CI/CD Pipeline / PR Build Worker Image (push) Has been skipped
CI/CD Pipeline / Build Staging API Image (push) Successful in 12m7s
CI/CD Pipeline / Build Staging Web Image (push) Successful in 1m15s
CI/CD Pipeline / Build Staging Worker Image (push) Successful in 4m6s
CI/CD Pipeline / Build Production API Image (push) Has been skipped
CI/CD Pipeline / Build Production Web Image (push) Has been skipped
CI/CD Pipeline / Build Production Worker Image (push) Has been skipped
CI/CD Pipeline / Deploy Production (push) Has been skipped
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (push) Successful in 1m34s
CI/CD Pipeline / Staging E2E Tests (push) Successful in 1m29s
CI/CD Pipeline / ACR Image Cleanup (push) Failing after 5s
CI/CD Pipeline / Staging API Integration Tests (push) Successful in 3m50s
CI/CD Pipeline / Canary Release to Production (push) Has been cancelled
2026-07-26 12:58:22 +08:00
xiaoxia 56ab6bcef4 fix(test): 修复Unit Tests收集失败 - dedup模块级mock污染sys.modules (#917)
CI/CD Pipeline / Canary Release to Production (push) Has been skipped
CI/CD Pipeline / Check if frontend-only change (push) Has been skipped
CI/CD Pipeline / Validate - Code Quality (push) Failing after 3m6s
CI/CD Pipeline / Validate - Type Check (mypy) (push) Successful in 1m8s
CI/CD Pipeline / Validate - Migration (alembic) (push) Successful in 1m6s
CI/CD Pipeline / Unit Tests (push) Failing after 5m4s
CI/CD Pipeline / Integration Tests (push) Successful in 2m39s
CI/CD Pipeline / Frontend Lint (push) Successful in 52s
CI/CD Pipeline / Frontend Unit Tests (push) Successful in 1m22s
CI/CD Pipeline / PR Build API Image (push) Has been skipped
CI/CD Pipeline / PR Build Web Image (push) Has been skipped
CI/CD Pipeline / PR Build Worker Image (push) Has been skipped
CI/CD Pipeline / Build Staging API Image (push) Successful in 13m15s
CI/CD Pipeline / Build Staging Web Image (push) Successful in 1m54s
CI/CD Pipeline / Build Staging Worker Image (push) Successful in 5m22s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (push) Successful in 1m55s
CI/CD Pipeline / Staging E2E Tests (push) Successful in 1m44s
CI/CD Pipeline / Staging API Integration Tests (push) Successful in 4m9s
CI/CD Pipeline / Build Production API Image (push) Has been skipped
CI/CD Pipeline / Build Production Web Image (push) Has been skipped
CI/CD Pipeline / Build Production Worker Image (push) Has been skipped
CI/CD Pipeline / Deploy Production (push) Has been skipped
CI/CD Pipeline / Production Browser E2E (push) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (push) Failing after 22s
根因: test_dedup_pure.py模块级mock了整个worker_app和sqlalchemy_impl包,导致pytest收集其他测试时导入子模块触发AttributeError: __spec__。

修复: 只mock具体叶子模块和session(真正触发DB连接的元凶),不mock整个父包。

fix #906
2026-07-26 09:29:38 +08:00
xiaoxia 8d280e3df9 fix(test): 修复test_dedup_pure收集阶段db连接副作用 + 完整mock (#894) 2026-07-25 20:53:22 +08:00
xiaoxia 2972f19954 test(ci): 修复10个ruff错误 + Unit Tests dedup收集失败 (#839 #881)
CI/CD Pipeline / Check if frontend-only change (push) Has been skipped
CI/CD Pipeline / PR Build API Image (push) Has been skipped
CI/CD Pipeline / PR Build Web Image (push) Has been skipped
CI/CD Pipeline / PR Build Worker Image (push) Has been skipped
CI/CD Pipeline / Frontend Lint (push) Successful in 58s
CI/CD Pipeline / Validate - Type Check (mypy) (push) Successful in 1m38s
CI/CD Pipeline / Build Production API Image (push) Has been skipped
CI/CD Pipeline / Build Production Web Image (push) Has been skipped
CI/CD Pipeline / Build Production Worker Image (push) Has been skipped
CI/CD Pipeline / Validate - Migration (alembic) (push) Successful in 1m46s
CI/CD Pipeline / Build Staging Web Image (push) Successful in 1m42s
CI/CD Pipeline / Deploy Production (push) Has been skipped
CI/CD Pipeline / Production Browser E2E (push) Has been skipped
CI/CD Pipeline / Frontend Unit Tests (push) Successful in 1m18s
CI/CD Pipeline / Build Staging Worker Image (push) Successful in 3m23s
CI/CD Pipeline / Validate - Code Quality (push) Successful in 5m27s
CI/CD Pipeline / Unit Tests (push) Failing after 5m54s
CI/CD Pipeline / Integration Tests (push) Successful in 2m26s
CI/CD Pipeline / Build Staging API Image (push) Successful in 11m39s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (push) Successful in 1m14s
CI/CD Pipeline / ACR Image Cleanup (push) Failing after 14s
CI/CD Pipeline / Staging E2E Tests (push) Failing after 2m3s
CI/CD Pipeline / Staging API Integration Tests (push) Successful in 4m14s
CI/CD Pipeline / Canary Release to Production (push) Has been skipped
2026-07-25 17:38:17 +08:00
xiaoxia dc62884c5f test(unit): 第70波 - 去重算法 + 字幕渲染 + TTS引擎纯逻辑 (+73) (#868)
CI/CD Pipeline / Check if frontend-only change (push) Has been skipped
CI/CD Pipeline / PR Build API Image (push) Has been skipped
CI/CD Pipeline / PR Build Web Image (push) Has been skipped
CI/CD Pipeline / PR Build Worker Image (push) Has been skipped
CI/CD Pipeline / Frontend Lint (push) Successful in 57s
CI/CD Pipeline / Validate - Type Check (mypy) (push) Successful in 1m30s
CI/CD Pipeline / Validate - Migration (alembic) (push) Successful in 1m31s
CI/CD Pipeline / Build Staging Web Image (push) Successful in 1m38s
CI/CD Pipeline / Build Production API Image (push) Has been skipped
CI/CD Pipeline / Build Production Web Image (push) Has been skipped
CI/CD Pipeline / Build Production Worker Image (push) Has been skipped
CI/CD Pipeline / Validate - Code Quality (push) Failing after 2m59s
CI/CD Pipeline / Frontend Unit Tests (push) Successful in 1m46s
CI/CD Pipeline / Build Staging Worker Image (push) Successful in 2m49s
CI/CD Pipeline / Unit Tests (push) Failing after 4m29s
CI/CD Pipeline / Deploy Production (push) Has been skipped
CI/CD Pipeline / Production Browser E2E (push) Has been skipped
CI/CD Pipeline / Integration Tests (push) Successful in 2m4s
CI/CD Pipeline / Build Staging API Image (push) Successful in 13m7s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (push) Successful in 1m16s
CI/CD Pipeline / ACR Image Cleanup (push) Failing after 15s
CI/CD Pipeline / Staging API Integration Tests (push) Successful in 5m13s
CI/CD Pipeline / Staging E2E Tests (push) Failing after 5m38s
2026-07-25 14:18:22 +08:00