bba8beffef6beee8d35fc1a8cd806b29261e01ac
4 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 xiaoxia
|
28b3010668 |
fix(dedup): 修复查重率恒为0%——指纹绕开降重裁剪+局部片段复用+阈值校准+3个单位bug (#1702) (#1703)
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (push) Successful in 2s
CI/CD Pipeline / Check push changed paths (push) Successful in 4s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 5s
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 7s
CI/CD Pipeline / Build Staging API Image (push) Successful in 38s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 18s
CI/CD Pipeline / Build Staging Worker Image (push) Successful in 43s
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 22s
CI/CD Pipeline / CI Gate (pull_request) Successful in 8s
CI/CD Pipeline / Retag skipped Staging Web Image (push) Successful in 28s
CI/CD Pipeline / Validate - Python (mypy + alembic) (push) Successful in 2m1s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 2m21s
CI/CD Pipeline / Integration Tests (push) Successful in 2m30s
AI Code Review / AI Code Review (pull_request) Failing after 2m33s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (push) Successful in 1m10s
CI/CD Pipeline / Validate - Style (push) Successful in 3m13s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 3m12s
CI/CD Pipeline / ACR Image Cleanup (push) Successful in 1m42s
CI/CD Pipeline / Validate - Security (push) Successful in 5m58s
CI/CD Pipeline / Frontend Unit Tests (push) Successful in 6m6s
CI/CD Pipeline / Staging API Integration Tests (push) Successful in 4m38s
CI/CD Pipeline / Unit Tests (push) Successful in 8m43s
CI/CD Pipeline / Production Browser E2E (push) Has been skipped
CI/CD Pipeline / Staging E2E Tests (push) Successful in 6m35s
CI/CD Pipeline / Canary Release to Production (push) Failing after 271h18m23s
CI/CD Pipeline / Deploy Production (push) Failing after 271h18m23s
CI/CD Pipeline / CI Gate (push) Failing after 271h18m27s
CI/CD Pipeline / Build Production Web Image (push) Failing after 271h18m27s
CI/CD Pipeline / Build Production API Image (push) Failing after 271h18m27s
PR Automation / Auto Merge on CI Green + Approved (pull_request) Failing after 271h23m45s
CI/CD Pipeline / Canary Release to Production (pull_request) Failing after 271h26m17s
CI/CD Pipeline / Retag skipped Staging Worker Image (push) Failing after 271h26m18s
CI/CD Pipeline / Retag skipped Staging API Image (push) Failing after 271h26m18s
CI/CD Pipeline / Deploy Production (pull_request) Failing after 271h26m23s
CI/CD Pipeline / Staging API Integration Tests (pull_request) Failing after 271h26m24s
CI/CD Pipeline / Staging E2E Tests (pull_request) Failing after 271h26m24s
CI/CD Pipeline / Build Production Web Image (pull_request) Failing after 271h26m26s
CI/CD Pipeline / Build Production API Image (pull_request) Failing after 271h26m27s
CI/CD Pipeline / Frontend Unit Tests (pull_request) Failing after 271h26m28s
CI/CD Pipeline / Frontend Lint (pull_request) Failing after 271h26m29s
CI/CD Pipeline / Integration Tests (pull_request) Failing after 271h26m29s
CI/CD Pipeline / Unit Tests (pull_request) Failing after 271h26m29s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Failing after 271h26m29s
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Failing after 271h27m5s
CI/CD Pipeline / Validate - Security (pull_request) Failing after 271h26m30s
CI/CD Pipeline / Build Staging Web Image (push) Failing after 271h27m8s
CI/CD Pipeline / Validate - Style (pull_request) Failing after 271h26m30s
CI/CD Pipeline / Build Staging Worker Image (pull_request) Failing after 271h27m10s
CI/CD Pipeline / PR Build Web Image (pull_request) Failing after 271h26m31s
CI/CD Pipeline / Build Staging API Image (pull_request) Failing after 271h27m10s
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Failing after 271h27m4s
CI/CD Pipeline / Frontend Lint (push) Failing after 271h27m11s
CI/CD Pipeline / Check push changed paths (pull_request) Failing after 271h27m13s
CI/CD Pipeline / PR Build Worker Image (push) Failing after 271h27m14s
CI/CD Pipeline / PR Build Web Image (push) Failing after 271h27m14s
CI/CD Pipeline / PR Build API Image (push) Failing after 271h27m14s
CI/CD Pipeline / Check if frontend-only change (push) Failing after 271h27m15s
CI/CD Pipeline / Build Production Worker Image (push) Failing after 271h52m56s
CI/CD Pipeline / ACR Image Cleanup (pull_request) Failing after 272h0m52s
CI/CD Pipeline / Build Production Worker Image (pull_request) Failing after 272h0m54s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Failing after 272h0m57s
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Failing after 272h1m33s
CI/CD Pipeline / Build Staging Web Image (pull_request) Failing after 272h1m39s
Co-authored-by: xiaoxia <dev@xiaoxiajianji.com> Co-committed-by: xiaoxia <dev@xiaoxiajianji.com> |
||
 CI Bot
|
a0d4f6e111 |
style: auto-format with black + isort + prettier [skip ci-format-check]
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 2s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 2s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 22s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 10s
CI/CD Pipeline / Unit Tests (pull_request) Successful in 1m20s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m27s
CI/CD Pipeline / Validate - Style (pull_request) Failing after 1m33s
AI Code Review / AI Code Review (pull_request) Failing after 2m11s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 2m19s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 2m43s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 4m35s
CI/CD Pipeline / Validate - Security (pull_request) Has been cancelled
CI/CD Pipeline / Build Production API Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Web Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been cancelled
CI/CD Pipeline / Deploy Production (pull_request) Has been cancelled
CI/CD Pipeline / Production Browser E2E (pull_request) Has been cancelled
CI/CD Pipeline / Canary Release to Production (pull_request) Has been cancelled
CI/CD Pipeline / CI Gate (pull_request) Has been cancelled
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 2m29s
CI/CD Pipeline / ACR Image Cleanup (pull_request) Failing after 304h19m57s
CI/CD Pipeline / Staging E2E Tests (pull_request) Failing after 304h19m57s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Failing after 304h20m0s
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Failing after 304h20m6s
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Failing after 304h20m6s
CI/CD Pipeline / PR Build Web Image (pull_request) Failing after 304h20m7s
CI/CD Pipeline / Frontend Lint (pull_request) Failing after 304h20m30s
CI/CD Pipeline / Build Staging Worker Image (pull_request) Failing after 304h20m35s
CI/CD Pipeline / Check push changed paths (pull_request) Failing after 304h20m37s
CI/CD Pipeline / Build Staging Web Image (pull_request) Failing after 304h20m35s
CI/CD Pipeline / Staging API Integration Tests (pull_request) Failing after 304h54m21s
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Failing after 304h54m30s
CI/CD Pipeline / Frontend Unit Tests (pull_request) Failing after 304h54m54s
CI/CD Pipeline / Build Staging API Image (pull_request) Failing after 304h54m59s
|
||
|
xiaoxia
|
fac80b1f77 |
feat(dedup): 动态抽帧 + 滑动窗口时序匹配 (#1659)
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 1s
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 2s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 22s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 9s
CI/CD Pipeline / Unit Tests (pull_request) Successful in 1m22s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m25s
AI Code Review / AI Code Review (pull_request) Failing after 1m50s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 2m39s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 2m44s
CI/CD Pipeline / Validate - Style (pull_request) Has been cancelled
CI/CD Pipeline / Validate - Security (pull_request) Has been cancelled
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Has been cancelled
CI/CD Pipeline / Build Production API Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Web Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been cancelled
CI/CD Pipeline / Deploy Production (pull_request) Has been cancelled
CI/CD Pipeline / Production Browser E2E (pull_request) Has been cancelled
CI/CD Pipeline / Canary Release to Production (pull_request) Has been cancelled
CI/CD Pipeline / CI Gate (pull_request) Has been cancelled
PR Automation / Auto Merge on CI Green + Approved (pull_request) Has been cancelled
CI/CD Pipeline / ACR Image Cleanup (pull_request) Failing after 304h23m6s
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Failing after 304h23m16s
CI/CD Pipeline / Staging API Integration Tests (pull_request) Failing after 304h23m6s
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Failing after 304h23m16s
CI/CD Pipeline / Staging E2E Tests (pull_request) Failing after 304h23m7s
CI/CD Pipeline / Build Staging Web Image (pull_request) Failing after 304h23m21s
CI/CD Pipeline / Build Staging API Image (pull_request) Failing after 304h23m21s
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Failing after 304h23m16s
CI/CD Pipeline / Frontend Unit Tests (pull_request) Failing after 304h23m43s
CI/CD Pipeline / Frontend Lint (pull_request) Failing after 304h23m43s
CI/CD Pipeline / Check push changed paths (pull_request) Failing after 304h23m53s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Failing after 304h57m36s
CI/CD Pipeline / Build Staging Worker Image (pull_request) Failing after 304h57m45s
CI/CD Pipeline / PR Build Web Image (pull_request) Failing after 304h57m45s
1. 动态抽帧策略 — detect_keyframe_timestamps() - 降采样到 320x240 逐帧灰度差异检测场景切换 - 最小间隔过滤（保留差异最大的候选帧） - 数量裁剪到 [MIN_KEYFRAMES=5, MAX_KEYFRAMES=30] - 长视频（>3分钟）每 30 秒分段保底 2. 滑动窗口时序匹配 — find_duplicate_segments() - 逐帧最佳匹配 → 连续 run 检测（允许 MAX_GAP=2 间隙） - 最少 MIN_CONSECUTIVE_MATCHES=5 帧才报告 - 返回 DuplicateSegment（query/target 时间范围 + 平均距离） 3. 查重算法升级 - 均值距离 → 中位数距离（抵抗异常值） - 新增帧匹配比例条件（match_ratio >= 0.7） - Bhattacharyya 系数替代余弦相似度 - pHash + 直方图加权融合（0.7/0.3） - 判定重复后附加 duplicate_segments 字段 4. 删除旧代码 - 移除 SHORT_VIDEO_CHUNK_SEC/LONG_VIDEO_CHUNK_SEC 固定间隔 - 移除 compute_chunk_interval() - 移除 _average_histogram_similarity() 5. 测试 - 新增 test_dedup_v2.py: 34 个测试 - 更新 test_dedup_engine.py/test_duplicate_rate.py/test_dedup_pure.py - 清理 test_fingerprint_chunks.py 中旧常量测试 |
||
|
saas-backend-agent
|
cbca0c3584 |
feat: 分片指纹存储改造 + 存量指纹重建脚本 #1657
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 3s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 2s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 2m19s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 2m43s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m22s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 8s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 7s
CI/CD Pipeline / Validate - Style (pull_request) Has been cancelled
CI/CD Pipeline / Validate - Security (pull_request) Has been cancelled
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Has been cancelled
CI/CD Pipeline / Unit Tests (pull_request) Has been cancelled
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been cancelled
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been cancelled
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been cancelled
CI/CD Pipeline / Build Production API Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Web Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been cancelled
CI/CD Pipeline / Deploy Production (pull_request) Has been cancelled
CI/CD Pipeline / Production Browser E2E (pull_request) Has been cancelled
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been cancelled
CI/CD Pipeline / Canary Release to Production (pull_request) Has been cancelled
CI/CD Pipeline / CI Gate (pull_request) Has been cancelled
AI Code Review / AI Code Review (pull_request) Has been cancelled
PR Automation / Auto Merge on CI Green + Approved (pull_request) Has been cancelled
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Failing after 306h35m11s
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Failing after 306h35m11s
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Failing after 306h35m11s
CI/CD Pipeline / Build Staging Worker Image (pull_request) Failing after 306h35m45s
CI/CD Pipeline / Build Staging Web Image (pull_request) Failing after 306h35m45s
CI/CD Pipeline / Frontend Lint (pull_request) Failing after 306h36m1s
CI/CD Pipeline / Build Staging API Image (pull_request) Failing after 306h35m46s
CI/CD Pipeline / PR Build Web Image (pull_request) Failing after 306h35m53s
CI/CD Pipeline / Check push changed paths (pull_request) Failing after 306h40m31s
CI/CD Pipeline / Frontend Unit Tests (pull_request) Failing after 307h10m24s
## 改动 ### 1. 新建 video_fingerprint_chunks 表（Migration 063） - 按时间分片存储 pHash + color_histogram - 索引：video_id, project_id, user_id ### 2. 新增 VideoFingerprintChunkModel - packages/adapters/sqlalchemy_impl/models.py ### 3. 改造 dedup.py 指纹计算 - compute_fingerprint() 改为按时间分片抽帧 - 短视频（≤60s）：每 2s 一片 - 长视频（>60s）：每 5s 一片 - VideoFingerprint 新增 chunks 字段（list of FingerprintChunk） - 向后兼容：keyframe_phashes/color_histograms 保留 - to_chunk_models() 方法转换为 SQLAlchemy Model - check_duplicate() 优先从分片表读取，回退到 JSON 字段 - check_duplicate_task() 写入分片表 ### 4. 改造 dedup_helpers.py - create_video_record_and_dedup() 同步写入分片表 ### 5. 存量指纹重建脚本 - apps/api/scripts/rebuild_fingerprint_chunks.py - 支持 --dry-run 和 --batch-size - 幂等：已有分片数据的视频跳过 ### 6. 单元测试（11 个） - 分片策略：60s→30片，120s→24片 - to_chunk_models() 输出正确 - _save_fingerprint_chunks 幂等性 - to_dict() 向后兼容 Closes #1657 |