fix(dedup): 修复查重率恒为0%——指纹绕开降重裁剪+局部片段复用+阈值校准+3个单位bug (#1702) #1703
Merged
auto-approve-bot
merged 1 commits from 2026-09-05 07:57:24 +08:00
fix/dedup-zero-rate-1702 into develop
1 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 deploy-release-bot
|
5fbe29b553 |
fix(dedup): 修复查重率恒为0%——指纹绕开降重裁剪+局部片段复用+阈值校准+3个单位bug (Issue #1702)
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 3s
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 3s
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 31s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 1m39s
CI/CD Pipeline / Unit Tests (pull_request) Successful in 1m54s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m54s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 1m57s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 1m57s
CI/CD Pipeline / Validate - Style (pull_request) Successful in 2m14s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 2m49s
CI/CD Pipeline / Validate - Security (pull_request) Successful in 3m46s
CI/CD Pipeline / Build Production API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / CI Gate (pull_request) Successful in 4s
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / Canary Release to Production (pull_request) Has been skipped
AI Code Review / AI Code Review (pull_request) Successful in 4m17s
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 1m36s
ACR Cleanup / ACR Image Cleanup (pull_request_target) Successful in 10s
Preview Cleanup / Cleanup Preview Environment (pull_request) Successful in 28s
P0: - 指纹改用固定1s间隔密集均匀采样(sample_fingerprint_timestamps)替代动态 场景检测抽帧：staging真实数据回归证明动态抽帧让同源两视频取帧时刻错位、 配对时序错乱是0检出主因；固定间隔采样后复用片段帧时刻天然对齐 - 指纹pHash/颜色直方图基于中心90%区域(center_crop_frame)，绕开成片强制 random_edge_crop(2-5%随机裁边)对指纹的污染；MD5仍基于原始帧 - find_duplicate_segments重构：全量距离矩阵+时序一致贪心对齐，支持局部 片段复用(各复用片段独立成run)；±1邻接窗口(NEIGHBOR_WINDOW)容忍切点 不一致；连续匹配门槛短视频自适应 min(5, max(2, n//2)) - pHash阈值经staging真实同源/异源指纹回归校准为12(同源密集采样min=8、 同帧两次随机裁剪距离4~10、异源min=24)，阈值常量统一模块级PHASH_THRESHOLD - 去掉frame_match_rate<0.3硬跳过；frame_match_rate分母改min(两视频分片数)； temporal_coverage为主指标，无连续片段时按匹配帧占比回退 - check_duplicate/check_batch/compute_duplicate_rate遍历所有候选取融合分 最高者(旧逻辑首个过阈即返回) P1(确定性bug): - 时长预过滤 fingerprint.duration/1000 单位错误(duration本身是秒)， dedup.py与dedup_helpers.py两处修复 - 颜色直方图每通道NORM_L1归一化；Bhattacharyya系数改为 Σ√(ab)/√(Σa·Σb)概率分布归一(旧L2三通道拼接算出~14.9，上限应1) - temporal_coverage量纲修正(duration_sec*1000) - 帧匹配与片段匹配阈值比较统一为<=PHASH_THRESHOLD P2: - 三个比对入口每候选落debug日志(min_distances/frame_match_rate/coverage/ median/fusion/segments)，0匹配落info日志 配套: - _save_fingerprint_chunks改为先删旧分片再写入(旧'有数据跳过'语义导致 recompute对存量视频不生效) - recompute-dedup接口加force参数(已有查重数据也可强制重算) - _is_bad_fingerprint修正: 旧'所有phash相同即黑屏'误杀短视频(<8帧不判坏) - 新增15个验收单测(同源裁剪检出/局部复用检出/异源不误报/N=1不回归等)， 同步更新受行为变化影响的旧测试 |