fix: 查重系统黑屏视频过滤 — 跳过坏指纹防止虚假匹配 #1664 #1688
Merged
auto-approve-bot
merged 1 commits from 2026-09-04 14:56:49 +08:00
fix/dedup-three-bugs-1664 into develop
1 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 saas-backend-agent
|
503d6ccfac |
fix: 查重系统黑屏视频过滤 — 跳过坏指纹防止虚假匹配 #1664
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 3s
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 3s
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 32s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 32s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 1m46s
CI/CD Pipeline / Unit Tests (pull_request) Successful in 1m45s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 2m4s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 2m4s
CI/CD Pipeline / Validate - Style (pull_request) Successful in 2m31s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 3m3s
AI Code Review / AI Code Review (pull_request) Successful in 6m19s
CI/CD Pipeline / Validate - Security (pull_request) Successful in 6m18s
CI/CD Pipeline / Build Production API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / Canary Release to Production (pull_request) Has been skipped
CI/CD Pipeline / CI Gate (pull_request) Successful in 3s
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 3m50s
ACR Cleanup / ACR Image Cleanup (pull_request_target) Successful in 10s
Preview Cleanup / Cleanup Preview Environment (pull_request) Successful in 31s
问题：1秒黑屏/纯色视频的所有关键帧phash几乎相同，与任何视频 比较都会得到虚假的'匹配'结果，污染查重池。 修复： - 新增 _is_bad_fingerprint() 静态方法检测低质量指纹： - 空 phash 列表 → 坏 - 多帧但所有 phash 完全相同 → 黑屏/纯色视频 - 多帧但所有 phash 汉明距离都 < 3 → 近似黑屏 - 单帧不视为坏指纹（兼容短视频/抽帧不足） - check_duplicate / compute_duplicate_rate / check_batch_duplicate 三个方法在 MD5 精确匹配之后、pHash 比较之前加过滤 - MD5 精确匹配优先级高于坏指纹过滤 测试：13 个新单测覆盖 - _is_bad_fingerprint 8 个用例（空/单帧/全同/近似/多样/混合/已知黑屏） - check_duplicate 3 个集成测试（黑屏跳过/正常不过滤/MD5优先） - compute_duplicate_rate 2 个集成测试（黑屏排除/全黑屏零查重率） 注：Bug #1（PHASH_THRESHOLD=40）和 Bug #2（visual_similarity覆盖） 已在 PR #1674/#1684 中修复：阈值已收紧到 8，visual_similarity 只在 dup_rate > max 时更新。 |