xiaoxia/xiaoxia-saas
1
0
Fork 0
Code Issues 11 Pull Requests 4 Actions 7 Packages Projects Releases 7 Wiki Activity

fix: 查重系统黑屏视频过滤 — 跳过坏指纹防止虚假匹配 #1664 #1688

Merged
auto-approve-bot merged 1 commits from fix/dedup-three-bugs-1664 into develop 2026-09-04 14:56:49 +08:00
Conversation 2 Commits 1 Files Changed 2
+368
xiaoxia commented 2026-09-04 14:49:45 +08:00
Owner
Copy Link
Copy Source

问题

1秒黑屏/纯色视频的所有关键帧 phash 几乎相同（如 9999999999999966），与任何视频比较都会得到虚假的"匹配"结果，污染查重池。

修复

_is_bad_fingerprint() 静态方法

新增指纹质量检测，在 check_duplicate / compute_duplicate_rate / check_batch_duplicate 三个方法的 MD5 精确匹配之后、pHash 比较之前加过滤：

条件 结果
空 phash 列表 坏指纹
多帧但所有 phash 完全相同 黑屏/纯色视频 → 跳过
多帧但所有 phash 汉明距离 < 3 近似黑屏 → 跳过
单帧（只有1个phash） 不视为坏指纹（兼容短视频）
有足够多样的 phash 有效指纹 → 正常比较

优先级

MD5 精确匹配 优先于 坏指纹过滤 — 即使两个视频 phash 都是黑屏特征，只要 MD5 完全一致就返回精确匹配。

关于三项 Bug 的核实结果

Bug 状态
Bug #1: PHASH_THRESHOLD=40 太宽松 ❌ 已修复 — PR #1674 已收紧到 8（比要求的 22 更严格）
Bug #2: visual_similarity 循环覆盖 ❌ 已修复 — PR #1684 已改为 if dup_rate > max 条件更新
Bug #3: 黑屏视频污染查重池 ✅ 本次修复

测试

13 个新单测（test_bad_fingerprint_filter.py）：

  • _is_bad_fingerprint 8 个用例
  • check_duplicate 3 个集成测试
  • compute_duplicate_rate 2 个集成测试

全部 110 个查重相关测试通过。

## 问题 1秒黑屏/纯色视频的所有关键帧 phash 几乎相同（如 `9999999999999966`），与任何视频比较都会得到虚假的"匹配"结果，污染查重池。 ## 修复 ### `_is_bad_fingerprint()` 静态方法 新增指纹质量检测，在 `check_duplicate` / `compute_duplicate_rate` / `check_batch_duplicate` 三个方法的 MD5 精确匹配之后、pHash 比较之前加过滤： | 条件 | 结果 | |------|------| | 空 phash 列表 | 坏指纹 | | 多帧但所有 phash 完全相同 | 黑屏/纯色视频 → 跳过 | | 多帧但所有 phash 汉明距离 < 3 | 近似黑屏 → 跳过 | | 单帧（只有1个phash） | **不视为坏指纹**（兼容短视频） | | 有足够多样的 phash | 有效指纹 → 正常比较 | ### 优先级 MD5 精确匹配 **优先于** 坏指纹过滤 — 即使两个视频 phash 都是黑屏特征，只要 MD5 完全一致就返回精确匹配。 ## 关于三项 Bug 的核实结果 | Bug | 状态 | |-----|------| | **Bug #1**: PHASH_THRESHOLD=40 太宽松 | ❌ 已修复 — PR #1674 已收紧到 **8**（比要求的 22 更严格） | | **Bug #2**: visual_similarity 循环覆盖 | ❌ 已修复 — PR #1684 已改为 `if dup_rate > max` 条件更新 | | **Bug #3**: 黑屏视频污染查重池 | ✅ 本次修复 | ## 测试 13 个新单测（`test_bad_fingerprint_filter.py`）： - `_is_bad_fingerprint` 8 个用例 - `check_duplicate` 3 个集成测试 - `compute_duplicate_rate` 2 个集成测试 全部 110 个查重相关测试通过。
xiaoxia added 1 commit 2026-09-04 14:49:46 +08:00
fix: 查重系统黑屏视频过滤 — 跳过坏指纹防止虚假匹配 #1664
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
Details
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 3s
Details
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
Details
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 3s
Details
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
Details
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
Details
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
Details
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
Details
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
Details
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
Details
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
Details
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
Details
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
Details
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
Details
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
Details
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
Details
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 32s
Details
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 32s
Details
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 1m46s
Details
CI/CD Pipeline / Unit Tests (pull_request) Successful in 1m45s
Details
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 2m4s
Details
CI/CD Pipeline / Integration Tests (pull_request) Successful in 2m4s
Details
CI/CD Pipeline / Validate - Style (pull_request) Successful in 2m31s
Details
PR Automation / Auto Approve on CI Green (pull_request) Successful in 3m3s
Details
AI Code Review / AI Code Review (pull_request) Successful in 6m19s
Details
CI/CD Pipeline / Validate - Security (pull_request) Successful in 6m18s
Details
CI/CD Pipeline / Build Production API Image (pull_request) Has been skipped
Details
CI/CD Pipeline / Build Production Web Image (pull_request) Has been skipped
Details
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been skipped
Details
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
Details
CI/CD Pipeline / Canary Release to Production (pull_request) Has been skipped
Details
CI/CD Pipeline / CI Gate (pull_request) Successful in 3s
Details
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
Details
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 3m50s
Details
ACR Cleanup / ACR Image Cleanup (pull_request_target) Successful in 10s
Details
Preview Cleanup / Cleanup Preview Environment (pull_request) Successful in 31s
Details
503d6ccfac 
问题：1秒黑屏/纯色视频的所有关键帧phash几乎相同，与任何视频
比较都会得到虚假的'匹配'结果，污染查重池。

修复：
- 新增 _is_bad_fingerprint() 静态方法检测低质量指纹：
  - 空 phash 列表 → 坏
  - 多帧但所有 phash 完全相同 → 黑屏/纯色视频
  - 多帧但所有 phash 汉明距离都 < 3 → 近似黑屏
  - 单帧不视为坏指纹（兼容短视频/抽帧不足）
- check_duplicate / compute_duplicate_rate / check_batch_duplicate
  三个方法在 MD5 精确匹配之后、pHash 比较之前加过滤
- MD5 精确匹配优先级高于坏指纹过滤

测试：13 个新单测覆盖
- _is_bad_fingerprint 8 个用例（空/单帧/全同/近似/多样/混合/已知黑屏）
- check_duplicate 3 个集成测试（黑屏跳过/正常不过滤/MD5优先）
- compute_duplicate_rate 2 个集成测试（黑屏排除/全黑屏零查重率）

注：Bug #1（PHASH_THRESHOLD=40）和 Bug #2（visual_similarity覆盖）
已在 PR #1674/#1684 中修复：阈值已收紧到 8，visual_similarity
只在 dup_rate > max 时更新。
gitea-actions bot commented 2026-09-04 14:51:32 +08:00
Copy Link
Copy Source

🚀 预览环境已部署

项目 详情
PR号 #1688
预览链接 https://pr-1688.preview.xiaoxiajianji.com
API环境 staging

💡 预览环境使用 staging API 数据，请勿在预览环境中操作重要数据。

🔄 每次提交新代码后预览环境会自动更新。

🗑️ PR 关闭或合并后，预览环境会自动清理。

🚀 **预览环境已部署** | 项目 | 详情 | |------|------| | PR号 | #1688 | | 预览链接 | [https://pr-1688.preview.xiaoxiajianji.com](https://pr-1688.preview.xiaoxiajianji.com) | | API环境 | staging | > 💡 预览环境使用 staging API 数据，请勿在预览环境中操作重要数据。 > > 🔄 每次提交新代码后预览环境会自动更新。 > > 🗑️ PR 关闭或合并后，预览环境会自动清理。
auto-approve-bot merged commit 4633126bb4 into develop 2026-09-04 14:56:49 +08:00
auto-approve-bot deleted branch fix/dedup-three-bugs-1664 2026-09-04 14:56:50 +08:00
gitea-actions bot commented 2026-09-04 14:57:16 +08:00
Copy Link
Copy Source

🗑️ 预览环境已清理

PR #1688 已关闭或合并，对应的预览环境已被清理。

如有需要，可以重新打开 PR 来重新生成预览环境。

🗑️ **预览环境已清理** PR #1688 已关闭或合并，对应的预览环境已被清理。 > 如有需要，可以重新打开 PR 来重新生成预览环境。
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
P0
阻塞性，必须立即处理
 P1
高优先级
 P2
中优先级
 P3
低优先级/后考虑
 architecture
架构设计
 backend
后端相关
 blocked
阻塞中
 bug
Bug修复
 ci-bug
CI本身的bug
 ci-optimization
CI优化类任务
 ci-pipeline
CI流水线/workflow相关
 cleanup
代码清理
 devops
运维/部署/CI
 documentation
文档相关
 feature
新功能
 frontend
前端相关
 question
需要讨论确认
 ready
PR CI通过，可合并
 refactor
代码重构
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
auto-approve-bot xiaoxia
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: xiaoxia/xiaoxia-saas#1688
Delete Branch "fix/dedup-three-bugs-1664"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?