xiaoxia/xiaoxia-saas
1
0
Fork 0
Code Issues 11 Pull Requests 3 Actions Packages Projects Releases 7 Wiki Activity

fix(dedup): pHash阈值二次校准12→16 + 时序对齐允许±1反向抖动，修复降重同源对漏检 (#1702) #1709

Merged
auto-approve-bot merged 1 commits from fix/dedup-threshold16-temporal-jitter-1702 into develop 2026-09-05 11:30:12 +08:00
Conversation 2 Commits 1 Files Changed 5
+176 -50
xiaoxia commented 2026-09-05 11:24:03 +08:00
Owner
Copy Link
Copy Source

背景

#1708（scope=user 去掉时长预过滤）部署后第四轮 staging 验证：A(20s) 已正确检出 C(11.75s) 重复 63.64%，但证据视频 B(11.75s) 对 A 仍 best_fusion=0 漏检。进 worker 容器用该用户全部 15 个真实成片指纹离线诊断，定位两个根因。

根因 1：阈值 12 对降重同源对过严

视频对（1s 密集采样） 中位数距离 <=12 命中 <=16 命中
B(同源) vs A 14 4/11=0.36 ❌ 8/11=0.73 ✓
C(同源) vs A 18 7/11=0.64 8/11=0.73 ✓
13 个异源候选 vs B/A 22~28 0 帧 0 帧（最近邻最小距离 18）

降重滤镜/字幕/画面扰动把同源帧距离从 8 推到 1416；阈值 16 检出同源 0.73+，与异源分布（最近邻 ≥18）仍有 ≥2bit 安全裕度，异源零误报。

根因 2：时序对齐只允许目标索引正向 delta

密集采样下相邻帧 pHash 接近，全局最近邻在目标相邻帧间正/负 1 跳变是正常的（切点错位、局部倒退）。旧逻辑 0 <= delta <= window+1 把同源连续匹配拆碎，短视频 min_consecutive=5 门槛够不上。改为 |delta| <= window+1；大跳跃（>window）仍正常断段，不误桥接。

改动

  • PHASH_THRESHOLD 12→16（SEGMENT_MATCH_THRESHOLD 统一跟随），注释记录两轮校准数据
  • find_duplicate_segments 连贯约束改为绝对值，docstring 同步
  • 新增 6 个回归测试：阈值边界（距离16检出/18零误报/0.73 比例）、反向抖动保持 run、大跳跃不桥接
  • 旧测试同步：不匹配哈希构造（aaaa↔bbbb 距离恰为 16 现算匹配，换成距离 ≥32/64 的哈希）、阈值断言 12→16

测试

  • 查重相关全量单测 290 passed；black/isort/ruff 全绿

验收

合并部署后重算证据视频，预期 B 检出 A（或更强匹配 C）is_duplicate=True，且对 13 个异源视频零误报。

## 背景 #1708（scope=user 去掉时长预过滤）部署后第四轮 staging 验证：A(20s) 已正确检出 C(11.75s) 重复 63.64%，但证据视频 B(11.75s) 对 A 仍 `best_fusion=0` 漏检。进 worker 容器用该用户**全部 15 个真实成片指纹**离线诊断，定位两个根因。 ## 根因 1：阈值 12 对降重同源对过严 | 视频对（1s 密集采样） | 中位数距离 | <=12 命中 | **<=16 命中** | |---|---|---|---| | B(同源) vs A | 14 | 4/11=0.36 ❌ | **8/11=0.73** ✓ | | C(同源) vs A | 18 | 7/11=0.64 | **8/11=0.73** ✓ | | 13 个异源候选 vs B/A | 22~28 | **0 帧** | **0 帧**（最近邻最小距离 18） | 降重滤镜/字幕/画面扰动把同源帧距离从 ~8 推到 14~16；阈值 16 检出同源 0.73+，与异源分布（最近邻 ≥18）仍有 ≥2bit 安全裕度，**异源零误报**。 ## 根因 2：时序对齐只允许目标索引正向 delta 密集采样下相邻帧 pHash 接近，全局最近邻在目标相邻帧间正/负 1 跳变是正常的（切点错位、局部倒退）。旧逻辑 `0 <= delta <= window+1` 把同源连续匹配拆碎，短视频 min_consecutive=5 门槛够不上。改为 `|delta| <= window+1`；大跳跃（>window）仍正常断段，不误桥接。 ## 改动 - `PHASH_THRESHOLD` 12→16（`SEGMENT_MATCH_THRESHOLD` 统一跟随），注释记录两轮校准数据 - `find_duplicate_segments` 连贯约束改为绝对值，docstring 同步 - 新增 6 个回归测试：阈值边界（距离16检出/18零误报/0.73 比例）、反向抖动保持 run、大跳跃不桥接 - 旧测试同步：不匹配哈希构造（aaaa↔bbbb 距离恰为 16 现算匹配，换成距离 ≥32/64 的哈希）、阈值断言 12→16 ## 测试 - 查重相关全量单测 **290 passed**；black/isort/ruff 全绿 ## 验收 合并部署后重算证据视频，预期 B 检出 A（或更强匹配 C）`is_duplicate=True`，且对 13 个异源视频零误报。
xiaoxia added 1 commit 2026-09-05 11:24:04 +08:00
fix(dedup): pHash阈值二次校准12→16+时序对齐允许±1反向抖动，修复降重对漏检 (Issue #1702)
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 1s
Details
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
Details
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 1s
Details
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
Details
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
Details
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
Details
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
Details
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
Details
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
Details
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
Details
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
Details
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
Details
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
Details
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
Details
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
Details
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
Details
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 32s
Details
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 29s
Details
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 1m21s
Details
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 1m43s
Details
CI/CD Pipeline / Unit Tests (pull_request) Successful in 1m44s
Details
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m44s
Details
CI/CD Pipeline / Validate - Style (pull_request) Successful in 2m6s
Details
PR Automation / Auto Approve on CI Green (pull_request) Successful in 3m10s
Details
CI/CD Pipeline / Validate - Security (pull_request) Successful in 5m22s
Details
CI/CD Pipeline / Build Production API Image (pull_request) Has been skipped
Details
CI/CD Pipeline / Build Production Web Image (pull_request) Has been skipped
Details
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been skipped
Details
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
Details
CI/CD Pipeline / CI Gate (pull_request) Successful in 2s
Details
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
Details
CI/CD Pipeline / Canary Release to Production (pull_request) Has been skipped
Details
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 2m42s
Details
ACR Cleanup / ACR Image Cleanup (pull_request_target) Successful in 9s
Details
AI Code Review / AI Code Review (pull_request) Successful in 6m17s
Details
Preview Cleanup / Cleanup Preview Environment (pull_request) Successful in 17s
Details
d37a5f6a19 
#1708 去掉时长预过滤后，staging 第四轮验证发现证据视频 B(11s) 对
A(20s) 仍 best_fusion=0 漏检。离线诊断（worker 容器，该用户全部
15 个真实成片指纹）：

1. 阈值 12 过严：同源降重对 B->A 中位数距离 14，<=12 仅 4/11 命中；
   <=16 命中 8/11=0.73。异源 13 个候选每帧全局最近邻最小距离 18，
   <=16 命中帧数全部为 0 → 16 检出同源且异源零误报（>=2bit 裕度）。
2. 时序对齐只允许目标索引正向 delta：密集 1s 采样下相邻帧 pHash
   接近，全局最近邻在目标相邻帧间正负 1 跳变属正常，旧逻辑把同源
   连续匹配拆碎，min_consecutive 门槛够不上。改为 |delta| <=
   neighbor_window+1（正/反向抖动均允许），大跳跃（>window）仍断段。

改动：
- PHASH_THRESHOLD 12→16（SEGMENT_MATCH_THRESHOLD 跟随统一），
  常量注释记录两轮校准数据
- find_duplicate_segments 时序连贯约束 0<=delta<=w+1 → |delta|<=w+1
- 新增 6 个回归测试：阈值 16 校准边界（16 检出/18 零误报/0.73 比例）、
  反向抖动保持 run、大跳跃不桥接
- 同步更新旧测试断言与不匹配哈希构造（a-b 距离 16 现算匹配）

测试：查重相关 290 passed；black/isort/ruff 全绿
gitea-actions bot commented 2026-09-05 11:25:23 +08:00
Copy Link
Copy Source

🚀 预览环境已部署

项目 详情
PR号 #1709
预览链接 https://pr-1709.preview.xiaoxiajianji.com
API环境 staging

💡 预览环境使用 staging API 数据，请勿在预览环境中操作重要数据。

🔄 每次提交新代码后预览环境会自动更新。

🗑️ PR 关闭或合并后，预览环境会自动清理。

🚀 **预览环境已部署** | 项目 | 详情 | |------|------| | PR号 | #1709 | | 预览链接 | [https://pr-1709.preview.xiaoxiajianji.com](https://pr-1709.preview.xiaoxiajianji.com) | | API环境 | staging | > 💡 预览环境使用 staging API 数据，请勿在预览环境中操作重要数据。 > > 🔄 每次提交新代码后预览环境会自动更新。 > > 🗑️ PR 关闭或合并后，预览环境会自动清理。
auto-approve-bot merged commit cf83c0df9f into develop 2026-09-05 11:30:12 +08:00
auto-approve-bot deleted branch fix/dedup-threshold16-temporal-jitter-1702 2026-09-05 11:30:12 +08:00
gitea-actions bot commented 2026-09-05 11:30:28 +08:00
Copy Link
Copy Source

🗑️ 预览环境已清理

PR #1709 已关闭或合并，对应的预览环境已被清理。

如有需要，可以重新打开 PR 来重新生成预览环境。

🗑️ **预览环境已清理** PR #1709 已关闭或合并，对应的预览环境已被清理。 > 如有需要，可以重新打开 PR 来重新生成预览环境。
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
P0
阻塞性，必须立即处理
 P1
高优先级
 P2
中优先级
 P3
低优先级/后考虑
 architecture
架构设计
 backend
后端相关
 blocked
阻塞中
 bug
Bug修复
 ci-bug
CI本身的bug
 ci-optimization
CI优化类任务
 ci-pipeline
CI流水线/workflow相关
 cleanup
代码清理
 devops
运维/部署/CI
 documentation
文档相关
 feature
新功能
 frontend
前端相关
 question
需要讨论确认
 ready
PR CI通过，可合并
 refactor
代码重构
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
auto-approve-bot xiaoxia
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: xiaoxia/xiaoxia-saas#1709
Delete Branch "fix/dedup-threshold16-temporal-jitter-1702"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?