refactor(dedup): 查重模块优化 — XOR 汉明距离 + 简化判定逻辑 #148

Merged
xiaoxia merged 5 commits from feature/duplication-optimization into develop 2026-07-01 16:18:05 +08:00

5 Commits

Author SHA1 Message Date
灵应 ed7de72b0c refactor(dedup): 优化查重算法 — XOR 汉明距离 + 简化判定逻辑
Deploy / Build Production Runtime Images (push) Has been skipped
Deploy / Deploy Production (push) Has been skipped
Deploy / Production Browser E2E (push) Has been skipped
CI/CD Pipeline / Frontend Lint (pull_request) Failing after 189h16m18s
CI/CD Pipeline / Validate Code Quality And Tests (pull_request) Failing after 189h16m21s
Deploy / Deploy Staging (push) Failing after 189h16m52s
CI/CD Pipeline / Frontend Lint (push) Failing after 189h17m22s
CI/CD Pipeline / Validate Code Quality And Tests (push) Failing after 189h17m26s
- hamming_distance: 替换 zfill+字符比较为 XOR bit 计数 (bin(h1^h2).count('1'))
- check_duplicate: 移除直方图融合和 best_match,返回第一个匹配
- compute_phash: 补充详细中文算法注释(DCT 步骤说明)
- 修复 similarity 精度:移除 round(x, 4) 保留完整浮点精度

所有 74 个测试通过(8 个 cv2 依赖跳过)。
2026-07-01 15:37:05 +08:00
灵应 112021c16c test: 补充查重模块单元测试与集成测试
- 查重引擎单元测试 (test_dedup_engine.py): 25 用例
  - hamming_distance XOR bit 计数验证
  - compute_phash / compute_color_histogram(需 cv2,无则跳过)
  - check_duplicate 相似度判定逻辑(MD5 精确匹配、pHash 阈值、首次匹配返回)
- 查重领域模型测试 (test_duplication_domain.py): 22 用例
  - DuplicationRecord.create() 工厂方法校验
  - 状态转换(pending → processing → completed/failed)
  - DuplicateSegment.create() 参数校验
- 查重用例层测试 (test_duplication_use_cases.py): 15 用例
  - UploadForDuplicationUseCase / ListDuplicationRecordsUseCase
  - GetDuplicationDetailUseCase / DeleteDuplicationRecordUseCase
  - RetryDuplicationUseCase 状态重置逻辑
- 查重 API 集成测试 (test_duplication_api.py): 20 用例
  - 列表/详情/删除/重试 4 个端点的正常流程与异常场景
  - 跨用户隔离验证
  - 跨端点组合场景测试

覆盖率: 422 passed, 8 skipped (cv2-dependent), 0 failures
2026-07-01 15:31:31 +08:00
灵应 6c3ddb018f fix: 修复 APIRouter 不支持 summary 参数的问题 2026-07-01 15:31:24 +08:00
灵应 b79c05377a fix: 重新注册 edit_plans_router 到 API Router
Deploy / Build Production Runtime Images (push) Has been skipped
Deploy / Deploy Production (push) Has been skipped
Deploy / Production Browser E2E (push) Has been skipped
Deploy / Deploy Staging (push) Failing after 189h50m15s
CI/CD Pipeline / Frontend Lint (push) Failing after 189h50m42s
CI/CD Pipeline / Validate Code Quality And Tests (push) Failing after 189h50m46s
2026-07-01 14:49:08 +08:00
灵应 ee8c60a143 perf(duplication): 查重模块代码优化 — 修复6个代码质量问题
1. 修复 hamming_distance 不等长哈希处理(hex() 前导零丢失)
2. 集成颜色直方图到 check_duplicate(此前计算但未使用,浪费 CPU)
3. check_duplicate 改为返回最佳匹配而非首个匹配
4. 修复仓库删除顺序(先删片段再删记录,防止孤儿数据)
5. 域模型添加 can_retry()/reset_for_retry(),仅 failed 状态允许重试
6. 列表接口暴露 offset/limit 分页参数
2026-07-01 14:48:15 +08:00