feat(dedup): 自动查重率扩大到用户全局 跨项目比对 #1602

Closed
xiaoxia wants to merge 0 commits from feat/auto-duplicate-rate-user-scope into develop
Owner

P0: compute_duplicate_rate 从 project_id 范围扩大到 user_id 全局范围。dedup_helpers 传 user_id。107 测试全过。

P0: compute_duplicate_rate 从 project_id 范围扩大到 user_id 全局范围。dedup_helpers 传 user_id。107 测试全过。
xiaoxia added 1 commit 2026-09-01 13:06:31 +08:00
feat(dedup): 自动查重率扩大到用户全局——跨项目比对
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 1s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 1s
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 3m54s
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 3m52s
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 4m46s
AI Code Review / AI Code Review (pull_request) Failing after 4m56s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 4m53s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 2m16s
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
PR Automation / Auto Approve on CI Green (pull_request) Successful in 6m54s
Preview Deploy / Deploy Preview Environment (pull_request) Failing after 7m2s
CI/CD Pipeline / Validate - Style (pull_request) Failing after 7m10s
CI/CD Pipeline / Validate - Security (pull_request) Has been cancelled
CI/CD Pipeline / Unit Tests (pull_request) Has been cancelled
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been cancelled
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been cancelled
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been cancelled
CI/CD Pipeline / Build Production API Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Web Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been cancelled
CI/CD Pipeline / Deploy Production (pull_request) Has been cancelled
CI/CD Pipeline / Production Browser E2E (pull_request) Has been cancelled
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been cancelled
CI/CD Pipeline / Canary Release to Production (pull_request) Has been cancelled
CI/CD Pipeline / CI Gate (pull_request) Has been cancelled
3a9a4334df
原 compute_duplicate_rate 仅按 project_id 比较,用户在不同项目/模板
下生成的视频互不可见,导致查重率始终为 0。

修改:
- dedup.py: compute_duplicate_rate 新增可选 user_id 参数;
  有 user_id 时按用户全局查询(跨项目),否则回退 project_id。
  查询上限从 100 提升到 200。
- dedup_helpers.py: create_video_record_and_dedup 传 user_id
  给 compute_duplicate_rate。

新增测试:
- test_duplicate_rate.py: +2 用例(跨项目匹配 / 空 user_id 回退)
- test_dedup_helpers_user_id.py: 3 用例(user_id 透传 / 空值兼容 /
  duplicate_rate 写回记录)

共 107 个相关测试全过。
auto-approve-bot added 1 commit 2026-09-01 13:13:45 +08:00
style: auto-format with black + isort + prettier [skip ci-format-check]
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 3s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 2s
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Validate - Style (pull_request) Has been cancelled
CI/CD Pipeline / Validate - Security (pull_request) Has been cancelled
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Has been cancelled
CI/CD Pipeline / Unit Tests (pull_request) Has been cancelled
CI/CD Pipeline / Integration Tests (pull_request) Has been cancelled
CI/CD Pipeline / Frontend Lint (pull_request) Has been cancelled
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been cancelled
CI/CD Pipeline / PR Build API Image (pull_request) Has been cancelled
CI/CD Pipeline / PR Build Web Image (pull_request) Has been cancelled
CI/CD Pipeline / PR Build Worker Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Staging API Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been cancelled
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been cancelled
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been cancelled
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been cancelled
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been cancelled
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been cancelled
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been cancelled
CI/CD Pipeline / Build Production API Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Web Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been cancelled
CI/CD Pipeline / Deploy Production (pull_request) Has been cancelled
CI/CD Pipeline / Production Browser E2E (pull_request) Has been cancelled
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been cancelled
CI/CD Pipeline / Canary Release to Production (pull_request) Has been cancelled
CI/CD Pipeline / CI Gate (pull_request) Has been cancelled
AI Code Review / AI Code Review (pull_request) Has been cancelled
PR Automation / Auto Approve on CI Green (pull_request) Has been cancelled
PR Automation / Auto Merge on CI Green + Approved (pull_request) Has been cancelled
Preview Deploy / Deploy Preview Environment (pull_request) Has been cancelled
1225e65cc3
xiaoxia added 1 commit 2026-09-01 13:17:40 +08:00
fix(dedup): AI Review 阻塞问题修复——DB 层排除自身 + 文档修正
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 3s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 3s
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 4m41s
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 4m51s
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Frontend Unit Tests (pull_request) Successful in 4m15s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 5m44s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 5m20s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 4m15s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 6m49s
CI/CD Pipeline / Frontend Lint (pull_request) Successful in 6m2s
CI/CD Pipeline / Unit Tests (pull_request) Successful in 6m32s
CI/CD Pipeline / PR Build Web Image (pull_request) Successful in 3m1s
CI/CD Pipeline / Validate - Style (pull_request) Has been cancelled
CI/CD Pipeline / Validate - Security (pull_request) Has been cancelled
CI/CD Pipeline / PR Build Worker Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production API Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Web Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been cancelled
CI/CD Pipeline / Deploy Production (pull_request) Has been cancelled
CI/CD Pipeline / Production Browser E2E (pull_request) Has been cancelled
CI/CD Pipeline / Canary Release to Production (pull_request) Has been cancelled
CI/CD Pipeline / CI Gate (pull_request) Has been cancelled
AI Code Review / AI Code Review (pull_request) Has been cancelled
9ad729d917
1. DB 查询层排除 current_video_id:
   原实现在 Python 循环里排除自身,但 GeneratedVideo 记录在查重前已
   写入 DB(dedup_helpers L80 create → L126 compute_duplicate_rate),
   查询结果会包含自身 → MD5 完全匹配 → duplicate_rate 恒为 100%。
   现在在 query 构建时即排除:query.filter(id != current_video_id)

2. 文档修正:limit(200) → "遍历最近 200 个"(与实际实现一致)

3. 测试 mock 更新:filter() 现在被调用两次(scope + self-exclusion),
   所有测试改用 chainable query_mock 模式。

🚀 预览环境已部署

项目 详情
PR号 #1602
预览链接 https://pr-1602.preview.xiaoxiajianji.com
API环境 staging

💡 预览环境使用 staging API 数据,请勿在预览环境中操作重要数据。

🔄 每次提交新代码后预览环境会自动更新。

🗑️ PR 关闭或合并后,预览环境会自动清理。

🚀 **预览环境已部署** | 项目 | 详情 | |------|------| | PR号 | #1602 | | 预览链接 | [https://pr-1602.preview.xiaoxiajianji.com](https://pr-1602.preview.xiaoxiajianji.com) | | API环境 | staging | > 💡 预览环境使用 staging API 数据,请勿在预览环境中操作重要数据。 > > 🔄 每次提交新代码后预览环境会自动更新。 > > 🗑️ PR 关闭或合并后,预览环境会自动清理。
auto-approve-bot added 1 commit 2026-09-01 13:25:09 +08:00
style: auto-format with black + isort + prettier [skip ci-format-check]
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 6s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 7s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 3m42s
CI/CD Pipeline / Frontend Unit Tests (pull_request) Successful in 4m49s
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 4m57s
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
CI/CD Pipeline / PR Build Web Image (pull_request) Successful in 4m52s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 5m56s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 6m9s
CI/CD Pipeline / Validate - Style (pull_request) Successful in 6m11s
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 6m43s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 3m17s
AI Code Review / AI Code Review (pull_request) Successful in 7m43s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 8m8s
CI/CD Pipeline / Validate - Security (pull_request) Successful in 8m43s
CI/CD Pipeline / Unit Tests (pull_request) Successful in 8m50s
CI/CD Pipeline / Frontend Lint (pull_request) Successful in 9m10s
CI/CD Pipeline / Build Production API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Web Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / Canary Release to Production (pull_request) Has been skipped
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / CI Gate (pull_request) Successful in 4s
ACR Cleanup / ACR Image Cleanup (pull_request_target) Waiting to run
Preview Cleanup / Cleanup Preview Environment (pull_request) Successful in 41s
1cb5991c4b
Collaborator

【阻塞级判定】

  • 是否存在阻塞级问题:否
  • 阻塞级问题数量:0 个

📊 审查概览

  • 整体评价:通过
  • 建议级问题数量:1 个

🔴 阻塞级问题(必须修复)

💡 改进建议(不阻塞合并)

  1. [tests/unit/test_duplicate_rate.py: 213] 测试断言缺失
    • 具体内容:在测试方法 test_user_id_empty_falls_back_to_project 中,代码注释写道“通过检查 filter 被调用时的参数来间接验证”,但实际上缺少对应的断言语句。目前的测试仅验证了函数不报错且返回值为 0.0,并未真正验证在 user_id 为空时是否回退到了 project_id 进行过滤。建议补充对 session.query.return_value.filter.call_args 的检查,确保过滤条件正确。

良好实践

  • 向后兼容性dedup.pycompute_duplicate_rate 方法新增 user_id 参数时设置了默认值 "",并使用关键字-only 参数(*,),确保了未传参时的向后兼容性,设计良好。
  • SQL 链式调用处理test_duplicate_rate.py 中正确更新了 Mock 设置,使 query_mock.filter 返回自身以支持链式调用,符合新代码逻辑。
  • 逻辑清晰dedup.py 中通过 if user_id 明确区分了全局查重和项目级查重的逻辑分支,代码可读性高。

格式检查通过 | 逻辑审查通过 | 性能良好


🤖 由 AI 代码审查机器人自动生成 | 2026-09-01 05:32:53 | 模型:

### 【阻塞级判定】 - 是否存在阻塞级问题:否 - 阻塞级问题数量:0 个 ### 📊 审查概览 - 整体评价:通过 - 建议级问题数量:1 个 ### 🔴 阻塞级问题(必须修复) 无 ### 💡 改进建议(不阻塞合并) 1. **[tests/unit/test_duplicate_rate.py: 213] 测试断言缺失** - 具体内容:在测试方法 `test_user_id_empty_falls_back_to_project` 中,代码注释写道“通过检查 filter 被调用时的参数来间接验证”,但实际上缺少对应的断言语句。目前的测试仅验证了函数不报错且返回值为 0.0,并未真正验证在 `user_id` 为空时是否回退到了 `project_id` 进行过滤。建议补充对 `session.query.return_value.filter.call_args` 的检查,确保过滤条件正确。 ### ✅ 良好实践 - **向后兼容性**:`dedup.py` 中 `compute_duplicate_rate` 方法新增 `user_id` 参数时设置了默认值 `""`,并使用关键字-only 参数(`*,`),确保了未传参时的向后兼容性,设计良好。 - **SQL 链式调用处理**:`test_duplicate_rate.py` 中正确更新了 Mock 设置,使 `query_mock.filter` 返回自身以支持链式调用,符合新代码逻辑。 - **逻辑清晰**:`dedup.py` 中通过 `if user_id` 明确区分了全局查重和项目级查重的逻辑分支,代码可读性高。 --- ✅ 格式检查通过 | ✅ 逻辑审查通过 | ✅ 性能良好 --- <sub>🤖 由 AI 代码审查机器人自动生成 | 2026-09-01 05:32:53 | 模型: </sub> <!-- AI_CODE_REVIEW_AUTO_COMMENT -->
auto-approve-bot approved these changes 2026-09-01 13:40:32 +08:00
auto-approve-bot left a comment
Collaborator

CI全绿,自动审批通过。

CI全绿,自动审批通过。
auto-approve-bot approved these changes 2026-09-01 13:40:32 +08:00
auto-approve-bot left a comment
Collaborator

CI全绿,自动审批通过。

CI全绿,自动审批通过。
xiaoxia closed this pull request 2026-09-03 15:54:29 +08:00

🗑️ 预览环境已清理

PR #1602 已关闭或合并,对应的预览环境已被清理。

如有需要,可以重新打开 PR 来重新生成预览环境。

🗑️ **预览环境已清理** PR #1602 已关闭或合并,对应的预览环境已被清理。 > 如有需要,可以重新打开 PR 来重新生成预览环境。
Some checks are pending
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 6s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 7s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 3m42s
CI/CD Pipeline / Frontend Unit Tests (pull_request) Successful in 4m49s
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 4m57s
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
CI/CD Pipeline / PR Build Web Image (pull_request) Successful in 4m52s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 5m56s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 6m9s
CI/CD Pipeline / Validate - Style (pull_request) Successful in 6m11s
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 6m43s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 3m17s
AI Code Review / AI Code Review (pull_request) Successful in 7m43s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 8m8s
CI/CD Pipeline / Validate - Security (pull_request) Successful in 8m43s
CI/CD Pipeline / Unit Tests (pull_request) Successful in 8m50s
CI/CD Pipeline / Frontend Lint (pull_request) Successful in 9m10s
CI/CD Pipeline / Build Production API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Web Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / Canary Release to Production (pull_request) Has been skipped
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / CI Gate (pull_request) Successful in 4s
ACR Cleanup / ACR Image Cleanup (pull_request_target) Waiting to run
Preview Cleanup / Cleanup Preview Environment (pull_request) Successful in 41s

Pull request closed

Sign in to join this conversation.