cbca0c3584ad0ae9cfdb6893573edc400836b8a4
1 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 saas-backend-agent
|
cbca0c3584 |
feat: 分片指纹存储改造 + 存量指纹重建脚本 #1657
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 3s
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 2s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 2m19s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 2m43s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m22s
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 8s
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 7s
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Validate - Style (pull_request) Has been cancelled
CI/CD Pipeline / Validate - Security (pull_request) Has been cancelled
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Has been cancelled
CI/CD Pipeline / Unit Tests (pull_request) Has been cancelled
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been cancelled
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been cancelled
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been cancelled
CI/CD Pipeline / Build Production API Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Web Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been cancelled
CI/CD Pipeline / Deploy Production (pull_request) Has been cancelled
CI/CD Pipeline / Production Browser E2E (pull_request) Has been cancelled
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been cancelled
CI/CD Pipeline / Canary Release to Production (pull_request) Has been cancelled
CI/CD Pipeline / CI Gate (pull_request) Has been cancelled
AI Code Review / AI Code Review (pull_request) Has been cancelled
PR Automation / Auto Merge on CI Green + Approved (pull_request) Has been cancelled
## 改动 ### 1. 新建 video_fingerprint_chunks 表（Migration 063） - 按时间分片存储 pHash + color_histogram - 索引：video_id, project_id, user_id ### 2. 新增 VideoFingerprintChunkModel - packages/adapters/sqlalchemy_impl/models.py ### 3. 改造 dedup.py 指纹计算 - compute_fingerprint() 改为按时间分片抽帧 - 短视频（≤60s）：每 2s 一片 - 长视频（>60s）：每 5s 一片 - VideoFingerprint 新增 chunks 字段（list of FingerprintChunk） - 向后兼容：keyframe_phashes/color_histograms 保留 - to_chunk_models() 方法转换为 SQLAlchemy Model - check_duplicate() 优先从分片表读取，回退到 JSON 字段 - check_duplicate_task() 写入分片表 ### 4. 改造 dedup_helpers.py - create_video_record_and_dedup() 同步写入分片表 ### 5. 存量指纹重建脚本 - apps/api/scripts/rebuild_fingerprint_chunks.py - 支持 --dry-run 和 --batch-size - 幂等：已有分片数据的视频跳过 ### 6. 单元测试（11 个） - 分片策略：60s→30片，120s→24片 - to_chunk_models() 输出正确 - _save_fingerprint_chunks 幂等性 - to_dict() 向后兼容 Closes #1657 |