feat(worker): 手动查重 worker task + visual_similarity/match_count 字段 #1661

- 新增 Celery task worker.process_duplication_check:
  OSS 下载上传视频 → 动态抽帧指纹 → 跨项目与用户所有成片比对
  (compute_duplicate_rate + find_duplicate_segments) → 更新 DuplicationRecord
- 状态流转 pending → processing → completed/failed,失败重试 3 次间隔 60s,
  重试耗尽标记 failed,临时文件 finally 清理
- 片段映射:worker 分片 DuplicateSegment(ms/汉明距离) → 领域 DuplicateSegment(秒/0-100 相似度)
- DuplicationRecord 实体新增 visual_similarity/match_count(可选 None,向后兼容),
  mark_completed()/reset_for_retry() 同步
- migration 065: duplication_records 表加两列
- repository/model/schemas/routes 读写映射与响应透传
- 上传接口与重试接口 enqueue task(此前记录一直 pending 的根因)
- 11 个新单测:成功/跳过/未找到/重试/失败标记/临时文件清理/片段映射/schema
This commit is contained in:
saas-backend-agent
2026-09-04 01:11:04 +08:00
parent df164ddf75
commit c6af6dd072
9 changed files with 638 additions and 1 deletions
@@ -25,6 +25,8 @@ class SQLAlchemyDuplicationRecordRepository:
status=record.status,
duplicate_rate=record.duplicate_rate,
duplicate_count=record.duplicate_count,
visual_similarity=record.visual_similarity,
match_count=record.match_count,
video_fingerprint=json.dumps(record.video_fingerprint) if record.video_fingerprint else None,
error_message=record.error_message,
created_at=record.created_at,
@@ -58,6 +60,8 @@ class SQLAlchemyDuplicationRecordRepository:
model.status = record.status
model.duplicate_rate = record.duplicate_rate
model.duplicate_count = record.duplicate_count
model.visual_similarity = record.visual_similarity
model.match_count = record.match_count
model.video_fingerprint = json.dumps(record.video_fingerprint) if record.video_fingerprint else None
model.error_message = record.error_message
model.updated_at = record.updated_at
@@ -121,6 +125,8 @@ class SQLAlchemyDuplicationRecordRepository:
status=model.status,
duplicate_rate=model.duplicate_rate,
duplicate_count=int(model.duplicate_count or 0),
visual_similarity=getattr(model, "visual_similarity", None),
match_count=getattr(model, "match_count", None),
video_fingerprint=json.loads(fp_raw) if fp_raw else None,
error_message=getattr(model, "error_message", ""),
segments=segments,
@@ -417,6 +417,9 @@ class DuplicationRecordModel(Base):
status = Column(String(20), nullable=False, default="pending", index=True)
duplicate_rate = Column(Float, nullable=True)
duplicate_count = Column(Integer, nullable=False, default=0)
# #1661 手动查重:视觉相似度(0~1)/ 匹配视频数
visual_similarity = Column(Float, nullable=True)
match_count = Column(Integer, nullable=True)
video_fingerprint = Column(Text, nullable=True)
error_message = Column(Text, nullable=False, default="")
created_at = Column(DateTime, nullable=False, default=lambda: datetime.now(timezone.utc))