feat(worker): 手动查重 worker task + visual_similarity/match_count 字段 #1661
- 新增 Celery task worker.process_duplication_check: OSS 下载上传视频 → 动态抽帧指纹 → 跨项目与用户所有成片比对 (compute_duplicate_rate + find_duplicate_segments) → 更新 DuplicationRecord - 状态流转 pending → processing → completed/failed,失败重试 3 次间隔 60s, 重试耗尽标记 failed,临时文件 finally 清理 - 片段映射:worker 分片 DuplicateSegment(ms/汉明距离) → 领域 DuplicateSegment(秒/0-100 相似度) - DuplicationRecord 实体新增 visual_similarity/match_count(可选 None,向后兼容), mark_completed()/reset_for_retry() 同步 - migration 065: duplication_records 表加两列 - repository/model/schemas/routes 读写映射与响应透传 - 上传接口与重试接口 enqueue task(此前记录一直 pending 的根因) - 11 个新单测:成功/跳过/未找到/重试/失败标记/临时文件清理/片段映射/schema
This commit is contained in:
@@ -25,6 +25,8 @@ class SQLAlchemyDuplicationRecordRepository:
|
||||
status=record.status,
|
||||
duplicate_rate=record.duplicate_rate,
|
||||
duplicate_count=record.duplicate_count,
|
||||
visual_similarity=record.visual_similarity,
|
||||
match_count=record.match_count,
|
||||
video_fingerprint=json.dumps(record.video_fingerprint) if record.video_fingerprint else None,
|
||||
error_message=record.error_message,
|
||||
created_at=record.created_at,
|
||||
@@ -58,6 +60,8 @@ class SQLAlchemyDuplicationRecordRepository:
|
||||
model.status = record.status
|
||||
model.duplicate_rate = record.duplicate_rate
|
||||
model.duplicate_count = record.duplicate_count
|
||||
model.visual_similarity = record.visual_similarity
|
||||
model.match_count = record.match_count
|
||||
model.video_fingerprint = json.dumps(record.video_fingerprint) if record.video_fingerprint else None
|
||||
model.error_message = record.error_message
|
||||
model.updated_at = record.updated_at
|
||||
@@ -121,6 +125,8 @@ class SQLAlchemyDuplicationRecordRepository:
|
||||
status=model.status,
|
||||
duplicate_rate=model.duplicate_rate,
|
||||
duplicate_count=int(model.duplicate_count or 0),
|
||||
visual_similarity=getattr(model, "visual_similarity", None),
|
||||
match_count=getattr(model, "match_count", None),
|
||||
video_fingerprint=json.loads(fp_raw) if fp_raw else None,
|
||||
error_message=getattr(model, "error_message", ""),
|
||||
segments=segments,
|
||||
|
||||
@@ -417,6 +417,9 @@ class DuplicationRecordModel(Base):
|
||||
status = Column(String(20), nullable=False, default="pending", index=True)
|
||||
duplicate_rate = Column(Float, nullable=True)
|
||||
duplicate_count = Column(Integer, nullable=False, default=0)
|
||||
# #1661 手动查重:视觉相似度(0~1)/ 匹配视频数
|
||||
visual_similarity = Column(Float, nullable=True)
|
||||
match_count = Column(Integer, nullable=True)
|
||||
video_fingerprint = Column(Text, nullable=True)
|
||||
error_message = Column(Text, nullable=False, default="")
|
||||
created_at = Column(DateTime, nullable=False, default=lambda: datetime.now(timezone.utc))
|
||||
|
||||
Reference in New Issue
Block a user