develop
7 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
df99305dd6 |
feat(worker): celery 队列隔离 + 孤儿任务消息作废 (#1714)
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 1s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 2s
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 29s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 29s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 49s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 1m39s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m44s
CI/CD Pipeline / Validate - Style (pull_request) Successful in 2m19s
AI Code Review / AI Code Review (pull_request) Failing after 2m52s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 2m54s
CI/CD Pipeline / Validate - Security (pull_request) Successful in 4m11s
CI/CD Pipeline / Unit Tests (pull_request) Failing after 6m23s
CI/CD Pipeline / Build Production API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI/CD Pipeline / CI Gate (pull_request) Failing after 1s
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / Canary Release to Production (pull_request) Has been skipped
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 3m39s
问题:素材转码与视频生成共用 celery 默认队列、worker 单进程消费, 20+ 转码积压会把用户生成任务堵 40 分钟以上;孤儿清理把任务标 failed 后 Redis 队列消息未作废,消息被重投导致 failed→running 非法转换, worker 打印 ERROR 后继续产出半成品。 队列隔离: - 新增 packages/shared/celery_queues.py:generation/transcode/celery 三队列与 task_routes(generate_video→generation;ingest_asset/ classify_asset/duplication→transcode),apply_queue_settings() - worker 入口改双进程:generation worker 独占队列并内嵌 beat (prefetch=1, GENERATION_CONCURRENCY 默认 2),transcode worker 消费 transcode,celery(并发=总-2,最小 1),任一退出则整体终止 - compose/部署脚本/ps1 同步新增 GENERATION_CONCURRENCY 与健康检查 消息作废: - 新增 packages/shared/celery_orphan_guard.py:终态守卫 ensure_task_claimable、Redis 队列消息物理清理(JSON 信封解析, 按业务 id + celery headers.id 双匹配,未命中 rpush 保序)、 revoke_and_purge(control.revoke + 物理清队列双保险) - 入队点(生成/上传/分片/重试)send_task 后持久化 celery_task_id 到 generation_tasks/ingest_jobs(新列,067 迁移,失败仅 warning) - generate_video/ingest_asset 执行前校验 DB 状态:终态直接 discarded 不进业务逻辑;mark_processing 返回 False(非法转换)安全中止 - 孤儿/超时清理标 failed 时同时 revoke + 清队列消息 - pending 超时阈值 15→45 分钟,与 running 孤儿(20min)区分 测试:新增 22 个单测(路由表/真实 Redis 消息清理/终态守卫/ 非法转换中止/标 failed 后消息不重投/入队持久化),全量 14301 passed;067 迁移隔离 DDL 验证 upgrade/downgrade 通过。 |
||
|
|
0a7ae8db4f |
fix(upload): complete 接口幂等去重 + HEVC 转码回写占位 asset 禁止兜底新建 (#1714) (#1715)
CI/CD Pipeline / Check if frontend-only change (push) Has been skipped
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (push) Successful in 1s
CI/CD Pipeline / Check push changed paths (push) Successful in 4s
CI/CD Pipeline / Frontend Lint (push) Has been skipped
CI/CD Pipeline / PR Build API Image (push) Has been skipped
CI/CD Pipeline / PR Build Web Image (push) Has been skipped
CI/CD Pipeline / PR Build Worker Image (push) Has been skipped
CI/CD Pipeline / Build Staging API Image (push) Successful in 41s
CI/CD Pipeline / Build Staging Worker Image (push) Successful in 32s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 2m5s
CI/CD Pipeline / Validate - Python (mypy + alembic) (push) Successful in 2m8s
AI Code Review / AI Code Review (pull_request) Failing after 2m24s
CI/CD Pipeline / Build Staging Web Image (push) Successful in 1m51s
CI/CD Pipeline / Validate - Style (push) Successful in 2m36s
CI/CD Pipeline / Integration Tests (push) Successful in 2m33s
CI/CD Pipeline / Retag skipped Staging API Image (push) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (push) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (push) Has been skipped
PR Automation / Auto Approve on CI Green (pull_request) Successful in 3m52s
PR Automation / Auto Merge on CI Green + Approved (pull_request) Has been skipped
CI/CD Pipeline / Frontend Unit Tests (push) Successful in 5m21s
CI/CD Pipeline / Validate - Security (push) Successful in 5m43s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (push) Successful in 3m1s
CI/CD Pipeline / Staging E2E Tests (push) Failing after 1m20s
CI/CD Pipeline / ACR Image Cleanup (push) Successful in 1m32s
CI/CD Pipeline / Unit Tests (push) Successful in 8m29s
CI/CD Pipeline / Build Production API Image (push) Has been skipped
CI/CD Pipeline / Build Production Web Image (push) Has been skipped
CI/CD Pipeline / Build Production Worker Image (push) Has been skipped
CI/CD Pipeline / CI Gate (push) Has been skipped
CI/CD Pipeline / Deploy Production (push) Has been skipped
CI/CD Pipeline / Production Browser E2E (push) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (push) Successful in 3m18s
CI/CD Pipeline / Canary Release to Production (push) Has been cancelled
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 1s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 1s
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 21s
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
CI/CD Pipeline / PR Build Web Image (pull_request) Successful in 24s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 28s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 1m43s
CI/CD Pipeline / Frontend Unit Tests (pull_request) Successful in 1m41s
CI/CD Pipeline / Frontend Lint (pull_request) Successful in 1m50s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 2m4s
CI/CD Pipeline / Validate - Style (pull_request) Successful in 2m25s
CI/CD Pipeline / Validate - Security (pull_request) Successful in 4m35s
CI/CD Pipeline / Unit Tests (pull_request) Successful in 7m16s
CI/CD Pipeline / Build Production API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI/CD Pipeline / CI Gate (pull_request) Successful in 17s
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / Canary Release to Production (pull_request) Has been skipped
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
Co-authored-by: xiaoxia <dev@xiaoxiajianji.com> Co-committed-by: xiaoxia <dev@xiaoxiajianji.com> |
||
|
|
4fee87c5e8 |
feat: 素材重复上传检测 + 批量生成视频
CI/CD Pipeline / Deploy Staging (push) Has been skipped
CI/CD Pipeline / Staging E2E Tests (push) Has been skipped
CI/CD Pipeline / Build Production Runtime Images (push) Has been skipped
CI/CD Pipeline / Deploy Production (push) Has been skipped
CI/CD Pipeline / Production Browser E2E (push) Has been skipped
CI/CD Pipeline / Frontend Lint (push) Failing after 45h56m35s
CI/CD Pipeline / Validate Code Quality And Tests (push) Failing after 45h56m48s
任务1: 素材重复上传检测 - 上传接口支持 file_hash 参数,通过 MD5+素材库ID 去重 - 命中去重直接返回已有 asset_id,不重复存 OSS - file_hash 透传: API → IngestJob → Asset 全链路 - 三条上传路径(表单/直传/分片)均支持去重 - Alembic 031: assets + ingest_jobs 加 file_hash 列+索引 - 6 个单元测试覆盖去重命中/未命中/空hash/透传 任务3: 批量生成视频 - POST /generations 支持 count 参数,一次创建多条生成任务 - 每条任务独立状态跟踪,响应返回 task_ids 列表 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
0422967229 |
refactor: Remove workspace concept - Projects now directly under User
BREAKING CHANGES:
- Removed Workspace, WorkspaceMember, WorkspaceInvitation entities
- Project now has owner_user_id instead of workspace_id
- Added shared_users list to Project for collaboration
- Subscription/quota moved from Workspace to User level
Changes:
- packages/domain/entities.py: Removed Workspace entities, updated Project
- packages/adapters/sqlalchemy_impl/models.py: Updated models
- packages/application/: Removed workspace use cases, updated other use cases
- packages/ports/: Removed workspace repository interfaces
- apps/api/: Updated routes, schemas, dependencies, router
- alembic/versions/007_remove_workspace_concept.py: Database migration
New APIs:
- POST /projects/{id}/share: Share project with user
- DELETE /projects/{id}/share/{user_id}: Unshare project
|
||
|
|
ee6b4a663a | feat(tasks): add project task center | ||
|
|
f0ee5c6184 | feat: persist ingest flow in postgres | ||
|
|
e4e2595e70 |
feat: add PostgreSQL persistence layer
- adapters: SQLAlchemy implementations for all 4 repositories (Project, AssetLibrary, Asset, IngestJob) - models: SQLAlchemy ORM models with proper schema - database: connection config and session management - tests: SQLAlchemy repository integration test (in-memory SQLite) - all 7 integration tests passing |