xiaoxia/xiaoxia-saas
1
0
Fork 0
Code Issues 10 Pull Requests 2 Actions Packages Projects Releases 7 Wiki Activity

fix(worker): OSS上传崩溃修复 - connect超时 + 分片上传 + 总超时保护 #238

Merged
xiaoxia merged 1 commits from fix/oss-upload-crash into develop 2026-07-13 09:33:13 +08:00
Conversation 0 Commits 1 Files Changed 2
+318 -10
xiaoxia commented 2026-07-13 07:51:53 +08:00
Owner
Copy Link
Copy Source

问题

staging worker 在 OSS 上传阶段崩溃/卡死：3 个 one_take 任务渲染完成后，日志停在「OSS上传开始」，之后无任何输出，worker 进程从容器内消失。

根因分析（按概率排序）

  1. OOM Kill：FFmpeg 渲染后内存未完全释放 + OSS SDK 内部缓冲 + 容器内存上限 → Linux OOM killer 直接杀进程
  2. OSS SDK 无超时挂死：oss_bucket() 未配置 connect_timeout，网络抖动时 TCP 握手阶段无限挂起
  3. 大文件一次性上传：put_object_from_file 一次性加载到内存，大文件场景内存峰值高

修复内容

1. oss_bucket() 增加 connect_timeout=10s

  • 防止网络抖动时 TCP 握手阶段无限挂死
  • 所有 OSS 操作（上传/下载/预签名）都受益

2. upload_to_oss() 大文件自动分片上传

  • = 100MB 自动走 oss2.resumable_upload 分片上传

  • 分片大小 8MB，3 线程并发
  • 大幅降低内存峰值，减少 OOM 风险

3. upload_to_oss() 总超时保护（300s）

  • 子线程执行上传 + 主线程 wait(timeout) 超时机制
  • 超过 300s 自动返回 None，调用方正常走失败逻辑
  • 避免 worker 进程无限挂死导致队列堵塞

测试

  • 7 个新增单元测试，全部通过
    • oss_bucket connect_timeout 配置验证
    • 小文件走 put_object_from_file
    • 大文件走 resumable_upload（分片参数校验）
    • 上传超时返回 None
    • 上传异常返回 None
    • 未配置 OSS 返回 None
## 问题 staging worker 在 OSS 上传阶段崩溃/卡死：3 个 one_take 任务渲染完成后，日志停在「OSS上传开始」，之后无任何输出，worker 进程从容器内消失。 ## 根因分析（按概率排序） 1. **OOM Kill**：FFmpeg 渲染后内存未完全释放 + OSS SDK 内部缓冲 + 容器内存上限 → Linux OOM killer 直接杀进程 2. **OSS SDK 无超时挂死**：`oss_bucket()` 未配置 `connect_timeout`，网络抖动时 TCP 握手阶段无限挂起 3. **大文件一次性上传**：`put_object_from_file` 一次性加载到内存，大文件场景内存峰值高 ## 修复内容 ### 1. oss_bucket() 增加 connect_timeout=10s - 防止网络抖动时 TCP 握手阶段无限挂死 - 所有 OSS 操作（上传/下载/预签名）都受益 ### 2. upload_to_oss() 大文件自动分片上传 - >= 100MB 自动走 `oss2.resumable_upload` 分片上传 - 分片大小 8MB，3 线程并发 - 大幅降低内存峰值，减少 OOM 风险 ### 3. upload_to_oss() 总超时保护（300s） - 子线程执行上传 + 主线程 wait(timeout) 超时机制 - 超过 300s 自动返回 None，调用方正常走失败逻辑 - 避免 worker 进程无限挂死导致队列堵塞 ## 测试 - 7 个新增单元测试，全部通过 - oss_bucket connect_timeout 配置验证 - 小文件走 put_object_from_file - 大文件走 resumable_upload（分片参数校验） - 上传超时返回 None - 上传异常返回 None - 未配置 OSS 返回 None
xiaoxia force-pushed fix/oss-upload-crash from 071535826c to 7b113d67eb 2026-07-13 08:17:31 +08:00 Compare
xiaoxia force-pushed fix/oss-upload-crash from 7b113d67eb to 0bcefc7458 2026-07-13 08:54:29 +08:00 Compare
xiaoxia added 1 commit 2026-07-13 08:59:07 +08:00
fix(worker): OSS上传崩溃修复 - connect超时 + 分片上传 + 总超时保护
CI/CD Pipeline / Frontend Lint (pull_request) Successful in 52s
Details
CI/CD Pipeline / Validate Code Quality And Tests (pull_request) Successful in 1m11s
Details
CI/CD Pipeline / Build & Push Staging (Watchtower auto-deploy) (pull_request) Has been skipped
Details
CI/CD Pipeline / Build Production Runtime Images (pull_request) Has been skipped
Details
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
Details
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
Details
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
Details
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
Details
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m49s
Details
9b55b88edb 
1. oss_bucket() 增加 connect_timeout=10s，防止TCP握手挂死
2. upload_to_oss() 大文件(>=100MB)自动走分片上传，降低内存峰值减少OOM风险
3. upload_to_oss() 增加总超时保护(300s)，线程超时自动返回失败
4. 7个单元测试覆盖：超时配置、分片策略、超时保护、异常处理
xiaoxia force-pushed fix/oss-upload-crash from 0bcefc7458 to 9b55b88edb 2026-07-13 08:59:07 +08:00 Compare
xiaoxia merged commit 4c5ab7f80e into develop 2026-07-13 09:33:13 +08:00
xiaoxia deleted branch fix/oss-upload-crash 2026-07-13 09:33:13 +08:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
P0
阻塞性，必须立即处理
 P1
高优先级
 P2
中优先级
 P3
低优先级/后考虑
 architecture
架构设计
 backend
后端相关
 blocked
阻塞中
 bug
Bug修复
 ci-bug
CI本身的bug
 ci-optimization
CI优化类任务
 ci-pipeline
CI流水线/workflow相关
 cleanup
代码清理
 devops
运维/部署/CI
 documentation
文档相关
 feature
新功能
 frontend
前端相关
 question
需要讨论确认
 ready
PR CI通过，可合并
 refactor
代码重构
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
auto-approve-bot xiaoxia
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: xiaoxia/xiaoxia-saas#238
Delete Branch "fix/oss-upload-crash"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?