xiaoxia/xiaoxia-saas
1
0
Fork 0
Code Issues 9 Pull Requests 3 Actions Packages Projects Releases 7 Wiki Activity

feat(ci): staging部署脚本增加本地自动回滚能力 #385

Merged
xiaoxia merged 4 commits from feat/staging-auto-rollback into develop 2026-07-16 21:26:44 +08:00
Conversation 1 Commits 4 Files Changed 1
+164 -9
xiaoxia commented 2026-07-16 08:18:20 +08:00
Owner
Copy Link
Copy Source

背景

当前staging部署只有CI侧公网健康检查失败后的SSH回滚（第二道防线）。但如果容器刚启动就挂了（比如启动命令错、依赖缺失），要等CI侧健康检查超时才回滚，浪费时间。

改动

在scripts/ci_staging_deploy.sh（staging服务器本地执行的部署脚本）中增加本地自动回滚能力：

  1. 部署前快照：记录API/Worker/Web三个服务当前运行的镜像tag
  2. rollback()函数：健康检查不通过时，用旧镜像重新启动容器
  3. 触发点：
    • 容器docker run失败 → 立即回滚
    • API健康检查超时（120s）→ 自动回滚
    • Web健康检查超时（30s）→ 自动回滚
  4. 数据库迁移失败不自动回滚：数据安全考虑，需人工介入判断
  5. 支持SKIP_ROLLBACK=true：调试时可关闭自动回滚

两道防线

  • 第一道：staging服务器本地回滚（快，容器启动阶段就失败的场景）
  • 第二道：CI侧公网健康检查回滚（全链路验证，服务真的对外可用）
## 背景 当前staging部署只有CI侧公网健康检查失败后的SSH回滚（第二道防线）。但如果容器刚启动就挂了（比如启动命令错、依赖缺失），要等CI侧健康检查超时才回滚，浪费时间。 ## 改动 在`scripts/ci_staging_deploy.sh`（staging服务器本地执行的部署脚本）中增加本地自动回滚能力： 1. **部署前快照**：记录API/Worker/Web三个服务当前运行的镜像tag 2. **rollback()函数**：健康检查不通过时，用旧镜像重新启动容器 3. **触发点**： - 容器`docker run`失败 → 立即回滚 - API健康检查超时（120s）→ 自动回滚 - Web健康检查超时（30s）→ 自动回滚 4. **数据库迁移失败不自动回滚**：数据安全考虑，需人工介入判断 5. **支持SKIP_ROLLBACK=true**：调试时可关闭自动回滚 ## 两道防线 - 第一道：staging服务器本地回滚（快，容器启动阶段就失败的场景） - 第二道：CI侧公网健康检查回滚（全链路验证，服务真的对外可用）
xiaoxia force-pushed feat/staging-auto-rollback from f544ee501f to e838b3595e 2026-07-16 08:49:01 +08:00 Compare
xiaoxia force-pushed feat/staging-auto-rollback from e838b3595e to 5a34e55134 2026-07-16 09:06:54 +08:00 Compare
xiaoxia added 1 commit 2026-07-16 09:22:10 +08:00
feat(ci): staging部署脚本增加本地自动回滚能力- 部署前记录当前三个服务的镜像版本（回滚目标）- 新增rollback()函数：容器启动失败/健康检查不通过时自动回滚- 本地健康检查失败立即回滚，不用等CI侧SSH回滚（更快）- 迁移失败不自动回滚（数据安全考虑，需人工介入）- 支持SKIP_ROLLBACK环境变量关闭自动回滚- 与CI侧公网健康检查回滚形成两道防线
CI Build & Deploy Pipeline / Build Staging API Image (push) Has been skipped
Details
CI Build & Deploy Pipeline / Build Staging Web Image (push) Has been skipped
Details
CI Build & Deploy Pipeline / Build Staging Worker Image (push) Has been skipped
Details
CI Build & Deploy Pipeline / Build Staging API Image (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Build Staging Web Image (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Build Staging Worker Image (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Deploy Staging (Watchtower auto-deploy) (push) Has been skipped
Details
CI Build & Deploy Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Staging API Integration Tests (push) Has been skipped
Details
CI Build & Deploy Pipeline / Staging E2E Tests (push) Has been skipped
Details
CI Build & Deploy Pipeline / Staging E2E Tests (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Staging API Integration Tests (pull_request) Has been skipped
Details
CI/CD Pipeline / Frontend Lint (pull_request) Successful in 1m40s
Details
CI/CD Pipeline / Frontend Lint (push) Successful in 1m41s
Details
CI Build & Deploy Pipeline / Build Production API Image (push) Has been skipped
Details
CI Build & Deploy Pipeline / Build Production Web Image (push) Has been skipped
Details
CI Build & Deploy Pipeline / Build Production Worker Image (push) Has been skipped
Details
CI Build & Deploy Pipeline / Build Production API Image (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Build Production Web Image (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Build Production Worker Image (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Deploy Production (push) Has been skipped
Details
CI Build & Deploy Pipeline / Production Browser E2E (push) Has been skipped
Details
CI Build & Deploy Pipeline / Deploy Production (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Production Browser E2E (pull_request) Has been skipped
Details
CI/CD Pipeline / Unit Tests (push) Successful in 3m0s
Details
CI/CD Pipeline / Unit Tests (pull_request) Successful in 3m8s
Details
CI/CD Pipeline / Validate Code Quality And Tests (pull_request) Failing after 13m16s
Details
CI/CD Pipeline / Validate Code Quality And Tests (push) Failing after 13m17s
Details
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m35s
Details
CI/CD Pipeline / Integration Tests (push) Successful in 1m34s
Details
6d4b9a0e51
xiaoxia force-pushed feat/staging-auto-rollback from 5a34e55134 to 6d4b9a0e51 2026-07-16 09:22:10 +08:00 Compare
xiaoxia added 1 commit 2026-07-16 09:50:35 +08:00
chore(ci): retry CI run to avoid flaky runner
CI Build & Deploy Pipeline / Build Staging API Image (push) Has been skipped
Details
CI Build & Deploy Pipeline / Build Staging Web Image (push) Has been skipped
Details
CI Build & Deploy Pipeline / Build Staging Worker Image (push) Has been skipped
Details
CI Build & Deploy Pipeline / Build Staging API Image (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Build Staging Web Image (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Deploy Staging (Watchtower auto-deploy) (push) Has been skipped
Details
CI Build & Deploy Pipeline / Build Staging Worker Image (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Staging E2E Tests (push) Has been skipped
Details
CI Build & Deploy Pipeline / Staging API Integration Tests (push) Has been skipped
Details
CI Build & Deploy Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Staging E2E Tests (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Staging API Integration Tests (pull_request) Has been skipped
Details
CI/CD Pipeline / Frontend Lint (push) Successful in 1m13s
Details
CI/CD Pipeline / Frontend Lint (pull_request) Successful in 1m25s
Details
CI/CD Pipeline / Unit Tests (push) Successful in 3m38s
Details
CI/CD Pipeline / Unit Tests (pull_request) Successful in 3m9s
Details
CI Build & Deploy Pipeline / Build Production API Image (push) Has been skipped
Details
CI Build & Deploy Pipeline / Build Production Web Image (push) Has been skipped
Details
CI Build & Deploy Pipeline / Build Production Worker Image (push) Has been skipped
Details
CI Build & Deploy Pipeline / Build Production API Image (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Deploy Production (push) Has been skipped
Details
CI Build & Deploy Pipeline / Build Production Web Image (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Production Browser E2E (push) Has been skipped
Details
CI Build & Deploy Pipeline / Build Production Worker Image (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Deploy Production (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Production Browser E2E (pull_request) Has been skipped
Details
CI/CD Pipeline / Validate Code Quality And Tests (pull_request) Failing after 15m43s
Details
CI/CD Pipeline / Validate Code Quality And Tests (push) Failing after 16m39s
Details
CI/CD Pipeline / Integration Tests (push) Successful in 2m16s
Details
CI/CD Pipeline / Integration Tests (pull_request) Successful in 2m4s
Details
58e4e4a18d
xiaoxia added 1 commit 2026-07-16 20:07:35 +08:00
Merge remote-tracking branch 'origin/develop' into feat/staging-auto-rollback
CI Build & Deploy Pipeline / Build Staging API Image (push) Has been skipped
Details
CI Build & Deploy Pipeline / Build Staging Web Image (push) Has been skipped
Details
CI Build & Deploy Pipeline / Build Staging Worker Image (push) Has been skipped
Details
CI Build & Deploy Pipeline / Build Staging API Image (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Build Staging Web Image (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Deploy Staging (Watchtower auto-deploy) (push) Has been skipped
Details
CI Build & Deploy Pipeline / Build Staging Worker Image (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Staging E2E Tests (push) Has been skipped
Details
CI Build & Deploy Pipeline / Staging API Integration Tests (push) Has been skipped
Details
CI Build & Deploy Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Staging E2E Tests (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Staging API Integration Tests (pull_request) Has been skipped
Details
CI/CD Pipeline / Frontend Lint (push) Successful in 2m11s
Details
CI/CD Pipeline / Validate Code Quality And Tests (push) Successful in 7m10s
Details
CI/CD Pipeline / Frontend Lint (pull_request) Successful in 2m0s
Details
CI/CD Pipeline / Unit Tests (push) Successful in 8m21s
Details
CI/CD Pipeline / Validate Code Quality And Tests (pull_request) Successful in 6m44s
Details
CI/CD Pipeline / Unit Tests (pull_request) Successful in 8m6s
Details
CI Build & Deploy Pipeline / Build Production API Image (push) Has been skipped
Details
CI Build & Deploy Pipeline / Build Production Web Image (push) Has been skipped
Details
CI Build & Deploy Pipeline / Build Production Worker Image (push) Has been skipped
Details
CI Build & Deploy Pipeline / Build Production API Image (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Build Production Web Image (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Build Production Worker Image (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Deploy Production (push) Has been skipped
Details
CI Build & Deploy Pipeline / Deploy Production (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Production Browser E2E (push) Has been skipped
Details
CI Build & Deploy Pipeline / Production Browser E2E (pull_request) Has been skipped
Details
CI/CD Pipeline / Integration Tests (push) Successful in 5m32s
Details
CI/CD Pipeline / Integration Tests (pull_request) Successful in 3m59s
Details
31d8ce765f
auto-approve-bot approved these changes 2026-07-16 20:52:42 +08:00
auto-approve-bot left a comment
Collaborator
Copy Link
Copy Source

CI全绿，自动审批通过。

CI全绿，自动审批通过。
xiaoxia added 1 commit 2026-07-16 21:05:46 +08:00
Merge remote-tracking branch 'origin/develop' into feat/staging-auto-rollback
CI Build & Deploy Pipeline / Build Staging API Image (push) Has been skipped
Details
CI Build & Deploy Pipeline / Build Staging Web Image (push) Has been skipped
Details
CI Build & Deploy Pipeline / Build Staging Worker Image (push) Has been skipped
Details
CI Build & Deploy Pipeline / Deploy Staging (Watchtower auto-deploy) (push) Has been skipped
Details
CI Build & Deploy Pipeline / Build Staging API Image (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Staging E2E Tests (push) Has been skipped
Details
CI Build & Deploy Pipeline / Build Staging Web Image (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Staging API Integration Tests (push) Has been skipped
Details
CI Build & Deploy Pipeline / Build Staging Worker Image (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Staging E2E Tests (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Staging API Integration Tests (pull_request) Has been skipped
Details
CI/CD Pipeline / Frontend Lint (push) Successful in 1m56s
Details
CI Build & Deploy Pipeline / Build Production API Image (push) Has been skipped
Details
CI Build & Deploy Pipeline / Build Production Web Image (push) Has been skipped
Details
CI Build & Deploy Pipeline / Build Production Worker Image (push) Has been skipped
Details
CI Build & Deploy Pipeline / Deploy Production (push) Has been skipped
Details
CI Build & Deploy Pipeline / Production Browser E2E (push) Has been skipped
Details
CI/CD Pipeline / Validate Code Quality And Tests (push) Successful in 7m37s
Details
CI Build & Deploy Pipeline / Build Production API Image (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Build Production Web Image (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Build Production Worker Image (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Deploy Production (pull_request) Has been skipped
Details
CI Build & Deploy Pipeline / Production Browser E2E (pull_request) Has been skipped
Details
CI/CD Pipeline / Frontend Lint (pull_request) Successful in 1m28s
Details
CI/CD Pipeline / Validate Code Quality And Tests (pull_request) Successful in 4m50s
Details
CI/CD Pipeline / Unit Tests (push) Successful in 6m47s
Details
CI/CD Pipeline / Unit Tests (pull_request) Successful in 3m39s
Details
CI/CD Pipeline / Integration Tests (push) Successful in 2m3s
Details
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m47s
Details
dec873d13b
xiaoxia merged commit 4c662cca32 into develop 2026-07-16 21:26:44 +08:00
Sign in to join this conversation.
Reviewers
No Reviewers
auto-approve-bot
Labels
Clear labels
P0
阻塞性，必须立即处理
 P1
高优先级
 P2
中优先级
 P3
低优先级/后考虑
 architecture
架构设计
 backend
后端相关
 blocked
阻塞中
 bug
Bug修复
 ci-bug
CI本身的bug
 ci-optimization
CI优化类任务
 ci-pipeline
CI流水线/workflow相关
 cleanup
代码清理
 devops
运维/部署/CI
 documentation
文档相关
 feature
新功能
 frontend
前端相关
 question
需要讨论确认
 ready
PR CI通过，可合并
 refactor
代码重构
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
auto-approve-bot xiaoxia
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: xiaoxia/xiaoxia-saas#385
Delete Branch "feat/staging-auto-rollback"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?