fix(web, P0): nginx entrypoint 在 CI bind mount 场景不应 rm default.conf(staging 502 阻塞部署) #1856

Merged
auto-approve-bot merged 2 commits from fix/nginx-entrypoint-bind-mount into develop 2026-09-11 11:11:49 +08:00
Owner

问题

Staging 部署(Run #49298、#49301)连续失败:

  • Web 容器 30s 健康检查超时
  • 日志反复出现:rm: can't remove '/etc/nginx/conf.d/default.conf': Resource busy
  • 自动回滚到老镜像 29ca51da(#1853 引入的版本,本身 ln -sf 有 busybox 问题)→ web 无限重启 → staging 永久 502

根因

PR #1853 把 nginx 配置烤入镜像并通过 nginx-entrypoint.sh 按 APP_ENV 切换 symlink,但忽略了 CI 部署脚本的实际行为:

scripts/ci_staging_deploy.sh:519 和 scripts/ci_production_deploy.sh:436:

-v "$NGINX_CONF_FILE:/etc/nginx/conf.d/default.conf:ro"

CI 部署时宿主机生成的 nginx 配置(含 resolver/docker upstream 代理)以 readonly bind mount 挂到容器内 /etc/nginx/conf.d/default.conf。容器启动后 entrypoint:

  • #1853 用 ln -sf → busybox 在 target 为普通文件时会在 target 目录下创建子链接,nginx 读不到正确配置
  • #1855 改为 rm -f 再 ln -s → 对 readonly bind mount 的文件 rm 返回 EBUSY ("Resource busy"),set -e 直接退出,nginx 根本没启动

两次"修复"都没命中真正原因:CI 部署路径下 default.conf 是外部注入的,entrypoint 不应动它。

修复

重写 infra/docker/nginx-entrypoint.sh,分三种场景:

  1. 已经是正确 symlink(本地重启容器、镜像原生):直接 exec nginx
  2. 镜像原生场景(无外部挂载):rm -f + ln -s 切换 symlink;rm 失败(bind mount EBUSY/EROFS)说明外部已注入配置,|| true 吞错跳过 ln
  3. 兜底:只要 conf.d 下有 .conf 文件(含 bind mount 进来的)就启动;完全缺失才报错退出

验证

本地模拟两种场景:

  • ✅ bind mount readonly(模拟 CI 部署):rm 失败不阻塞,使用外部挂载的配置启动
  • ✅ 无挂载(本地 docker-compose / 直接 run):成功删除镜像默认 default.conf 并建立指向 nginx-{staging,production}.conf 的 symlink

影响

  • 修复后新镜像部署时 entrypoint 不会再报错退出,nginx 正常读取 bind mount 的配置
  • 回滚到老镜像问题仍存在(老镜像 entrypoint 有 bug),但只要这版部署成功就不再回滚
  • 不影响生产部署(prod 脚本同样用 bind mount 方式挂 default.conf)
## 问题 Staging 部署(Run #49298、#49301)连续失败: - Web 容器 30s 健康检查超时 - 日志反复出现:`rm: can't remove '/etc/nginx/conf.d/default.conf': Resource busy` - 自动回滚到老镜像 `29ca51da`(#1853 引入的版本,本身 `ln -sf` 有 busybox 问题)→ web 无限重启 → staging 永久 502 ## 根因 PR #1853 把 nginx 配置烤入镜像并通过 `nginx-entrypoint.sh` 按 APP_ENV 切换 symlink,但**忽略了 CI 部署脚本的实际行为**: `scripts/ci_staging_deploy.sh:519` 和 `scripts/ci_production_deploy.sh:436`: ```bash -v "$NGINX_CONF_FILE:/etc/nginx/conf.d/default.conf:ro" ``` CI 部署时宿主机生成的 nginx 配置(含 resolver/docker upstream 代理)以 **readonly bind mount** 挂到容器内 `/etc/nginx/conf.d/default.conf`。容器启动后 entrypoint: - #1853 用 `ln -sf` → busybox 在 target 为普通文件时会在 target 目录下创建子链接,nginx 读不到正确配置 - #1855 改为 `rm -f` 再 `ln -s` → 对 readonly bind mount 的文件 `rm` 返回 EBUSY ("Resource busy"),`set -e` 直接退出,nginx 根本没启动 两次"修复"都没命中真正原因:**CI 部署路径下 default.conf 是外部注入的,entrypoint 不应动它**。 ## 修复 重写 `infra/docker/nginx-entrypoint.sh`,分三种场景: 1. **已经是正确 symlink**(本地重启容器、镜像原生):直接 exec nginx 2. **镜像原生场景**(无外部挂载):rm -f + ln -s 切换 symlink;rm 失败(bind mount EBUSY/EROFS)说明外部已注入配置,`|| true` 吞错跳过 ln 3. **兜底**:只要 conf.d 下有 .conf 文件(含 bind mount 进来的)就启动;完全缺失才报错退出 ## 验证 本地模拟两种场景: - ✅ bind mount readonly(模拟 CI 部署):rm 失败不阻塞,使用外部挂载的配置启动 - ✅ 无挂载(本地 docker-compose / 直接 run):成功删除镜像默认 default.conf 并建立指向 nginx-{staging,production}.conf 的 symlink ## 影响 - 修复后新镜像部署时 entrypoint 不会再报错退出,nginx 正常读取 bind mount 的配置 - 回滚到老镜像问题仍存在(老镜像 entrypoint 有 bug),但只要这版部署成功就不再回滚 - 不影响生产部署(prod 脚本同样用 bind mount 方式挂 default.conf)
xiaoxia added 2 commits 2026-09-11 11:03:51 +08:00
fix(web, P0): nginx entrypoint 在 CI bind mount 场景不应 rm default.conf
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Successful in 1s
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 1s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 31s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 31s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 1m27s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m53s
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Successful in 1m55s
CI/CD Pipeline / Validate - Style (pull_request) Successful in 2m21s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 2m54s
AI Code Review / AI Code Review (pull_request) Successful in 6m21s
CI/CD Pipeline / Validate - Security (pull_request) Successful in 6m56s
CI/CD Pipeline / Unit Tests (pull_request) Successful in 7m14s
CI/CD Pipeline / CI Gate (pull_request) Successful in 1s
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 5m0s
ACR Cleanup / ACR Image Cleanup (pull_request_target) Successful in 7s
Preview Cleanup / Cleanup Preview Environment (pull_request) Successful in 29s
CI/CD Pipeline / Deploy Production (pull_request) Failing after 124h12m41s
CI/CD Pipeline / Frontend Unit Tests (pull_request) Failing after 124h19m57s
CI/CD Pipeline / Build Production API Image (pull_request) Failing after 124h12m42s
CI/CD Pipeline / Build Staging API Image (pull_request) Failing after 124h19m58s
CI/CD Pipeline / Staging E2E Tests (pull_request) Failing after 124h19m50s
CI/CD Pipeline / Build Staging Worker Image (pull_request) Failing after 124h19m42s
CI/CD Pipeline / Build Staging Web Image (pull_request) Failing after 124h19m44s
CI/CD Pipeline / ACR Image Cleanup (pull_request) Failing after 124h19m34s
CI/CD Pipeline / Build Production Web Image (pull_request) Failing after 124h12m27s
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Failing after 124h19m40s
CI/CD Pipeline / PR Build Web Image (pull_request) Failing after 124h19m41s
CI/CD Pipeline / Canary Release to Production (pull_request) Failing after 124h12m26s
CI/CD Pipeline / Build Production Worker Image (pull_request) Failing after 124h12m27s
CI/CD Pipeline / Staging API Integration Tests (pull_request) Failing after 124h19m34s
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Failing after 124h19m39s
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Failing after 124h19m41s
CI/CD Pipeline / Frontend Lint (pull_request) Failing after 124h19m42s
CI/CD Pipeline / Check push changed paths (pull_request) Failing after 124h19m45s
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Failing after 124h54m48s
4ece137786
部署脚本(ci_staging_deploy.sh / ci_production_deploy.sh)通过
-v 宿主机nginx配置:/etc/nginx/conf.d/default.conf:ro 把配置 bind mount
进容器。#1853 引入 entrypoint 按 APP_ENV 切换 symlink 的逻辑,但没考虑
CI 部署路径:
- #1853 用 ln -sf,busybox 在 target 为普通文件时会把子链接建到 target 目录里,
  nginx 读不到配置,web 容器循环重启。
- #1855 改为先 rm -f 再 ln -s,但对 bind mount readonly 的文件 rm 会报
  'Resource busy' (EBUSY),set -e 下脚本直接退出,nginx 没起,
  30s 健康检查失败 → 自动回滚 → 回滚到老镜像(29ca51da)也有同样问题
  → staging 永久 502。

修复:entrypoint 分三种场景处理:
1. default.conf 已经是指向目标 conf 的 symlink:什么都不做直接 exec nginx;
2. 镜像原生场景(无外部挂载):rm -f + ln -s 切换 symlink;
   rm 失败(bind mount readonly, EBUSY/EROFS)说明外部已注入配置,
   用 || true 吞错,跳过 ln;
3. 兜底:只要 conf.d 下有 .conf 文件就启动,缺失才报错退出。

本地模拟两种路径均验证通过:
- bind mount readonly 场景:rm 失败不阻塞,使用外部配置启动;
- 无挂载本地/开发场景:成功 rm 旧文件并建立正确 symlink。

🚀 预览环境已部署

项目 详情
PR号 #1856
预览链接 https://pr-1856.preview.xiaoxiajianji.com
API环境 staging

💡 预览环境使用 staging API 数据,请勿在预览环境中操作重要数据。

🔄 每次提交新代码后预览环境会自动更新。

🗑️ PR 关闭或合并后,预览环境会自动清理。

🚀 **预览环境已部署** | 项目 | 详情 | |------|------| | PR号 | #1856 | | 预览链接 | [https://pr-1856.preview.xiaoxiajianji.com](https://pr-1856.preview.xiaoxiajianji.com) | | API环境 | staging | > 💡 预览环境使用 staging API 数据,请勿在预览环境中操作重要数据。 > > 🔄 每次提交新代码后预览环境会自动更新。 > > 🗑️ PR 关闭或合并后,预览环境会自动清理。
auto-approve-bot merged commit e7ab963ae3 into develop 2026-09-11 11:11:49 +08:00
auto-approve-bot deleted branch fix/nginx-entrypoint-bind-mount 2026-09-11 11:11:49 +08:00

🗑️ 预览环境已清理

PR #1856 已关闭或合并,对应的预览环境已被清理。

如有需要,可以重新打开 PR 来重新生成预览环境。

🗑️ **预览环境已清理** PR #1856 已关闭或合并,对应的预览环境已被清理。 > 如有需要,可以重新打开 PR 来重新生成预览环境。
Sign in to join this conversation.