fix(ci): buildx本地缓存损坏自动检测+清理重试,根治snapshot not found连锁失败 #523

Merged
auto-approve-bot merged 2 commits from fix/buildx-local-cache-corruption into develop 2026-07-18 16:15:40 +08:00
Owner

问题

Web构建连续多轮失败,错误:parent snapshot sha256:xxx does not exist

根因是runner本地 /tmp/buildx-cache/ 目录里的buildx本地缓存(snapshot)损坏了,每次构建都从本地坏缓存读,直接炸。

API和Worker没挂是因为它们的缓存没坏,但Web那个缓存目录从#518那次开始坏了,之后所有Web构建连锁失败。

修复

scripts/ci/docker_build_push.sh 新增 build_with_cache_retry() 函数:

  1. 自动检测:识别缓存损坏类错误(parent snapshot not found / cache corrupt / failed to compute cache key)
  2. 自动清理+重试:检测到后删除本地缓存目录 + prune buildx builder内部状态,然后重试一次
  3. 最终降级:还是失败就完全不用本地缓存,只用registry缓存构建

非缓存类错误(代码问题等)直接透传,不误重试。

效果

  • 缓存损坏不再导致构建全挂,自动自愈
  • 不影响正常构建的速度(缓存好的时候跟之前一样快)
  • 只在真正遇到缓存损坏时多花一次构建时间
## 问题 Web构建连续多轮失败,错误:`parent snapshot sha256:xxx does not exist` 根因是runner本地 `/tmp/buildx-cache/` 目录里的buildx本地缓存(snapshot)损坏了,每次构建都从本地坏缓存读,直接炸。 API和Worker没挂是因为它们的缓存没坏,但Web那个缓存目录从#518那次开始坏了,之后所有Web构建连锁失败。 ## 修复 `scripts/ci/docker_build_push.sh` 新增 `build_with_cache_retry()` 函数: 1. **自动检测**:识别缓存损坏类错误(parent snapshot not found / cache corrupt / failed to compute cache key) 2. **自动清理+重试**:检测到后删除本地缓存目录 + prune buildx builder内部状态,然后重试一次 3. **最终降级**:还是失败就完全不用本地缓存,只用registry缓存构建 非缓存类错误(代码问题等)直接透传,不误重试。 ## 效果 - 缓存损坏不再导致构建全挂,自动自愈 - 不影响正常构建的速度(缓存好的时候跟之前一样快) - 只在真正遇到缓存损坏时多花一次构建时间
xiaoxia added 2 commits 2026-07-18 16:11:25 +08:00
fix(ci): Web构建改用layer缓存替代cache mount,避免node_modules缓存损坏导致构建全挂
CI Build & Deploy Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Staging API Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Production API Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Production Web Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI Build & Deploy Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI Build & Deploy Pipeline / Deploy Production (pull_request) Has been skipped
CI Build & Deploy Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 22s
CI/CD Pipeline / Frontend Lint (pull_request) Successful in 51s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 56s
CI/CD Pipeline / Validate Code Quality And Tests (pull_request) Successful in 2m51s
CI/CD Pipeline / Unit Tests (pull_request) Successful in 3m4s
Auto Approve CI PRs / Auto Approve on CI Green (pull_request) Successful in 3m33s
AI Code Review / AI Code Review (pull_request) Successful in 4m5s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m28s
Auto Merge CI PRs / Auto Merge on CI Green + Approved (pull_request) Successful in 4m49s
Preview Cleanup / Cleanup Preview Environment (pull_request) Failing after 1s
c9839b265c
- 移除node_modules的cache mount,改为写入镜像layer走ACR缓存
- 保留/root/.npm cache mount加速下载(不影响正确性)
- 保留.tscache cache mount加速TS增量编译(坏了不影响构建)
- layer缓存由ACR跨runner共享,内容完整性由layer哈希保证
fix(ci): buildx本地缓存损坏自动检测+清理重试,根治snapshot not found连锁失败
CI Build & Deploy Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Staging API Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Production Web Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Production API Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Build Production Worker Image (pull_request) Has been skipped
CI Build & Deploy Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI Build & Deploy Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI Build & Deploy Pipeline / Deploy Production (pull_request) Has been skipped
CI Build & Deploy Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 21s
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 1m13s
CI/CD Pipeline / Frontend Lint (pull_request) Successful in 1m15s
CI/CD Pipeline / Validate Code Quality And Tests (pull_request) Successful in 2m11s
CI/CD Pipeline / Unit Tests (pull_request) Successful in 3m19s
Auto Approve CI PRs / Auto Approve on CI Green (pull_request) Successful in 3m47s
CI/CD Pipeline / Integration Tests (pull_request) Successful in 1m36s
Auto Merge CI PRs / Auto Merge on CI Green + Approved (pull_request) Successful in 4m14s
Preview Cleanup / Cleanup Preview Environment (pull_request) Successful in 20s
AI Code Review / AI Code Review (pull_request) Successful in 7m58s
7ec43b3b28
- 新增build_with_cache_retry()函数:检测缓存损坏类错误自动清理本地缓存并重试
- 覆盖错误模式:parent snapshot not found、cache corrupt、failed to compute cache key
- 两级降级:本地缓存重试1次 → 完全不用本地缓存只用registry缓存
- 同时清理buildx builder内部snapshot状态(docker buildx prune -a)
- 非缓存类错误直接透传,不误重试

🚀 预览环境已部署

项目 详情
PR号 #523
预览链接 https://pr-523.preview.xiaoxiajianji.com
API环境 staging

💡 预览环境使用 staging API 数据,请勿在预览环境中操作重要数据。

🔄 每次提交新代码后预览环境会自动更新。

🗑️ PR 关闭或合并后,预览环境会自动清理。

🚀 **预览环境已部署** | 项目 | 详情 | |------|------| | PR号 | #523 | | 预览链接 | [https://pr-523.preview.xiaoxiajianji.com](https://pr-523.preview.xiaoxiajianji.com) | | API环境 | staging | > 💡 预览环境使用 staging API 数据,请勿在预览环境中操作重要数据。 > > 🔄 每次提交新代码后预览环境会自动更新。 > > 🗑️ PR 关闭或合并后,预览环境会自动清理。
auto-approve-bot approved these changes 2026-07-18 16:15:12 +08:00
auto-approve-bot left a comment
Collaborator

CI全绿,自动审批通过。

CI全绿,自动审批通过。
auto-approve-bot approved these changes 2026-07-18 16:15:12 +08:00
auto-approve-bot left a comment
Collaborator

CI全绿,自动审批通过。

CI全绿,自动审批通过。
auto-approve-bot merged commit 12b285c42a into develop 2026-07-18 16:15:40 +08:00
auto-approve-bot deleted branch fix/buildx-local-cache-corruption 2026-07-18 16:15:40 +08:00

🗑️ 预览环境已清理

PR #523 已关闭或合并,对应的预览环境已被清理。

如有需要,可以重新打开 PR 来重新生成预览环境。

🗑️ **预览环境已清理** PR #523 已关闭或合并,对应的预览环境已被清理。 > 如有需要,可以重新打开 PR 来重新生成预览环境。
Sign in to join this conversation.