Files
xiaoxia-saas/docs/RELEASE-AUTOMATION-RETROSPECTIVE-2026-06-22.md
T

93 lines
3.9 KiB
Markdown

# v0.1.9 Release Automation Retrospective
Date: 2026-06-22
Branch: `develop`
Production release: `v0.1.9`
## Result
`v0.1.9` is the first verified end-to-end automatic production release for 小虾 SaaS:
1. Gitea tag push starts production release workflow.
2. Dedicated `runtime-builder-01` builds API/Worker runtime images.
3. Dedicated builder builds Web `apps/web/dist` in a Node Docker container.
4. Dedicated builder uploads both artifacts to production.
5. Dedicated builder drives production deployment over SSH.
6. Production host only extracts artifacts, loads runtime images, runs migrations, restarts containers, and health checks.
7. Public Web/API smoke tests pass.
## Verified Production State
After `v0.1.9`:
- `xiaoxia-api-production` runs `xiaoxia-saas-api:v0.1.9` and is healthy.
- `xiaoxia-worker-production` runs `xiaoxia-saas-worker:v0.1.9`.
- `xiaoxia-web-production` is healthy and serves prebuilt Web dist.
- `/var/lib/xiaoxia-saas-production/runtime-images-v0.1.9.tar` is present, about 325 MB.
- `/var/lib/xiaoxia-saas-production/release-v0.1.9.tar.gz` is present, about 883 KB.
Public smoke results:
```text
python scripts\smoke_public_auth_flow.py
public_auth_flow=ok
python scripts\smoke_public_upload_flow.py
public_upload_flow=ok
public project detail smoke
public_project_get=ok
```
## What Repeated
The repeated tags were not feature releases; they were release-chain verification tags.
- `v0.1.6`: runtime-builder checkout/token fix worked, but Gitea skipped downstream production deploy after cross-runner `needs`.
- `v0.1.7`: production deploy moved to `runtime-builder` over SSH; build/upload succeeded, but deploy failed because `deploy-production.sh` defaulted to `/host` paths while SSH uses host paths directly.
- `v0.1.8`: SSH deploy used `HOST_PREFIX=`, but release artifact packaging used broad `--exclude=dist`, which removed `apps/web/dist` and correctly failed the Web artifact gate.
- `v0.1.9`: artifact packaging changed to `--exclude=./dist`, preserving `apps/web/dist`; automatic release succeeded.
Usable rollback tags:
- `v0.1.5`: previously verified stable production release.
- `v0.1.9`: current verified automatic production release.
Do not use `v0.1.6`, `v0.1.7`, or `v0.1.8` as rollback targets unless intentionally reproducing release-chain failures.
## Root Causes Fixed
1. `GITHUB_TOKEN` was not implicitly available in act_runner shell scripts.
- Fix: checkout steps explicitly bind `GITHUB_TOKEN: ${{ github.token }}`.
2. Gitea `needs` with production deploy on a different runner path skipped the downstream deploy job.
- Fix: production deploy also runs on `runtime-builder`, then SSHes to production.
3. Production deploy script defaults to `/host` for Docker-in-Docker CI execution.
- Fix: SSH deploy passes `HOST_PREFIX=` so the script uses normal host paths.
4. Release source artifact accidentally included build output tarballs.
- Fix: exclude root `./dist` from release tar.
5. A broad `--exclude=dist` also excluded Web dist.
- Fix: use `--exclude=./dist`, not `--exclude=dist`.
## Current Release Contract
For production `v*` tag releases:
- Production host must not build API or Worker images.
- Production host must not build frontend assets.
- `runtime-builder-01` is the build and deployment driver.
- `runtime-images-<tag>.tar` must exist before production deploy.
- `release-<tag>.tar.gz` must include `apps/web/dist/index.html`.
- Production deploy must fail if either artifact is missing.
- Production smoke is required before saying release is done.
## Follow-Ups
- Add an explicit dry-run workflow or staging release rehearsal to reduce tag-based trial and error.
- Keep `v0.1.6` to `v0.1.8` documented as failed release-chain probes.
- Consider adding an API version endpoint that reports the Docker image tag; current `/health` version text can lag image tags.
- Review temporary credential posture for builder SSH key and old registration tokens.