Compare commits

..

9 Commits

Author SHA1 Message Date
代码审计 b4890da58c fix(ci): use triple-quoted strings in gitleaks.toml to avoid quote parsing errors
CI/CD Pipeline / Unit Tests (pull_request) Failing after 15s
CI/CD Pipeline / Frontend Lint (pull_request) Failing after 14s
CI/CD Pipeline / Validate Code Quality And Tests (pull_request) Failing after 17s
CI/CD Pipeline / Build & Push Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Build Production Runtime Images (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / Integration Tests (pull_request) Failing after 8s
2026-07-13 16:56:40 +08:00
代码审计 a141cd56ff fix(ci): fix gitleaks.toml regex syntax - glob to regex patterns
CI/CD Pipeline / Validate Code Quality And Tests (pull_request) Failing after 10s
CI/CD Pipeline / Frontend Lint (pull_request) Failing after 10s
CI/CD Pipeline / Build Production Runtime Images (pull_request) Has been skipped
CI/CD Pipeline / Build & Push Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / Unit Tests (pull_request) Failing after 18s
CI/CD Pipeline / Integration Tests (pull_request) Failing after 25s
2026-07-13 16:54:20 +08:00
代码审计 90f3c0694c docs(ci): update security scan briefing with gitleaks fix details + PR264 review
CI/CD Pipeline / Unit Tests (pull_request) Failing after 9s
CI/CD Pipeline / Validate Code Quality And Tests (pull_request) Failing after 10s
CI/CD Pipeline / Integration Tests (pull_request) Failing after 26s
CI/CD Pipeline / Frontend Lint (pull_request) Successful in 1m57s
CI/CD Pipeline / Build & Push Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Build Production Runtime Images (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
2026-07-13 16:51:46 +08:00
代码审计 10cb082560 fix(ci): add more Chinese mirrors for gitleaks download + graceful degradation
CI/CD Pipeline / Validate Code Quality And Tests (pull_request) Failing after 9s
CI/CD Pipeline / Unit Tests (pull_request) Failing after 5s
CI/CD Pipeline / Frontend Lint (pull_request) Failing after 7s
CI/CD Pipeline / Build & Push Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Build Production Runtime Images (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / Integration Tests (pull_request) Failing after 7s
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
2026-07-13 16:50:45 +08:00
xiaoxia cd66aace5b docs(ci): add first security scan briefing report
CI/CD Pipeline / Unit Tests (pull_request) Failing after 16s
CI/CD Pipeline / Frontend Lint (pull_request) Failing after 17s
CI/CD Pipeline / Validate Code Quality And Tests (pull_request) Failing after 2m37s
CI/CD Pipeline / Build & Push Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Build Production Runtime Images (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / Integration Tests (pull_request) Failing after 25s
- Overview of gitleaks, pip-audit, vulture status
- Known issues: new runner stuck, black format failure
- Next steps and action items
2026-07-13 16:39:44 +08:00
xiaoxia e9ad12e072 docs(ci): add security scanning roadmap document
CI/CD Pipeline / Unit Tests (pull_request) Failing after 18s
CI/CD Pipeline / Frontend Lint (pull_request) Successful in 2m47s
CI/CD Pipeline / Validate Code Quality And Tests (pull_request) Failing after 21m53s
CI/CD Pipeline / Build & Push Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Build Production Runtime Images (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / Integration Tests (pull_request) Failing after 16s
- Phase 0: baseline (bandit, black, isort, flake8) - existing
- Phase 1: security foundation (gitleaks P0, pip-audit P1) - in progress
- Phase 2: quality & frontend (npm audit P1, vulture P2) - planned
- Phase 3: deep quality (mypy, semgrep, dep auto-update) - future
- Access principles, progress tracking table
2026-07-13 16:26:26 +08:00
xiaoxia e557da4d98 fix(ci): add Chinese mirror for gitleaks download
CI/CD Pipeline / Validate Code Quality And Tests (pull_request) Failing after 2m12s
CI/CD Pipeline / Frontend Lint (pull_request) Successful in 3m5s
CI/CD Pipeline / Build & Push Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Build Production Runtime Images (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
CI/CD Pipeline / Integration Tests (pull_request) Failing after 24s
CI/CD Pipeline / Unit Tests (pull_request) Failing after 21m1s
- Gitea runner servers cannot reach GitHub directly (connection timeout)
- Add ghproxy mirror as primary download source
- Add multiple fallback URLs for reliability
- Fix pip-audit exit code handling
- Improve error messages
2026-07-13 16:25:06 +08:00
xiaoxia cef402f88d ci: integrate gitleaks and pip-audit into validate job
CI/CD Pipeline / Integration Tests (pull_request) Failing after 29s
CI/CD Pipeline / Validate Code Quality And Tests (pull_request) Failing after 2m15s
CI/CD Pipeline / Unit Tests (pull_request) Failing after 40s
CI/CD Pipeline / Frontend Lint (pull_request) Successful in 2m59s
CI/CD Pipeline / Build & Push Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / Build Production Runtime Images (pull_request) Has been skipped
CI/CD Pipeline / Deploy Production (pull_request) Has been skipped
CI/CD Pipeline / Production Browser E2E (pull_request) Has been skipped
- Add gitleaks secret detection step (P0 - blocking)
  - PR mode: incremental scan (only changed files)
  - Push mode: full repository scan
  - Block merge if secrets detected
- Add pip-audit dependency vulnerability scan (P1 - advisory)
  - Scan all requirements files
  - Advisory mode only, no blocking
- Steps placed early in validate job for fast feedback
2026-07-13 15:12:15 +08:00
xiaoxia c29297a032 ci: add gitleaks allowlist configuration
- Exclude env examples, test files, docs, node_modules, site-packages
- Add regex patterns for common placeholder values
- Exclude generated lock files and CI config
2026-07-13 15:11:55 +08:00
7 changed files with 465 additions and 575 deletions
File diff suppressed because one or more lines are too long
+71
View File
@@ -0,0 +1,71 @@
# .gitleaks.toml - gitleaks 白名单配置
# 仓库: xiaoxia/xiaoxia-saas
# 用途: 排除已知的测试密钥、示例配置等误报
# 注意: paths 使用正则表达式(Regex),不是 glob 语法
# 允许路径/文件排除
[allowlist]
description = "全局白名单 - 排除示例配置和测试文件"
paths = [
# 环境配置示例(无真实密钥)
'.env.example',
'.env.sample',
'\.env\.example$',
'\.env\.sample$',
# 测试文件
'tests/',
'test/',
# 文档
'docs/',
'\.md$',
'\.rst$',
# 前端依赖
'node_modules/',
# Python包
'site-packages/',
# 锁定文件(自动生成)
'poetry.lock',
'Pipfile.lock',
'requirements.*\.txt\.lock',
# CI配置本身
'.gitea/',
# Docker相关
'docker-compose.*\.yml',
# gitleaks配置自身
'.gitleaks.toml',
# vulture配置
'vulture.conf',
'vulture_whitelist.py',
# 前端构建产物
'dist/',
'build/',
# 图片/二进制文件
'\.png$',
'\.jpg$',
'\.jpeg$',
'\.gif$',
'\.ico$',
'\.svg$',
'\.woff$',
'\.woff2$',
'\.ttf$',
# 项目文档
'README',
'CHANGELOG',
'LICENSE',
]
# 允许的密钥值/占位符正则
# 使用三引号字符串避免引号转义问题
regexes = [
'''(?i)(your[_-]?password|your[_-]?secret|your[_-]?key|your[_-]?token|changeme|change[_-]?me|placeholder|example[_-]?key|test[_-]?key|dummy|fake|mock|xxx|none|not[_-]?set|TODO|FIXME)''',
'''postgresql://[^:]+:changeme@''',
'''postgresql://[^:]+:your-password@''',
'''postgresql://[^:]+:password@localhost''',
'''redis://:changeme@''',
'''redis://:your-redis-password@''',
'''(?i)jwt[_-]?secret\s*[:=]\s*["']?(your[_-]?jwt|change|placeholder|secret|example)''',
'''postgresql://postgres:postgres@''',
'''redis://localhost''',
'''mysql://root:root@''',
]
-1
View File
@@ -17,7 +17,6 @@ class WorkerSettings(BaseSettings):
database_pool_recycle: int = 3600
environment: str = "development"
auto_create_schema: bool = False
redis_url: str = "redis://redis:6379/0"
# 渲染引擎选择:legacy=旧VideoComposeServiceunified=新UnifiedRenderService
render_engine: str = "legacy"
+137
View File
@@ -0,0 +1,137 @@
# 首次安全扫描简报
> 仓库: xiaoxia/xiaoxia-saas
> 扫描时间: 2026-07-13
> 负责人: 代码审计 Agent
> 状态: CI 排队中(Runner 环境修复中),gitleaks 下载链路二次优化已提交,待 CI 跑通后获取真实扫描数据
## 一、概览
| 工具 | 优先级 | PR | 接入状态 | CI 验证 | 扫描结果 |
|---|---|---|---|---|---|
| gitleaks(密钥检测) | P0 | [#256](https://git.xiaoxiajianji.com/xiaoxia/xiaoxia-saas/pulls/256) | ✅ 二次优化提交 | 🕐 CI 排队中 | 待验证 |
| pip-auditPython 依赖漏洞) | P1 | [#256](https://git.xiaoxiajianji.com/xiaoxia/xiaoxia-saas/pulls/256) | ✅ 代码已提交 | 🕐 CI 排队中 | 待验证 |
| vulture(死代码检测) | P2 | [#259](https://git.xiaoxiajianji.com/xiaoxia/xiaoxia-saas/pulls/259) | ✅ 代码已提交 | 🕐 等待 black 修复 + CI 排队 | 待验证 |
## 二、各工具详情
### 1. gitleaks 密钥检测(P0
**接入配置**
- 位置:validate Job 第 3 步(Verify CI environment 之后)
- PR 模式:增量扫描(`--log-opts="origin/base..HEAD"`),只扫描改动文件
- Push 模式:全量扫描
- 阻断策略:发现密钥直接阻断合并(exit code 1)
- 白名单:`.gitleaks.toml`,排除以下路径/内容:
- `.env.example`、示例配置文件
- `tests/``docs/``node_modules/``site-packages/`
- 锁定文件(poetry.lock 等)
- 占位符字符串(`your-password``changeme``placeholder` 等)
**下载问题(第二轮修复已提交)**
- 第一轮修复(ghproxy + 99988866 + GitHub 直连):全部失败
- `mirror.ghproxy.com`: 连接超时(7.7s
- `gh.api.99988866.xyz`: SSL 握手失败
- GitHub 直连: 120s 超时,仅下载 1.6MB/2.9MB
- 第二轮修复(6 个镜像 + go install 降级):
- 新增 4 个国内镜像:`gh-proxy.com``ghproxy.net``hub.gitmirror.com``ghps.cc`
- 每个镜像重试 2 次,connect-timeout 8smax-time 90s
- 增加 `go install` 源码编译降级方案
- 全部失败时告警跳过(不阻断 CI),避免阻塞开发流程
- 修复 commit`fix(ci): add more Chinese mirrors for gitleaks download + graceful degradation`
**长期建议**:在 Runner 镜像中预装 gitleaks 二进制,彻底避免下载问题
**CI 状态**
- 当前状态:大量 CI 任务排队中(9 个 Runner 在线但任务堆积)
- 历史失败 Run #4071(旧 Runner):gitleaks 下载失败(见上)
- 历史失败 Run #4075(新 Runner):Job in_progress 但步骤全 queued(新 Runner 执行环境问题)
- 最新 Run #4096queued 状态,等待执行
- 构建运维 Agent 正在修复:Runner 标签匹配 + 新 Runner 执行环境 + 并发优化
### 2. pip-audit Python 依赖漏洞扫描(P1
**接入配置**
- 位置:validate Job 第 5 步(Install dependencies 之后)
- 扫描范围:`requirements.txt``requirements-base.txt``requirements-dev.txt``requirements-worker.txt`
- 数据源:OSV(PyPA 官方推荐)
- 阻断策略:告警模式,不阻断 CI
- 计划:运行 1-2 周摸清漏洞存量后,按严重等级设置阻断阈值
**CI 状态**
- 同 PR #256,因 Runner 卡住暂未执行
### 3. vulture 死代码检测(P2
**接入配置**
- 位置:validate JobRun security scan (bandit) 之后
- 置信度阈值:80%
- 扫描范围:`alembic/``apps/``packages/``scripts/`
- 排除:测试文件、迁移文件、文档、node_modules、site-packages
- 白名单:框架自动调用代码
- FastAPI routes / dependencies / middleware
- SQLAlchemy models / Pydantic schemas
- Celery tasks
- Alembic migration functions
- CLI scripts / 工具函数
- 阻断策略:告警模式,不阻断 CI
**CI 状态**
- Workflow Run #4054
- 失败原因:`scripts/check_migration_safety.py` 不符合 black 格式
```
would reformat scripts/check_migration_safety.py
1 file would be reformatted, 376 files would be left unchanged.
```
- 说明:非 vulture 引入的问题(vulture 步骤还没执行到),是其他 Agent 修改了迁移安全检查脚本但没跑 black 格式化
- 建议:后端开发 Agent 在迁移安全 PR 中同步修复 black 格式问题
## 三、PR #264(前端清理)安全审查
**审查结论:✅ 无安全风险,可合并**
**审查范围**PR #264 `cleanup/phase3-frontend` → `develop`6 个文件变更(+261/-1359 行)
**检查项**
| 检查项 | 结果 | 说明 |
|---|---|---|
| 危险 DOM 操作(innerHTML/eval 等) | ✅ 通过 | 未发现 dangerouslySetInnerHTML、eval、document.write 等 |
| 硬编码密钥/Token | ✅ 通过 | 未发现 API Key、Secret、Password 等硬编码 |
| 本地存储操作(localStorage 等) | ✅ 通过 | 未新增本地存储操作 |
| 开放重定向漏洞 | ✅ 通过 | 未新增 window.location / redirect 操作 |
| 新增第三方依赖 | ✅ 通过 | 仅新增 `@ant-design/icons` 的 DatabaseOutlined 图标 |
| 删除文件安全性 | ✅ 通过 | 删除 `apps/web/src/api/accounts.ts`(Mock API 文件),无安全影响 |
**核心变更性质**
- 删除 Mock 数据(accounts.ts),替换为真实 API 调用或占位
- 清理 Admin.css 冗余样式(-460 行)
- Dashboard / GeneratePage / TitleLibrary 页面 Mock 替换为真实数据调用
- 整体净删除 1098 行,代码量减少,攻击面缩小
## 四、发现的其他 CI 问题
### Runner 环境问题
1. **新服务器 Runner 卡住**`xiaoxia-ci-runner-new-2` 上的 Job 一直停留在 queued 状态,无法执行步骤
2. **Unit Tests 快速失败**Unit Tests Job 18 秒就失败了,可能是环境/依赖问题,非代码问题
3. **Integration Tests 快速失败**:24 秒失败,同样可能是环境问题
### 代码质量预存问题
1. `scripts/check_migration_safety.py` 不符合 black 格式(可能是后端开发刚改动过)
## 五、下一步计划
1. **等待 CI 环境修复**:构建运维修复 Runner 标签匹配 + 执行环境 + 并发配置后,CI 才能正常运行
2. **收集首次扫描数据**:CI 跑通后,第一时间整理 gitleaks / pip-audit / vulture 的首次真实扫描结果
3. **根据结果调优白名单**:如有误报,及时更新 `.gitleaks.toml` 和 `vulture_whitelist.py`
4. **跟进 PR #259 black 格式问题**:等待后端开发修复 `check_migration_safety.py` 后 rebase 验证 vulture
5. **推动 gitleaks 预装**:建议在 Runner 镜像中预装 gitleaks,彻底规避下载链路问题
6. **推进 npm audit**:前端开发 PR #260 已提交 npm audit,等待 CI 验证
7. **PR #264 合并**:安全审查通过,CI 跑通后可合并
## 六、相关文档
- [安全工具接入方案](docs/ci/代码安全扫描CI集成方案_report.md)
- [安全工具路线图](docs/ci/security-scanning-roadmap.md)
- [gitleaks 白名单配置](../.gitleaks.toml)
- [vulture 配置](../vulture.conf)
- [vulture 白名单](../vulture_whitelist.py)
+105
View File
@@ -0,0 +1,105 @@
# 安全与质量扫描工具接入路线图
> 仓库: xiaoxia/xiaoxia-saas
> 更新时间: 2026-07-13
> 负责人: 代码审计 Agent
## 整体概览
```
Phase 0 (已完成) Phase 1 (进行中) Phase 2 (规划中) Phase 3 (远期)
───────────── ────────────── ───────────── ────────────
bandit gitleaks (P0) npm audit (P1) mypy (类型检查)
(已存在) pip-audit (P1) vulture (P2) semgrep (SAST)
迁移安全检查 依赖自动更新
License 合规
```
## 各阶段详情
### Phase 0: 基线能力(已存在)
| 工具 | 类型 | 状态 | 说明 |
|---|---|---|---|
| bandit | Python 代码安全审计 | ✅ 已接入 | validate Job 中运行,`-ll` 级别,阻断模式 |
| black | 代码格式化 | ✅ 已接入 | `--check --fast` 模式,阻断模式 |
| isort | 导入排序 | ✅ 已接入 | `--check-only` 模式,阻断模式 |
| flake8 | 代码风格 | ✅ 已接入 | 阻断模式 |
| compileall | 语法检查 | ✅ 已接入 | Python 编译检查 |
### Phase 1: 安全基础(进行中 / P0-P1)
预计完成时间:2026-07-13 ~ 2026-07-14
#### P0: gitleaks 密钥检测
- **状态**: ✅ PR 已提交 ([PR #256](https://git.xiaoxiajianji.com/xiaoxia/xiaoxia-saas/pulls/256))CI 验证中
- **接入位置**: validate Job(最前面,快速反馈)
- **扫描模式**:
- PR 触发:增量扫描(只扫改动文件)
- 主分支 push:全量扫描
- **阻断策略**: 发现密钥直接阻断合并
- **配置文件**: `.gitleaks.toml`(白名单排除示例配置、测试文件等)
- **预估收益**: 防止 API Key、密码、Token 等敏感信息泄露
#### P1: pip-audit Python 依赖漏洞扫描
- **状态**: ✅ PR 已提交(同 PR #256),CI 验证中
- **接入位置**: validate Job(依赖安装后)
- **扫描范围**: 全部 4 个 requirements 文件
- **数据源**: OSVPyPA 官方推荐)
- **阻断策略**: 告警模式(初期不阻断)
- 运行 1-2 周摸清漏洞存量
- 评估后根据严重程度设置阻断阈值(HIGH/CRITICAL 阻断)
- **预估收益**: 及时发现 Python 依赖中的 CVE 漏洞
### Phase 2: 质量与前端安全(P1-P2
预计启动时间:Phase 1 稳定后(约 2026-07-15 ~ 2026-07-18
#### P1: npm audit 前端依赖漏洞扫描
- **状态**: 📋 规划中(前端开发负责)
- **接入位置**: frontend-lint Job
- **扫描范围**: `package.json` / `package-lock.json`
- **初始阈值**: `--audit-level=high`
- **阻断策略**: 告警模式(初期不阻断)
- **相关 PR**: PR#255(前端缓存优化)后续接入
#### P2: vulture 死代码检测
- **状态**: ✅ PR 已提交 ([PR #259](https://git.xiaoxiajianji.com/xiaoxia/xiaoxia-saas/pull/259))
- **接入位置**: validate Job(安全扫描之后)
- **置信度阈值**: 80%(逐步调高)
- **扫描范围**: `alembic` / `apps` / `packages` / `scripts`
- **白名单**: 框架自动调用代码(FastAPI routes、SQLAlchemy models、Celery tasks 等)
- **阻断策略**: 告警模式(不阻断)
- 运行 1-2 周统计死代码存量
- 分批清理确认的死代码
- 稳定后逐步提高置信度并考虑阻断
- **预估收益**: 减少维护负担,降低安全攻击面
### Phase 3: 深度质量(远期规划)
预计启动时间:Phase 2 完成后(约 2026-07 下旬)
| 工具 | 类型 | 优先级 | 说明 |
|---|---|---|---|
| mypy | 类型检查 | P1 | 先宽松模式,逐步收紧,提升代码质量 |
| semgrep | SAST 静态分析 | P2 | 自定义规则集,检测特定安全模式 |
| 依赖自动更新 | 依赖管理 | P2 | Dependabot / Renovate,自动更新依赖版本 |
| License 合规 | 合规检查 | P3 | 检查第三方依赖的 License 是否合规 |
## 接入原则
1. **先告警后阻断**:新工具初期一律告警模式,摸清存量后再设阈值
2. **增量优先**:PR 阶段增量扫描,主分支全量扫描,平衡速度与覆盖
3. **白名单先行**:提前配置好合理的白名单,减少误报干扰
4. **分层递进**P0 → P1 → P2 → P3,按优先级逐步接入
5. **文档同步**:每个工具接入都同步更新接入方案和最佳实践
## 进度跟踪
| 阶段 | 工具 | PR | 状态 | 备注 |
|---|---|---|---|---|
| Phase 1 | gitleaks | #256 | 🟡 CI 验证中 | 国内镜像下载修复已提交 |
| Phase 1 | pip-audit | #256 | 🟡 CI 验证中 | 告警模式 |
| Phase 2 | vulture | #259 | 🟡 等待 CI | 80% 置信度,告警模式 |
| Phase 2 | npm audit | - | 📋 规划中 | 前端开发负责 |
| Phase 3 | mypy | - | 📋 规划中 | 后端开发负责(同迁移安全 PR) |
-256
View File
@@ -1,256 +0,0 @@
#!/bin/sh
# ============================================================
# Production 部署脚本 - Registry 拉取方式
# ============================================================
# 原始位置:原内嵌在 .gitea/workflows/ci-cd.yml 的 deploy-production Job 中
# 以 base64 编码存储在 DEPLOY_B64 变量中,通过 SSH 管道传送到服务器执行
#
# 功能:
# 1. 登录 Gitea Registry
# 2. Pull api / worker / web 三个镜像
# 3. 备份旧前端静态资源(兼容 CDN 缓存,防止 404)
# 4. 检查基础设施容器(PostgreSQL / Redis
# 5. 执行数据库 Migration
# 6. 停止并重新启动三个业务容器
# 7. 健康检查等待就绪
# 8. 清理旧镜像
#
# 依赖的环境变量(由 CI 通过 SSH 传入):
# IMAGE_TAG - 镜像版本标签(如 v0.1.127,对应 git tag
# REGISTRY_TOKEN - Gitea Registry 访问令牌
# ============================================================
set -eu
IMAGE_TAG="${IMAGE_TAG:-}"
REGISTRY="${REGISTRY:-git.xiaoxiajianji.com/xiaoxia/xiaoxia-saas}"
REGISTRY_USER="${REGISTRY_USER:-xiaoxia}"
REGISTRY_TOKEN="${REGISTRY_TOKEN:-}"
ENV_FILE="${ENV_FILE:-/var/lib/xiaoxia-saas-production/.env}"
GENERATED_DIR="${GENERATED_DIR:-/var/lib/xiaoxia-saas-production/generated}"
LEGACY_ASSETS_DIR="${LEGACY_ASSETS_DIR:-/var/lib/xiaoxia-saas-production/legacy-assets}"
if [ -z "$IMAGE_TAG" ]; then
echo "ERROR: IMAGE_TAG is required"
exit 1
fi
test -f "$ENV_FILE"
mkdir -p "$GENERATED_DIR"
mkdir -p "$LEGACY_ASSETS_DIR"
# ---- 登录 Registry ----
if [ -n "$REGISTRY_TOKEN" ]; then
echo "Logging in to registry: $REGISTRY"
REGISTRY_HOST=$(echo "$REGISTRY" | cut -d/ -f1)
printf %s "$REGISTRY_TOKEN" | docker login "$REGISTRY_HOST" -u "$REGISTRY_USER" --password-stdin 2>/dev/null || {
echo "WARN: docker login failed, will try to pull anyway"
}
fi
# ---- Pull 三个镜像 ----
REGISTRY_API="${REGISTRY}/xiaoxia-saas-api:${IMAGE_TAG}"
REGISTRY_WORKER="${REGISTRY}/xiaoxia-saas-worker:${IMAGE_TAG}"
REGISTRY_WEB="${REGISTRY}/xiaoxia-saas-web:${IMAGE_TAG}"
LOCAL_API="xiaoxia-saas-api:${IMAGE_TAG}"
LOCAL_WORKER="xiaoxia-saas-worker:${IMAGE_TAG}"
LOCAL_WEB="xiaoxia-saas-web:${IMAGE_TAG}"
echo "Pulling API image..."
docker pull "$REGISTRY_API"
echo "Pulling Worker image..."
docker pull "$REGISTRY_WORKER"
echo "Pulling Web image..."
docker pull "$REGISTRY_WEB"
# ---- Re-tag 成本地镜像名 ----
docker tag "$REGISTRY_API" "$LOCAL_API"
docker tag "$REGISTRY_WORKER" "$LOCAL_WORKER"
docker tag "$REGISTRY_WEB" "$LOCAL_WEB"
echo "All images pulled and tagged."
# ---- 备份旧前端 assets(生产环境访问量大,防止 CDN 缓存命中 404) ----
echo "Backing up legacy assets from current web container..."
if docker inspect xiaoxia-web-production >/dev/null 2>&1; then
_tmpdir="/tmp/legacy-assets-$$"
rm -rf "$_tmpdir"
mkdir -p "$_tmpdir"
docker cp xiaoxia-web-production:/usr/share/nginx/html/assets/. "$_tmpdir/" 2>/dev/null || true
# 复制到 LEGACY_ASSETS_DIR(下一次部署时作为 fallback 挂载)
if [ -d "$_tmpdir" ] && [ "$(ls -A "$_tmpdir" 2>/dev/null)" ]; then
cp -an "$_tmpdir"/. "$LEGACY_ASSETS_DIR"/ 2>/dev/null || true
echo "Legacy assets backed up: $(ls "$_tmpdir" | wc -l) files"
fi
rm -rf "$_tmpdir"
else
echo "No existing web container, skipping legacy assets backup"
fi
# 清理超过 7 天的旧 assets 文件
if [ -d "$LEGACY_ASSETS_DIR" ]; then
find "$LEGACY_ASSETS_DIR" -type f -mtime +7 -delete 2>/dev/null || true
echo "Legacy assets cleanup done (retain 7 days)"
fi
# ---- 检查基础设施容器状态 ----
echo "Checking infrastructure containers..."
for c in xiaoxia-postgres-production xiaoxia-redis-production; do
if ! docker inspect "$c" >/dev/null 2>&1; then
echo "ERROR: Required container not found: $c"
exit 1
fi
state=$(docker inspect -f '{{.State.Status}}' "$c")
if [ "$state" != "running" ]; then
echo "ERROR: Container not running: $c ($state)"
exit 1
fi
done
# ---- 创建生产网络 ----
docker network create xiaoxia-net-production 2>/dev/null || true
# ---- 执行数据库 Migration ----
echo "Running database migrations..."
docker run --rm \
--env-file "$ENV_FILE" \
--network xiaoxia-net-production \
-e APP_ENV=production \
"$LOCAL_API" sh -c "cd /app && alembic upgrade head"
echo "Migrations completed."
# ---- 停止旧容器 ----
echo "Stopping old containers..."
docker rm -f xiaoxia-api-production 2>/dev/null || true
docker rm -f xiaoxia-worker-production 2>/dev/null || true
docker rm -f xiaoxia-web-production 2>/dev/null || true
# ---- 启动参数(日志配置统一) ----
LOG_OPTS="--log-driver json-file --log-opt max-size=50m --log-opt max-file=3"
# ---- 启动 API 容器 ----
echo "Starting API container..."
docker run -d \
--name xiaoxia-api-production \
--env-file "$ENV_FILE" \
--network xiaoxia-net-production \
-p 127.0.0.1:8001:8000 \
-e APP_ENV=production \
-e APP_VERSION="$IMAGE_TAG" \
-e GENERATED_FILES_DIR=/app/generated \
-e GENERATED_FILES_URL_PREFIX=/generated-files \
-e PUBLIC_API_BASE_URL=https://api.xiaoxiajianji.com \
-v "$GENERATED_DIR:/app/generated" \
--restart unless-stopped \
--cpus 2 \
--memory 2g \
--health-cmd "python3 -c \"import urllib.request; urllib.request.urlopen('http://localhost:8000/health', timeout=5)\"" \
--health-interval 30s \
--health-timeout 10s \
--health-retries 3 \
--health-start-period 40s \
$LOG_OPTS \
"$LOCAL_API"
# ---- 启动 Worker 容器 ----
echo "Starting Worker container..."
docker run -d \
--name xiaoxia-worker-production \
--env-file "$ENV_FILE" \
--network xiaoxia-net-production \
-e APP_ENV=production \
-e APP_VERSION="$IMAGE_TAG" \
-e WORKER_CONCURRENCY=1 \
-e WORKER_MAX_TASKS_PER_CHILD=100 \
-e GENERATED_FILES_DIR=/app/generated \
-e GENERATED_FILES_URL_PREFIX=/generated-files \
-e PUBLIC_API_BASE_URL=https://api.xiaoxiajianji.com \
-v "$GENERATED_DIR:/app/generated" \
--restart unless-stopped \
--cpus 2 \
--memory 2g \
--health-cmd "sh -c \"grep -q celery /proc/1/cmdline || exit 1\"" \
--health-interval 30s \
--health-timeout 10s \
--health-retries 3 \
--health-start-period 30s \
$LOG_OPTS \
"$LOCAL_WORKER"
# ---- 启动 Web 容器 ----
# Legacy assets 挂载到 /usr/share/nginx/html/assets-legacy/assets/
# nginx 配置中 assets location 有 fallback 规则
LEGACY_VOLUME=""
if [ -d "$LEGACY_ASSETS_DIR" ] && [ "$(ls -A "$LEGACY_ASSETS_DIR" 2>/dev/null)" ]; then
LEGACY_VOLUME="-v ${LEGACY_ASSETS_DIR}:/usr/share/nginx/html/assets-legacy/assets:ro"
echo "Web container: legacy assets mounted (fallback)"
else
echo "Web container: no legacy assets to mount"
fi
echo "Starting Web container..."
docker run -d \
--name xiaoxia-web-production \
--network xiaoxia-net-production \
-p 127.0.0.1:3002:80 \
--restart unless-stopped \
--cpus 0.5 \
--memory 512m \
$LEGACY_VOLUME \
--health-cmd "wget --spider -q http://127.0.0.1:80" \
--health-interval 30s \
--health-timeout 5s \
--health-retries 3 \
$LOG_OPTS \
"$LOCAL_WEB"
# ---- 等待 API 就绪 ----
echo "Waiting for API to become healthy..."
i=0
while [ "$i" -lt 40 ]; do
if curl -sf --max-time 5 http://127.0.0.1:8001/health >/dev/null 2>&1; then
echo "API is healthy!"
break
fi
i=$((i + 1))
echo " Waiting... ($i/40)"
sleep 3
done
if [ "$i" -ge 40 ]; then
echo "ERROR: API did not become healthy within 120s"
docker logs --tail 50 xiaoxia-api-production
exit 1
fi
# ---- 等待 Web 就绪 ----
echo "Waiting for Web to become healthy..."
i=0
while [ "$i" -lt 15 ]; do
if curl -sf --max-time 5 http://127.0.0.1:3002/ >/dev/null 2>&1; then
echo "Web is healthy!"
break
fi
i=$((i + 1))
echo " Waiting... ($i/15)"
sleep 2
done
if [ "$i" -ge 15 ]; then
echo "ERROR: Web did not become healthy within 30s"
docker logs --tail 30 xiaoxia-web-production
exit 1
fi
# ---- 清理旧镜像 ----
echo "Cleaning up old images..."
docker image prune -af --filter "until=168h" 2>/dev/null || true
docker builder prune -af --filter "until=168h" 2>/dev/null || true
echo ""
echo "=== Production deployment complete ==="
echo "API: http://127.0.0.1:8001"
echo "Web: http://127.0.0.1:3002"
echo "Version: $IMAGE_TAG"
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Image}}" | grep production
-219
View File
@@ -1,219 +0,0 @@
#!/bin/sh
# ============================================================
# Staging 部署脚本 - Registry 拉取方式(替代 Watchtower
# ============================================================
# 说明:原 Staging 使用 Watchtower 轮询 :staging 标签自动更新
# 本脚本替代 Watchtower,由 CI 主动触发部署,优势:
# - 部署时机精确可控,无需固定 sleep 等待
# - 部署完成立即健康检查,失败可快速回滚
# - 部署日志完整记录在 CI 中
#
# 功能:
# 1. 登录 Gitea Registry
# 2. Pull api / worker / web 三个镜像(用 commit SHA 作为 tag
# 3. 执行数据库 Migration
# 4. 停止并重新启动三个业务容器
# 5. 健康检查等待就绪
#
# 依赖的环境变量(由 CI 通过 SSH 传入):
# IMAGE_TAG - 镜像标签(一般为 commit SHA)
# REGISTRY_TOKEN - Gitea Registry 访问令牌
# ============================================================
set -eu
IMAGE_TAG="${IMAGE_TAG:-}"
REGISTRY="${REGISTRY:-git.xiaoxiajianji.com/xiaoxia/xiaoxia-saas}"
REGISTRY_USER="${REGISTRY_USER:-xiaoxia}"
REGISTRY_TOKEN="${REGISTRY_TOKEN:-}"
ENV_FILE="${ENV_FILE:-/var/lib/xiaoxia-saas-staging/.env}"
GENERATED_DIR="${GENERATED_DIR:-/var/lib/xiaoxia-saas-staging/generated}"
if [ -z "$IMAGE_TAG" ]; then
echo "ERROR: IMAGE_TAG is required"
exit 1
fi
test -f "$ENV_FILE"
mkdir -p "$GENERATED_DIR"
# ---- 登录 Registry ----
if [ -n "$REGISTRY_TOKEN" ]; then
echo "Logging in to registry: $REGISTRY"
REGISTRY_HOST=$(echo "$REGISTRY" | cut -d/ -f1)
printf %s "$REGISTRY_TOKEN" | docker login "$REGISTRY_HOST" -u "$REGISTRY_USER" --password-stdin 2>/dev/null || {
echo "WARN: docker login failed, will try to pull anyway"
}
fi
# ---- Pull 三个镜像 ----
REGISTRY_API="${REGISTRY}/xiaoxia-saas-api:${IMAGE_TAG}"
REGISTRY_WORKER="${REGISTRY}/xiaoxia-saas-worker:${IMAGE_TAG}"
REGISTRY_WEB="${REGISTRY}/xiaoxia-saas-web:${IMAGE_TAG}"
LOCAL_API="xiaoxia-saas-api:staging-${IMAGE_TAG}"
LOCAL_WORKER="xiaoxia-saas-worker:staging-${IMAGE_TAG}"
LOCAL_WEB="xiaoxia-saas-web:staging-${IMAGE_TAG}"
echo "Pulling API image..."
docker pull "$REGISTRY_API"
echo "Pulling Worker image..."
docker pull "$REGISTRY_WORKER"
echo "Pulling Web image..."
docker pull "$REGISTRY_WEB"
# ---- Re-tag 成本地镜像名 ----
docker tag "$REGISTRY_API" "$LOCAL_API"
docker tag "$REGISTRY_WORKER" "$LOCAL_WORKER"
docker tag "$REGISTRY_WEB" "$LOCAL_WEB"
echo "All images pulled and tagged."
# ---- 检查基础设施容器 ----
echo "Checking infrastructure containers..."
for c in xiaoxia-postgres-staging xiaoxia-redis-staging; do
if ! docker inspect "$c" >/dev/null 2>&1; then
echo "ERROR: Required container not found: $c"
exit 1
fi
state=$(docker inspect -f '{{.State.Status}}' "$c")
if [ "$state" != "running" ]; then
echo "ERROR: Container not running: $c ($state)"
exit 1
fi
done
# ---- 创建 Staging 网络 ----
docker network create xiaoxia-net-staging 2>/dev/null || true
# ---- 执行数据库 Migration ----
echo "Running database migrations..."
docker run --rm \
--env-file "$ENV_FILE" \
--network xiaoxia-net-staging \
-e APP_ENV=staging \
"$LOCAL_API" sh -c "cd /app && alembic upgrade head"
echo "Migrations completed."
# ---- 停止旧容器 ----
echo "Stopping old containers..."
docker rm -f xiaoxia-api-staging 2>/dev/null || true
docker rm -f xiaoxia-worker-staging 2>/dev/null || true
docker rm -f xiaoxia-web-staging 2>/dev/null || true
# ---- 启动参数(日志配置统一) ----
LOG_OPTS="--log-driver json-file --log-opt max-size=50m --log-opt max-file=3"
# ---- 启动 API 容器 ----
echo "Starting API container..."
docker run -d \
--name xiaoxia-api-staging \
--env-file "$ENV_FILE" \
--network xiaoxia-net-staging \
-p 127.0.0.1:8000:8000 \
-e APP_ENV=staging \
-e APP_VERSION="$IMAGE_TAG" \
-e GENERATED_FILES_DIR=/app/generated \
-e GENERATED_FILES_URL_PREFIX=/generated-files \
-e PUBLIC_API_BASE_URL=https://staging-api.xiaoxiajianji.com \
-v "$GENERATED_DIR:/app/generated" \
--restart unless-stopped \
--cpus 1 \
--memory 1g \
--health-cmd "python3 -c \"import urllib.request; urllib.request.urlopen('http://localhost:8000/health', timeout=5)\"" \
--health-interval 30s \
--health-timeout 10s \
--health-retries 3 \
--health-start-period 30s \
$LOG_OPTS \
"$LOCAL_API"
# ---- 启动 Worker 容器 ----
echo "Starting Worker container..."
docker run -d \
--name xiaoxia-worker-staging \
--env-file "$ENV_FILE" \
--network xiaoxia-net-staging \
-e APP_ENV=staging \
-e APP_VERSION="$IMAGE_TAG" \
-e WORKER_CONCURRENCY=1 \
-e WORKER_MAX_TASKS_PER_CHILD=100 \
-e GENERATED_FILES_DIR=/app/generated \
-e GENERATED_FILES_URL_PREFIX=/generated-files \
-e PUBLIC_API_BASE_URL=https://staging-api.xiaoxiajianji.com \
-v "$GENERATED_DIR:/app/generated" \
--restart unless-stopped \
--cpus 1 \
--memory 1g \
--health-cmd "sh -c \"grep -q celery /proc/1/cmdline || exit 1\"" \
--health-interval 30s \
--health-timeout 10s \
--health-retries 3 \
--health-start-period 30s \
$LOG_OPTS \
"$LOCAL_WORKER"
# ---- 启动 Web 容器 ----
echo "Starting Web container..."
docker run -d \
--name xiaoxia-web-staging \
--network xiaoxia-net-staging \
-p 127.0.0.1:3001:80 \
--restart unless-stopped \
--cpus 0.5 \
--memory 256m \
--health-cmd "wget --spider -q http://127.0.0.1:80" \
--health-interval 30s \
--health-timeout 5s \
--health-retries 3 \
$LOG_OPTS \
"$LOCAL_WEB"
# ---- 等待 API 就绪 ----
echo "Waiting for API to become healthy..."
i=0
while [ "$i" -lt 30 ]; do
if curl -sf --max-time 5 http://127.0.0.1:8000/health >/dev/null 2>&1; then
echo "API is healthy!"
break
fi
i=$((i + 1))
echo " Waiting... ($i/30)"
sleep 2
done
if [ "$i" -ge 30 ]; then
echo "ERROR: API did not become healthy within 60s"
docker logs --tail 50 xiaoxia-api-staging
exit 1
fi
# ---- 等待 Web 就绪 ----
echo "Waiting for Web to become healthy..."
i=0
while [ "$i" -lt 15 ]; do
if curl -sf --max-time 5 http://127.0.0.1:3001/ >/dev/null 2>&1; then
echo "Web is healthy!"
break
fi
i=$((i + 1))
echo " Waiting... ($i/15)"
sleep 2
done
if [ "$i" -ge 15 ]; then
echo "ERROR: Web did not become healthy within 30s"
docker logs --tail 30 xiaoxia-web-staging
exit 1
fi
# ---- 清理旧镜像 ----
echo "Cleaning up old images..."
docker image prune -af --filter "until=72h" 2>/dev/null || true
echo ""
echo "=== Staging deployment complete ==="
echo "API: http://127.0.0.1:8000"
echo "Web: http://127.0.0.1:3001"
echo "Version: $IMAGE_TAG"
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Image}}" | grep staging