fix(scripts): #1894/#1968 抖音 yt-dlp cookies 支持 + 旧档位归一化补测 + CI 覆盖率修复
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 26s
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 1m28s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 1m23s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 2m3s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 3m11s
AI Code Review / AI Code Review (pull_request) Successful in 6m49s
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 10m42s
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Failing after 14m49s
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
CI/CD Pipeline / Integration Tests (pull_request) Successful in 2m30s
CI/CD Pipeline / Unit Tests (pull_request) Has been cancelled
CI/CD Pipeline / Build Production API Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Web Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been cancelled
CI/CD Pipeline / Deploy Production (pull_request) Has been cancelled
CI/CD Pipeline / Production Browser E2E (pull_request) Has been cancelled
CI/CD Pipeline / Canary Release to Production (pull_request) Has been cancelled
CI/CD Pipeline / CI Gate (pull_request) Has been cancelled
CI/CD Pipeline / Validate - Style (pull_request) Has been cancelled
CI/CD Pipeline / Validate - Security (pull_request) Has been cancelled
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Has been cancelled
CI/CD Pipeline / Check push changed paths (pull_request) Has been skipped
CI/CD Pipeline / Build Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Build Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging API Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Web Image (pull_request) Has been skipped
CI/CD Pipeline / Retag skipped Staging Worker Image (pull_request) Has been skipped
CI/CD Pipeline / Deploy Staging (Watchtower auto-deploy) (pull_request) Has been skipped
CI/CD Pipeline / Staging E2E Tests (pull_request) Has been skipped
CI/CD Pipeline / Staging API Integration Tests (pull_request) Has been skipped
CI/CD Pipeline / ACR Image Cleanup (pull_request) Has been skipped
CI/CD Pipeline / Check if frontend-only change (pull_request) Successful in 26s
CI/CD Pipeline / PR Build Web Image (pull_request) Has been skipped
Preview Deploy / Deploy Preview Environment (pull_request) Successful in 1m28s
CI/CD Pipeline / PR Build API Image (pull_request) Successful in 1m23s
CI/CD Pipeline / PR Build Worker Image (pull_request) Successful in 2m3s
PR Automation / Auto Approve on CI Green (pull_request) Successful in 3m11s
AI Code Review / AI Code Review (pull_request) Successful in 6m49s
PR Automation / Auto Merge on CI Green + Approved (pull_request) Successful in 10m42s
CI/CD Pipeline / Dedup Check - skip PR tests when covered by push pipeline (pull_request) Failing after 14m49s
CI/CD Pipeline / Frontend Lint (pull_request) Has been skipped
CI/CD Pipeline / Frontend Unit Tests (pull_request) Has been skipped
CI/CD Pipeline / Integration Tests (pull_request) Successful in 2m30s
CI/CD Pipeline / Unit Tests (pull_request) Has been cancelled
CI/CD Pipeline / Build Production API Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Web Image (pull_request) Has been cancelled
CI/CD Pipeline / Build Production Worker Image (pull_request) Has been cancelled
CI/CD Pipeline / Deploy Production (pull_request) Has been cancelled
CI/CD Pipeline / Production Browser E2E (pull_request) Has been cancelled
CI/CD Pipeline / Canary Release to Production (pull_request) Has been cancelled
CI/CD Pipeline / CI Gate (pull_request) Has been cancelled
CI/CD Pipeline / Validate - Style (pull_request) Has been cancelled
CI/CD Pipeline / Validate - Security (pull_request) Has been cancelled
CI/CD Pipeline / Validate - Python (mypy + alembic) (pull_request) Has been cancelled
- scripts_ai.py: 支持通过 DOUYIN_COOKIES_FILE 指定 Netscape cookies 文件;
cookies 缺失/过期类错误('fresh cookies' 等)返回 503 + 友好文案
('抖音链接解析暂时不可用,请稍后重试或手动输入文案'),
不再暴露 yt-dlp 原始错误;普通下载失败统一文案,不带内部异常文本
- ci_staging_deploy.sh: API 容器挂载 /var/lib/xiaoxia-saas-staging/configs/douyin_cookies.txt
到 /app/configs/douyin_cookies.txt;自动 mkdir + 空占位兜底
- infra/docker/compose.yml: API 服务挂载 ../../deploy/configs:/app/configs:ro
- deploy/configs/.env.{staging,production}: 新增 DOUYIN_COOKIES_FILE 环境变量
- deploy/configs/douyin_cookies.txt: Netscape 格式占位文件(真实 cookies 部署时替换)
- 新增 test_helpers_get_user_plan.py 覆盖 _helpers.get_user_plan 旧档位归一化分支
- 新增 test_subscription_routes_legacy.py 覆盖 _build_subscription_info 旧档位归一化/
cancel_subscription 免费用户分支/change_plan 参数校验/toggle-auto-renew 等
- 扩展 test_extract_from_douyin_errors.py: 新增 5 个 cookies 场景测试(503 友好文案/
cookiefile 注入/错误信息脱敏等)
- 增量覆盖率从 59% 提升到 60%+,解决 CI Gate 阻塞
This commit is contained in:
@@ -38,6 +38,51 @@ logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
# 抖音 cookies 文件路径(Netscape 格式),由环境变量 DOUYIN_COOKIES_FILE 覆盖
|
||||
# 默认路径与 deploy/configs/douyin_cookies.txt 对应(容器内挂载到 /app/configs/)
|
||||
DOUYIN_COOKIES_FILE = os.environ.get(
|
||||
"DOUYIN_COOKIES_FILE",
|
||||
"/app/configs/douyin_cookies.txt",
|
||||
)
|
||||
|
||||
# cookies 失效/需要刷新的错误关键词
|
||||
_COOKIES_ERROR_KEYWORDS = (
|
||||
"fresh cookies",
|
||||
"cookies (not necessarily logged in)",
|
||||
"cookies are needed",
|
||||
"need cookies",
|
||||
"cookie is expired",
|
||||
"login required",
|
||||
"sign in to continue",
|
||||
"未登录",
|
||||
"需要登录",
|
||||
"cookies过期",
|
||||
)
|
||||
|
||||
|
||||
def _cookies_file_exists() -> bool:
|
||||
"""检查抖音 cookies 文件是否存在且非空."""
|
||||
try:
|
||||
return bool(DOUYIN_COOKIES_FILE) and os.path.isfile(DOUYIN_COOKIES_FILE) and os.path.getsize(DOUYIN_COOKIES_FILE) > 100
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
|
||||
def _is_cookies_related_error(msg: str) -> bool:
|
||||
"""判断 yt-dlp 的错误是否与 cookies 缺失/过期有关."""
|
||||
low = msg.lower()
|
||||
return any(kw in low for kw in _COOKIES_ERROR_KEYWORDS)
|
||||
|
||||
|
||||
# 启动时记录 cookies 状态,便于排查
|
||||
if _cookies_file_exists():
|
||||
logger.info("抖音 cookies 文件已加载: %s (%d bytes)", DOUYIN_COOKIES_FILE, os.path.getsize(DOUYIN_COOKIES_FILE))
|
||||
else:
|
||||
logger.warning(
|
||||
"抖音 cookies 文件未找到或无效: path=%s 抖音提取功能可能因 cookies 缺失失败",
|
||||
DOUYIN_COOKIES_FILE,
|
||||
)
|
||||
|
||||
# 抖音 URL 校验:支持短链 v.douyin.com 和长链 www.douyin.com/video/
|
||||
_DOUYIN_URL_RE = re.compile(
|
||||
r"^(https?://)?(v\.douyin\.com/\S+|www\.douyin\.com/video/\S+)$",
|
||||
@@ -104,26 +149,45 @@ def extract_from_douyin(
|
||||
"noplaylist": True,
|
||||
}
|
||||
|
||||
# 如果存在抖音 cookies 文件,传给 yt-dlp 绕过反爬
|
||||
if _cookies_file_exists():
|
||||
ydl_opts["cookiefile"] = DOUYIN_COOKIES_FILE
|
||||
logger.debug("使用抖音 cookies 文件: %s", DOUYIN_COOKIES_FILE)
|
||||
|
||||
try:
|
||||
ydl = yt_dlp.YoutubeDL(ydl_opts)
|
||||
info = ydl.extract_info(url_for_download, download=True)
|
||||
except yt_dlp.utils.DownloadError as exc:
|
||||
# yt-dlp 官方异常类型:HTTP 错误、短链失效、视频下架等
|
||||
# yt-dlp 官方异常类型:HTTP 错误、短链失效、视频下架、cookies 过期等
|
||||
msg = str(exc)
|
||||
logger.warning("抖音下载失败: url=%s error=%s", source_url, msg)
|
||||
# 404/视频不存在/不可下载 → 400;网络问题/上游异常 → 502
|
||||
# 404/视频不存在/不可下载 → 400
|
||||
is_bad_url = any(
|
||||
kw in msg.lower() for kw in ("404", "not found", "unable to download webpage", "unsupported url", "no video formats")
|
||||
kw in msg.lower() for kw in ("404", "not found", "unable to download webpage", "unsupported url", "no video formats", "video unavailable", "this video isn't available")
|
||||
)
|
||||
# cookies 缺失/过期 → 返回友好提示,不暴露 yt-dlp 原始错误
|
||||
if _is_cookies_related_error(msg):
|
||||
logger.error("抖音 cookies 失效或缺失,需要刷新: %s", msg[:300])
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
|
||||
detail="抖音链接解析暂时不可用,请稍后重试或手动输入文案",
|
||||
) from exc
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST if is_bad_url else status.HTTP_502_BAD_GATEWAY,
|
||||
detail=("无法解析该抖音链接,请确认链接有效且视频未被下架" if is_bad_url else f"视频下载失败: {msg[:200]}"),
|
||||
detail=("无法解析该抖音链接,请确认链接有效且视频未被下架" if is_bad_url else "视频下载失败,请稍后重试"),
|
||||
) from exc
|
||||
except Exception as exc:
|
||||
logger.exception("抖音视频下载异常: url=%s", source_url)
|
||||
msg = str(exc)
|
||||
logger.exception("抖音视频下载异常: url=%s error=%s", source_url, msg)
|
||||
# cookies 相关的未知异常也走友好提示
|
||||
if _is_cookies_related_error(msg):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
|
||||
detail="抖音链接解析暂时不可用,请稍后重试或手动输入文案",
|
||||
) from exc
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_502_BAD_GATEWAY,
|
||||
detail=f"视频下载失败: {str(exc)[:200]}",
|
||||
detail="视频下载失败,请稍后重试",
|
||||
) from exc
|
||||
|
||||
if info is None:
|
||||
|
||||
Reference in New Issue
Block a user