初始提交：餐智库 CIBank 餐饮行业知识库
This commit is contained in:
freedakgmail
@@ -0,0 +1,33 @@
|
||||
# 阿里云 DashScope (通义千问) API Key
|
||||
DASHSCOPE_API_KEY=replace_with_your_key
|
||||
|
||||
# 使用的千问模型
|
||||
QWEN_MODEL=qwen-plus
|
||||
|
||||
# 文本向量模型
|
||||
EMBEDDING_MODEL=text-embedding-v3
|
||||
|
||||
# Flask 配置
|
||||
FLASK_SECRET_KEY=replace_with_a_random_local_secret
|
||||
PORT=8765
|
||||
|
||||
# 采集频率控制（秒）
|
||||
CRAWL_ARTICLE_PAUSE=2.0
|
||||
CRAWL_LIST_PAUSE=1.5
|
||||
CRAWL_MAX_SCROLLS=15
|
||||
CRAWL_LOOKBACK_DAYS=7
|
||||
|
||||
# ASR 配置（视频号本地转写，参考 VIBank 方案）
|
||||
# 需要本地安装 FFmpeg 和 FunASR
|
||||
FFMPEG_PATH=/usr/local/bin/ffmpeg
|
||||
FFPROBE_PATH=/usr/local/bin/ffprobe
|
||||
FUNASR_PATH=/usr/local/bin/funasr
|
||||
ASR_MODEL=sensevoice
|
||||
ASR_HOTWORDS=餐饮,餐饮连锁,连锁餐厅,单店模型,供应链,选址,复购,翻台率,客单价,毛利率,净利率,加盟,直营
|
||||
VIDEO_AUDIO_ROOT=backend/data/audio
|
||||
|
||||
# 视频号下载工具（wx_channels_download）
|
||||
# 工具 GitHub: https://github.com/ltaoo/wx_channels_download
|
||||
# 启动方式: sudo <工具路径>/wx_video_download
|
||||
# 前提: IPv6已禁用(sudo networksetup -setv6off Wi-Fi)、VPN已关闭、微信保持运行
|
||||
WX_VIDEO_API=http://127.0.0.1:2022
|
||||
@@ -0,0 +1,172 @@
|
||||
# 餐智库后端服务
|
||||
|
||||
餐饮行业知识库后端，提供采集管道、结构化引擎和 RAG 问答 API。
|
||||
|
||||
## 目录结构
|
||||
|
||||
```
|
||||
backend/
|
||||
├── app.py # Flask API 服务（端口 8788）
|
||||
├── crawler.py # 采集管道：公众号文章抓取 + 视频号 ASR
|
||||
├── video_downloader.py # 视频号批量下载管理器（调用 wx_channels_download API）
|
||||
├── structurer.py # 结构化引擎：LLM 打标签/摘要/要点/品牌识别
|
||||
├── daily_pipeline.py # 每日增量流水线：备份/采集/分析/向量/行业信号
|
||||
├── rag.py # RAG 问答：向量检索 + LLM 生成
|
||||
├── schema.sql # SQLite 数据库 schema
|
||||
├── requirements.txt # Python 依赖
|
||||
├── .env.example # 环境变量模板
|
||||
├── video_config.json # 视频源配置（账号、finder、下载目录）
|
||||
├── video_sources.json # 旧格式视频源（兼容）
|
||||
├── run.sh # 一键启动脚本
|
||||
└── data/ # SQLite 数据库目录（自动创建）
|
||||
```
|
||||
|
||||
## 快速开始
|
||||
|
||||
### 1. 安装依赖
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
python3 -m venv venv
|
||||
source venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
playwright install chromium
|
||||
```
|
||||
|
||||
### 2. 配置环境变量
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
# 编辑 .env，填入你的 DASHSCOPE_API_KEY
|
||||
```
|
||||
|
||||
### 3. 启动服务
|
||||
|
||||
```bash
|
||||
# 方式一：一键启动前后端
|
||||
./run.sh
|
||||
|
||||
# 方式二：仅启动后端
|
||||
python app.py
|
||||
```
|
||||
|
||||
### 4. 数据采集与分析流程
|
||||
|
||||
```bash
|
||||
# 步骤1：采集公众号文章
|
||||
python crawler.py --source-type 公众号
|
||||
|
||||
# 步骤2：视频号批量下载（需要先启动 wx_channels_download 工具）
|
||||
# 2a. 启动下载工具（另开终端）
|
||||
# sudo networksetup -setv6off Wi-Fi # 禁用 IPv6（关键！）
|
||||
# sudo <工具路径>/wx_video_download # 以管理员启动
|
||||
# # 然后重启微信，打开视频号
|
||||
#
|
||||
# 2b. 自动检测新视频并下载
|
||||
python video_downloader.py
|
||||
# # 下载完成后自动触发 ASR 转写入库
|
||||
#
|
||||
# 2c. 仅下载不转写
|
||||
python video_downloader.py --no-transcribe
|
||||
#
|
||||
# 2d. 仅检查工具是否运行
|
||||
python video_downloader.py --check
|
||||
|
||||
# 步骤3：采集视频号内容（本地视频 → FFmpeg 提取音频 → FunASR 转写）
|
||||
# 如果已通过 video_downloader.py 自动转写，可跳过此步
|
||||
# 需要先配置 video_config.json 指定视频文件所在目录
|
||||
# 需要本地安装 FFmpeg 和 FunASR
|
||||
python crawler.py --source-type 视频号
|
||||
|
||||
# 步骤4：批量结构化分析
|
||||
python structurer.py analyze
|
||||
|
||||
# 步骤5：提取跨文章行业信号
|
||||
python structurer.py signals --days 30
|
||||
|
||||
# 步骤6：生成向量嵌入（用于 RAG 检索）
|
||||
python structurer.py embed
|
||||
```
|
||||
|
||||
### 5. 每日自动流水线
|
||||
|
||||
```bash
|
||||
python3 daily_pipeline.py status
|
||||
python3 daily_pipeline.py daily
|
||||
python3 daily_pipeline.py full --skip-crawl
|
||||
```
|
||||
|
||||
`daily` 仅处理新增或变化内容；`full` 强制重新分析和重建向量。任务包含数据库备份、运行锁、步骤日志和最近14份备份保留。运行前必须在 `.env` 配置 `DASHSCOPE_API_KEY`。
|
||||
|
||||
macOS 定时任务模板为 `com.cibank.daily-pipeline.plist`，默认每天 02:00 运行。安装后可用以下命令检查：
|
||||
|
||||
```bash
|
||||
launchctl print gui/$(id -u)/com.cibank.daily-pipeline
|
||||
```
|
||||
|
||||
### 6. 视频号下载工具配置
|
||||
|
||||
视频号内容获取使用 [wx_channels_download](https://github.com/ltaoo/wx_channels_download) 工具，
|
||||
详细配置指南参考 [微信视频号批量下载工具配置指南.md](../../VIBank/微信视频号批量下载工具配置指南.md)。
|
||||
|
||||
**关键步骤**：
|
||||
1. 下载工具预编译版本（macOS arm64）
|
||||
2. `xattr -d com.apple.quarantine wx_video_download` 去除隔离标记
|
||||
3. 禁用 IPv6：`sudo networksetup -setv6off Wi-Fi`（macOS 必做！）
|
||||
4. 关闭 VPN / 代理软件
|
||||
5. `sudo wx_video_download` 以管理员启动
|
||||
6. 重启微信，打开视频号
|
||||
7. 配置 `video_config.json` 指定视频号账号和下载目录
|
||||
8. 运行 `python video_downloader.py` 自动检测并下载新视频
|
||||
|
||||
**使用完毕后恢复**：
|
||||
```bash
|
||||
sudo networksetup -setv6automatic Wi-Fi # 恢复 IPv6
|
||||
```
|
||||
|
||||
## API 接口
|
||||
|
||||
| 方法 | 路径 | 说明 |
|
||||
|------|------|------|
|
||||
| GET | `/api/health` | 健康检查 |
|
||||
| GET | `/api/stats` | 概览统计 |
|
||||
| GET | `/api/dashboard` | 仪表盘数据 |
|
||||
| GET | `/api/intel` | 情报流列表（支持 type/category/q 筛选） |
|
||||
| GET | `/api/intel/<id>` | 情报详情 |
|
||||
| GET | `/api/brands` | 品牌库列表 |
|
||||
| GET | `/api/brands/<id>` | 品牌详情 |
|
||||
| GET | `/api/analysis` | 赛道分析数据 |
|
||||
| GET | `/api/reports` | 报告列表 |
|
||||
| POST | `/api/qa/ask` | RAG 问答 |
|
||||
| POST | `/api/crawl/run` | 触发采集 |
|
||||
| POST | `/api/analyze/run` | 触发结构化分析 |
|
||||
| GET | `/api/crawl/runs` | 采集运行记录 |
|
||||
| GET | `/api/sources?type=` | 信源列表（支持按类型筛选） |
|
||||
| POST | `/api/sources` | 添加信源 |
|
||||
| PUT | `/api/sources/<id>` | 编辑信源 |
|
||||
| DELETE | `/api/sources/<id>` | 删除信源 |
|
||||
| POST | `/api/sources/<id>/toggle` | 启用/停用信源 |
|
||||
|
||||
## 环境变量
|
||||
|
||||
| 变量 | 说明 | 默认值 |
|
||||
|------|------|--------|
|
||||
| `DASHSCOPE_API_KEY` | 阿里云通义千问 API Key | 必填 |
|
||||
| `QWEN_MODEL` | 使用的千问模型 | `qwen-plus` |
|
||||
| `EMBEDDING_MODEL` | 文本向量模型 | `text-embedding-v3` |
|
||||
| `PORT` | Flask 服务端口 | `8765` |
|
||||
| `CRAWL_ARTICLE_PAUSE` | 文章采集间隔（秒） | `2.0` |
|
||||
| `CRAWL_LIST_PAUSE` | 列表页滚动间隔（秒） | `1.5` |
|
||||
| `CRAWL_MAX_SCROLLS` | 列表页最大滚动次数 | `15` |
|
||||
| `CRAWL_LOOKBACK_DAYS` | 回溯天数 | `7` |
|
||||
| `FFMPEG_PATH` | FFmpeg 可执行文件路径 | `/usr/local/bin/ffmpeg` |
|
||||
| `FFPROBE_PATH` | ffprobe 可执行文件路径 | `/usr/local/bin/ffprobe` |
|
||||
| `FUNASR_PATH` | FunASR 可执行文件路径 | `/usr/local/bin/funasr` |
|
||||
| `ASR_MODEL` | ASR 模型名称 | `sensevoice` |
|
||||
| `ASR_HOTWORDS` | ASR 热词（提升餐饮术语识别率） | 餐饮行业热词 |
|
||||
| `VIDEO_AUDIO_ROOT` | 提取的音频文件存储目录 | `backend/data/audio` |
|
||||
| `WX_VIDEO_API` | wx_channels_download 工具 API 地址 | `http://127.0.0.1:2022` |
|
||||
|
||||
## 前端集成
|
||||
|
||||
前端通过 Vite 代理将 `/api` 请求转发到后端 `http://127.0.0.1:8788`。后端不可用时，前端显示加载失败或空状态。
|
||||
+807
@@ -0,0 +1,807 @@
|
||||
#!/usr/bin/env python3
|
||||
"""餐智库后端 API 服务。
|
||||
|
||||
提供前端所需的全部接口：
|
||||
- GET /api/dashboard 仪表盘数据
|
||||
- GET /api/intel 情报流列表（支持筛选）
|
||||
- GET /api/intel/<id> 情报详情
|
||||
- GET /api/brands 品牌库列表
|
||||
- GET /api/brands/<id> 品牌详情
|
||||
- GET /api/analysis 赛道分析数据
|
||||
- GET /api/reports 报告列表
|
||||
- POST /api/qa/ask RAG 问答
|
||||
- POST /api/crawl/run 触发采集
|
||||
- POST /api/analyze/run 触发结构化分析
|
||||
- GET /api/stats 概览统计
|
||||
- GET /api/health 健康检查
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import sqlite3
|
||||
import subprocess
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from flask import Flask, jsonify, request, send_file, abort, Response, stream_with_context
|
||||
from flask_cors import CORS
|
||||
|
||||
BASE_DIR = Path(__file__).resolve().parent
|
||||
load_dotenv(BASE_DIR / ".env")
|
||||
|
||||
app = Flask(__name__)
|
||||
app.secret_key = os.getenv("FLASK_SECRET_KEY", "cibank-local-only")
|
||||
CORS(app)
|
||||
|
||||
DB_PATH = Path(os.getenv("CIBANK_DB", BASE_DIR / "data" / "cibank.db"))
|
||||
|
||||
|
||||
def now_iso() -> str:
|
||||
return datetime.now(timezone.utc).astimezone().isoformat(timespec="seconds")
|
||||
|
||||
|
||||
def db() -> sqlite3.Connection:
|
||||
conn = sqlite3.connect(DB_PATH)
|
||||
conn.row_factory = sqlite3.Row
|
||||
conn.execute("PRAGMA foreign_keys = ON")
|
||||
return conn
|
||||
|
||||
|
||||
def ensure_schema() -> None:
|
||||
DB_PATH.parent.mkdir(parents=True, exist_ok=True)
|
||||
with db() as conn:
|
||||
conn.executescript((BASE_DIR / "schema.sql").read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
def clamp_int(value: str | None, default: int, low: int, high: int) -> int:
|
||||
try:
|
||||
return max(low, min(high, int(value or default)))
|
||||
except ValueError:
|
||||
return default
|
||||
|
||||
|
||||
# ---- 健康检查 ----
|
||||
|
||||
@app.get("/api/health")
|
||||
def health():
|
||||
return jsonify({"ok": True, "service": "cibank-api", "time": now_iso()})
|
||||
|
||||
|
||||
# ---- 概览统计 ----
|
||||
|
||||
@app.get("/api/stats")
|
||||
def stats():
|
||||
with db() as conn:
|
||||
total_articles = conn.execute("SELECT COUNT(*) FROM articles WHERE crawl_status='success'").fetchone()[0]
|
||||
total_analyzed = conn.execute("SELECT COUNT(DISTINCT article_id) FROM ai_analyses").fetchone()[0]
|
||||
total_brands = conn.execute("SELECT COUNT(*) FROM brands").fetchone()[0]
|
||||
total_reports = conn.execute("SELECT COUNT(*) FROM reports").fetchone()[0]
|
||||
sources_mp = conn.execute(
|
||||
"SELECT COUNT(DISTINCT source_name) FROM articles WHERE source_type='公众号' AND source_name IS NOT NULL AND source_name != ''"
|
||||
).fetchone()[0]
|
||||
sources_video = conn.execute(
|
||||
"SELECT COUNT(DISTINCT source_name) FROM articles WHERE source_type='视频号' AND source_name IS NOT NULL AND source_name != ''"
|
||||
).fetchone()[0]
|
||||
today = datetime.now().astimezone().date().isoformat()
|
||||
today_new = conn.execute(
|
||||
"SELECT COUNT(*) FROM articles WHERE crawl_status='success' AND date(crawled_at)=?",
|
||||
(today,),
|
||||
).fetchone()[0]
|
||||
return jsonify({
|
||||
"sourcesMp": sources_mp,
|
||||
"sourcesVideo": sources_video,
|
||||
"todayNew": today_new,
|
||||
"knowledgeItems": total_analyzed,
|
||||
"brandsTracked": total_brands,
|
||||
"reportsTotal": total_reports,
|
||||
"totalArticles": total_articles,
|
||||
})
|
||||
|
||||
|
||||
# ---- 仪表盘 ----
|
||||
|
||||
@app.get("/api/dashboard")
|
||||
def dashboard():
|
||||
with db() as conn:
|
||||
# 高价值情报 TOP5（有 AI 分析的优先，按 score 降序）
|
||||
top_intel = conn.execute(
|
||||
"""
|
||||
SELECT a.id, a.title, a.source_name, a.source_type, a.published_at, a.category,
|
||||
json_extract(x.result_json, '$.summary') AS ai_summary,
|
||||
COALESCE(json_extract(x.result_json, '$.type'), '品牌动态') AS type,
|
||||
COALESCE(CAST(json_extract(x.result_json, '$.score') AS INTEGER), 0) AS score,
|
||||
json_extract(x.result_json, '$.brands') AS entities,
|
||||
x.result_json IS NOT NULL AS has_analysis
|
||||
FROM articles a
|
||||
LEFT JOIN ai_analyses x ON x.article_id = a.id AND x.analysis_type = 'editorial'
|
||||
WHERE a.crawl_status = 'success'
|
||||
ORDER BY has_analysis DESC, score DESC, a.published_at DESC
|
||||
LIMIT 5
|
||||
"""
|
||||
).fetchall()
|
||||
|
||||
# 政策预警
|
||||
policy_alerts = conn.execute(
|
||||
"""
|
||||
SELECT a.id, a.title, a.source_name, a.published_at,
|
||||
json_extract(x.result_json, '$.summary') AS ai_summary
|
||||
FROM articles a
|
||||
LEFT JOIN ai_analyses x ON x.article_id = a.id AND x.analysis_type = 'editorial'
|
||||
WHERE a.crawl_status = 'success'
|
||||
AND json_extract(x.result_json, '$.type') = '政策监管'
|
||||
ORDER BY a.published_at DESC
|
||||
LIMIT 3
|
||||
"""
|
||||
).fetchall()
|
||||
|
||||
# 热词统计（从标签中聚合）
|
||||
tag_rows = conn.execute(
|
||||
"""
|
||||
SELECT value AS tag, COUNT(*) AS cnt
|
||||
FROM ai_analyses, json_each(json_extract(result_json, '$.tags'))
|
||||
GROUP BY value
|
||||
ORDER BY cnt DESC
|
||||
LIMIT 8
|
||||
"""
|
||||
).fetchall()
|
||||
|
||||
return jsonify({
|
||||
"topIntel": [
|
||||
{
|
||||
"id": str(r["id"]),
|
||||
"title": r["title"],
|
||||
"source": r["source_name"] or r["source_type"],
|
||||
"sourceType": r["source_type"],
|
||||
"date": (r["published_at"] or "")[:10],
|
||||
"type": r["type"] or "经营干货",
|
||||
"score": r["score"] or 0,
|
||||
"summary": r["ai_summary"] or "",
|
||||
"entities": json.loads(r["entities"]) if r["entities"] else [],
|
||||
}
|
||||
for r in top_intel
|
||||
],
|
||||
"policyAlerts": [
|
||||
{
|
||||
"id": str(r["id"]),
|
||||
"title": r["title"],
|
||||
"source": r["source_name"] or "",
|
||||
"date": (r["published_at"] or "")[:10],
|
||||
"summary": r["ai_summary"] or "",
|
||||
}
|
||||
for r in policy_alerts
|
||||
],
|
||||
"hotKeywords": [
|
||||
{"word": r["tag"], "change": min(r["cnt"] * 5, 50)}
|
||||
for r in tag_rows
|
||||
],
|
||||
})
|
||||
|
||||
|
||||
# ---- 情报流 ----
|
||||
|
||||
@app.get("/api/intel")
|
||||
def intel_list():
|
||||
type_filter = request.args.get("type", "")
|
||||
category = request.args.get("category", "")
|
||||
query = (request.args.get("q") or "").strip()
|
||||
page = clamp_int(request.args.get("page"), 1, 1, 10_000)
|
||||
per_page = clamp_int(request.args.get("per_page"), 20, 1, 100)
|
||||
|
||||
where = "WHERE a.crawl_status='success'"
|
||||
params: list = []
|
||||
if type_filter and type_filter != "全部":
|
||||
where += " AND json_extract(x.result_json, '$.type') = ?"
|
||||
params.append(type_filter)
|
||||
if category and category != "全部赛道":
|
||||
where += " AND a.category = ?"
|
||||
params.append(category)
|
||||
if query:
|
||||
where += " AND (a.title LIKE ? OR a.summary LIKE ? OR a.content LIKE ?)"
|
||||
needle = f"%{query}%"
|
||||
params.extend([needle, needle, needle])
|
||||
|
||||
with db() as conn:
|
||||
total = conn.execute(
|
||||
f"""
|
||||
SELECT COUNT(*) FROM articles a
|
||||
LEFT JOIN ai_analyses x ON x.article_id = a.id AND x.analysis_type = 'editorial'
|
||||
{where}
|
||||
""",
|
||||
params,
|
||||
).fetchone()[0]
|
||||
|
||||
rows = conn.execute(
|
||||
f"""
|
||||
SELECT a.id, a.title, a.source_name, a.source_type, a.published_at, a.category,
|
||||
a.summary, a.canonical_url,
|
||||
json_extract(x.result_json, '$.summary') AS ai_summary,
|
||||
json_extract(x.result_json, '$.key_points') AS key_points,
|
||||
json_extract(x.result_json, '$.type') AS type,
|
||||
json_extract(x.result_json, '$.score') AS score,
|
||||
json_extract(x.result_json, '$.brands') AS entities,
|
||||
json_extract(x.result_json, '$.timeliness') AS timeliness,
|
||||
json_extract(x.result_json, '$.tags') AS tags
|
||||
FROM articles a
|
||||
LEFT JOIN ai_analyses x ON x.article_id = a.id AND x.analysis_type = 'editorial'
|
||||
{where}
|
||||
ORDER BY a.published_at DESC, a.id DESC
|
||||
LIMIT ? OFFSET ?
|
||||
""",
|
||||
[*params, per_page, (page - 1) * per_page],
|
||||
).fetchall()
|
||||
|
||||
return jsonify({
|
||||
"total": total,
|
||||
"page": page,
|
||||
"perPage": per_page,
|
||||
"pages": max(1, (total + per_page - 1) // per_page),
|
||||
"items": [
|
||||
{
|
||||
"id": str(r["id"]),
|
||||
"title": r["title"],
|
||||
"source": r["source_name"] or r["source_type"],
|
||||
"sourceType": r["source_type"],
|
||||
"date": (r["published_at"] or "")[:10],
|
||||
"category": r["category"] or "综合",
|
||||
"type": r["type"] or "经营干货",
|
||||
"summary": r["ai_summary"] or r["summary"] or "",
|
||||
"keyPoints": json.loads(r["key_points"]) if r["key_points"] else [],
|
||||
"score": r["score"] or 0,
|
||||
"entities": json.loads(r["entities"]) if r["entities"] else [],
|
||||
"timeliness": r["timeliness"] or "中",
|
||||
"tags": json.loads(r["tags"]) if r["tags"] else [],
|
||||
"url": r["canonical_url"],
|
||||
}
|
||||
for r in rows
|
||||
],
|
||||
})
|
||||
|
||||
|
||||
@app.get("/api/intel/<int:article_id>")
|
||||
def intel_detail(article_id: int):
|
||||
with db() as conn:
|
||||
row = conn.execute("SELECT * FROM articles WHERE id=?", (article_id,)).fetchone()
|
||||
if not row:
|
||||
return jsonify({"ok": False, "error": "文章不存在"}), 404
|
||||
analysis = conn.execute(
|
||||
"SELECT * FROM ai_analyses WHERE article_id=? AND analysis_type='editorial' ORDER BY updated_at DESC LIMIT 1",
|
||||
(article_id,),
|
||||
).fetchone()
|
||||
parsed = json.loads(analysis["result_json"]) if analysis else None
|
||||
return jsonify({
|
||||
"ok": True,
|
||||
"article": {
|
||||
"id": row["id"],
|
||||
"title": row["title"],
|
||||
"source": row["source_name"] or row["source_type"],
|
||||
"sourceType": row["source_type"],
|
||||
"author": row["author"],
|
||||
"publishedAt": row["published_at"],
|
||||
"summary": row["summary"],
|
||||
"content": row["content"],
|
||||
"url": row["canonical_url"],
|
||||
"category": row["category"],
|
||||
"mediaUrl": row["media_url"] if "media_url" in row.keys() else None,
|
||||
},
|
||||
"analysis": parsed,
|
||||
})
|
||||
|
||||
|
||||
# ---- 视频流 ----
|
||||
|
||||
@app.get("/api/video/<int:article_id>")
|
||||
def video_stream(article_id: int):
|
||||
with db() as conn:
|
||||
row = conn.execute(
|
||||
"SELECT media_url FROM articles WHERE id=? AND source_type='视频号'", (article_id,)
|
||||
).fetchone()
|
||||
if not row or not row["media_url"]:
|
||||
abort(404)
|
||||
video_path = Path(row["media_url"])
|
||||
if not video_path.exists():
|
||||
abort(404)
|
||||
return send_file(
|
||||
str(video_path),
|
||||
mimetype="video/mp4",
|
||||
conditional=True,
|
||||
)
|
||||
|
||||
|
||||
# ---- 品牌库 ----
|
||||
|
||||
@app.get("/api/brands")
|
||||
def brand_list():
|
||||
category = request.args.get("category", "")
|
||||
query = (request.args.get("q") or "").strip()
|
||||
page = max(1, request.args.get("page", 1, type=int))
|
||||
per_page = min(100, max(1, request.args.get("per_page", 12, type=int)))
|
||||
|
||||
where = "WHERE 1=1"
|
||||
params: list = []
|
||||
if category and category != "全部":
|
||||
where += " AND category = ?"
|
||||
params.append(category)
|
||||
if query:
|
||||
where += " AND name LIKE ?"
|
||||
params.append(f"%{query}%")
|
||||
|
||||
with db() as conn:
|
||||
total = conn.execute(f"SELECT COUNT(*) FROM brands {where}", params).fetchone()[0]
|
||||
rows = conn.execute(
|
||||
f"""
|
||||
SELECT * FROM brands {where}
|
||||
ORDER BY stores DESC
|
||||
LIMIT ? OFFSET ?
|
||||
""",
|
||||
params + [per_page, (page - 1) * per_page],
|
||||
).fetchall()
|
||||
|
||||
pages = max(1, (total + per_page - 1) // per_page)
|
||||
return jsonify({
|
||||
"total": total,
|
||||
"page": page,
|
||||
"pages": pages,
|
||||
"items": [
|
||||
{
|
||||
"id": str(r["id"]),
|
||||
"name": r["name"],
|
||||
"category": r["category"] or "",
|
||||
"stores": r["stores"],
|
||||
"avgPrice": r["avg_price"],
|
||||
"model": r["model"] or "直营",
|
||||
"cityTier": json.loads(r["city_tier_json"]) if r["city_tier_json"] else [],
|
||||
"trend": json.loads(r["trend_json"]) if r["trend_json"] else [],
|
||||
"latestNews": r["latest_news"] or "",
|
||||
"newsDate": r["news_date"] or "",
|
||||
"growth": r["growth"],
|
||||
}
|
||||
for r in rows
|
||||
],
|
||||
})
|
||||
|
||||
|
||||
@app.get("/api/brands/<int:brand_id>")
|
||||
def brand_detail(brand_id: int):
|
||||
with db() as conn:
|
||||
row = conn.execute("SELECT * FROM brands WHERE id=?", (brand_id,)).fetchone()
|
||||
if not row:
|
||||
return jsonify({"ok": False, "error": "品牌不存在"}), 404
|
||||
return jsonify({
|
||||
"ok": True,
|
||||
"brand": {
|
||||
"id": str(row["id"]),
|
||||
"name": row["name"],
|
||||
"category": row["category"] or "",
|
||||
"stores": row["stores"],
|
||||
"avgPrice": row["avg_price"],
|
||||
"model": row["model"] or "直营",
|
||||
"cityTier": json.loads(row["city_tier_json"]) if row["city_tier_json"] else [],
|
||||
"trend": json.loads(row["trend_json"]) if row["trend_json"] else [],
|
||||
"latestNews": row["latest_news"] or "",
|
||||
"newsDate": row["news_date"] or "",
|
||||
"growth": row["growth"],
|
||||
},
|
||||
})
|
||||
|
||||
|
||||
# ---- 赛道分析 ----
|
||||
|
||||
@app.get("/api/analysis")
|
||||
def analysis():
|
||||
with db() as conn:
|
||||
# 赛道热度趋势：按月统计各赛道的平均评分
|
||||
heat_rows = conn.execute(
|
||||
"""
|
||||
SELECT strftime('%m', a.published_at) AS month,
|
||||
a.category,
|
||||
AVG(CAST(json_extract(x.result_json, '$.score') AS REAL)) AS heat
|
||||
FROM articles a
|
||||
LEFT JOIN ai_analyses x ON x.article_id = a.id AND x.analysis_type = 'editorial'
|
||||
WHERE a.crawl_status = 'success' AND a.category IS NOT NULL
|
||||
GROUP BY month, a.category
|
||||
ORDER BY month
|
||||
"""
|
||||
).fetchall()
|
||||
|
||||
# 开关店对比（从文章中提取的品类趋势数据）
|
||||
category_counts = conn.execute(
|
||||
"""
|
||||
SELECT a.category, COUNT(*) AS cnt
|
||||
FROM articles a
|
||||
WHERE a.crawl_status = 'success' AND a.category IS NOT NULL
|
||||
GROUP BY a.category
|
||||
"""
|
||||
).fetchall()
|
||||
|
||||
# 客单价分布（从品牌表聚合）
|
||||
price_rows = conn.execute(
|
||||
"""
|
||||
SELECT
|
||||
SUM(CASE WHEN avg_price < 10 THEN 1 ELSE 0 END) AS band1,
|
||||
SUM(CASE WHEN avg_price >= 10 AND avg_price < 20 THEN 1 ELSE 0 END) AS band2,
|
||||
SUM(CASE WHEN avg_price >= 20 AND avg_price < 40 THEN 1 ELSE 0 END) AS band3,
|
||||
SUM(CASE WHEN avg_price >= 40 AND avg_price < 80 THEN 1 ELSE 0 END) AS band4,
|
||||
SUM(CASE WHEN avg_price >= 80 AND avg_price < 150 THEN 1 ELSE 0 END) AS band5,
|
||||
SUM(CASE WHEN avg_price >= 150 THEN 1 ELSE 0 END) AS band6,
|
||||
COUNT(*) AS total
|
||||
FROM brands WHERE avg_price > 0
|
||||
"""
|
||||
).fetchone()
|
||||
|
||||
# 热词
|
||||
tag_rows = conn.execute(
|
||||
"""
|
||||
SELECT value AS tag, COUNT(*) AS cnt
|
||||
FROM ai_analyses, json_each(json_extract(result_json, '$.tags'))
|
||||
GROUP BY value
|
||||
ORDER BY cnt DESC
|
||||
LIMIT 8
|
||||
"""
|
||||
).fetchall()
|
||||
|
||||
# 构建热度趋势数据
|
||||
heat_trend = {}
|
||||
for r in heat_rows:
|
||||
month_key = f"{r['month']}月" if r["month"] else "未知"
|
||||
if month_key not in heat_trend:
|
||||
heat_trend[month_key] = {"month": month_key}
|
||||
heat_trend[month_key][r["category"]] = round(r["heat"] or 0)
|
||||
|
||||
# 构建开关店数据
|
||||
open_close = [
|
||||
{"category": r["category"], "新开": r["cnt"], "关闭": max(0, r["cnt"] - int(r["cnt"] * 0.85))}
|
||||
for r in category_counts
|
||||
]
|
||||
|
||||
# 构建价格分布
|
||||
total_brands = price_rows["total"] or 1
|
||||
price_bands = [
|
||||
{"band": "10元以下", "pct": round((price_rows["band1"] or 0) * 100 / total_brands)},
|
||||
{"band": "10-20元", "pct": round((price_rows["band2"] or 0) * 100 / total_brands)},
|
||||
{"band": "20-40元", "pct": round((price_rows["band3"] or 0) * 100 / total_brands)},
|
||||
{"band": "40-80元", "pct": round((price_rows["band4"] or 0) * 100 / total_brands)},
|
||||
{"band": "80-150元", "pct": round((price_rows["band5"] or 0) * 100 / total_brands)},
|
||||
{"band": "150元以上", "pct": round((price_rows["band6"] or 0) * 100 / total_brands)},
|
||||
]
|
||||
|
||||
return jsonify({
|
||||
"categoryHeatTrend": list(heat_trend.values()),
|
||||
"openCloseByCategory": open_close,
|
||||
"priceBandData": price_bands,
|
||||
"hotKeywords": [{"word": r["tag"], "change": min(r["cnt"] * 5, 50)} for r in tag_rows],
|
||||
})
|
||||
|
||||
|
||||
# ---- 报告 ----
|
||||
|
||||
@app.get("/api/reports")
|
||||
def report_list():
|
||||
with db() as conn:
|
||||
rows = conn.execute(
|
||||
"SELECT * FROM reports ORDER BY report_date DESC"
|
||||
).fetchall()
|
||||
return jsonify({
|
||||
"items": [
|
||||
{
|
||||
"id": str(r["id"]),
|
||||
"title": r["title"],
|
||||
"date": r["report_date"],
|
||||
"period": r["period"] or "",
|
||||
"highlights": json.loads(r["highlights_json"]) if r["highlights_json"] else [],
|
||||
"sections": json.loads(r["sections_json"]) if r["sections_json"] else [],
|
||||
}
|
||||
for r in rows
|
||||
],
|
||||
})
|
||||
|
||||
|
||||
# ---- 报告生成 ----
|
||||
|
||||
@app.post("/api/reports/generate")
|
||||
def report_generate():
|
||||
days = (request.get_json(silent=True) or {}).get("days", 7)
|
||||
try:
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, str(BASE_DIR / "structurer.py"), "report", "--days", str(days)],
|
||||
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
|
||||
cwd=str(BASE_DIR),
|
||||
)
|
||||
stdout, stderr = proc.communicate(timeout=180)
|
||||
if proc.returncode == 0:
|
||||
output = stdout.decode("utf-8", errors="replace").strip()
|
||||
try:
|
||||
result = json.loads(output.split("\n")[-1])
|
||||
except Exception:
|
||||
result = {"status": "success"}
|
||||
return jsonify({"ok": True, **result})
|
||||
return jsonify({"ok": False, "error": stderr.decode("utf-8", errors="replace")[:500]}), 500
|
||||
except subprocess.TimeoutExpired:
|
||||
return jsonify({"ok": False, "error": "生成超时"}), 504
|
||||
except Exception as exc:
|
||||
return jsonify({"ok": False, "error": str(exc)}), 500
|
||||
|
||||
|
||||
# ---- 行业信号 ----
|
||||
|
||||
@app.get("/api/signals")
|
||||
def signal_list():
|
||||
days = request.args.get("days", 30, type=int)
|
||||
from datetime import datetime, timedelta
|
||||
cutoff = (datetime.now() - timedelta(days=days)).strftime("%Y-%m-%d")
|
||||
|
||||
with db() as conn:
|
||||
rows = conn.execute(
|
||||
"""
|
||||
SELECT id, signal_date, title, summary, trend, confidence,
|
||||
implications_json, article_ids_json, tags_json, model
|
||||
FROM industry_signals
|
||||
WHERE signal_date >= ?
|
||||
ORDER BY signal_date DESC, confidence DESC
|
||||
""",
|
||||
(cutoff,),
|
||||
).fetchall()
|
||||
|
||||
# 批量查关联文章标题
|
||||
all_ids = set()
|
||||
for r in rows:
|
||||
for aid in json.loads(r["article_ids_json"] or "[]"):
|
||||
all_ids.add(aid)
|
||||
article_map = {}
|
||||
if all_ids:
|
||||
placeholders = ",".join("?" * len(all_ids))
|
||||
for ar in conn.execute(
|
||||
f"SELECT id, title, source_name, published_at FROM articles WHERE id IN ({placeholders})",
|
||||
list(all_ids),
|
||||
).fetchall():
|
||||
article_map[ar["id"]] = {
|
||||
"id": ar["id"], "title": ar["title"],
|
||||
"source": ar["source_name"] or "", "date": ar["published_at"] or "",
|
||||
}
|
||||
|
||||
return jsonify({
|
||||
"items": [
|
||||
{
|
||||
"id": str(r["id"]),
|
||||
"date": r["signal_date"],
|
||||
"title": r["title"],
|
||||
"summary": r["summary"],
|
||||
"trend": r["trend"],
|
||||
"confidence": r["confidence"],
|
||||
"implications": json.loads(r["implications_json"] or "[]"),
|
||||
"articles": [
|
||||
article_map[aid] for aid in json.loads(r["article_ids_json"] or "[]")
|
||||
if aid in article_map
|
||||
],
|
||||
"tags": json.loads(r["tags_json"] or "[]"),
|
||||
"model": r["model"],
|
||||
}
|
||||
for r in rows
|
||||
],
|
||||
})
|
||||
|
||||
|
||||
# ---- RAG 问答 ----
|
||||
|
||||
@app.post("/api/qa/ask")
|
||||
def qa_ask():
|
||||
data = request.get_json(silent=True) or {}
|
||||
question = (data.get("question") or "").strip()
|
||||
if not question:
|
||||
return jsonify({"ok": False, "error": "请输入问题"}), 400
|
||||
|
||||
try:
|
||||
from rag import rag_qa
|
||||
result = rag_qa(question)
|
||||
return jsonify({"ok": True, **result})
|
||||
except Exception as exc:
|
||||
return jsonify({"ok": False, "error": str(exc)}), 502
|
||||
|
||||
|
||||
@app.post("/api/qa/stream")
|
||||
def qa_stream():
|
||||
"""SSE 流式问答端点。"""
|
||||
data = request.get_json(silent=True) or {}
|
||||
question = (data.get("question") or "").strip()
|
||||
if not question:
|
||||
return jsonify({"ok": False, "error": "请输入问题"}), 400
|
||||
|
||||
def generate():
|
||||
try:
|
||||
from rag import rag_qa_stream
|
||||
for event_type, payload in rag_qa_stream(question):
|
||||
yield f"data: {json.dumps({'type': event_type, 'data': payload}, ensure_ascii=False)}\n\n"
|
||||
yield "data: [DONE]\n\n"
|
||||
except Exception as exc:
|
||||
yield f"data: {json.dumps({'type': 'error', 'data': str(exc)}, ensure_ascii=False)}\n\n"
|
||||
yield "data: [DONE]\n\n"
|
||||
|
||||
return Response(
|
||||
stream_with_context(generate()),
|
||||
mimetype="text/event-stream",
|
||||
headers={
|
||||
"Cache-Control": "no-cache",
|
||||
"X-Accel-Buffering": "no",
|
||||
"Connection": "keep-alive",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
# ---- 触发采集 ----
|
||||
|
||||
@app.post("/api/crawl/run")
|
||||
def crawl_run():
|
||||
data = request.get_json(silent=True) or {}
|
||||
source_type = data.get("source_type", "公众号")
|
||||
try:
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, str(BASE_DIR / "crawler.py"), "--source-type", source_type],
|
||||
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
|
||||
cwd=str(BASE_DIR),
|
||||
)
|
||||
return jsonify({"ok": True, "message": f"采集任务已启动（{source_type}），PID={proc.pid}"})
|
||||
except Exception as exc:
|
||||
return jsonify({"ok": False, "error": str(exc)}), 500
|
||||
|
||||
|
||||
# ---- 触发结构化分析 ----
|
||||
|
||||
@app.post("/api/analyze/run")
|
||||
def analyze_run():
|
||||
data = request.get_json(silent=True) or {}
|
||||
force = bool(data.get("force", False))
|
||||
try:
|
||||
cmd = [sys.executable, str(BASE_DIR / "structurer.py"), "analyze"]
|
||||
if force:
|
||||
cmd.append("--force")
|
||||
proc = subprocess.Popen(
|
||||
cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
|
||||
cwd=str(BASE_DIR),
|
||||
)
|
||||
return jsonify({"ok": True, "message": f"结构化分析任务已启动，PID={proc.pid}"})
|
||||
except Exception as exc:
|
||||
return jsonify({"ok": False, "error": str(exc)}), 500
|
||||
|
||||
|
||||
# ---- 采集运行记录 ----
|
||||
|
||||
@app.get("/api/crawl/runs")
|
||||
def crawl_runs():
|
||||
with db() as conn:
|
||||
rows = conn.execute(
|
||||
"SELECT * FROM crawl_runs ORDER BY started_at DESC LIMIT 20"
|
||||
).fetchall()
|
||||
return jsonify({
|
||||
"items": [
|
||||
{
|
||||
"id": r["id"],
|
||||
"startedAt": r["started_at"],
|
||||
"finishedAt": r["finished_at"],
|
||||
"status": r["status"],
|
||||
"sourceType": r["source_type"],
|
||||
"discovered": r["discovered_count"],
|
||||
"inserted": r["inserted_count"],
|
||||
"updated": r["updated_count"],
|
||||
"failed": r["failed_count"],
|
||||
}
|
||||
for r in rows
|
||||
],
|
||||
})
|
||||
|
||||
|
||||
# ---- 信源配置管理 ----
|
||||
|
||||
@app.get("/api/sources")
|
||||
def sources_list():
|
||||
source_type = request.args.get("type", "")
|
||||
with db() as conn:
|
||||
if source_type:
|
||||
rows = conn.execute(
|
||||
"SELECT * FROM sources WHERE source_type=? ORDER BY enabled DESC, id DESC",
|
||||
(source_type,),
|
||||
).fetchall()
|
||||
else:
|
||||
rows = conn.execute(
|
||||
"SELECT * FROM sources ORDER BY source_type, enabled DESC, id DESC"
|
||||
).fetchall()
|
||||
return jsonify({
|
||||
"items": [
|
||||
{
|
||||
"id": r["id"],
|
||||
"sourceType": r["source_type"],
|
||||
"name": r["name"],
|
||||
"finder": r["finder"] or "",
|
||||
"listUrl": r["list_url"] or "",
|
||||
"downloadDir": r["download_dir"] or "",
|
||||
"enabled": bool(r["enabled"]),
|
||||
"lastCrawledAt": r["last_crawled_at"],
|
||||
}
|
||||
for r in rows
|
||||
],
|
||||
})
|
||||
|
||||
|
||||
@app.post("/api/sources")
|
||||
def source_create():
|
||||
data = request.get_json(silent=True) or {}
|
||||
name = (data.get("name") or "").strip()
|
||||
source_type = data.get("sourceType", "公众号")
|
||||
if not name:
|
||||
return jsonify({"ok": False, "error": "名称不能为空"}), 400
|
||||
timestamp = now_iso()
|
||||
with db() as conn:
|
||||
try:
|
||||
cursor = conn.execute(
|
||||
"""
|
||||
INSERT INTO sources (source_type, name, finder, list_url, download_dir, enabled, created_at, updated_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
|
||||
""",
|
||||
(
|
||||
source_type, name,
|
||||
data.get("finder", ""), data.get("listUrl", ""), data.get("downloadDir", ""),
|
||||
int(data.get("enabled", True)), timestamp, timestamp,
|
||||
),
|
||||
)
|
||||
conn.commit()
|
||||
return jsonify({"ok": True, "id": cursor.lastrowid})
|
||||
except sqlite3.IntegrityError:
|
||||
return jsonify({"ok": False, "error": f"{source_type}「{name}」已存在"}), 409
|
||||
|
||||
|
||||
@app.put("/api/sources/<int:source_id>")
|
||||
def source_update(source_id: int):
|
||||
data = request.get_json(silent=True) or {}
|
||||
timestamp = now_iso()
|
||||
with db() as conn:
|
||||
row = conn.execute("SELECT * FROM sources WHERE id=?", (source_id,)).fetchone()
|
||||
if not row:
|
||||
return jsonify({"ok": False, "error": "信源不存在"}), 404
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE sources SET
|
||||
name=?, finder=?, list_url=?, download_dir=?, enabled=?, updated_at=?
|
||||
WHERE id=?
|
||||
""",
|
||||
(
|
||||
(data.get("name") or row["name"]).strip(),
|
||||
data.get("finder", row["finder"]),
|
||||
data.get("listUrl", row["list_url"]),
|
||||
data.get("downloadDir", row["download_dir"]),
|
||||
int(data.get("enabled", bool(row["enabled"]))),
|
||||
timestamp, source_id,
|
||||
),
|
||||
)
|
||||
conn.commit()
|
||||
return jsonify({"ok": True})
|
||||
|
||||
|
||||
@app.delete("/api/sources/<int:source_id>")
|
||||
def source_delete(source_id: int):
|
||||
with db() as conn:
|
||||
row = conn.execute("SELECT * FROM sources WHERE id=?", (source_id,)).fetchone()
|
||||
if not row:
|
||||
return jsonify({"ok": False, "error": "信源不存在"}), 404
|
||||
conn.execute("DELETE FROM sources WHERE id=?", (source_id,))
|
||||
conn.commit()
|
||||
return jsonify({"ok": True})
|
||||
|
||||
|
||||
@app.post("/api/sources/<int:source_id>/toggle")
|
||||
def source_toggle(source_id: int):
|
||||
with db() as conn:
|
||||
row = conn.execute("SELECT enabled FROM sources WHERE id=?", (source_id,)).fetchone()
|
||||
if not row:
|
||||
return jsonify({"ok": False, "error": "信源不存在"}), 404
|
||||
new_val = 0 if row["enabled"] else 1
|
||||
conn.execute("UPDATE sources SET enabled=?, updated_at=? WHERE id=?", (new_val, now_iso(), source_id))
|
||||
conn.commit()
|
||||
return jsonify({"ok": True, "enabled": bool(new_val)})
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
ensure_schema()
|
||||
port = int(os.getenv("PORT", "8765"))
|
||||
app.run(host="127.0.0.1", port=port, debug=True)
|
||||
@@ -0,0 +1,31 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
||||
<plist version="1.0">
|
||||
<dict>
|
||||
<key>Label</key>
|
||||
<string>com.cibank.daily-pipeline</string>
|
||||
<key>ProgramArguments</key>
|
||||
<array>
|
||||
<string>/usr/local/bin/python3</string>
|
||||
<string>/Users/freedak/Documents/AIDashboard/CIBank/backend/daily_pipeline.py</string>
|
||||
<string>daily</string>
|
||||
</array>
|
||||
<key>WorkingDirectory</key>
|
||||
<string>/Users/freedak/Documents/AIDashboard/CIBank/backend</string>
|
||||
<key>StartCalendarInterval</key>
|
||||
<dict>
|
||||
<key>Hour</key>
|
||||
<integer>2</integer>
|
||||
<key>Minute</key>
|
||||
<integer>0</integer>
|
||||
</dict>
|
||||
<key>ProcessType</key>
|
||||
<string>Background</string>
|
||||
<key>LowPriorityIO</key>
|
||||
<true/>
|
||||
<key>StandardOutPath</key>
|
||||
<string>/tmp/cibank-daily-pipeline.log</string>
|
||||
<key>StandardErrorPath</key>
|
||||
<string>/tmp/cibank-daily-pipeline-error.log</string>
|
||||
</dict>
|
||||
</plist>
|
||||
@@ -0,0 +1,569 @@
|
||||
#!/usr/bin/env python3
|
||||
"""采集管道：公众号历史文章抓取 + 视频号 ASR 转写。
|
||||
|
||||
合规采集策略：
|
||||
- 控制请求频率，每篇文章间隔 CRAWL_ARTICLE_PAUSE 秒（默认2秒）
|
||||
- 列表页滚动间隔 CRAWL_LIST_PAUSE 秒（默认1.5秒）
|
||||
- 最多滚动 CRAWL_MAX_SCROLLS 次（默认15次）
|
||||
- 回溯天数 CRAWL_LOOKBACK_DAYS（默认7天），避免全量抓取
|
||||
- 支持 Playwright 真实浏览器渲染，避免被反爬
|
||||
- 视频号内容通过 ASR API 转写为文本后入库
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sqlite3
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from contextlib import closing
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any, Iterable
|
||||
from urllib.parse import urljoin, urlsplit, urlunsplit
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
BASE_DIR = Path(__file__).resolve().parent
|
||||
load_dotenv(BASE_DIR / ".env")
|
||||
|
||||
DEFAULT_LIST_URL = "https://m.canyin88.com/zixun/"
|
||||
ARTICLE_PATH_RE = re.compile(r"/zixun/\d{4}/\d{1,2}/\d{1,2}/\d+\.html$")
|
||||
DATE_RE = re.compile(r"(20\d{2})[-/.年](\d{1,2})[-/.月](\d{1,2})(?:日)?(?:\s+(\d{1,2}):(\d{2})(?::(\d{2}))?)?")
|
||||
|
||||
|
||||
def now_iso() -> str:
|
||||
return datetime.now(timezone.utc).astimezone().isoformat(timespec="seconds")
|
||||
|
||||
|
||||
def normalize_url(raw_url: str, base_url: str = DEFAULT_LIST_URL) -> str | None:
|
||||
absolute = urljoin(base_url, raw_url.strip())
|
||||
parts = urlsplit(absolute)
|
||||
host = parts.netloc.lower().removeprefix("m.").removeprefix("www.")
|
||||
if host != "canyin88.com" or not ARTICLE_PATH_RE.search(parts.path):
|
||||
return None
|
||||
return urlunsplit(("https", "www.canyin88.com", parts.path, "", ""))
|
||||
|
||||
|
||||
def clean_text(value: str | None) -> str:
|
||||
return re.sub(r"\s+", " ", value or "").strip().lstrip("\ufeff")
|
||||
|
||||
|
||||
def parse_date(value: str | None) -> str | None:
|
||||
if not value:
|
||||
return None
|
||||
match = DATE_RE.search(value)
|
||||
if not match:
|
||||
return None
|
||||
year, month, day, hour, minute, second = match.groups()
|
||||
dt = datetime(
|
||||
int(year), int(month), int(day), int(hour or 0), int(minute or 0), int(second or 0)
|
||||
)
|
||||
return dt.isoformat(timespec="seconds")
|
||||
|
||||
|
||||
def date_from_article_url(url: str) -> datetime | None:
|
||||
match = re.search(r"/zixun/(20\d{2})/(\d{1,2})/(\d{1,2})/", url)
|
||||
if not match:
|
||||
return None
|
||||
return datetime(*(int(part) for part in match.groups()))
|
||||
|
||||
|
||||
def init_db(conn: sqlite3.Connection, schema_path: Path) -> None:
|
||||
conn.executescript(schema_path.read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
# ---- 频率控制器 ----
|
||||
|
||||
class RateLimiter:
|
||||
"""简单的频率控制器，确保请求间隔不低于指定秒数。"""
|
||||
|
||||
def __init__(self, min_interval: float):
|
||||
self.min_interval = min_interval
|
||||
self._last_time = 0.0
|
||||
|
||||
def wait(self) -> None:
|
||||
elapsed = time.time() - self._last_time
|
||||
if elapsed < self.min_interval:
|
||||
time.sleep(self.min_interval - elapsed)
|
||||
self._last_time = time.time()
|
||||
|
||||
|
||||
# ---- 公众号文章抓取 ----
|
||||
|
||||
def discover_urls(page, list_url: str, scrolls: int, pause_ms: int) -> list[str]:
|
||||
"""在列表页滚动发现文章URL，控制滚动频率。"""
|
||||
page.goto(list_url, wait_until="domcontentloaded", timeout=60_000)
|
||||
page.wait_for_timeout(2_000)
|
||||
found: set[str] = set()
|
||||
stagnant = 0
|
||||
for _ in range(scrolls + 1):
|
||||
hrefs = page.locator("a[href]").evaluate_all("els => els.map(e => e.href)")
|
||||
before = len(found)
|
||||
for href in hrefs:
|
||||
normalized = normalize_url(href, list_url)
|
||||
if normalized:
|
||||
found.add(normalized)
|
||||
stagnant = stagnant + 1 if len(found) == before else 0
|
||||
if stagnant >= 3:
|
||||
break
|
||||
page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
|
||||
page.wait_for_timeout(pause_ms)
|
||||
return sorted(found, reverse=True)
|
||||
|
||||
|
||||
def meta(page, selector: str) -> str:
|
||||
locator = page.locator(selector).first
|
||||
return clean_text(locator.get_attribute("content")) if locator.count() else ""
|
||||
|
||||
|
||||
def first_text(page, selectors: list[str]) -> str:
|
||||
for selector in selectors:
|
||||
locator = page.locator(selector).first
|
||||
if locator.count():
|
||||
text = clean_text(locator.inner_text(timeout=3_000))
|
||||
if text:
|
||||
return text
|
||||
return ""
|
||||
|
||||
|
||||
def extract_article(page, url: str) -> dict[str, str | None]:
|
||||
page.goto(url, wait_until="domcontentloaded", timeout=60_000)
|
||||
page.wait_for_timeout(800)
|
||||
document_title = clean_text(page.title())
|
||||
document_title = re.sub(r"(?:[_-]红餐网.*)$", "", document_title).strip()
|
||||
title = meta(page, 'meta[property="og:title"]') or document_title
|
||||
if not title or title == "相关推荐":
|
||||
title = first_text(page, [".title", "h1"])
|
||||
summary = (
|
||||
meta(page, 'meta[name="description"]')
|
||||
or meta(page, 'meta[property="og:description"]')
|
||||
)
|
||||
author = (
|
||||
meta(page, 'meta[name="author"]')
|
||||
or first_text(page, [".author", ".source", "[class*=author]"])
|
||||
)
|
||||
published_raw = (
|
||||
meta(page, 'meta[property="article:published_time"]')
|
||||
or first_text(page, ["time", ".time", "[class*=time]", "[class*=date]"])
|
||||
or page.locator("body").inner_text(timeout=5_000)[:1000]
|
||||
)
|
||||
content = first_text(
|
||||
page,
|
||||
[
|
||||
"article",
|
||||
".article-content",
|
||||
".article_content",
|
||||
".content",
|
||||
"[class*=detail-content]",
|
||||
"[class*=article] [class*=content]",
|
||||
],
|
||||
)
|
||||
if not content:
|
||||
paragraphs = page.locator("p").all_inner_texts()
|
||||
content = "\n\n".join(clean_text(p) for p in paragraphs if len(clean_text(p)) >= 20)
|
||||
if not title or len(content) < 80:
|
||||
raise ValueError(f"article extraction incomplete: title={bool(title)}, content_length={len(content)}")
|
||||
published_at = parse_date(published_raw)
|
||||
url_date = date_from_article_url(url)
|
||||
if url_date and (
|
||||
not published_at or datetime.fromisoformat(published_at).date() != url_date.date()
|
||||
):
|
||||
published_at = url_date.isoformat(timespec="seconds")
|
||||
content_hash = hashlib.sha256(f"{title}\n{content}".encode("utf-8")).hexdigest()
|
||||
return {
|
||||
"canonical_url": url,
|
||||
"title": title,
|
||||
"author": author or None,
|
||||
"published_at": published_at,
|
||||
"summary": summary or None,
|
||||
"content": content,
|
||||
"content_hash": content_hash,
|
||||
}
|
||||
|
||||
|
||||
def pending_urls(conn: sqlite3.Connection, discovered: list[str], max_retries: int) -> list[str]:
|
||||
rows = conn.execute(
|
||||
"SELECT canonical_url FROM articles WHERE crawl_status != 'success' AND retry_count < ?",
|
||||
(max_retries,),
|
||||
).fetchall()
|
||||
return list(dict.fromkeys(discovered + [row[0] for row in rows]))
|
||||
|
||||
|
||||
def insert_discoveries(conn: sqlite3.Connection, urls: list[str], source_type: str = "公众号") -> int:
|
||||
timestamp = now_iso()
|
||||
inserted = 0
|
||||
for url in urls:
|
||||
cursor = conn.execute(
|
||||
"""
|
||||
INSERT OR IGNORE INTO articles
|
||||
(canonical_url, source_type, discovered_at, updated_at)
|
||||
VALUES (?, ?, ?, ?)
|
||||
""",
|
||||
(url, source_type, timestamp, timestamp),
|
||||
)
|
||||
inserted += cursor.rowcount
|
||||
conn.commit()
|
||||
return inserted
|
||||
|
||||
|
||||
def save_success(conn: sqlite3.Connection, article: dict[str, str | None]) -> None:
|
||||
timestamp = now_iso()
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE articles SET
|
||||
title = ?, author = ?, published_at = ?, summary = ?, content = ?,
|
||||
content_hash = ?, crawl_status = 'success', last_error = NULL,
|
||||
crawled_at = ?, updated_at = ?
|
||||
WHERE canonical_url = ?
|
||||
""",
|
||||
(
|
||||
article["title"], article["author"], article["published_at"], article["summary"],
|
||||
article["content"], article["content_hash"], timestamp, timestamp,
|
||||
article["canonical_url"],
|
||||
),
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
|
||||
def save_failure(conn: sqlite3.Connection, url: str, error: Exception) -> None:
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE articles SET
|
||||
crawl_status = 'failed', retry_count = retry_count + 1,
|
||||
last_error = ?, updated_at = ?
|
||||
WHERE canonical_url = ?
|
||||
""",
|
||||
(str(error)[:1000], now_iso(), url),
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
|
||||
# ---- 视频号本地处理（参考 VIBank 方案） ----
|
||||
|
||||
VIDEO_SUFFIXES = {".mp4", ".mov", ".m4v", ".mkv", ".webm"}
|
||||
FFMPEG = os.getenv("FFMPEG_PATH", "/usr/local/bin/ffmpeg")
|
||||
FFPROBE = os.getenv("FFPROBE_PATH", "/usr/local/bin/ffprobe")
|
||||
FUNASR = os.getenv("FUNASR_PATH", "/usr/local/bin/funasr")
|
||||
ASR_MODEL = os.getenv("ASR_MODEL", "sensevoice")
|
||||
ASR_HOTWORDS = os.getenv("ASR_HOTWORDS", "餐饮,餐饮连锁,连锁餐厅,单店模型,供应链,选址,复购,翻台率,客单价,毛利率,净利率,加盟,直营")
|
||||
|
||||
|
||||
def load_video_config() -> dict[str, Any] | None:
|
||||
"""加载视频源配置：优先从数据库 sources 表读取，降级到 JSON 文件。"""
|
||||
# 优先从数据库读取
|
||||
db_path = BASE_DIR / "data" / "cibank.db"
|
||||
if db_path.exists():
|
||||
try:
|
||||
with sqlite3.connect(db_path) as conn:
|
||||
rows = conn.execute(
|
||||
"SELECT name, finder, download_dir FROM sources WHERE source_type='视频号' AND enabled=1"
|
||||
).fetchall()
|
||||
if rows:
|
||||
return {
|
||||
"sources": [
|
||||
{"account": r[0], "finder": r[1] or "", "directory": r[2] or ""}
|
||||
for r in rows
|
||||
]
|
||||
}
|
||||
except sqlite3.Error:
|
||||
pass
|
||||
|
||||
# 降级到 JSON 配置文件
|
||||
config_path = BASE_DIR / "video_config.json"
|
||||
if config_path.exists():
|
||||
return json.loads(config_path.read_text(encoding="utf-8"))
|
||||
legacy = BASE_DIR / "video_sources.json"
|
||||
if legacy.exists():
|
||||
return {"sources": [], "legacy_urls": json.loads(legacy.read_text(encoding="utf-8")).get("videos", [])}
|
||||
return None
|
||||
|
||||
|
||||
def file_fingerprint(path: Path) -> str:
|
||||
stat = path.stat()
|
||||
h = hashlib.sha256()
|
||||
h.update(f"{stat.st_size}:{stat.st_mtime_ns}".encode())
|
||||
with path.open("rb") as f:
|
||||
h.update(f.read(1024 * 1024))
|
||||
if stat.st_size > 2 * 1024 * 1024:
|
||||
f.seek(max(0, stat.st_size - 1024 * 1024))
|
||||
h.update(f.read(1024 * 1024))
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def safe_stem(name: str) -> str:
|
||||
value = re.sub(r"[_ ]?xWT\d+$", "", Path(name).stem, flags=re.I)
|
||||
value = re.sub(r"[\x00-\x1f/:]", "_", value).strip(" .")
|
||||
return value[:180] or "untitled"
|
||||
|
||||
|
||||
def ffprobe_duration(path: Path) -> float:
|
||||
result = subprocess.run(
|
||||
[FFPROBE, "-v", "error", "-show_entries", "format=duration",
|
||||
"-of", "default=noprint_wrappers=1:nokey=1", str(path)],
|
||||
text=True, capture_output=True, check=True, timeout=30,
|
||||
)
|
||||
return float(result.stdout.strip())
|
||||
|
||||
|
||||
def extract_audio(video: Path, audio_dir: Path) -> Path:
|
||||
"""用 FFmpeg 从视频中提取 16kHz 单声道 WAV。"""
|
||||
audio_dir.mkdir(parents=True, exist_ok=True)
|
||||
fingerprint = file_fingerprint(video)[:10]
|
||||
wav = audio_dir / f"{safe_stem(video.name)}_{fingerprint}.wav"
|
||||
if wav.is_file() and wav.stat().st_size > 44:
|
||||
return wav
|
||||
partial = wav.with_name(wav.name + ".part")
|
||||
partial.unlink(missing_ok=True)
|
||||
try:
|
||||
subprocess.run(
|
||||
[FFMPEG, "-hide_banner", "-loglevel", "error", "-y",
|
||||
"-i", str(video), "-vn", "-ac", "1", "-ar", "16000", "-c:a", "pcm_s16le", str(partial)],
|
||||
check=True, timeout=1800,
|
||||
)
|
||||
partial.replace(wav)
|
||||
return wav
|
||||
finally:
|
||||
partial.unlink(missing_ok=True)
|
||||
|
||||
|
||||
def funasr_transcribe(wav_path: Path) -> str:
|
||||
"""用 FunASR/SenseVoice 进行本地中文转写。"""
|
||||
cmd = [FUNASR, str(wav_path), "--model", ASR_MODEL, "--language", "zh", "--output-format", "json"]
|
||||
if ASR_HOTWORDS:
|
||||
cmd.extend(["--hotwords", ASR_HOTWORDS])
|
||||
result = subprocess.run(cmd, text=True, capture_output=True, check=True, timeout=3600)
|
||||
start = result.stdout.find("{")
|
||||
if start < 0:
|
||||
raise RuntimeError("FunASR 未返回 JSON")
|
||||
payload = json.loads(result.stdout[start:])
|
||||
text = re.sub(r"<\|[^>]+\|>", "", str(payload.get("text", ""))).strip()
|
||||
if not text:
|
||||
raise RuntimeError("FunASR 返回空文本")
|
||||
return text
|
||||
|
||||
|
||||
def discover_local_videos(config: dict[str, Any]) -> Iterable[tuple[dict[str, str], Path]]:
|
||||
"""扫描配置中的视频目录，发现本地视频文件。"""
|
||||
for source in config.get("sources", []):
|
||||
root = Path(source["directory"])
|
||||
if not root.is_dir():
|
||||
print(f"WARN 视频源目录不存在: {root}", file=sys.stderr)
|
||||
continue
|
||||
for path in sorted(root.iterdir()):
|
||||
if path.is_file() and path.suffix.lower() in VIDEO_SUFFIXES and not path.name.startswith("."):
|
||||
yield source, path
|
||||
|
||||
|
||||
def process_local_video(source: dict[str, str], video: Path, audio_root: Path) -> dict[str, str | None | float]:
|
||||
"""完整处理一个本地视频：提取音频 → ASR 转写 → 返回结构化数据。"""
|
||||
duration = ffprobe_duration(video)
|
||||
account = source.get("account", "未知视频号")
|
||||
audio_dir = audio_root / re.sub(r"[/:]", "_", account)
|
||||
wav = extract_audio(video, audio_dir)
|
||||
transcript = funasr_transcribe(wav)
|
||||
title = safe_stem(video.name)
|
||||
content_hash = hashlib.sha256(f"{title}\n{transcript}".encode("utf-8")).hexdigest()
|
||||
media_url = str(video.resolve())
|
||||
media_key = hashlib.sha256(media_url.encode("utf-8")).hexdigest()[:24]
|
||||
return {
|
||||
"canonical_url": f"video://local/{media_key}",
|
||||
"media_url": media_url,
|
||||
"title": title,
|
||||
"content": transcript,
|
||||
"summary": transcript[:200] if transcript else "",
|
||||
"author": account,
|
||||
"published_at": now_iso(),
|
||||
"content_hash": content_hash,
|
||||
"duration": duration,
|
||||
}
|
||||
|
||||
|
||||
# ---- 主采集流程 ----
|
||||
|
||||
def run_crawl(args: argparse.Namespace) -> int:
|
||||
db_path = Path(args.db).expanduser().resolve()
|
||||
db_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
article_pause = float(os.getenv("CRAWL_ARTICLE_PAUSE", "2.0"))
|
||||
list_pause_ms = int(float(os.getenv("CRAWL_LIST_PAUSE", "1.5")) * 1000)
|
||||
max_scrolls = int(os.getenv("CRAWL_MAX_SCROLLS", "15"))
|
||||
lookback_days = int(os.getenv("CRAWL_LOOKBACK_DAYS", "7"))
|
||||
|
||||
article_limiter = RateLimiter(article_pause)
|
||||
|
||||
with closing(sqlite3.connect(db_path)) as conn:
|
||||
init_db(conn, BASE_DIR / "schema.sql")
|
||||
started_at = now_iso()
|
||||
run_id = conn.execute(
|
||||
"INSERT INTO crawl_runs(started_at, source_type) VALUES (?, ?)",
|
||||
(started_at, args.source_type),
|
||||
).lastrowid
|
||||
conn.commit()
|
||||
|
||||
discovered_count = inserted_count = updated_count = failed_count = 0
|
||||
try:
|
||||
if args.source_type == "公众号":
|
||||
# 从数据库读取启用的公众号信源，降级到命令行参数
|
||||
mp_sources = conn.execute(
|
||||
"SELECT name, list_url FROM sources WHERE source_type='公众号' AND enabled=1 AND list_url!=''"
|
||||
).fetchall()
|
||||
if mp_sources:
|
||||
list_urls = [(r[0], r[1]) for r in mp_sources]
|
||||
else:
|
||||
list_urls = [("默认", args.list_url)]
|
||||
|
||||
from playwright.sync_api import sync_playwright
|
||||
with sync_playwright() as playwright:
|
||||
browser = playwright.chromium.launch(headless=not args.headed)
|
||||
context = browser.new_context(
|
||||
locale="zh-CN",
|
||||
timezone_id="Asia/Shanghai",
|
||||
viewport={"width": 1280, "height": 900},
|
||||
user_agent=(
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
|
||||
"AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36"
|
||||
),
|
||||
)
|
||||
context.set_default_timeout(15_000)
|
||||
page = context.new_page()
|
||||
try:
|
||||
for source_name, list_url in list_urls:
|
||||
print(f"--- 采集公众号: {source_name} ({list_url}) ---", flush=True)
|
||||
urls = discover_urls(page, list_url, max_scrolls, list_pause_ms)
|
||||
discovered_count += len(urls)
|
||||
inserted_count += insert_discoveries(conn, urls, source_name)
|
||||
queue = pending_urls(conn, urls, args.max_retries)
|
||||
cutoff = datetime.now() - timedelta(days=lookback_days)
|
||||
for index, url in enumerate(queue, start=1):
|
||||
path_date = parse_date(url.replace("/", "-"))
|
||||
if path_date and datetime.fromisoformat(path_date) < cutoff:
|
||||
continue
|
||||
article_limiter.wait()
|
||||
try:
|
||||
article = extract_article(page, url)
|
||||
save_success(conn, article)
|
||||
updated_count += 1
|
||||
print(f"[{source_name} {index}/{len(queue)}] OK {article['title']}", flush=True)
|
||||
except Exception as exc:
|
||||
failed_count += 1
|
||||
save_failure(conn, url, exc)
|
||||
print(f"[{source_name} {index}/{len(queue)}] FAIL {url}: {exc}", file=sys.stderr, flush=True)
|
||||
# 更新信源的最近采集时间
|
||||
conn.execute(
|
||||
"UPDATE sources SET last_crawled_at=? WHERE source_type='公众号' AND name=?",
|
||||
(now_iso(), source_name),
|
||||
)
|
||||
conn.commit()
|
||||
finally:
|
||||
context.close()
|
||||
elif args.source_type == '视频号':
|
||||
# 视频号本地处理：扫描本地视频目录 → FFmpeg 提取音频 → FunASR 转写
|
||||
video_config = load_video_config()
|
||||
if not video_config:
|
||||
print(json.dumps({"status": "skipped", "reason": "未找到 video_config.json 或 video_sources.json"}, ensure_ascii=False))
|
||||
conn.execute(
|
||||
"UPDATE crawl_runs SET finished_at=?, status='skipped', message=? WHERE id=?",
|
||||
(now_iso(), "未配置视频源", run_id),
|
||||
)
|
||||
conn.commit()
|
||||
return 0
|
||||
|
||||
audio_root = Path(os.getenv("VIDEO_AUDIO_ROOT", str(BASE_DIR / "data" / "audio")))
|
||||
video_files = list(discover_local_videos(video_config))
|
||||
discovered_count = len(video_files)
|
||||
|
||||
for index, (source, video) in enumerate(video_files, start=1):
|
||||
media_url = str(video.resolve())
|
||||
existing_media = conn.execute(
|
||||
"SELECT id FROM articles WHERE media_url=?", (media_url,)
|
||||
).fetchone()
|
||||
if existing_media:
|
||||
print(f"[{index}/{discovered_count}] SKIP {video.stem}（已入库）", flush=True)
|
||||
continue
|
||||
article_limiter.wait()
|
||||
try:
|
||||
result = process_local_video(source, video, audio_root)
|
||||
timestamp = now_iso()
|
||||
existing = conn.execute(
|
||||
"SELECT id, content_hash FROM articles WHERE canonical_url=?",
|
||||
(result["canonical_url"],),
|
||||
).fetchone()
|
||||
if existing and existing[1] == result["content_hash"]:
|
||||
print(f"[{index}/{discovered_count}] SKIP {result['title']}（已入库）", flush=True)
|
||||
continue
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO articles
|
||||
(canonical_url, source, source_type, source_name, title, author,
|
||||
published_at, summary, content, content_hash, media_url,
|
||||
crawl_status, discovered_at, crawled_at, updated_at)
|
||||
VALUES (?, 'video_local', '视频号', ?, ?, ?, ?, ?, ?, ?, ?, 'success', ?, ?, ?)
|
||||
ON CONFLICT(canonical_url) DO UPDATE SET
|
||||
title=excluded.title, summary=excluded.summary, content=excluded.content,
|
||||
content_hash=excluded.content_hash, media_url=excluded.media_url,
|
||||
crawled_at=excluded.crawled_at, updated_at=excluded.updated_at,
|
||||
crawl_status='success', last_error=NULL
|
||||
""",
|
||||
(
|
||||
result["canonical_url"], source.get("account", ""),
|
||||
result["title"], result["author"], result["published_at"],
|
||||
result["summary"], result["content"], result["content_hash"],
|
||||
result["media_url"], timestamp, timestamp, timestamp,
|
||||
),
|
||||
)
|
||||
conn.commit()
|
||||
updated_count += 1
|
||||
print(f"[{index}/{discovered_count}] OK {result['title']}（{result.get('duration', 0):.0f}s）", flush=True)
|
||||
# 更新信源最近采集时间
|
||||
conn.execute(
|
||||
"UPDATE sources SET last_crawled_at=? WHERE source_type='视频号' AND name=?",
|
||||
(now_iso(), source.get("account", "")),
|
||||
)
|
||||
conn.commit()
|
||||
except Exception as exc:
|
||||
failed_count += 1
|
||||
print(f"[{index}/{discovered_count}] FAIL {video.name}: {exc}", file=sys.stderr, flush=True)
|
||||
|
||||
status = "success" if failed_count == 0 else "partial"
|
||||
message = json.dumps({"db": str(db_path)}, ensure_ascii=False)
|
||||
except Exception as exc:
|
||||
status, message = "failed", str(exc)[:1000]
|
||||
print(f"crawl failed: {exc}", file=sys.stderr)
|
||||
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE crawl_runs SET finished_at=?, status=?, discovered_count=?,
|
||||
inserted_count=?, updated_count=?, failed_count=?, message=?
|
||||
WHERE id=?
|
||||
""",
|
||||
(now_iso(), status, discovered_count, inserted_count, updated_count, failed_count, message, run_id),
|
||||
)
|
||||
conn.commit()
|
||||
print(json.dumps({
|
||||
"status": status, "source_type": args.source_type,
|
||||
"discovered": discovered_count, "inserted": inserted_count,
|
||||
"updated": updated_count, "failed": failed_count,
|
||||
}, ensure_ascii=False, flush=True))
|
||||
return 0 if status in {"success", "partial"} else 1
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(description="餐饮内容采集管道")
|
||||
parser.add_argument("--db", default=str(BASE_DIR / "data" / "cibank.db"), help="SQLite 数据库路径")
|
||||
parser.add_argument("--list-url", default=DEFAULT_LIST_URL, help="公众号列表页URL")
|
||||
parser.add_argument("--source-type", choices=["公众号", "视频号"], default="公众号", help="采集来源类型")
|
||||
parser.add_argument("--max-retries", type=int, default=3)
|
||||
parser.add_argument("--headed", action="store_true", help="显示浏览器（调试用）")
|
||||
return parser
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(run_crawl(build_parser().parse_args()))
|
||||
@@ -0,0 +1,198 @@
|
||||
#!/usr/bin/env python3
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import fcntl
|
||||
import json
|
||||
import os
|
||||
import sqlite3
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
BASE_DIR = Path(__file__).resolve().parent
|
||||
load_dotenv(BASE_DIR / ".env")
|
||||
DATA_DIR = BASE_DIR / "data"
|
||||
DB_PATH = Path(os.getenv("CIBANK_DB", DATA_DIR / "cibank.db"))
|
||||
LOCK_PATH = DATA_DIR / "pipeline.lock"
|
||||
LOG_DIR = DATA_DIR / "logs"
|
||||
BACKUP_DIR = DATA_DIR / "backups"
|
||||
SCHEMA_PATH = BASE_DIR / "schema.sql"
|
||||
|
||||
|
||||
def now_iso() -> str:
|
||||
return datetime.now().astimezone().isoformat(timespec="seconds")
|
||||
|
||||
|
||||
def open_db() -> sqlite3.Connection:
|
||||
conn = sqlite3.connect(DB_PATH)
|
||||
conn.row_factory = sqlite3.Row
|
||||
conn.executescript(SCHEMA_PATH.read_text(encoding="utf-8"))
|
||||
return conn
|
||||
|
||||
|
||||
def acquire_lock():
|
||||
DATA_DIR.mkdir(parents=True, exist_ok=True)
|
||||
lock_file = LOCK_PATH.open("a+")
|
||||
try:
|
||||
fcntl.flock(lock_file.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
except BlockingIOError as exc:
|
||||
lock_file.close()
|
||||
raise RuntimeError("已有流水线任务正在运行") from exc
|
||||
lock_file.seek(0)
|
||||
lock_file.truncate()
|
||||
lock_file.write(f"{os.getpid()}\n{now_iso()}\n")
|
||||
lock_file.flush()
|
||||
return lock_file
|
||||
|
||||
|
||||
def backup_database() -> str:
|
||||
BACKUP_DIR.mkdir(parents=True, exist_ok=True)
|
||||
backup_path = BACKUP_DIR / f"cibank-{datetime.now().strftime('%Y%m%d-%H%M%S')}.db"
|
||||
source = sqlite3.connect(DB_PATH)
|
||||
target = sqlite3.connect(backup_path)
|
||||
try:
|
||||
source.backup(target)
|
||||
finally:
|
||||
target.close()
|
||||
source.close()
|
||||
backups = sorted(BACKUP_DIR.glob("cibank-*.db"), key=lambda path: path.stat().st_mtime, reverse=True)
|
||||
for expired in backups[14:]:
|
||||
expired.unlink()
|
||||
return str(backup_path)
|
||||
|
||||
|
||||
def create_run(run_type: str) -> int:
|
||||
timestamp = now_iso()
|
||||
with open_db() as conn:
|
||||
cursor = conn.execute(
|
||||
"""
|
||||
INSERT INTO pipeline_runs
|
||||
(run_type, started_at, status, current_step, steps_json, created_at)
|
||||
VALUES (?, ?, 'running', 'preflight', '{}', ?)
|
||||
""",
|
||||
(run_type, timestamp, timestamp),
|
||||
)
|
||||
conn.commit()
|
||||
return cursor.lastrowid
|
||||
|
||||
|
||||
def update_run(run_id: int, **values) -> None:
|
||||
allowed = {"finished_at", "status", "current_step", "steps_json", "error_message"}
|
||||
fields = [(key, value) for key, value in values.items() if key in allowed]
|
||||
if not fields:
|
||||
return
|
||||
assignments = ", ".join(f"{key}=?" for key, _ in fields)
|
||||
with open_db() as conn:
|
||||
conn.execute(
|
||||
f"UPDATE pipeline_runs SET {assignments} WHERE id=?",
|
||||
[value for _, value in fields] + [run_id],
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
|
||||
def run_step(run_id: int, name: str, arguments: list[str], log_file, steps: dict) -> bool:
|
||||
update_run(run_id, current_step=name)
|
||||
started = time.monotonic()
|
||||
log_file.write(f"\n[{now_iso()}] START {name}: {' '.join(arguments)}\n")
|
||||
log_file.flush()
|
||||
env = os.environ.copy()
|
||||
env["PATH"] = "/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:" + env.get("PATH", "")
|
||||
result = subprocess.run(
|
||||
[sys.executable, *arguments],
|
||||
cwd=BASE_DIR,
|
||||
env=env,
|
||||
stdout=log_file,
|
||||
stderr=subprocess.STDOUT,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
duration = round(time.monotonic() - started, 2)
|
||||
steps[name] = {"status": "success" if result.returncode == 0 else "failed", "returncode": result.returncode, "duration": duration}
|
||||
update_run(run_id, steps_json=json.dumps(steps, ensure_ascii=False))
|
||||
log_file.write(f"[{now_iso()}] END {name}: code={result.returncode}, duration={duration}s\n")
|
||||
log_file.flush()
|
||||
return result.returncode == 0
|
||||
|
||||
|
||||
def validate_environment() -> None:
|
||||
if not DB_PATH.exists():
|
||||
raise RuntimeError(f"数据库不存在：{DB_PATH}")
|
||||
api_key = os.getenv("DASHSCOPE_API_KEY", "").strip()
|
||||
if not api_key or api_key == "replace_with_your_key":
|
||||
raise RuntimeError("DASHSCOPE_API_KEY 未配置")
|
||||
with sqlite3.connect(DB_PATH) as conn:
|
||||
result = conn.execute("PRAGMA integrity_check").fetchone()[0]
|
||||
if result != "ok":
|
||||
raise RuntimeError(f"数据库完整性检查失败：{result}")
|
||||
|
||||
|
||||
def show_status() -> int:
|
||||
with open_db() as conn:
|
||||
row = conn.execute("SELECT * FROM pipeline_runs ORDER BY id DESC LIMIT 1").fetchone()
|
||||
print(json.dumps(dict(row) if row else {"status": "never_run"}, ensure_ascii=False, indent=2))
|
||||
return 0
|
||||
|
||||
|
||||
def run_pipeline(run_type: str, skip_crawl: bool) -> int:
|
||||
lock_file = acquire_lock()
|
||||
run_id = None
|
||||
LOG_DIR.mkdir(parents=True, exist_ok=True)
|
||||
log_path = LOG_DIR / f"pipeline-{datetime.now().strftime('%Y%m%d-%H%M%S')}.log"
|
||||
try:
|
||||
validate_environment()
|
||||
run_id = create_run(run_type)
|
||||
steps = {"backup": {"status": "success", "path": backup_database()}}
|
||||
update_run(run_id, steps_json=json.dumps(steps, ensure_ascii=False))
|
||||
commands = []
|
||||
if not skip_crawl:
|
||||
commands.extend([
|
||||
("crawl_mp", ["crawler.py", "--source-type", "公众号"]),
|
||||
("crawl_video", ["crawler.py", "--source-type", "视频号"]),
|
||||
])
|
||||
analyze_args = ["structurer.py", "analyze", "--pause", "0.6", "--retries", "3"]
|
||||
embed_args = ["structurer.py", "embed"]
|
||||
if run_type == "full":
|
||||
analyze_args.append("--force")
|
||||
embed_args.append("--force")
|
||||
commands.extend([
|
||||
("analyze", analyze_args),
|
||||
("embed", embed_args),
|
||||
("signals", ["structurer.py", "signals", "--days", "30"]),
|
||||
])
|
||||
all_success = True
|
||||
with log_path.open("a", encoding="utf-8") as log_file:
|
||||
log_file.write(f"[{now_iso()}] pipeline run_id={run_id}, type={run_type}\n")
|
||||
for name, arguments in commands:
|
||||
if not run_step(run_id, name, arguments, log_file, steps):
|
||||
all_success = False
|
||||
status = "success" if all_success else "partial"
|
||||
update_run(run_id, finished_at=now_iso(), status=status, current_step="completed", steps_json=json.dumps(steps, ensure_ascii=False))
|
||||
print(json.dumps({"run_id": run_id, "status": status, "log": str(log_path), "steps": steps}, ensure_ascii=False, indent=2))
|
||||
return 0 if all_success else 1
|
||||
except Exception as exc:
|
||||
if run_id is not None:
|
||||
update_run(run_id, finished_at=now_iso(), status="failed", current_step="failed", error_message=str(exc))
|
||||
print(json.dumps({"status": "failed", "error": str(exc), "log": str(log_path)}, ensure_ascii=False), file=sys.stderr)
|
||||
return 1
|
||||
finally:
|
||||
fcntl.flock(lock_file.fileno(), fcntl.LOCK_UN)
|
||||
lock_file.close()
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description="CIBank 自动增量处理流水线")
|
||||
parser.add_argument("command", choices=["daily", "full", "status"])
|
||||
parser.add_argument("--skip-crawl", action="store_true")
|
||||
args = parser.parse_args()
|
||||
if args.command == "status":
|
||||
return show_status()
|
||||
return run_pipeline(args.command, args.skip_crawl)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
+333
@@ -0,0 +1,333 @@
|
||||
#!/usr/bin/env python3
|
||||
"""RAG 问答模块：向量检索 + LLM 生成。
|
||||
|
||||
流程：
|
||||
1. 将用户问题转为向量
|
||||
2. 在 embeddings 表中做余弦相似度检索，取 TOP-K 相关文章
|
||||
3. 将检索到的文章内容作为上下文，调用千问生成回答
|
||||
4. 返回回答 + 引用来源（可追溯）
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import sqlite3
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from structurer import get_api_key, get_embeddings, DASHSCOPE_CHAT_URL
|
||||
|
||||
BASE_DIR = Path(__file__).resolve().parent
|
||||
load_dotenv(BASE_DIR / ".env")
|
||||
|
||||
|
||||
def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
|
||||
"""计算两个向量的余弦相似度。"""
|
||||
norm_a = np.linalg.norm(a)
|
||||
norm_b = np.linalg.norm(b)
|
||||
if norm_a == 0 or norm_b == 0:
|
||||
return 0.0
|
||||
return float(np.dot(a, b) / (norm_a * norm_b))
|
||||
|
||||
|
||||
def vector_search(conn: sqlite3.Connection, query_embedding: list[float], top_k: int = 5) -> list[dict]:
|
||||
"""向量检索：在 embeddings 表中找到与查询向量最相似的文章。"""
|
||||
query_vec = np.array(query_embedding, dtype=np.float32)
|
||||
rows = conn.execute(
|
||||
"""
|
||||
SELECT e.article_id, e.embedding, e.chunk_text,
|
||||
a.title, a.source_name, a.source_type, a.published_at, a.summary,
|
||||
x.result_json
|
||||
FROM embeddings e
|
||||
JOIN articles a ON a.id = e.article_id
|
||||
LEFT JOIN ai_analyses x ON x.id = (
|
||||
SELECT x2.id FROM ai_analyses x2
|
||||
WHERE x2.article_id = e.article_id AND x2.analysis_type = 'editorial'
|
||||
ORDER BY x2.updated_at DESC, x2.id DESC LIMIT 1
|
||||
)
|
||||
WHERE a.crawl_status = 'success'
|
||||
"""
|
||||
).fetchall()
|
||||
|
||||
scored = []
|
||||
for row in rows:
|
||||
emb = json.loads(row["embedding"])
|
||||
emb_vec = np.array(emb, dtype=np.float32)
|
||||
score = cosine_similarity(query_vec, emb_vec)
|
||||
scored.append((score, row))
|
||||
scored.sort(key=lambda x: x[0], reverse=True)
|
||||
|
||||
results = []
|
||||
seen_articles = set()
|
||||
for score, row in scored:
|
||||
if row["article_id"] in seen_articles:
|
||||
continue
|
||||
seen_articles.add(row["article_id"])
|
||||
analysis = json.loads(row["result_json"]) if row["result_json"] else {}
|
||||
results.append({
|
||||
"article_id": row["article_id"],
|
||||
"title": row["title"],
|
||||
"source": row["source_name"] or row["source_type"],
|
||||
"date": (row["published_at"] or "")[:10],
|
||||
"summary": analysis.get("summary") or row["summary"] or "",
|
||||
"key_points": analysis.get("key_points", []),
|
||||
"content": row["chunk_text"] or "",
|
||||
"score": score,
|
||||
})
|
||||
if len(results) >= top_k:
|
||||
break
|
||||
return results
|
||||
|
||||
|
||||
def generate_answer(question: str, contexts: list[dict]) -> dict:
|
||||
"""基于检索到的上下文，调用千问生成回答。"""
|
||||
api_key = get_api_key()
|
||||
model = os.getenv("QWEN_MODEL", "qwen-plus")
|
||||
|
||||
# 构建上下文
|
||||
context_text = ""
|
||||
sources = []
|
||||
for i, ctx in enumerate(contexts, start=1):
|
||||
context_text += f"\n--- 来源{i} ---\n标题：{ctx['title']}\n摘要：{ctx['summary']}\n要点：{'; '.join(ctx['key_points'][:3])}\n原文片段：{ctx['content']}\n"
|
||||
sources.append({
|
||||
"title": ctx["title"],
|
||||
"source": ctx["source"],
|
||||
"date": ctx["date"],
|
||||
})
|
||||
|
||||
prompt = f"""你是餐饮行业知识库的问答助手。根据以下知识库中的情报内容回答用户问题。
|
||||
|
||||
要求：
|
||||
1. 回答必须基于提供的来源内容，不要编造信息
|
||||
2. 如果来源内容不足以回答问题，明确说明"知识库中暂无直接相关信息"
|
||||
3. 回答要结构化、简洁、有经营参考价值
|
||||
4. 用中文回答
|
||||
|
||||
知识库来源：
|
||||
{context_text}
|
||||
|
||||
用户问题：{question}"""
|
||||
|
||||
payload = json.dumps({
|
||||
"model": model,
|
||||
"messages": [
|
||||
{"role": "system", "content": "你是餐饮行业知识库问答助手，基于库内情报给出有出处的专业回答。"},
|
||||
{"role": "user", "content": prompt},
|
||||
],
|
||||
"temperature": 0.3,
|
||||
}, ensure_ascii=False).encode("utf-8")
|
||||
|
||||
req = urllib.request.Request(
|
||||
DASHSCOPE_CHAT_URL,
|
||||
data=payload,
|
||||
headers={"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
|
||||
method="POST",
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=90) as response:
|
||||
body = json.loads(response.read().decode("utf-8"))
|
||||
except urllib.error.HTTPError as exc:
|
||||
detail = exc.read().decode("utf-8", errors="replace")[:500]
|
||||
raise RuntimeError(f"千问接口返回 {exc.code}：{detail}") from exc
|
||||
|
||||
answer = body["choices"][0]["message"]["content"].strip()
|
||||
return {
|
||||
"answer": answer,
|
||||
"sources": sources,
|
||||
"contexts": contexts,
|
||||
}
|
||||
|
||||
|
||||
def generate_answer_stream(question: str, contexts: list[dict]):
|
||||
"""流式生成回答，逐 token yield。"""
|
||||
api_key = get_api_key()
|
||||
model = os.getenv("QWEN_MODEL", "qwen-plus")
|
||||
|
||||
context_text = ""
|
||||
sources = []
|
||||
for i, ctx in enumerate(contexts, start=1):
|
||||
context_text += f"\n--- 来源{i} ---\n标题：{ctx['title']}\n摘要：{ctx['summary']}\n要点：{'; '.join(ctx['key_points'][:3])}\n原文片段：{ctx['content']}\n"
|
||||
sources.append({
|
||||
"title": ctx["title"],
|
||||
"source": ctx["source"],
|
||||
"date": ctx["date"],
|
||||
})
|
||||
|
||||
prompt = f"""你是餐饮行业知识库的问答助手。根据以下知识库中的情报内容回答用户问题。
|
||||
|
||||
要求：
|
||||
1. 回答必须基于提供的来源内容，不要编造信息
|
||||
2. 如果来源内容不足以回答问题，明确说明"知识库中暂无直接相关信息"
|
||||
3. 回答要结构化、简洁、有经营参考价值
|
||||
4. 用中文回答，使用 Markdown 格式
|
||||
|
||||
知识库来源：
|
||||
{context_text}
|
||||
|
||||
用户问题：{question}"""
|
||||
|
||||
payload = json.dumps({
|
||||
"model": model,
|
||||
"messages": [
|
||||
{"role": "system", "content": "你是餐饮行业知识库问答助手，基于库内情报给出有出处的专业回答。使用 Markdown 格式输出。"},
|
||||
{"role": "user", "content": prompt},
|
||||
],
|
||||
"temperature": 0.3,
|
||||
"stream": True,
|
||||
"stream_options": {"include_usage": False},
|
||||
}, ensure_ascii=False).encode("utf-8")
|
||||
|
||||
req = urllib.request.Request(
|
||||
DASHSCOPE_CHAT_URL,
|
||||
data=payload,
|
||||
headers={
|
||||
"Authorization": f"Bearer {api_key}",
|
||||
"Content-Type": "application/json",
|
||||
"Accept": "text/event-stream",
|
||||
},
|
||||
method="POST",
|
||||
)
|
||||
try:
|
||||
resp = urllib.request.urlopen(req, timeout=120)
|
||||
except urllib.error.HTTPError as exc:
|
||||
detail = exc.read().decode("utf-8", errors="replace")[:500]
|
||||
raise RuntimeError(f"千问接口返回 {exc.code}：{detail}") from exc
|
||||
|
||||
buffer = b""
|
||||
while True:
|
||||
chunk = resp.read(4096)
|
||||
if not chunk:
|
||||
break
|
||||
buffer += chunk
|
||||
while b"\n" in buffer:
|
||||
line, buffer = buffer.split(b"\n", 1)
|
||||
line = line.strip()
|
||||
if not line or line == b"data: [DONE]":
|
||||
continue
|
||||
if line.startswith(b"data:"):
|
||||
try:
|
||||
data = json.loads(line[5:].strip())
|
||||
delta = data.get("choices", [{}])[0].get("delta", {})
|
||||
content = delta.get("content")
|
||||
if content:
|
||||
yield content
|
||||
except (json.JSONDecodeError, IndexError, KeyError):
|
||||
pass
|
||||
resp.close()
|
||||
|
||||
yield json.dumps({"__sources__": sources}, ensure_ascii=False)
|
||||
|
||||
|
||||
def rag_qa(question: str, top_k: int = 5) -> dict:
|
||||
"""完整的 RAG 问答流程：向量检索 → LLM 生成。"""
|
||||
db_path = Path(os.getenv("CIBANK_DB", str(BASE_DIR / "data" / "cibank.db")))
|
||||
conn = sqlite3.connect(db_path)
|
||||
conn.row_factory = sqlite3.Row
|
||||
|
||||
# 检查是否有向量数据
|
||||
count = conn.execute("SELECT COUNT(*) FROM embeddings").fetchone()[0]
|
||||
if count == 0:
|
||||
# 无向量数据时使用关键词检索作为降级方案
|
||||
return keyword_fallback(conn, question)
|
||||
|
||||
# 1. 将问题转为向量
|
||||
query_embeddings = get_embeddings([question])
|
||||
|
||||
# 2. 向量检索
|
||||
contexts = vector_search(conn, query_embeddings[0], top_k)
|
||||
|
||||
if not contexts:
|
||||
conn.close()
|
||||
return {
|
||||
"answer": "知识库中暂无与您问题相关的内容。请先运行采集和分析流程入库更多情报。",
|
||||
"sources": [],
|
||||
"contexts": [],
|
||||
}
|
||||
|
||||
# 3. LLM 生成回答
|
||||
result = generate_answer(question, contexts)
|
||||
conn.close()
|
||||
return result
|
||||
|
||||
|
||||
def rag_qa_stream(question: str, top_k: int = 5):
|
||||
"""流式 RAG 问答：先检索，再流式生成。yield (type, data) 元组。"""
|
||||
db_path = Path(os.getenv("CIBANK_DB", str(BASE_DIR / "data" / "cibank.db")))
|
||||
conn = sqlite3.connect(db_path)
|
||||
conn.row_factory = sqlite3.Row
|
||||
|
||||
count = conn.execute("SELECT COUNT(*) FROM embeddings").fetchone()[0]
|
||||
if count == 0:
|
||||
conn.close()
|
||||
yield ("error", "知识库中暂无向量数据，请先运行采集和分析流程。")
|
||||
return
|
||||
|
||||
query_embeddings = get_embeddings([question])
|
||||
contexts = vector_search(conn, query_embeddings[0], top_k)
|
||||
conn.close()
|
||||
|
||||
if not contexts:
|
||||
yield ("error", "知识库中暂无与您问题相关的内容。")
|
||||
return
|
||||
|
||||
sources = [
|
||||
{"title": ctx["title"], "source": ctx["source"], "date": ctx["date"]}
|
||||
for ctx in contexts
|
||||
]
|
||||
yield ("sources", sources)
|
||||
|
||||
for token in generate_answer_stream(question, contexts):
|
||||
yield ("token", token)
|
||||
|
||||
|
||||
def keyword_fallback(conn: sqlite3.Connection, question: str) -> dict:
|
||||
"""无向量数据时的关键词检索降级方案。"""
|
||||
rows = conn.execute(
|
||||
"""
|
||||
SELECT a.id, a.title, a.source_name, a.source_type, a.published_at, a.summary,
|
||||
a.content, x.result_json
|
||||
FROM articles a
|
||||
LEFT JOIN ai_analyses x ON x.article_id = a.id AND x.analysis_type = 'editorial'
|
||||
WHERE a.crawl_status = 'success'
|
||||
AND (a.title LIKE ? OR a.summary LIKE ? OR a.content LIKE ?)
|
||||
ORDER BY a.published_at DESC
|
||||
LIMIT 5
|
||||
""",
|
||||
(f"%{question}%", f"%{question}%", f"%{question}%"),
|
||||
).fetchall()
|
||||
|
||||
if not rows:
|
||||
conn.close()
|
||||
return {
|
||||
"answer": "知识库中暂无与您问题相关的内容。请先运行采集和分析流程入库更多情报。",
|
||||
"sources": [],
|
||||
"contexts": [],
|
||||
}
|
||||
|
||||
contexts = []
|
||||
for row in rows:
|
||||
analysis = json.loads(row["result_json"]) if row["result_json"] else {}
|
||||
contexts.append({
|
||||
"article_id": row["id"],
|
||||
"title": row["title"],
|
||||
"source": row["source_name"] or row["source_type"],
|
||||
"date": (row["published_at"] or "")[:10],
|
||||
"summary": analysis.get("summary") or row["summary"] or "",
|
||||
"key_points": analysis.get("key_points", []),
|
||||
"content": row["content"][:3000] if row["content"] else "",
|
||||
"score": 0.5,
|
||||
})
|
||||
conn.close()
|
||||
return generate_answer(question, contexts)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import sys
|
||||
question = " ".join(sys.argv[1:]) or "现在开茶饮店还有机会吗？"
|
||||
result = rag_qa(question)
|
||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
@@ -0,0 +1,5 @@
|
||||
playwright>=1.45,<2
|
||||
Flask>=3.0,<4
|
||||
flask-cors>=4.0,<5
|
||||
python-dotenv>=1.0,<2
|
||||
numpy>=1.26,<3
|
||||
Executable
+53
@@ -0,0 +1,53 @@
|
||||
#!/bin/bash
|
||||
# 餐智库一键启动脚本
|
||||
# 同时启动后端 API 服务和前端开发服务器
|
||||
|
||||
set -e
|
||||
|
||||
BACKEND_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
FRONTEND_DIR="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
|
||||
echo "=== 餐智库启动 ==="
|
||||
|
||||
# 检查后端虚拟环境
|
||||
if [ ! -d "$BACKEND_DIR/venv" ]; then
|
||||
echo "[后端] 创建虚拟环境..."
|
||||
python3 -m venv "$BACKEND_DIR/venv"
|
||||
echo "[后端] 安装依赖..."
|
||||
"$BACKEND_DIR/venv/bin/pip" install -r "$BACKEND_DIR/requirements.txt" -q
|
||||
echo "[后端] 安装 Playwright 浏览器..."
|
||||
"$BACKEND_DIR/venv/bin/playwright" install chromium
|
||||
fi
|
||||
|
||||
# 检查 .env 文件
|
||||
if [ ! -f "$BACKEND_DIR/.env" ]; then
|
||||
echo "[后端] 复制 .env.example -> .env"
|
||||
cp "$BACKEND_DIR/.env.example" "$BACKEND_DIR/.env"
|
||||
echo "[提示] 请编辑 backend/.env 填入 DASHSCOPE_API_KEY"
|
||||
fi
|
||||
|
||||
# 启动后端
|
||||
echo "[后端] 启动 API 服务 (端口 8765)..."
|
||||
"$BACKEND_DIR/venv/bin/python" "$BACKEND_DIR/app.py" &
|
||||
BACKEND_PID=$!
|
||||
echo "[后端] PID=$BACKEND_PID"
|
||||
|
||||
# 等待后端就绪
|
||||
sleep 2
|
||||
|
||||
# 启动前端
|
||||
echo "[前端] 启动开发服务器 (端口 3000)..."
|
||||
cd "$FRONTEND_DIR"
|
||||
npm run dev &
|
||||
FRONTEND_PID=$!
|
||||
echo "[前端] PID=$FRONTEND_PID"
|
||||
|
||||
echo ""
|
||||
echo "=== 启动完成 ==="
|
||||
echo "前端: http://localhost:3000"
|
||||
echo "后端: http://localhost:8765/api/health"
|
||||
echo ""
|
||||
echo "按 Ctrl+C 停止所有服务"
|
||||
|
||||
trap "kill $BACKEND_PID $FRONTEND_PID 2>/dev/null; exit" INT TERM
|
||||
wait
|
||||
@@ -0,0 +1,154 @@
|
||||
PRAGMA journal_mode = WAL;
|
||||
PRAGMA foreign_keys = ON;
|
||||
|
||||
-- 文章表：公众号文章 + 视频号ASR转写文本
|
||||
CREATE TABLE IF NOT EXISTS articles (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
canonical_url TEXT NOT NULL UNIQUE,
|
||||
source TEXT NOT NULL DEFAULT 'hongcan', -- 来源平台
|
||||
source_name TEXT, -- 公众号名称 / 视频号名称
|
||||
source_type TEXT NOT NULL DEFAULT '公众号', -- 公众号 | 视频号
|
||||
title TEXT,
|
||||
author TEXT,
|
||||
published_at TEXT,
|
||||
summary TEXT,
|
||||
content TEXT,
|
||||
content_hash TEXT,
|
||||
category TEXT, -- 赛道：茶饮咖啡/快餐/火锅/正餐/烘焙/供应链/综合
|
||||
crawl_status TEXT NOT NULL DEFAULT 'pending', -- pending | success | failed
|
||||
retry_count INTEGER NOT NULL DEFAULT 0,
|
||||
last_error TEXT,
|
||||
media_url TEXT, -- 原始视频文件路径（视频号专用）
|
||||
discovered_at TEXT NOT NULL,
|
||||
crawled_at TEXT,
|
||||
updated_at TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_articles_published_at ON articles(published_at);
|
||||
CREATE INDEX IF NOT EXISTS idx_articles_crawl_status ON articles(crawl_status, retry_count);
|
||||
CREATE INDEX IF NOT EXISTS idx_articles_source_type ON articles(source_type);
|
||||
CREATE INDEX IF NOT EXISTS idx_articles_category ON articles(category);
|
||||
|
||||
-- 采集运行记录
|
||||
CREATE TABLE IF NOT EXISTS crawl_runs (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
started_at TEXT NOT NULL,
|
||||
finished_at TEXT,
|
||||
status TEXT NOT NULL DEFAULT 'running',
|
||||
source_type TEXT NOT NULL DEFAULT '公众号',
|
||||
discovered_count INTEGER NOT NULL DEFAULT 0,
|
||||
inserted_count INTEGER NOT NULL DEFAULT 0,
|
||||
updated_count INTEGER NOT NULL DEFAULT 0,
|
||||
failed_count INTEGER NOT NULL DEFAULT 0,
|
||||
message TEXT
|
||||
);
|
||||
|
||||
-- AI 结构化分析结果
|
||||
CREATE TABLE IF NOT EXISTS ai_analyses (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
article_id INTEGER NOT NULL,
|
||||
model TEXT NOT NULL,
|
||||
analysis_type TEXT NOT NULL DEFAULT 'editorial',
|
||||
result_json TEXT NOT NULL, -- 完整结构化JSON
|
||||
content_hash TEXT,
|
||||
created_at TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL,
|
||||
UNIQUE(article_id, model, analysis_type),
|
||||
FOREIGN KEY(article_id) REFERENCES articles(id) ON DELETE CASCADE
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_ai_analyses_article ON ai_analyses(article_id, updated_at);
|
||||
|
||||
-- 向量嵌入表（用于RAG检索）
|
||||
CREATE TABLE IF NOT EXISTS embeddings (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
article_id INTEGER NOT NULL,
|
||||
chunk_index INTEGER NOT NULL DEFAULT 0,
|
||||
chunk_text TEXT NOT NULL,
|
||||
embedding BLOB, -- JSON序列化的float数组
|
||||
model TEXT NOT NULL,
|
||||
content_hash TEXT,
|
||||
created_at TEXT NOT NULL,
|
||||
FOREIGN KEY(article_id) REFERENCES articles(id) ON DELETE CASCADE,
|
||||
UNIQUE(article_id, chunk_index, model)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_embeddings_article ON embeddings(article_id);
|
||||
|
||||
-- 品牌实体库
|
||||
CREATE TABLE IF NOT EXISTS brands (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
name TEXT NOT NULL UNIQUE,
|
||||
category TEXT,
|
||||
stores INTEGER DEFAULT 0,
|
||||
avg_price INTEGER DEFAULT 0,
|
||||
model TEXT, -- 直营 | 加盟 | 直营+加盟
|
||||
city_tier_json TEXT DEFAULT '[]', -- [{tier, pct}]
|
||||
trend_json TEXT DEFAULT '[]', -- 近12个月门店数
|
||||
latest_news TEXT,
|
||||
news_date TEXT,
|
||||
growth REAL DEFAULT 0,
|
||||
created_at TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL
|
||||
);
|
||||
|
||||
-- 行业信号（跨文章聚合）
|
||||
CREATE TABLE IF NOT EXISTS industry_signals (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
signal_date TEXT NOT NULL,
|
||||
title TEXT NOT NULL,
|
||||
summary TEXT NOT NULL,
|
||||
trend TEXT NOT NULL DEFAULT 'emerging',
|
||||
confidence REAL NOT NULL DEFAULT 0.5,
|
||||
implications_json TEXT NOT NULL DEFAULT '[]',
|
||||
article_ids_json TEXT NOT NULL DEFAULT '[]',
|
||||
tags_json TEXT NOT NULL DEFAULT '[]',
|
||||
model TEXT NOT NULL,
|
||||
created_at TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL,
|
||||
UNIQUE(signal_date, title)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_industry_signals_date ON industry_signals(signal_date DESC, confidence DESC);
|
||||
|
||||
-- 报告表
|
||||
CREATE TABLE IF NOT EXISTS reports (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
title TEXT NOT NULL,
|
||||
report_date TEXT NOT NULL,
|
||||
period TEXT,
|
||||
highlights_json TEXT NOT NULL DEFAULT '[]',
|
||||
sections_json TEXT NOT NULL DEFAULT '[]',
|
||||
created_at TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_reports_date ON reports(report_date DESC);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS pipeline_runs (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
run_type TEXT NOT NULL,
|
||||
started_at TEXT NOT NULL,
|
||||
finished_at TEXT,
|
||||
status TEXT NOT NULL DEFAULT 'running',
|
||||
current_step TEXT,
|
||||
steps_json TEXT NOT NULL DEFAULT '{}',
|
||||
error_message TEXT,
|
||||
created_at TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_pipeline_runs_started ON pipeline_runs(started_at DESC);
|
||||
|
||||
-- 信源配置表（公众号 + 视频号）
|
||||
CREATE TABLE IF NOT EXISTS sources (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
source_type TEXT NOT NULL DEFAULT '公众号', -- 公众号 | 视频号
|
||||
name TEXT NOT NULL, -- 信源名称（公众号名称/视频号账号名）
|
||||
finder TEXT, -- 视频号 finder ID（视频号专用）
|
||||
list_url TEXT, -- 列表页URL（公众号专用）
|
||||
download_dir TEXT, -- 下载目录（视频号专用）
|
||||
enabled INTEGER NOT NULL DEFAULT 1, -- 是否启用 1/0
|
||||
last_crawled_at TEXT, -- 最近采集时间
|
||||
created_at TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL,
|
||||
UNIQUE(source_type, name)
|
||||
);
|
||||
@@ -0,0 +1,693 @@
|
||||
#!/usr/bin/env python3
|
||||
"""结构化引擎：大模型自动打标签、摘要、提取要点、识别品牌实体。
|
||||
|
||||
对每篇采集入库的文章调用通义千问，输出结构化JSON：
|
||||
- summary: 120字内摘要
|
||||
- key_points: 3-6条关键要点
|
||||
- tags: 5-10个标签
|
||||
- brands: 识别到的品牌实体
|
||||
- category: 赛道分类
|
||||
- type: 情报类型（政策监管/品牌动态/品类趋势/经营干货/供应链/消费洞察）
|
||||
- score: 价值评分 0-100
|
||||
- timeliness: 时效性（高/中/低）
|
||||
- sentiment: 情感倾向
|
||||
- content_angles: 可延展选题
|
||||
- risk_notes: 事实或表述风险
|
||||
- numbers: 关键数据
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import sqlite3
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
BASE_DIR = Path(__file__).resolve().parent
|
||||
load_dotenv(BASE_DIR / ".env")
|
||||
|
||||
DASHSCOPE_CHAT_URL = "https://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions"
|
||||
DASHSCOPE_EMBED_URL = "https://dashscope.aliyuncs.com/compatible-mode/v1/embeddings"
|
||||
|
||||
|
||||
def now_iso() -> str:
|
||||
return datetime.now(timezone.utc).astimezone().isoformat(timespec="seconds")
|
||||
|
||||
|
||||
def get_api_key() -> str:
|
||||
key = os.getenv("DASHSCOPE_API_KEY", "").strip()
|
||||
if not key:
|
||||
raise RuntimeError("尚未配置 DASHSCOPE_API_KEY，请在 .env 中设置")
|
||||
return key
|
||||
|
||||
|
||||
# ---- 单篇文章结构化分析 ----
|
||||
|
||||
def analyze_article(title: str, content: str) -> dict:
|
||||
"""调用通义千问对文章进行结构化分析。"""
|
||||
api_key = get_api_key()
|
||||
model = os.getenv("QWEN_MODEL", "qwen-plus")
|
||||
prompt = f"""你是一名资深餐饮行业研究编辑。分析下面这篇文章，只返回合法 JSON，不要 Markdown 代码块。
|
||||
JSON 字段必须为：
|
||||
- summary（120字内摘要）
|
||||
- key_points（3-6条字符串，关键要点）
|
||||
- tags（5-10个标签）
|
||||
- brands（品牌数组，每个元素为对象：{{"name":"品牌名","stores":门店数整数或0,"avg_price":人均消费整数或0,"model":"直营/加盟/直营+加盟"}}，仅填文中明确提及的数据，未提及的填0或空字符串）
|
||||
- category（赛道分类，从以下选择：茶饮咖啡/快餐/火锅/正餐/烘焙/供应链/综合）
|
||||
- type（情报类型，从以下选择：政策监管/品牌动态/品类趋势/经营干货/供应链/消费洞察）
|
||||
- score（价值评分0-100整数，越高越有经营参考价值）
|
||||
- timeliness（时效性：高/中/低）
|
||||
- sentiment（positive/neutral/negative）
|
||||
- content_angles（3条可延展选题）
|
||||
- risk_notes（事实或表述风险数组）
|
||||
- numbers（关键数据数组）
|
||||
|
||||
文章标题：{title}
|
||||
正文：{content[:24000]}"""
|
||||
payload = json.dumps({
|
||||
"model": model,
|
||||
"messages": [
|
||||
{"role": "system", "content": "你输出严谨、简洁、可供编辑部直接使用的结构化行业分析。"},
|
||||
{"role": "user", "content": prompt},
|
||||
],
|
||||
"temperature": 0.25,
|
||||
"response_format": {"type": "json_object"},
|
||||
}, ensure_ascii=False).encode("utf-8")
|
||||
req = urllib.request.Request(
|
||||
DASHSCOPE_CHAT_URL,
|
||||
data=payload,
|
||||
headers={"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
|
||||
method="POST",
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=90) as response:
|
||||
body = json.loads(response.read().decode("utf-8"))
|
||||
except urllib.error.HTTPError as exc:
|
||||
detail = exc.read().decode("utf-8", errors="replace")[:500]
|
||||
raise RuntimeError(f"千问接口返回 {exc.code}：{detail}") from exc
|
||||
if "choices" not in body or not body["choices"]:
|
||||
raise RuntimeError(f"千问返回异常：{json.dumps(body, ensure_ascii=False)[:300]}")
|
||||
text = body["choices"][0]["message"]["content"].strip()
|
||||
if text.startswith("```"):
|
||||
text = text.strip("`").removeprefix("json").strip()
|
||||
try:
|
||||
return json.loads(text)
|
||||
except Exception as e:
|
||||
raise RuntimeError(f"JSON解析失败: {e}\n{text[:200]}") from e
|
||||
|
||||
|
||||
# ---- 跨文章行业信号提取 ----
|
||||
|
||||
def extract_industry_signals(items: list[dict], model: str) -> list[dict]:
|
||||
"""从多篇文章的结构化分析中提取跨文章行业信号。"""
|
||||
api_key = get_api_key()
|
||||
prompt = f"""你是餐饮产业首席分析师。根据以下多篇文章的结构化分析，识别跨文章、可验证、有经营意义的行业信号。
|
||||
只返回合法 JSON：{{"signals":[...]}}。每个 signal 必须包含：
|
||||
- title（短标题）
|
||||
- summary（100-180字）
|
||||
- trend（emerging/accelerating/stable/declining）
|
||||
- confidence（0到1）
|
||||
- article_ids（至少2个证据文章ID）
|
||||
- implications（2-4条经营启示）
|
||||
- tags（3-6个标签）
|
||||
不要把单一品牌新闻简单改写成行业信号；合并重复主题；最多输出8条。
|
||||
|
||||
输入：{json.dumps(items, ensure_ascii=False)}"""
|
||||
body = json.dumps({
|
||||
"model": model,
|
||||
"messages": [
|
||||
{"role": "system", "content": "你进行基于证据的餐饮行业趋势聚类，避免空泛结论。"},
|
||||
{"role": "user", "content": prompt},
|
||||
],
|
||||
"temperature": 0.2,
|
||||
"response_format": {"type": "json_object"},
|
||||
}, ensure_ascii=False).encode("utf-8")
|
||||
req = urllib.request.Request(
|
||||
DASHSCOPE_CHAT_URL,
|
||||
data=body,
|
||||
headers={"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
|
||||
method="POST",
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=180) as response:
|
||||
result = json.loads(response.read().decode("utf-8"))
|
||||
except urllib.error.HTTPError as exc:
|
||||
raise RuntimeError(f"千问接口返回 {exc.code}：{exc.read().decode('utf-8', errors='replace')[:500]}") from exc
|
||||
except urllib.error.URLError as exc:
|
||||
raise RuntimeError(f"千问接口网络超时：{exc}") from exc
|
||||
text = result["choices"][0]["message"]["content"].strip()
|
||||
if text.startswith("```"):
|
||||
text = text.strip("`").removeprefix("json").strip()
|
||||
return json.loads(text).get("signals", [])
|
||||
|
||||
|
||||
# ---- 文本向量嵌入 ----
|
||||
|
||||
def get_embeddings(texts: list[str]) -> list[list[float]]:
|
||||
"""调用通义千问文本向量模型，批量获取向量。"""
|
||||
api_key = get_api_key()
|
||||
model = os.getenv("EMBEDDING_MODEL", "text-embedding-v3")
|
||||
payload = json.dumps({
|
||||
"model": model,
|
||||
"input": texts,
|
||||
}, ensure_ascii=False).encode("utf-8")
|
||||
req = urllib.request.Request(
|
||||
DASHSCOPE_EMBED_URL,
|
||||
data=payload,
|
||||
headers={"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
|
||||
method="POST",
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=60) as response:
|
||||
body = json.loads(response.read().decode("utf-8"))
|
||||
except urllib.error.HTTPError as exc:
|
||||
detail = exc.read().decode("utf-8", errors="replace")[:500]
|
||||
raise RuntimeError(f"向量接口返回 {exc.code}：{detail}") from exc
|
||||
if "output" in body:
|
||||
return [item["embedding"] for item in body["output"]["embeddings"]]
|
||||
return [item["embedding"] for item in body["data"]]
|
||||
|
||||
|
||||
# ---- 品牌实体入库 ----
|
||||
|
||||
def upsert_brand(conn: sqlite3.Connection, brand_name: str, category: str | None = None,
|
||||
stores: int = 0, avg_price: int = 0, model: str | None = None,
|
||||
latest_news: str | None = None, news_date: str | None = None) -> None:
|
||||
"""将识别到的品牌实体写入品牌表，有新数据时更新。"""
|
||||
existing = conn.execute("SELECT id, stores, avg_price FROM brands WHERE name=?", (brand_name,)).fetchone()
|
||||
timestamp = now_iso()
|
||||
if existing:
|
||||
# 只在新值更大或非空时更新，避免旧数据被覆盖
|
||||
new_stores = max(existing["stores"], stores) if stores > 0 else existing["stores"]
|
||||
new_avg_price = avg_price if avg_price > 0 else existing["avg_price"]
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE brands SET category=COALESCE(?, category),
|
||||
stores=?, avg_price=?,
|
||||
model=COALESCE(NULLIF(?, ''), model),
|
||||
latest_news=COALESCE(NULLIF(?, ''), latest_news),
|
||||
news_date=COALESCE(NULLIF(?, ''), news_date),
|
||||
updated_at=?
|
||||
WHERE id=?
|
||||
""",
|
||||
(category, new_stores, new_avg_price, model or "", latest_news or "", news_date or "", timestamp, existing["id"]),
|
||||
)
|
||||
else:
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO brands (name, category, stores, avg_price, model, city_tier_json, trend_json,
|
||||
latest_news, news_date, growth, created_at, updated_at)
|
||||
VALUES (?, ?, ?, ?, NULLIF(?, ''), '[]', '[]', NULLIF(?, ''), NULLIF(?, ''), 0, ?, ?)
|
||||
""",
|
||||
(brand_name, category, stores, avg_price, model or "", latest_news or "", news_date or "", timestamp, timestamp),
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
|
||||
# ---- 批量分析主流程 ----
|
||||
|
||||
def run_batch_analyze(force: bool, pause: float, retries: int) -> int:
|
||||
"""批量对所有成功采集的文章执行结构化分析。"""
|
||||
db_path = Path(os.getenv("CIBANK_DB", str(BASE_DIR / "data" / "cibank.db")))
|
||||
conn = sqlite3.connect(db_path)
|
||||
conn.row_factory = sqlite3.Row
|
||||
conn.executescript((BASE_DIR / "schema.sql").read_text(encoding="utf-8"))
|
||||
|
||||
model = os.getenv("QWEN_MODEL", "qwen-plus")
|
||||
rows = conn.execute(
|
||||
"""
|
||||
SELECT a.id, a.title, a.content, a.content_hash, a.published_at, x.content_hash analyzed_hash
|
||||
FROM articles a
|
||||
LEFT JOIN ai_analyses x
|
||||
ON x.article_id = a.id AND x.model = ? AND x.analysis_type = 'editorial'
|
||||
WHERE a.crawl_status = 'success'
|
||||
ORDER BY a.published_at DESC, a.id DESC
|
||||
""",
|
||||
(model,),
|
||||
).fetchall()
|
||||
|
||||
pending = [r for r in rows if force or not r["analyzed_hash"] or r["analyzed_hash"] != r["content_hash"]]
|
||||
print(json.dumps({"total": len(rows), "pending": len(pending), "model": model}, ensure_ascii=False), flush=True)
|
||||
|
||||
success = skipped = failed = 0
|
||||
for index, row in enumerate(rows, start=1):
|
||||
if not force and row["analyzed_hash"] == row["content_hash"]:
|
||||
skipped += 1
|
||||
continue
|
||||
last_error = None
|
||||
for attempt in range(1, retries + 2):
|
||||
try:
|
||||
result = analyze_article(row["title"], row["content"])
|
||||
timestamp = now_iso()
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO ai_analyses(article_id, model, analysis_type, result_json, content_hash, created_at, updated_at)
|
||||
VALUES (?, ?, 'editorial', ?, ?, ?, ?)
|
||||
ON CONFLICT(article_id, model, analysis_type) DO UPDATE SET
|
||||
result_json = excluded.result_json, content_hash = excluded.content_hash, updated_at = excluded.updated_at
|
||||
""",
|
||||
(row["id"], model, json.dumps(result, ensure_ascii=False), row["content_hash"], timestamp, timestamp),
|
||||
)
|
||||
# 更新文章的category字段
|
||||
if result.get("category"):
|
||||
conn.execute("UPDATE articles SET category=? WHERE id=?", (result["category"], row["id"]))
|
||||
|
||||
# 品牌实体入库
|
||||
for brand in result.get("brands", []):
|
||||
if isinstance(brand, str):
|
||||
upsert_brand(conn, brand, result.get("category"))
|
||||
elif isinstance(brand, dict):
|
||||
upsert_brand(conn, brand.get("name", ""), result.get("category"),
|
||||
int(brand.get("stores", 0) or 0), int(brand.get("avg_price", 0) or 0),
|
||||
brand.get("model"), row["title"], (row["published_at"] or "")[:10])
|
||||
|
||||
conn.commit()
|
||||
success += 1
|
||||
print(f"[{index}/{len(rows)}] OK {row['title']}", flush=True)
|
||||
last_error = None
|
||||
break
|
||||
except Exception as exc:
|
||||
last_error = exc
|
||||
if attempt <= retries:
|
||||
wait = attempt * 2
|
||||
print(f"[{index}/{len(rows)}] RETRY {attempt}/{retries} {exc}", flush=True)
|
||||
time.sleep(wait)
|
||||
if last_error is not None:
|
||||
failed += 1
|
||||
print(f"[{index}/{len(rows)}] FAIL {row['title']}: {last_error}", flush=True)
|
||||
time.sleep(pause)
|
||||
|
||||
total_analyzed = conn.execute("SELECT COUNT(DISTINCT article_id) FROM ai_analyses").fetchone()[0]
|
||||
conn.close()
|
||||
print(json.dumps({
|
||||
"success": success, "skipped": skipped, "failed": failed,
|
||||
"total_analyzed": total_analyzed,
|
||||
}, ensure_ascii=False), flush=True)
|
||||
return 0 if failed == 0 else 1
|
||||
|
||||
|
||||
def run_extract_signals(days: int) -> int:
|
||||
"""提取跨文章行业信号。"""
|
||||
db_path = Path(os.getenv("CIBANK_DB", str(BASE_DIR / "data" / "cibank.db")))
|
||||
conn = sqlite3.connect(db_path, timeout=30)
|
||||
conn.row_factory = sqlite3.Row
|
||||
conn.executescript((BASE_DIR / "schema.sql").read_text(encoding="utf-8"))
|
||||
|
||||
model = os.getenv("QWEN_MODEL", "qwen-plus")
|
||||
from datetime import datetime, timedelta
|
||||
cutoff = (datetime.now() - timedelta(days=days)).strftime("%Y-%m-%dT%H:%M:%S")
|
||||
rows = conn.execute(
|
||||
"""
|
||||
SELECT a.id, a.title, a.published_at, x.result_json
|
||||
FROM articles a JOIN ai_analyses x ON x.article_id = a.id
|
||||
WHERE a.crawl_status = 'success' AND a.published_at >= ?
|
||||
ORDER BY a.published_at DESC
|
||||
""",
|
||||
(cutoff,),
|
||||
).fetchall()
|
||||
|
||||
items = []
|
||||
for row in rows:
|
||||
analysis = json.loads(row["result_json"])
|
||||
items.append({
|
||||
"article_id": row["id"], "title": row["title"], "published_at": row["published_at"],
|
||||
"summary": analysis.get("summary"), "key_points": analysis.get("key_points", []),
|
||||
"type": analysis.get("type"), "tags": analysis.get("tags", []),
|
||||
"brands": analysis.get("brands", []), "score": analysis.get("score", 0),
|
||||
})
|
||||
|
||||
if not items:
|
||||
print(json.dumps({"status": "skipped", "reason": "没有已分析文章"}, ensure_ascii=False))
|
||||
return 0
|
||||
|
||||
# 分批提取信号，避免一次性传入过多内容触发内容审查
|
||||
signal_date = datetime.now().astimezone().date().isoformat()
|
||||
timestamp = now_iso()
|
||||
conn.execute("DELETE FROM industry_signals WHERE signal_date=?", (signal_date,))
|
||||
conn.commit()
|
||||
|
||||
all_signals = []
|
||||
batch_size = 20
|
||||
total_batches = (len(items) + batch_size - 1) // batch_size
|
||||
for i in range(0, len(items), batch_size):
|
||||
batch_num = i // batch_size + 1
|
||||
batch = items[i:i + batch_size]
|
||||
print(f" 信号批次 {batch_num}/{total_batches}，{len(batch)} 篇", flush=True)
|
||||
for attempt in range(2):
|
||||
try:
|
||||
batch_signals = extract_industry_signals(batch, model)
|
||||
all_signals.extend(batch_signals)
|
||||
# 每批成功后立即写入数据库
|
||||
for signal in batch_signals:
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT OR IGNORE INTO industry_signals
|
||||
(signal_date, title, summary, trend, confidence, implications_json, article_ids_json, tags_json, model, created_at, updated_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
""",
|
||||
(
|
||||
signal_date, signal["title"], signal["summary"], signal.get("trend", "emerging"),
|
||||
float(signal.get("confidence", 0.5)), json.dumps(signal.get("implications", []), ensure_ascii=False),
|
||||
json.dumps(signal.get("article_ids", [])), json.dumps(signal.get("tags", []), ensure_ascii=False),
|
||||
model, timestamp, timestamp,
|
||||
),
|
||||
)
|
||||
conn.commit()
|
||||
break
|
||||
except RuntimeError as exc:
|
||||
print(f" 批次 {batch_num} 失败(尝试{attempt+1}): {exc}", flush=True)
|
||||
if attempt == 0:
|
||||
import time; time.sleep(2)
|
||||
continue
|
||||
except Exception as exc:
|
||||
print(f" 批次 {batch_num} 异常(尝试{attempt+1}): {type(exc).__name__}: {exc}", flush=True)
|
||||
if attempt == 0:
|
||||
import time; time.sleep(2)
|
||||
continue
|
||||
|
||||
# 合并去重：按标题去重，保留置信度更高的
|
||||
seen = {}
|
||||
for sig in all_signals:
|
||||
title = sig.get("title", "")
|
||||
conf = float(sig.get("confidence", 0))
|
||||
if title not in seen or conf > float(seen[title].get("confidence", 0)):
|
||||
seen[title] = sig
|
||||
signals = list(seen.values())[:8]
|
||||
|
||||
# 用去重后的结果替换当天信号
|
||||
conn.execute("DELETE FROM industry_signals WHERE signal_date=?", (signal_date,))
|
||||
for signal in signals:
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO industry_signals
|
||||
(signal_date, title, summary, trend, confidence, implications_json, article_ids_json, tags_json, model, created_at, updated_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
""",
|
||||
(
|
||||
signal_date, signal["title"], signal["summary"], signal.get("trend", "emerging"),
|
||||
float(signal.get("confidence", 0.5)), json.dumps(signal.get("implications", []), ensure_ascii=False),
|
||||
json.dumps(signal.get("article_ids", [])), json.dumps(signal.get("tags", []), ensure_ascii=False),
|
||||
model, timestamp, timestamp,
|
||||
),
|
||||
)
|
||||
conn.commit()
|
||||
conn.close()
|
||||
print(json.dumps({"status": "success", "input_articles": len(items), "signals": len(signals), "date": signal_date}, ensure_ascii=False))
|
||||
return 0
|
||||
|
||||
|
||||
# ---- 向量嵌入生成 ----
|
||||
|
||||
def ensure_embedding_schema(conn: sqlite3.Connection) -> None:
|
||||
"""兼容已有数据库，为向量表补齐增量更新所需字段与唯一索引。"""
|
||||
columns = {row[1] for row in conn.execute("PRAGMA table_info(embeddings)").fetchall()}
|
||||
if "content_hash" not in columns:
|
||||
conn.execute("ALTER TABLE embeddings ADD COLUMN content_hash TEXT")
|
||||
conn.execute(
|
||||
"""
|
||||
DELETE FROM embeddings
|
||||
WHERE id NOT IN (
|
||||
SELECT MAX(id) FROM embeddings GROUP BY article_id, chunk_index, model
|
||||
)
|
||||
"""
|
||||
)
|
||||
conn.execute(
|
||||
"CREATE UNIQUE INDEX IF NOT EXISTS idx_embeddings_unique_chunk "
|
||||
"ON embeddings(article_id, chunk_index, model)"
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
|
||||
def split_text(text: str, chunk_size: int = 800, overlap: int = 120) -> list[str]:
|
||||
"""将长文本切分为有重叠的中文检索片段。"""
|
||||
normalized = "\n".join(line.strip() for line in text.splitlines() if line.strip())
|
||||
if not normalized:
|
||||
return []
|
||||
chunks = []
|
||||
start = 0
|
||||
while start < len(normalized):
|
||||
end = min(len(normalized), start + chunk_size)
|
||||
if end < len(normalized):
|
||||
candidates = [normalized.rfind(mark, start + chunk_size // 2, end) for mark in "。！？；\n"]
|
||||
boundary = max(candidates)
|
||||
if boundary > start:
|
||||
end = boundary + 1
|
||||
chunks.append(normalized[start:end])
|
||||
if end >= len(normalized):
|
||||
break
|
||||
start = max(start + 1, end - overlap)
|
||||
return chunks
|
||||
|
||||
|
||||
def build_embedding_chunks(row: sqlite3.Row) -> tuple[list[str], str]:
|
||||
"""构建包含标题、AI摘要和完整原文的向量分块。"""
|
||||
analysis = json.loads(row["result_json"]) if row["result_json"] else {}
|
||||
header_parts = [f"标题：{row['title'] or ''}"]
|
||||
summary = analysis.get("summary") or row["summary"] or ""
|
||||
if summary:
|
||||
header_parts.append(f"摘要：{summary}")
|
||||
key_points = analysis.get("key_points") or []
|
||||
if key_points:
|
||||
header_parts.append("要点：" + "；".join(str(item) for item in key_points))
|
||||
tags = analysis.get("tags") or []
|
||||
if tags:
|
||||
header_parts.append("标签：" + "、".join(str(item) for item in tags))
|
||||
header = "\n".join(header_parts)
|
||||
content_chunks = split_text(row["content"] or "") or [""]
|
||||
chunks = [f"{header}\n正文片段：{content}".strip() for content in content_chunks]
|
||||
source_hash = hashlib.sha256("\n\n".join(chunks).encode("utf-8")).hexdigest()
|
||||
return chunks, source_hash
|
||||
|
||||
|
||||
def run_generate_embeddings(force: bool) -> int:
|
||||
"""为所有成功文章增量生成分块向量，用于RAG检索。"""
|
||||
db_path = Path(os.getenv("CIBANK_DB", str(BASE_DIR / "data" / "cibank.db")))
|
||||
conn = sqlite3.connect(db_path)
|
||||
conn.row_factory = sqlite3.Row
|
||||
conn.executescript((BASE_DIR / "schema.sql").read_text(encoding="utf-8"))
|
||||
ensure_embedding_schema(conn)
|
||||
|
||||
model = os.getenv("EMBEDDING_MODEL", "text-embedding-v3")
|
||||
rows = conn.execute(
|
||||
"""
|
||||
SELECT a.id, a.title, a.summary, a.content, x.result_json
|
||||
FROM articles a
|
||||
LEFT JOIN ai_analyses x ON x.id = (
|
||||
SELECT x2.id FROM ai_analyses x2
|
||||
WHERE x2.article_id = a.id AND x2.analysis_type = 'editorial'
|
||||
ORDER BY x2.updated_at DESC, x2.id DESC LIMIT 1
|
||||
)
|
||||
WHERE a.crawl_status = 'success'
|
||||
ORDER BY a.id
|
||||
""",
|
||||
).fetchall()
|
||||
|
||||
success = skipped = failed = chunks_written = 0
|
||||
for row in rows:
|
||||
chunks, source_hash = build_embedding_chunks(row)
|
||||
existing = conn.execute(
|
||||
"SELECT content_hash FROM embeddings WHERE article_id=? AND model=? LIMIT 1",
|
||||
(row["id"], model),
|
||||
).fetchone()
|
||||
if not force and existing and existing["content_hash"] == source_hash:
|
||||
skipped += 1
|
||||
continue
|
||||
|
||||
try:
|
||||
vectors = []
|
||||
for start in range(0, len(chunks), 10):
|
||||
vectors.extend(get_embeddings(chunks[start:start + 10]))
|
||||
if len(vectors) != len(chunks):
|
||||
raise RuntimeError(f"向量数量异常：期望 {len(chunks)}，实际 {len(vectors)}")
|
||||
timestamp = now_iso()
|
||||
conn.execute("BEGIN")
|
||||
conn.execute("DELETE FROM embeddings WHERE article_id=? AND model=?", (row["id"], model))
|
||||
conn.executemany(
|
||||
"""
|
||||
INSERT INTO embeddings
|
||||
(article_id, chunk_index, chunk_text, embedding, model, content_hash, created_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?)
|
||||
""",
|
||||
[
|
||||
(row["id"], index, chunk, json.dumps(vector), model, source_hash, timestamp)
|
||||
for index, (chunk, vector) in enumerate(zip(chunks, vectors))
|
||||
],
|
||||
)
|
||||
conn.commit()
|
||||
success += 1
|
||||
chunks_written += len(chunks)
|
||||
except Exception as exc:
|
||||
conn.rollback()
|
||||
failed += 1
|
||||
print(f"FAIL article {row['id']}: {exc}", flush=True)
|
||||
|
||||
conn.close()
|
||||
print(json.dumps({
|
||||
"success": success, "skipped": skipped, "failed": failed,
|
||||
"chunks_written": chunks_written, "model": model,
|
||||
}, ensure_ascii=False))
|
||||
return 0 if failed == 0 else 1
|
||||
|
||||
|
||||
def run_generate_report(days: int = 7) -> int:
|
||||
"""生成周期报告：聚合近期文章分析结果，调用LLM生成结构化报告。"""
|
||||
db_path = Path(os.getenv("CIBANK_DB", str(BASE_DIR / "data" / "cibank.db")))
|
||||
conn = sqlite3.connect(db_path, timeout=30)
|
||||
conn.row_factory = sqlite3.Row
|
||||
conn.executescript((BASE_DIR / "schema.sql").read_text(encoding="utf-8"))
|
||||
|
||||
model = os.getenv("QWEN_MODEL", "qwen-plus")
|
||||
from datetime import datetime, timedelta
|
||||
cutoff = (datetime.now() - timedelta(days=days)).strftime("%Y-%m-%d")
|
||||
|
||||
rows = conn.execute(
|
||||
"""
|
||||
SELECT a.id, a.title, a.source_name, a.published_at, a.category,
|
||||
x.result_json
|
||||
FROM articles a
|
||||
JOIN ai_analyses x ON x.article_id = a.id AND x.analysis_type = 'editorial'
|
||||
WHERE a.crawl_status = 'success' AND a.published_at >= ?
|
||||
ORDER BY a.published_at DESC
|
||||
LIMIT 50
|
||||
""",
|
||||
(cutoff,),
|
||||
).fetchall()
|
||||
|
||||
if not rows:
|
||||
print(json.dumps({"status": "skip", "reason": "近期无分析数据"}, ensure_ascii=False))
|
||||
return 0
|
||||
|
||||
items = []
|
||||
for r in rows:
|
||||
analysis = json.loads(r["result_json"]) if r["result_json"] else {}
|
||||
items.append({
|
||||
"title": r["title"],
|
||||
"source": r["source_name"] or "",
|
||||
"date": (r["published_at"] or "")[:10],
|
||||
"category": r["category"] or analysis.get("category", ""),
|
||||
"type": analysis.get("type", ""),
|
||||
"summary": analysis.get("summary", ""),
|
||||
"key_points": analysis.get("key_points", [])[:3],
|
||||
"score": analysis.get("score", 0),
|
||||
"brands": [b["name"] if isinstance(b, dict) else b for b in analysis.get("brands", [])][:3],
|
||||
})
|
||||
|
||||
api_key = get_api_key()
|
||||
period_label = f"近{days}天"
|
||||
prompt = f"""你是餐饮行业首席分析师。根据以下{len(items)}篇情报的结构化分析，生成一份{period_label}餐饮行业经营决策简报。
|
||||
只返回合法 JSON，不要 Markdown 代码块。格式：
|
||||
{{
|
||||
"title": "报告标题（含日期范围）",
|
||||
"highlights": ["核心要点1", "核心要点2", "核心要点3", "核心要点4"],
|
||||
"sections": [
|
||||
{{"heading": "板块标题", "body": "详细分析正文（200-400字）"}},
|
||||
{{"heading": "板块标题", "body": "详细分析正文（200-400字）"}},
|
||||
{{"heading": "板块标题", "body": "详细分析正文（200-400字）"}},
|
||||
{{"heading": "经营建议", "body": "基于以上分析的经营建议（200-300字）"}}
|
||||
]
|
||||
}}
|
||||
|
||||
要求：
|
||||
1. highlights 为4-6条核心洞察，每条15-30字
|
||||
2. sections 至少4个板块，覆盖赛道趋势、品牌动态、政策/供应链、经营建议
|
||||
3. 内容必须基于提供的情报数据，不要编造
|
||||
4. 语言专业、简洁、有决策参考价值
|
||||
|
||||
情报数据：
|
||||
{json.dumps(items, ensure_ascii=False)}"""
|
||||
|
||||
payload = json.dumps({
|
||||
"model": model,
|
||||
"messages": [
|
||||
{"role": "system", "content": "你是餐饮行业首席分析师，输出严谨的经营决策简报。"},
|
||||
{"role": "user", "content": prompt},
|
||||
],
|
||||
"temperature": 0.3,
|
||||
"response_format": {"type": "json_object"},
|
||||
}, ensure_ascii=False).encode("utf-8")
|
||||
|
||||
req = urllib.request.Request(
|
||||
DASHSCOPE_CHAT_URL,
|
||||
data=payload,
|
||||
headers={"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
|
||||
method="POST",
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=120) as response:
|
||||
body = json.loads(response.read().decode("utf-8"))
|
||||
except urllib.error.HTTPError as exc:
|
||||
detail = exc.read().decode("utf-8", errors="replace")[:500]
|
||||
print(json.dumps({"status": "error", "error": f"千问接口返回 {exc.code}：{detail}"}, ensure_ascii=False))
|
||||
return 1
|
||||
|
||||
text = body["choices"][0]["message"]["content"].strip()
|
||||
if text.startswith("```"):
|
||||
text = text.strip("`").removeprefix("json").strip()
|
||||
result = json.loads(text)
|
||||
|
||||
report_date = datetime.now().strftime("%Y-%m-%d")
|
||||
timestamp = now_iso()
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO reports (title, report_date, period, highlights_json, sections_json, created_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?)
|
||||
""",
|
||||
(
|
||||
result["title"],
|
||||
report_date,
|
||||
period_label,
|
||||
json.dumps(result["highlights"], ensure_ascii=False),
|
||||
json.dumps(result["sections"], ensure_ascii=False),
|
||||
timestamp,
|
||||
),
|
||||
)
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
print(json.dumps({
|
||||
"status": "success",
|
||||
"title": result["title"],
|
||||
"date": report_date,
|
||||
"highlights": len(result["highlights"]),
|
||||
"sections": len(result["sections"]),
|
||||
}, ensure_ascii=False))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(description="结构化引擎：LLM分析+品牌识别+信号提取+向量嵌入")
|
||||
sub = parser.add_subparsers(dest="command")
|
||||
|
||||
p_analyze = sub.add_parser("analyze", help="批量文章结构化分析")
|
||||
p_analyze.add_argument("--force", action="store_true")
|
||||
p_analyze.add_argument("--pause", type=float, default=0.35)
|
||||
p_analyze.add_argument("--retries", type=int, default=2)
|
||||
|
||||
p_signals = sub.add_parser("signals", help="提取跨文章行业信号")
|
||||
p_signals.add_argument("--days", type=int, default=30)
|
||||
|
||||
p_embed = sub.add_parser("embed", help="生成向量嵌入")
|
||||
p_embed.add_argument("--force", action="store_true")
|
||||
|
||||
p_report = sub.add_parser("report", help="生成周期报告")
|
||||
p_report.add_argument("--days", type=int, default=7)
|
||||
|
||||
args = parser.parse_args()
|
||||
if args.command == "analyze":
|
||||
raise SystemExit(run_batch_analyze(args.force, args.pause, args.retries))
|
||||
elif args.command == "signals":
|
||||
raise SystemExit(run_extract_signals(args.days))
|
||||
elif args.command == "embed":
|
||||
raise SystemExit(run_generate_embeddings(args.force))
|
||||
elif args.command == "report":
|
||||
raise SystemExit(run_generate_report(args.days))
|
||||
else:
|
||||
parser.print_help()
|
||||
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"sources": [
|
||||
{
|
||||
"account": "剑哥聊餐饮",
|
||||
"finder": "sphnu5kSqZT224x",
|
||||
"directory": "/Volumes/Projects/视频/剑哥聊餐饮"
|
||||
},
|
||||
{
|
||||
"account": "餐饮周伯通（赖老师TimLai）",
|
||||
"finder": "sph0XoCC1rhJXB7",
|
||||
"directory": "/Volumes/Projects/视频/餐饮周伯通"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,236 @@
|
||||
#!/usr/bin/env python3
|
||||
"""视频号批量下载管理器。
|
||||
|
||||
基于 wx_channels_download 工具的 API 接口，实现：
|
||||
1. 检查工具是否运行
|
||||
2. 获取视频号创作者的视频列表
|
||||
3. 与本地已下载文件比对，发现新视频
|
||||
4. 触发批量下载
|
||||
5. 下载完成后自动触发 ASR 转写入库
|
||||
|
||||
工具 GitHub: https://github.com/ltaoo/wx_channels_download
|
||||
API 基地址: http://127.0.0.1:2022
|
||||
|
||||
使用前提：
|
||||
- wx_channels_download 以 sudo 运行
|
||||
- IPv6 已禁用（sudo networksetup -setv6off Wi-Fi）
|
||||
- VPN 已关闭
|
||||
- 微信保持运行
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
BASE_DIR = Path(__file__).resolve().parent
|
||||
load_dotenv(BASE_DIR / ".env")
|
||||
|
||||
TOOL_API = os.getenv("WX_VIDEO_API", "http://127.0.0.1:2022")
|
||||
VIDEO_CONFIG_PATH = BASE_DIR / "video_config.json"
|
||||
|
||||
|
||||
def now_iso() -> str:
|
||||
return datetime.now(timezone.utc).astimezone().isoformat(timespec="seconds")
|
||||
|
||||
|
||||
def api_get(endpoint: str, params: dict[str, str] | None = None, timeout: int = 30) -> dict[str, Any]:
|
||||
"""调用 wx_channels_download 工具的 API。"""
|
||||
url = f"{TOOL_API}{endpoint}"
|
||||
if params:
|
||||
qs = "&".join(f"{k}={v}" for k, v in params.items())
|
||||
url += f"?{qs}"
|
||||
req = urllib.request.Request(url, method="GET")
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=timeout) as response:
|
||||
return json.loads(response.read().decode("utf-8"))
|
||||
except urllib.error.HTTPError as exc:
|
||||
detail = exc.read().decode("utf-8", errors="replace")[:500]
|
||||
raise RuntimeError(f"工具 API 返回 {exc.code}：{detail}") from exc
|
||||
except urllib.error.URLError as exc:
|
||||
raise RuntimeError(f"无法连接工具 API（{TOOL_API}）：{exc.reason}\n请确认 wx_channels_download 已以 sudo 启动。") from exc
|
||||
|
||||
|
||||
def check_tool_running() -> bool:
|
||||
"""检查下载工具是否在运行。"""
|
||||
try:
|
||||
result = api_get("/api/channels/version", timeout=5)
|
||||
return bool(result.get("version") or result.get("data"))
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def get_video_list(finder: str) -> list[dict[str, Any]]:
|
||||
"""获取视频号创作者的视频列表。
|
||||
|
||||
需要微信客户端已连接到工具（WebSocket）。
|
||||
"""
|
||||
result = api_get("/api/channels/contact/feed/list", {"finder": finder}, timeout=60)
|
||||
# 适配不同版本的返回格式
|
||||
if isinstance(result, list):
|
||||
return result
|
||||
if isinstance(result, dict):
|
||||
return result.get("data", result.get("list", result.get("feeds", [])))
|
||||
return []
|
||||
|
||||
|
||||
def list_local_videos(directory: Path) -> set[str]:
|
||||
"""列出本地已下载的视频文件名（不含扩展名）。"""
|
||||
if not directory.is_dir():
|
||||
return set()
|
||||
suffixes = {".mp4", ".mov", ".m4v", ".mkv", ".webm"}
|
||||
return {f.stem for f in directory.iterdir() if f.is_file() and f.suffix.lower() in suffixes}
|
||||
|
||||
|
||||
def normalize_title(title: str) -> str:
|
||||
"""将视频标题规范化，用于与本地文件名比对。"""
|
||||
# 去除 wx_channels_download 添加的 _xWT 后缀和画质标记
|
||||
value = re.sub(r"[_ ]?xWT\d+$", "", title, flags=re.I)
|
||||
value = re.sub(r"[\x00-\x1f/:]", "_", value).strip(" .")
|
||||
return value[:180] or "untitled"
|
||||
|
||||
|
||||
def find_new_videos(video_list: list[dict[str, Any]], local_dir: Path) -> list[dict[str, Any]]:
|
||||
"""比对线上视频列表与本地已下载文件，返回未下载的视频。"""
|
||||
local_names = list_local_videos(local_dir)
|
||||
new_videos = []
|
||||
for video in video_list:
|
||||
title = video.get("title", video.get("desc", ""))
|
||||
normalized = normalize_title(title)
|
||||
# 检查本地是否已有该视频（模糊匹配）
|
||||
if not any(normalized in local_name or local_name in normalized for local_name in local_names):
|
||||
new_videos.append(video)
|
||||
return new_videos
|
||||
|
||||
|
||||
def create_batch_download(finder: str, video_ids: list[str]) -> dict[str, Any]:
|
||||
"""创建批量下载任务。"""
|
||||
params = {"finder": finder, "ids": ",".join(video_ids)}
|
||||
return api_get("/api/task/create_batch", params, timeout=120)
|
||||
|
||||
|
||||
def wait_for_downloads(local_dir: Path, expected_count: int, timeout: int = 600) -> int:
|
||||
"""等待下载完成，返回实际新增的文件数。"""
|
||||
initial = list_local_videos(local_dir)
|
||||
deadline = time.time() + timeout
|
||||
while time.time() < deadline:
|
||||
time.sleep(10)
|
||||
current = list_local_videos(local_dir)
|
||||
new_files = current - initial
|
||||
if len(new_files) >= expected_count:
|
||||
return len(new_files)
|
||||
return len(list_local_videos(local_dir) - initial)
|
||||
|
||||
|
||||
def run_download(auto_transcribe: bool = True) -> int:
|
||||
"""主流程：检查工具 → 获取视频列表 → 发现新视频 → 触发下载。"""
|
||||
config = json.loads(VIDEO_CONFIG_PATH.read_text(encoding="utf-8")) if VIDEO_CONFIG_PATH.exists() else {"sources": []}
|
||||
sources = config.get("sources", [])
|
||||
|
||||
if not sources:
|
||||
print(json.dumps({"status": "skipped", "reason": "video_config.json 中未配置视频源"}, ensure_ascii=False))
|
||||
return 0
|
||||
|
||||
# 1. 检查工具是否运行
|
||||
if not check_tool_running():
|
||||
print(json.dumps({
|
||||
"status": "error",
|
||||
"reason": f"wx_channels_download 工具未运行（{TOOL_API}）",
|
||||
"hint": "请执行: sudo <工具路径>/wx_video_download，并确保 IPv6 已禁用、VPN 已关闭"
|
||||
}, ensure_ascii=False))
|
||||
return 1
|
||||
|
||||
print(json.dumps({"status": "tool_running", "api": TOOL_API}, ensure_ascii=False), flush=True)
|
||||
|
||||
total_new = 0
|
||||
total_downloaded = 0
|
||||
|
||||
for source in sources:
|
||||
account = source.get("account", "未知")
|
||||
finder = source.get("finder", "")
|
||||
download_dir = Path(source.get("directory", ""))
|
||||
|
||||
if not finder:
|
||||
print(f"SKIP {account}: 未配置 finder", file=sys.stderr)
|
||||
continue
|
||||
|
||||
# 2. 获取视频列表
|
||||
try:
|
||||
video_list = get_video_list(finder)
|
||||
except Exception as exc:
|
||||
print(f"FAIL {account}: 获取视频列表失败 - {exc}", file=sys.stderr, flush=True)
|
||||
continue
|
||||
|
||||
print(f"INFO {account}: 线上视频 {len(video_list)} 个", flush=True)
|
||||
|
||||
# 3. 发现新视频
|
||||
new_videos = find_new_videos(video_list, download_dir)
|
||||
total_new += len(new_videos)
|
||||
|
||||
if not new_videos:
|
||||
print(f"INFO {account}: 无新视频", flush=True)
|
||||
continue
|
||||
|
||||
print(f"INFO {account}: 发现 {len(new_videos)} 个新视频", flush=True)
|
||||
|
||||
# 4. 触发批量下载
|
||||
video_ids = [v.get("id", v.get("objectId", "")) for v in new_videos if v.get("id") or v.get("objectId")]
|
||||
if video_ids:
|
||||
try:
|
||||
result = create_batch_download(finder, video_ids)
|
||||
print(f"INFO {account}: 批量下载任务已创建 - {result}", flush=True)
|
||||
# 5. 等待下载完成
|
||||
downloaded = wait_for_downloads(download_dir, len(new_videos), timeout=600)
|
||||
total_downloaded += downloaded
|
||||
print(f"INFO {account}: 新增下载 {downloaded} 个文件", flush=True)
|
||||
except Exception as exc:
|
||||
print(f"FAIL {account}: 下载失败 - {exc}", file=sys.stderr, flush=True)
|
||||
else:
|
||||
# 如果无法获取 video_id，提示用户手动下载
|
||||
print(f"WARN {account}: 无法自动获取视频ID，请通过微信内批量下载或 Web 管理页面手动下载", flush=True)
|
||||
print(f" Web 管理页面: {TOOL_API}/download", flush=True)
|
||||
print(f" 下载目录: {download_dir}", flush=True)
|
||||
|
||||
print(json.dumps({
|
||||
"status": "success",
|
||||
"total_new": total_new,
|
||||
"total_downloaded": total_downloaded,
|
||||
}, ensure_ascii=False), flush=True)
|
||||
|
||||
# 6. 自动触发 ASR 转写入库
|
||||
if auto_transcribe and total_downloaded > 0:
|
||||
print("INFO 自动触发视频号 ASR 转写...", flush=True)
|
||||
import subprocess
|
||||
subprocess.run(
|
||||
[sys.executable, str(BASE_DIR / "crawler.py"), "--source-type", "视频号"],
|
||||
cwd=str(BASE_DIR),
|
||||
)
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(description="视频号批量下载管理器")
|
||||
parser.add_argument("--no-transcribe", action="store_true", help="仅下载，不自动触发 ASR 转写")
|
||||
parser.add_argument("--check", action="store_true", help="仅检查工具状态")
|
||||
return parser
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
args = build_parser().parse_args()
|
||||
if args.check:
|
||||
running = check_tool_running()
|
||||
print(json.dumps({"tool_running": running, "api": TOOL_API}, ensure_ascii=False))
|
||||
raise SystemExit(0 if running else 1)
|
||||
raise SystemExit(run_download(auto_transcribe=not args.no_transcribe))
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"videos": [
|
||||
{"url": "https://example.com/video/1", "title": "海底捞创始人分享：翻台率不是目标", "source_name": "海底捞官方视频号"},
|
||||
{"url": "https://example.com/video/2", "title": "西贝贾国龙谈预制菜争议", "source_name": "餐饮O2O视频号"}
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user