feat: 新增db/pipeline数据导入管线,适配5月数据格式

- 整合所有导入脚本到db/pipeline/目录
- config.py: 按月份动态配置数据源路径,支持4月/5月不同目录结构
- import_bill_records_may.py: 5月专用账单导入(51列全渠道订单明细→bill_records + 50列品项销售明细→dish_sales_details)
- import_salary_attendance.py: 添加--salary-file/--attendance-file参数支持动态文件路径
- import_may_data.sh: 5月一键导入脚本(12步全流程)
- run_all.py: 一键全量导入+物化视图刷新编排
- refresh_materialized_views.py: 按依赖顺序刷新38个物化视图
- verify.py/verify_import.py: 数据一致性验证
- README.md: 管线文档
This commit is contained in:
freedakgmail
2026-08-18 22:17:26 +08:00
parent 9b3a9cdb4b
commit 2fc1b96f44
14 changed files with 4096 additions and 65 deletions
+67
View File
@@ -0,0 +1,67 @@
# 数据导入与物化刷新流水线
## 目录结构
```
db/pipeline/
├── README.md ← 本文件
├── config.py ← 数据源路径与数据库配置
├── run_all.py ← 一键全量导入+物化刷新
├── refresh_materialized_views.py ← 物化视图按依赖顺序刷新
├── verify.py ← 测试库与正式库行数对比验证
├── verify_import.py ← 详细数据对比验证
├── import_bill_records.py ← 账单查询原始数据导入(4月格式,196列宽表)
├── import_bill_records_may.py ← 5月专用账单导入(51列全渠道订单明细→bill_records + 50列品项销售明细→dish_sales_details
├── import_monthly_data.py ← 月度业务数据导入(库存/销售/成本/费用/中央厨房/配送)
├── import_salary_attendance.py ← 薪资考勤数据导入
├── import_store_location.py ← 门店位置与映射表导入
└── import_derived_data.py ← 派生分析数据生成
```
## 导入脚本
| 脚本 | 数据表 | 数据源 |
|------|--------|--------|
| `import_bill_records.py` | bill_records, bill_columns | 账单查询 Excel 目录(17个文件) |
| `import_monthly_data.py` | inventory_cost_records, dish_sales_details, dish_cost_analysis_*, operating_expense_records, central_kitchen_*, distribution_detail_records | 各业务 Excel 文件 |
| `import_salary_attendance.py` | salary_detail_records, attendance_records | 薪资/考勤 Excel |
| `import_store_location.py` | store_location_master, store_location_source_rows, sales_store_location_mapping, store_name_mapping, inventory_store_mapping, operating_expense_store_mapping | 各店信息 Excel |
| `import_derived_data.py` | dish_diagnosis_snapshot, central_kitchen_manufacturing_cost_pool | 从已有表派生 |
## 物化视图刷新顺序(按依赖分层)
| 层级 | 物化视图 | 依赖 |
|------|----------|------|
| L0 | bill_fact | 基础表 |
| L1 | mv_daily_revenue, mv_dish_basket_monthly, mv_dish_pair_summary_monthly, mv_dish_sku_summary_monthly, mv_dish_store_summary_monthly, mv_district_site_benchmark_monthly, mv_inventory_cost_classified_monthly, mv_site_segment_benchmark_monthly, mv_store_*_monthly (12个), mv_store_scorecard | bill_fact |
| L1 | dish_sales_april | bill_fact |
| L2 | dish_basket_april, dish_category_summary_april, dish_member_sku_april, dish_sku_summary_april, dish_store_sku_april | dish_sales_april |
| L2 | mv_dish_sku_abc_monthly | mv_dish_sku_summary_monthly |
| L2 | v_store_benchmark, v_store_category_mix, v_store_platform_economics | bill_fact |
| L3 | dish_store_summary_april | dish_basket_april |
| L3 | v_store_action_list | v_store_benchmark, v_store_category_mix, v_store_platform_economics |
| L3 | mv_store_action_priority_deep_april, mv_store_theoretical_actual_cost_april | bill_fact / dish_store_summary_april |
| L4 | v_store_execution_priority | v_store_action_list |
## 使用方法
### 一键全量导入
```bash
python3 db/pipeline/run_all.py --month 2026-04-01 --db bill_query_test
```
### 仅刷新物化视图
```bash
python3 db/pipeline/refresh_materialized_views.py --db bill_query_test
```
### 验证数据一致性
```bash
python3 db/pipeline/verify.py --db bill_query_test
```
### 单独导入某类数据
参见各脚本 `--help`,例如:
```bash
python3 db/pipeline/import_monthly_data.py --dataset inventory --month 2026-04-01 --file <path> --db bill_query_test
```