freedakgmail
2fc1b96f44
- 整合所有导入脚本到db/pipeline/目录 - config.py: 按月份动态配置数据源路径，支持4月/5月不同目录结构 - import_bill_records_may.py: 5月专用账单导入(51列全渠道订单明细→bill_records + 50列品项销售明细→dish_sales_details) - import_salary_attendance.py: 添加--salary-file/--attendance-file参数支持动态文件路径 - import_may_data.sh: 5月一键导入脚本(12步全流程) - run_all.py: 一键全量导入+物化视图刷新编排 - refresh_materialized_views.py: 按依赖顺序刷新38个物化视图 - verify.py/verify_import.py: 数据一致性验证 - README.md: 管线文档
68 lines
3.5 KiB
Markdown
68 lines
3.5 KiB
Markdown
# 数据导入与物化刷新流水线
|
||
|
||
## 目录结构
|
||
|
||
```
|
||
db/pipeline/
|
||
├── README.md ← 本文件
|
||
├── config.py ← 数据源路径与数据库配置
|
||
├── run_all.py ← 一键全量导入+物化刷新
|
||
├── refresh_materialized_views.py ← 物化视图按依赖顺序刷新
|
||
├── verify.py ← 测试库与正式库行数对比验证
|
||
├── verify_import.py ← 详细数据对比验证
|
||
├── import_bill_records.py ← 账单查询原始数据导入（4月格式，196列宽表）
|
||
├── import_bill_records_may.py ← 5月专用账单导入（51列全渠道订单明细→bill_records + 50列品项销售明细→dish_sales_details）
|
||
├── import_monthly_data.py ← 月度业务数据导入（库存/销售/成本/费用/中央厨房/配送）
|
||
├── import_salary_attendance.py ← 薪资考勤数据导入
|
||
├── import_store_location.py ← 门店位置与映射表导入
|
||
└── import_derived_data.py ← 派生分析数据生成
|
||
```
|
||
|
||
## 导入脚本
|
||
|
||
| 脚本 | 数据表 | 数据源 |
|
||
|------|--------|--------|
|
||
| `import_bill_records.py` | bill_records, bill_columns | 账单查询 Excel 目录（17个文件） |
|
||
| `import_monthly_data.py` | inventory_cost_records, dish_sales_details, dish_cost_analysis_*, operating_expense_records, central_kitchen_*, distribution_detail_records | 各业务 Excel 文件 |
|
||
| `import_salary_attendance.py` | salary_detail_records, attendance_records | 薪资/考勤 Excel |
|
||
| `import_store_location.py` | store_location_master, store_location_source_rows, sales_store_location_mapping, store_name_mapping, inventory_store_mapping, operating_expense_store_mapping | 各店信息 Excel |
|
||
| `import_derived_data.py` | dish_diagnosis_snapshot, central_kitchen_manufacturing_cost_pool | 从已有表派生 |
|
||
|
||
## 物化视图刷新顺序（按依赖分层）
|
||
|
||
| 层级 | 物化视图 | 依赖 |
|
||
|------|----------|------|
|
||
| L0 | bill_fact | 基础表 |
|
||
| L1 | mv_daily_revenue, mv_dish_basket_monthly, mv_dish_pair_summary_monthly, mv_dish_sku_summary_monthly, mv_dish_store_summary_monthly, mv_district_site_benchmark_monthly, mv_inventory_cost_classified_monthly, mv_site_segment_benchmark_monthly, mv_store_*_monthly (12个), mv_store_scorecard | bill_fact |
|
||
| L1 | dish_sales_april | bill_fact |
|
||
| L2 | dish_basket_april, dish_category_summary_april, dish_member_sku_april, dish_sku_summary_april, dish_store_sku_april | dish_sales_april |
|
||
| L2 | mv_dish_sku_abc_monthly | mv_dish_sku_summary_monthly |
|
||
| L2 | v_store_benchmark, v_store_category_mix, v_store_platform_economics | bill_fact |
|
||
| L3 | dish_store_summary_april | dish_basket_april |
|
||
| L3 | v_store_action_list | v_store_benchmark, v_store_category_mix, v_store_platform_economics |
|
||
| L3 | mv_store_action_priority_deep_april, mv_store_theoretical_actual_cost_april | bill_fact / dish_store_summary_april |
|
||
| L4 | v_store_execution_priority | v_store_action_list |
|
||
|
||
## 使用方法
|
||
|
||
### 一键全量导入
|
||
```bash
|
||
python3 db/pipeline/run_all.py --month 2026-04-01 --db bill_query_test
|
||
```
|
||
|
||
### 仅刷新物化视图
|
||
```bash
|
||
python3 db/pipeline/refresh_materialized_views.py --db bill_query_test
|
||
```
|
||
|
||
### 验证数据一致性
|
||
```bash
|
||
python3 db/pipeline/verify.py --db bill_query_test
|
||
```
|
||
|
||
### 单独导入某类数据
|
||
参见各脚本 `--help`，例如：
|
||
```bash
|
||
python3 db/pipeline/import_monthly_data.py --dataset inventory --month 2026-04-01 --file <path> --db bill_query_test
|
||
```
|