 darkpowerxo
|
7357458e45
|
feat: add LoRA/DoRA adapter layers for TimesFM 2.5 (PyTorch)
Implement production-grade PEFT adapters targeting the 2.5 architecture:
- LoRALinear: low-rank A/B decomposition with scaling (alpha/rank)
- DoRALinear: weight-decomposed LoRA (magnitude + direction)
- inject_adapters(): freezes base weights, wraps target nn.Linear modules
- Supports fused QKV (qkv_proj), attention output, and FFN layers
- num_adapter_layers controls how many top layers get adapters (0=all)
- target_modules selects 'all', 'attention', or 'ffn'
- merge_adapters(): folds adapter deltas back into base nn.Linear
- save/load_adapter_weights(): safetensors adapter-only checkpoints
- PEFTConfig dataclass with all hyperparameters
References:
LoRA — https://arxiv.org/abs/2106.09685
DoRA — https://arxiv.org/abs/2402.09353
|
2026-04-08 13:51:55 -04:00 |
|
|
siriuz42
|
7d8f3d971d
|
2.0.0 initial
|
2025-09-12 00:18:08 +00:00 |
|
|
Tanmay Shishodia
|
55f71de98a
|
Create PEFT README.md
|
2024-08-04 12:27:56 +05:30 |
|
|
tanmayshishodia
|
d72ff835fb
|
keep only a single bash script
|
2024-08-04 09:56:11 +05:30 |
|
|
tanmayshishodia
|
18da73a783
|
remove helper and fix early stopping logic
|
2024-07-18 19:18:56 +00:00 |
|
|
tanmayshishodia
|
5901805e7e
|
add bash scripts for running diff FT strategies
|
2024-07-17 21:39:14 +00:00 |
|
|
tanmayshishodia
|
a59979d3b4
|
add example dora.sh file
|
2024-07-16 19:20:04 +00:00 |
|
|
tanmayshishodia
|
845661dfa3
|
update usage tutorial
|
2024-07-16 18:24:50 +00:00 |
|
|
tanmayshishodia
|
461c2cd194
|
update checkpoint dir name
|
2024-07-16 16:26:30 +00:00 |
|
|
tanmayshishodia
|
71d980267d
|
add parameter efficient finetuning pipeline
Why? this commit adds a generic finetuning pipeline with LoRA and DoRA support
|
2024-07-16 01:26:36 +00:00 |
|