 darkpowerxo
|
7357458e45
|
feat: add LoRA/DoRA adapter layers for TimesFM 2.5 (PyTorch)
Implement production-grade PEFT adapters targeting the 2.5 architecture:
- LoRALinear: low-rank A/B decomposition with scaling (alpha/rank)
- DoRALinear: weight-decomposed LoRA (magnitude + direction)
- inject_adapters(): freezes base weights, wraps target nn.Linear modules
- Supports fused QKV (qkv_proj), attention output, and FFN layers
- num_adapter_layers controls how many top layers get adapters (0=all)
- target_modules selects 'all', 'attention', or 'ffn'
- merge_adapters(): folds adapter deltas back into base nn.Linear
- save/load_adapter_weights(): safetensors adapter-only checkpoints
- PEFTConfig dataclass with all hyperparameters
References:
LoRA — https://arxiv.org/abs/2106.09685
DoRA — https://arxiv.org/abs/2402.09353
|
2026-04-08 13:51:55 -04:00 |
|