refactor: replace custom PEFT pipeline with Transformers+PEFT example
Remove the custom peft/ directory (LoRA/DoRA adapters, trainer, data pipeline) in favor of a lightweight fine-tuning example that uses the standard HuggingFace Transformers + PEFT ecosystem. The new example at timesfm-forecasting/examples/finetuning/ demonstrates LoRA fine-tuning via TimesFm2_5ModelForPrediction and the peft library, based on the approach by @kashif at HuggingFace. - Remove peft/ (8 files) - Add timesfm-forecasting/examples/finetuning/finetune_lora.py - Add timesfm-forecasting/examples/finetuning/README.md - Update README.md to reference new example - Clean up .gitignore (remove peft_checkpoints/)
This commit is contained in:
@@ -0,0 +1,102 @@
|
||||
# Fine-Tuning TimesFM 2.5 with LoRA
|
||||
|
||||
Parameter-efficient fine-tuning of
|
||||
[TimesFM 2.5](https://huggingface.co/google/timesfm-2.5-200m-transformers)
|
||||
using **HuggingFace Transformers** and **PEFT (LoRA)**.
|
||||
|
||||
This approach is based on the fine-tuning workflow by
|
||||
[@kashif](https://github.com/kashif) at HuggingFace
|
||||
([notebook](https://github.com/huggingface/notebooks/blob/main/examples/timesfm2_5.ipynb)).
|
||||
|
||||
## How It Works
|
||||
|
||||
TimesFM 2.5 is available as a standard
|
||||
[Transformers](https://github.com/huggingface/transformers) model
|
||||
(`TimesFm2_5ModelForPrediction`). This means it supports the full Transformers
|
||||
ecosystem out of the box, including:
|
||||
|
||||
- **PEFT adapters** — LoRA, QLoRA, etc. via the
|
||||
[`peft`](https://github.com/huggingface/peft) library
|
||||
- **All attention backends** — eager, SDPA, Flash Attention 2/3, Flex Attention
|
||||
- **Standard `from_pretrained` / `save_pretrained` workflow**
|
||||
|
||||
The model's forward pass natively computes a training loss when `future_values`
|
||||
are provided, so fine-tuning requires nothing more than a standard PyTorch
|
||||
training loop.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Install
|
||||
|
||||
```bash
|
||||
pip install transformers accelerate peft pandas pyarrow scikit-learn
|
||||
```
|
||||
|
||||
### Train
|
||||
|
||||
```bash
|
||||
# Fine-tune with default settings on the retail sales dataset
|
||||
python finetune_lora.py
|
||||
|
||||
# Custom hyperparameters
|
||||
python finetune_lora.py \
|
||||
--epochs 20 \
|
||||
--batch_size 64 \
|
||||
--lr 5e-5 \
|
||||
--lora_r 8 \
|
||||
--lora_alpha 16 \
|
||||
--context_len 64 \
|
||||
--horizon_len 13 \
|
||||
--output_dir my-retail-adapter
|
||||
```
|
||||
|
||||
### Evaluate
|
||||
|
||||
```bash
|
||||
# Evaluate a previously trained adapter (skip training)
|
||||
python finetune_lora.py --eval_only --output_dir timesfm2_5-retail-lora
|
||||
```
|
||||
|
||||
## Key Concepts
|
||||
|
||||
### No External Normalisation
|
||||
|
||||
TimesFM 2.5 applies its own internal instance normalisation (RevIN). **Do not**
|
||||
normalise your data externally — feed raw values and let the model handle it.
|
||||
|
||||
### Random Window Sampling
|
||||
|
||||
Following [Chronos-2](https://github.com/amazon-science/chronos-forecasting),
|
||||
each training example is a random `(context, horizon)` window sliced from one of
|
||||
the input series. This is more data-efficient than always using the same
|
||||
fixed window per series.
|
||||
|
||||
### LoRA Target Modules
|
||||
|
||||
Using `target_modules="all-linear"` applies LoRA to every linear layer in the
|
||||
model. With `r=4` this adds only ~0.6% trainable parameters (~1.4M out of
|
||||
~232M), which is enough to meaningfully adapt the model to a new domain.
|
||||
|
||||
## CLI Options
|
||||
|
||||
| Flag | Default | Description |
|
||||
|------|---------|-------------|
|
||||
| `--model_id` | `google/timesfm-2.5-200m-transformers` | HuggingFace model ID |
|
||||
| `--context_len` | `64` | Context length for training windows |
|
||||
| `--horizon_len` | `13` | Forecast horizon in time steps |
|
||||
| `--epochs` | `10` | Training epochs |
|
||||
| `--batch_size` | `32` | Batch size |
|
||||
| `--lr` | `1e-4` | Learning rate |
|
||||
| `--lora_r` | `4` | LoRA rank |
|
||||
| `--lora_alpha` | `8` | LoRA alpha |
|
||||
| `--lora_dropout` | `0.05` | LoRA dropout |
|
||||
| `--num_samples` | `5000` | Random training windows to pre-sample |
|
||||
| `--output_dir` | `timesfm2_5-retail-lora` | Where to save the adapter |
|
||||
| `--seed` | `42` | Random seed |
|
||||
| `--eval_only` | — | Skip training; evaluate existing adapter |
|
||||
|
||||
## Acknowledgements
|
||||
|
||||
The Transformers integration and fine-tuning approach were developed by
|
||||
[@kashif](https://github.com/kashif) at HuggingFace. See the original notebook:
|
||||
<https://github.com/huggingface/notebooks/blob/main/examples/timesfm2_5.ipynb>
|
||||
Reference in New Issue
Block a user