updating readme with new results

experiments/environment.yml

@@ -20,5 +20,8 @@ dependencies:
     - git+https://github.com/amazon-science/chronos-forecasting.git
     - praxis
     - paxml
-    - jax[cuda12]==0.4.26
+    - jax[cuda12]
     - einshape
+    - python-dotenv
+    - nixtla>=0.5.1
+    - rich

experiments/environment_cpu.yml

@@ -20,5 +20,8 @@ dependencies:
     - git+https://github.com/amazon-science/chronos-forecasting.git
     - praxis
     - paxml
-    - jax[cpu]==0.4.26
+    - jax[cpu]
     - einshape
+    - python-dotenv
+    - nixtla>=0.5.1
+    - rich

experiments/extended_benchmarks/README.md

@@ -2,7 +2,6 @@
 
 The benchmark setting has been borrowed from Nixtla's original [benchmarking](https://github.com/AzulGarza/nixtla/tree/main/experiments/amazon-chronos) of time-series foundation models against a strong statistical ensemble. Later more datasets were added by the Chronos team in this [pull request](https://github.com/shchur/nixtla/tree/chronos-full-eval/experiments/amazon-chronos). We compare on all the datasets in this extended benchmarks.
 
-All experiments were performed on a [g2-standard-32](https://cloud.google.com/compute/docs/gpus). 
 
 ## Running TimesFM on the benchmark
 
@@ -19,7 +18,11 @@ Note: In the current version of TimesFM we focus on point forecasts and therefor
 
 ## Benchmark Results
 
-![Benchmark Results Table](./tfm_results.png)
+![Benchmark Results Table](./tfm_extended_new.png)
+
+__Update:__ We have added TimeGPT-1 to the benchmark results. We had to remove the Dominick dataset as we were not able to run TimeGPT-1 on this benchmark. Note that the previous results including Dominick remain available at `./tfm_results.png`. In order to reproduce the results for TimeGPT-1, please run `run_timegpt.py`.
+
+_Remark:_ All baselines except the ones involving TimeGPT were run performed on a [g2-standard-32](https://cloud.google.com/compute/docs/gpus). Since TimeGPT-1 can only be accessed by an API, the time column might not reflect the true speed of the model as it also includes the communication cost. Moreover, we are not sure about the exact backend hardware for TimeGPT.
 
 We can see that TimesFM performs the best in terms of both mase and smape. More importantly it is much faster than the other methods, in particular it is more than 600x faster than StatisticalEnsemble and 80x faster than Chronos (Large).
 

experiments/extended_benchmarks/run_timegpt.py

@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-"""Evaluation script for timesfm."""
+"""Evaluation script for timegpt."""
 
 import os
 import sys
@@ -57,7 +57,9 @@ dataset_names = [
 ]
 
 _MODEL_NAME = flags.DEFINE_string(
-    "model_name", "timegpt-1-long-horizon", "Path to model"
+    "model_name",
+    "timegpt-1-long-horizon",
+    "Path to model, can also be set to timegpt-1",
 )
 _SAVE_DIR = flags.DEFINE_string("save_dir", "./results", "Save directory")
 

pyproject.toml

@@ -8,7 +8,7 @@ dependencies = [
     "einshape>=1.0.0",
     "paxml>=1.4.0",
     "praxis>=1.4.0",
-    "jax==0.4.26",
+    "jax>=0.4.26",
     "numpy>=1.26.4",
     "pandas>=2.1.4",
 ]