HART: Hybrid Attention-RWKV Time series Foundation Model
Authors: Qingwen Lin, Boyan Xu, Shifeng Xie, Zhifeng Hao, Keli Zhang, Ruichu Cai
Organizations: School of Computer Science, Guangdong University of Technology · Huawei Noah’s Ark Lab · LIPADE, Université Paris Cité, Paris, France · College of Science, Shantou University · Peng Cheng Laboratory
Non-autoregressive multi-step forecasting (NAF) enables time series foundation models to predict an entire future horizon in a single forward pass. Despite its strong sequence modeling capabilities, RWKV-7 exhibits long-horizon degradation in our NAF experiments. In this setup, future patches are represented as missing inputs, requiring the recurrent state to evolve across the prediction horizon without new observations. Our analysis of RWKV-7's generalized delta rule suggests a mechanism we term Placeholder Idling Decay (PID): repeated state transitions can attenuate historical information when placeholder-driven writes provide insufficient compensation. Layer-wise analysis reveals reduced state-injection norms in most layers, while transition-matrix norms remain similar between historical and prediction regions, consistent with PID. Motivated by this analysis, we propose HART, a Hybrid Attention-RWKV Time-series Foundation Model that alternates attention and RWKV blocks, combining direct access to historical tokens with recurrent temporal modeling. Controlled experiments on GIFT-Eval and TIME-Benchmark show that HART improves over pure RWKV at medium and long horizons while maintaining comparable short-horizon performance, and achieves better aggregate performance than a comparably sized pure Transformer. With 36M parameters, HART achieves long-horizon MASE comparable to Toto-2.0-1B on GIFT-Eval (0.752 vs. 0.754), using approximately 1/28 of its parameters.
Figures & tables
Figure 1: Qualitative NAF predictions on a synthetic signal. Gray: history and ground truth; colored: median predictions with quantile.
Figure 2: Architecture of the proposed hybrid backbone. (a) Overall pipeline: time-patch embedding feeds into an alternating stack of Attention and RWKV blocks ( L times), followed by the output head. (b) Attention block: RMSNorm, gated self-attention, and feed-forward with residual connections. (c) RWKV block: LayerNorm, RWKV-TimeMIX (generalized delta rule), and feed-forward with residual connections.
Kt
Mt
Layer
Hist.
Pred.
Ratio
Hist.
Pred.
Ratio
1
8.78
6.91
0.79
6.77
6.98
1.03
2
4.72
3.10
0.66
7.20
7.29
1.01
3
4.72
2.93
0.62
7.26
7.39
1.02
4
6.00
5.25
0.87
7.24
7.32
1.01
5
7.15
4.29
0.60
7.17
7.27
1.01
Table 1: Layer-wise mean Frobenius norms of Kt and Mt in the historical and prediction regions, averaged over 4,096 sequences. Lower ratios indicate smaller prediction-region norms relative to the historical region.
Model
GIFT-Eval
TIME-Benchmark
Short
Medium
Long
All
Short
Medium
Long
All
RWKV-36M
0.689
0.749 †
0.770 †
0.719 †
0.620
0.727 †
0.755 †
0.676 †
Transformer-35M
0.695
0.747
0.763 †
0.721 †
0.618
0.725 †
0.735
0.670 †
HART-36M
0.689
0.727
0.752
0.710
0.618
0.716
0.723
0.666
Table 2: Self-developed architectures on GIFT-Eval and TIME-Benchmark (MASE, geometric mean). Lower is better; bold marks the lowest displayed value in each column, including ties. †indicates that the paired median difference between HART and the baseline has a 95% bootstrap confidence interval excluding zero.
Model
GIFT-Eval
TIME-Benchmark
Short
Medium
Long
All
Short
Medium
Long
All
YingLong-110M Wang et al. (2025)
0.766
0.856
0.885
0.809
–
–
–
–
YingLong-300M Wang et al. (2025)
0.758
0.841
0.867
0.798
–
–
–
–
TimesFM-2.0-500M Das et al. (2023)
0.704
0.810
0.860
0.758
0.652
0.791
0.797
0.718
Sundial-128M Liu et al. (2025c)
0.732
0.764
0.785
0.750
0.700
0.813
0.831
0.757
Xihe-1.5B Sun et al. (2025)
0.689
0.718
0.763
0.711
–
–
–
–
Table 3: External baselines on GIFT-Eval and TIME-Benchmark (MASE, geometric mean). Lower is better; bold marks the lowest displayed value in each column, including ties.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Model
bitbrains_fast_storage/5T
bitbrains_rnd/5T
bizitobs_application/10S
bizitobs_l2c/5T
bizitobs_l2c/H
bizitobs_service/10S
electricity/15T
electricity/H
ett1/15T
ett1/H
ett2/15T
ett2/H
jena_weather/10T
jena_weather/H
kdd_cup_2018/H
loop_seattle/5T
loop_seattle/H
m_dense/H
solar/10T
solar/H
sz_taxi/15T
HART-36M
0.784
0.947
0.969
0.531
0.394
0.982
0.742
0.741
0.853
1.049
0.889
0.970
1.025
0.618
0.709
0.604
0.562
0.451
0.909
0.814
0.784
RWKV-36M
0.797
0.951
1.031
0.471
0.420
0.959
0.785
0.766
0.933
1.062
0.945
0.937
1.011
0.619
0.714
0.691
0.582
0.486
0.924
0.834
0.785
Transformer-35M
0.783
0.947
0.977
0.636
0.397
1.090
0.755
0.744
0.847
1.029
0.902
0.883
0.979
0.626
0.736
0.621
0.567
0.454
0.935
0.820
0.784
Kairos-23M
0.859
0.976
0.947
0.764
0.390
1.015
0.818
0.845
0.849
0.903
0.896
0.894
0.863
0.842
0.765
0.701
0.611
0.568
0.926
0.857
0.768
TiRex-35M
0.805
0.957
1.059
0.417
0.410
1.093
0.801
0.810
0.857
0.893
0.884
1.050
0.852
0.777
0.756
0.761
0.566
0.470
0.954
0.771
0.737
Toto-2.0-22M
0.736
0.934
1.037
0.693
0.402
0.967
0.787
0.821
0.838
0.914
0.875
0.925
0.852
0.837
0.762
0.714
0.583
0.523
0.954
0.859
0.741
Appendix
Table 4: Per-dataset MASE on GIFT-Eval long-term datasets (21 datasets).
Model
bitbrains_fast_storage/5T
bitbrains_rnd/5T
bizitobs_application/10S
bizitobs_l2c/5T
bizitobs_l2c/H
bizitobs_service/10S
electricity/15T
electricity/H
ett1/15T
ett1/H
ett2/15T
ett2/H
jena_weather/10T
jena_weather/H
kdd_cup_2018/H
loop_seattle/5T
loop_seattle/H
m_dense/H
solar/10T
solar/H
sz_taxi/15T
HART-36M
0.805
0.965
0.765
0.463
0.331
0.817
0.717
0.745
0.845
0.889
0.830
0.898
0.932
0.840
0.691
0.619
0.620
0.424
0.903
0.890
0.774
RWKV-36M
0.815
0.968
0.906
0.484
0.340
0.867
0.737
0.756
0.895
0.879
0.883
0.857
0.947
0.854
0.690
0.700
0.632
0.452
0.880
0.902
0.773
Transformer-35M
0.804
0.967
0.962
0.595
0.341
0.931
0.724
0.753
0.834
0.904
0.847
0.801
0.919
0.861
0.696
0.629
0.613
0.434
0.870
0.903
0.771
Kairos-23M
0.886
0.989
0.936
0.605
0.325
0.906
0.788
0.818
0.854
0.840
0.865
0.814
0.882
0.944
0.726
0.702
0.661
0.511
0.812
0.900
0.781
TiRex-35M
0.822
0.972
0.982
0.498
0.344
0.889
0.771
0.782
0.854
0.788
0.833
0.916
0.846
0.853
0.725
0.763
0.620
0.445
0.924
0.838
0.753
Toto-2.0-22M
0.763
0.955
0.919
0.512
0.331
0.830
0.764
0.799
0.837
0.807
0.842
0.842
0.792
0.942
0.733
0.722
0.634
0.470
0.938
0.932
0.756
Appendix
Table 5: Per-dataset MASE on GIFT-Eval medium-term datasets (21 datasets).
Model
bizitobs_application
bizitobs_l2c/5T
bizitobs_l2c/H
bizitobs_service
car_parts/M
covid_deaths/D
electricity/15T
electricity/D
electricity/H
electricity/W
ett1/15T
ett1/D
ett1/H
ett1/W
ett2/15T
ett2/D
ett2/H
ett2/W
hierarchical_sales/D
hierarchical_sales/W
hospital/M
jena_weather/10T
jena_weather/D
jena_weather/H
kdd_cup_2018/D
kdd_cup_2018/H
loop_seattle/5T
loop_seattle/D
HART-36M
0.372
0.265
0.366
0.585
0.711
0.960
0.584
0.714
0.661
0.694
0.707
0.976
0.846
0.938
0.675
0.940
0.772
0.976
0.664
0.700
0.908
0.356
0.649
0.700
0.790
0.638
0.728
0.522
RWKV-36M
0.422
0.238
0.365
0.627
0.710
0.955
0.591
0.717
0.678
0.691
0.728
0.971
0.862
0.909
0.676
0.918
0.776
0.927
0.660
0.702
0.907
0.367
0.620
0.706
0.793
0.638
0.761
0.518
Transformer-35M
0.488
0.270
0.361
0.635
0.705
0.972
0.569
0.713
0.668
0.704
0.710
0.972
0.853
0.938
0.669
0.898
0.774
0.978
0.657
0.705
0.907
0.373
0.638
0.707
0.784
0.627
0.749
0.525
Kairos-23M
0.573
0.288
0.396
0.685
0.819
0.874
0.649
0.730
0.732
0.678
0.761
0.921
0.873
0.806
0.749
0.935
0.809
1.252
0.663
0.721
0.846
0.387
0.758
0.733
0.791
0.706
0.741
0.508
TiRex-35M
0.537
0.283
0.407
0.674
0.705
0.727
0.617
0.710
0.670
0.690
0.759
0.944
0.836
0.904
0.706
0.943
0.784
0.983
0.657
0.704
0.821
0.377
0.645
0.724
0.802
0.701
0.735
0.505
Toto-2.0-22M
0.542
0.254
0.340
0.631
0.735
0.632
0.596
0.737
0.779
0.773
0.713
0.930
0.871
0.857
0.669
0.926
0.802
1.054
0.661
0.711
0.823
0.351
0.711
0.729
0.801
0.706
0.730
0.514
Appendix
Table 6: Per-dataset MASE on GIFT-Eval short-term datasets (Part 1: A to M, 28 datasets).
Model
loop_seattle/H
m4_daily/D
m4_hourly/H
m4_monthly/M
m4_quarterly/Q
m4_weekly/W
m4_yearly/A
m_dense/D
m_dense/H
restaurant/D
saugeen/D
saugeen/M
saugeen/W
solar/10T
solar/D
solar/H
solar/W
sz_taxi/15T
sz_taxi/H
temperature_rain/D
us_births/D
us_births/M
us_births/W
bitbrains_fast_storage/5T
bitbrains_fast_storage/H
bitbrains_rnd/5T
bitbrains_rnd/H
HART-36M
0.654
1.005
0.558
0.834
0.874
0.703
1.100
0.407
0.520
0.671
0.824
0.797
0.566
0.991
0.841
0.867
0.675
0.717
0.770
0.656
0.193
1.243
0.731
0.589
0.827
0.828
0.970
RWKV-36M
0.665
0.996
0.550
0.824
0.865
0.698
1.112
0.412
0.512
0.672
0.789
0.778
0.562
0.797
0.844
0.878
0.670
0.717
0.770
0.710
0.202
1.541
0.652
0.591
0.830
0.829
0.969
Transformer-35M
0.664
0.998
0.554
0.826
0.860
0.706
1.097
0.396
0.535
0.674
0.846
0.808
0.593
0.913
0.838
0.884
0.761
0.718
0.773
0.656
0.183
1.458
0.676
0.595
0.826
0.833
0.970
Kairos-23M
0.667
1.011
0.882
0.771
0.743
0.892
0.752
0.407
0.571
0.692
0.895
0.715
0.623
1.071
0.859
0.912
0.715
0.727
0.773
0.699
0.234
1.100
0.794
0.711
0.854
0.889
0.980
TiRex-35M
0.638
0.986
0.605
0.731
0.715
0.717
0.848
0.405
0.526
0.674
0.879
0.767
0.588
0.984
0.827
0.782
0.707
0.710
0.763
0.660
0.237
0.791
0.669
0.608
0.785
0.840
0.963
Toto-2.0-22M
0.680
0.978
0.691
0.728
0.730
0.728
0.826
0.415
0.539
0.700
0.878
0.732
0.630
0.822
0.842
0.912
0.827
0.715
0.765
0.682
0.208
0.927
0.695
0.557
0.707
0.817
0.959
Appendix
Table 7: Per-dataset MASE on GIFT-Eval short-term datasets (Part 2: M–Z, 27 datasets).
Model
Australia_Solar/H
CPHL/15T
CPHL/30T
CPHL/H
Coastal_T_S/15T
Coastal_T_S/20T
Coastal_T_S/5T
Coastal_T_S/H
EWELD_Load/15T
Finland_Traffic/15T
MetroPT-3/5T
NE_China_Wind/H
OpenElectricity_NEM/5T
SG_Carpark/15T
SG_PM25/H
SG_Weather/D
Smart_Manufacturing/H
Water_Quality_Darwin/15T
current_velocity/10T
current_velocity/15T
current_velocity/20T
current_velocity/5T
current_velocity/H
epf_electricity_price/H
HART-36M
0.834
0.779
0.806
0.805
0.860
0.796
0.813
0.885
0.664
0.479
0.748
0.725
0.690
0.492
0.696
0.710
0.666
0.869
0.667
0.752
0.790
0.663
0.671
0.687
RWKV-36M
0.847
0.764
0.846
0.815
0.857
0.881
0.858
0.868
0.723
0.595
0.759
0.713
0.766
0.598
0.700
0.710
0.668
0.848
0.718
0.765
0.805
0.794
0.655
0.680
Transformer-35M
0.861
0.738
0.788
0.798
0.853
0.810
0.902
0.839
0.664
0.491
0.749
0.744
0.669
0.542
0.711
0.719
0.666
0.832
0.689
0.759
0.796
0.787
0.694
0.694
TiRex-35M
0.830
0.751
0.804
0.858
0.846
0.837
0.852
0.885
0.715
0.568
0.743
0.796
0.862
0.623
0.705
0.715
0.667
0.905
0.729
0.763
0.809
0.737
0.698
0.679
Toto-2.0-22M
0.890
0.750
0.812
0.846
0.854
0.761
0.820
0.802
0.718
0.520
0.751
0.844
0.727
0.624
0.693
0.711
0.669
0.811
0.676
0.735
0.754
0.709
0.662
0.704
Toto-2.0-313M
0.835
0.730
0.792
0.826
0.846
0.662
0.811
0.771
0.663
0.467
0.801
0.803
0.701
0.567
0.684
0.705
0.669
0.787
0.619
0.723
0.726
0.603
0.633
0.681
Appendix
Table 8: Per-dataset MASE on TIME-Benchmark long-term tasks (24 tasks).
Model
Australia_Solar/H
CPHL/15T
CPHL/30T
CPHL/H
Coastal_T_S/15T
Coastal_T_S/20T
Coastal_T_S/5T
Coastal_T_S/H
EWELD_Load/15T
Finland_Traffic/15T
MetroPT-3/5T
NE_China_Wind/H
OpenElectricity_NEM/5T
SG_Carpark/15T
SG_PM25/H
SG_Weather/D
Smart_Manufacturing/H
Water_Quality_Darwin/15T
current_velocity/10T
current_velocity/15T
current_velocity/20T
current_velocity/5T
current_velocity/H
epf_electricity_price/H
HART-36M
0.773
0.758
0.813
0.808
0.843
0.780
0.774
0.799
0.666
0.682
0.755
0.750
0.532
0.534
0.758
0.729
0.683
0.834
0.667
0.725
0.740
0.646
0.624
0.642
RWKV-36M
0.787
0.765
0.824
0.800
0.847
0.780
0.776
0.825
0.685
0.706
0.760
0.725
0.610
0.588
0.762
0.731
0.686
0.804
0.698
0.740
0.739
0.646
0.607
0.639
Transformer-35M
0.787
0.733
0.820
0.792
0.841
0.739
0.799
0.814
0.674
0.707
0.754
0.768
0.530
0.568
0.770
0.735
0.684
0.824
0.680
0.731
0.744
0.702
0.641
0.662
TiRex-35M
0.805
0.758
0.834
0.826
0.851
0.837
0.805
0.857
0.694
0.811
0.743
0.789
0.689
0.703
0.766
0.735
0.685
0.876
0.725
0.736
0.748
0.750
0.636
0.629
Toto-2.0-22M
0.826
0.744
0.804
0.847
0.848
0.694
0.784
0.743
0.681
0.762
0.749
0.811
0.583
0.718
0.756
0.731
0.687
0.762
0.681
0.724
0.715
0.663
0.616
0.655
Toto-2.0-313M
0.771
0.751
0.802
0.832
0.844
0.659
0.772
0.719
0.651
0.646
0.778
0.794
0.523
0.564
0.742
0.726
0.684
0.734
0.645
0.728
0.695
0.639
0.593
0.632
Appendix
Table 9: Per-dataset MASE on TIME-Benchmark medium-term tasks (24 tasks).
Model
Australia_Solar/H
Auto_Production_SF/M
CPHL/15T
CPHL/30T
CPHL/H
Coastal_T_S/15T
Coastal_T_S/20T
Coastal_T_S/5T
Coastal_T_S/H
Commodity_Import/M
Commodity_Production/M
Crypto/D
ECDC_COVID/D
ECDC_COVID/W
EWELD_Load/15T
Finland_Traffic/15T
Global_Influenza/W
Global_Price/Q
Housing_Inventory/M
JOLTS/M
Job_Claims/W
MetroPT-3/5T
NE_China_Wind/H
Oil_Price/B
Online_Retail_2_UCI/D
HART-36M
0.811
0.797
0.457
0.710
0.796
0.758
0.332
0.619
0.722
0.831
0.787
1.016
0.975
0.882
0.548
0.497
0.601
0.804
0.934
0.653
1.070
0.708
0.559
0.887
0.271
RWKV-36M
0.815
0.719
0.466
0.717
0.798
0.760
0.328
0.617
0.724
0.818
0.768
1.003
0.983
0.904
0.554
0.475
0.598
0.806
0.957
0.657
1.044
0.713
0.572
0.902
0.291
Transformer-35M
0.797
0.841
0.464
0.712
0.805
0.760
0.324
0.621
0.726
0.802
0.781
1.019
0.893
0.955
0.553
0.469
0.615
0.801
0.909
0.627
1.015
0.701
0.574
0.870
0.271
TiRex-35M
0.839
0.763
0.470
0.727
0.804
0.762
0.342
0.626
0.729
0.802
0.733
0.985
0.815
0.808
0.578
0.586
0.649
0.820
0.672
0.609
0.854
0.702
0.576
0.889
0.284
Toto-2.0-22M
0.892
0.698
0.468
0.704
0.790
0.760
0.319
0.624
0.719
0.802
0.742
1.052
0.847
0.996
0.566
0.532
0.679
0.771
0.707
0.571
0.873
0.735
0.600
0.877
0.311
Toto-2.0-313M
0.781
0.600
0.463
0.700
0.792
0.757
0.292
0.616
0.704
0.783
0.706
1.143
0.807
0.985
0.543
0.440
0.650
0.780
0.591
0.517
0.862
0.723
0.583
0.865
0.292
Appendix
Table 10: Per-dataset MASE on TIME-Benchmark short-term tasks (Part 1: A to M, 25 tasks).
Model
OpenElectricity_NEM/5T
Port_Activity/D
Port_Activity/W
SG_Carpark/15T
SG_PM25/H
SG_Weather/D
Smart_Manufacturing/H
Supply_Chain_Customer/D
Supply_Chain_Location/D
US_Labor/M
US_Term_Structure/B
Uncertainty_1M/M
Vehicle_Sales/M
Vehicle_Supply/M
WUI_Global/Q
Water_Quality_Darwin/15T
azure2019_D/5T
azure2019_I/5T
azure2019_U/5T
current_velocity/10T
current_velocity/15T
current_velocity/20T
current_velocity/5T
current_velocity/H
epf_electricity_price/H
HART-36M
0.364
0.578
0.766
0.372
0.736
0.779
0.710
0.340
0.393
0.414
0.872
0.318
0.581
0.794
0.720
0.559
0.686
0.688
0.601
0.494
0.499
0.464
0.396
0.663
0.669
RWKV-36M
0.385
0.577
0.769
0.389
0.739
0.779
0.714
0.343
0.397
0.438
0.846
0.338
0.588
0.798
0.716
0.545
0.669
0.680
0.595
0.513
0.501
0.463
0.391
0.651
0.665
Transformer-35M
0.361
0.576
0.769
0.380
0.742
0.786
0.712
0.338
0.389
0.416
0.863
0.319
0.576
0.797
0.721
0.556
0.689
0.696
0.604
0.500
0.501
0.464
0.423
0.665
0.687
TiRex-35M
0.414
0.576
0.753
0.434
0.743
0.780
0.713
0.356
0.406
0.443
0.849
0.351
0.570
0.714
0.721
0.567
0.697
0.719
0.581
0.512
0.500
0.467
0.408
0.654
0.658
Toto-2.0-22M
0.374
0.574
0.756
0.432
0.732
0.776
0.728
0.337
0.389
0.441
0.935
0.358
0.536
0.704
0.715
0.517
0.658
0.670
0.571
0.515
0.501
0.458
0.406
0.665
0.676
Toto-2.0-313M
0.354
0.573
0.754
0.379
0.725
0.775
0.722
0.324
0.379
0.451
0.869
0.370
0.516
0.690
0.713
0.493
0.619
0.623
0.549
0.505
0.495
0.446
0.423
0.642
0.659
Appendix
Table 11: Per-dataset MASE on TIME-Benchmark short-term tasks (Part 2: O–Z, 25 tasks).
Figure 3: Temporal evolution of ∥Kt∥F across all 12 layers. Blue: historical region; orange: prediction region; red dashed line marks the boundary. Prediction-region averages are lower in Layers 1 to 11, with transient increases in some layers; Layer 12 has a higher prediction-region average.
Figure 4: Temporal evolution of ∥Mt∥F across all 12 layers. Blue: historical region; orange: prediction region; red dashed line marks the boundary. The Mt norms remain similar across the boundary and throughout the prediction region, with small increases in some layers.
Experiment
H
RWKV-AR
RWKV-NAF
HART-AR
HART-NAF
Transformer-AR
Transformer-NAF
SE1 Regime Switch
S
0.162
0.163
0.161
0.161
0.162
0.162
M
0.126
0.140
0.125
0.126
0.126
0.127
L
0.131
0.191
0.130
0.131
0.131
0.133
SE2 Ultra-Long Cycle
S
0.488
0.490
0.481
0.483
0.482
0.483
M
0.205
0.246
0.200
0.226
0.199
0.215
L
0.201
0.296
0.198
0.228
0.196
0.212
Appendix
Table 12: NAF versus teacher-forced AR at 16k context length (MASE). AR uses real future inputs as a diagnostic control. Lower is better.
Time series forecasting predicts future values from ordered observations using statistical, neural, and probabilistic models to support decisions under uncertainty.
Time Series Foundation Models (TSFMs) have borrowed the long context paradigm from natural language processing under the premise that feeding more history into the model improves forecast quality. But in stochastic domains, distant history is often just high-frequency noise, not signal. Hence, the proposed work tests whether this premise actually holds by running continuous context architectures (PatchTST included) through the ETTh1 benchmark. The obtained results contradict the premise: an inverse scaling law shows up clearly, with forecasting error rising as context gets longer. A 3,000-step window causes performance to drop by over 68%, evidence that attention mechanisms are poor at ignoring irrelevant historical volatility. Retrieval-Augmented Forecasting (RAFT) is evaluated as an alternative. RAFT achieves a mean squared error (MSE) of 0.379 with a fixed 720-step window and selective retrieval, outperforming both long-context configurations and zero-shot foundation models (Chronos, Moirai) despite requiring far less computation. In addition, the retrieval step injects only the most relevant historical segments as dynamic exogenous variables, which gives the model a context-informed inductive bias it cannot build on its own from raw sequences. Therefore, foundation models going forward need to shift architecturally toward selective retrieval.
Rishi Ahuja, Kumar Prateek, Simranjit Singh +1
Department of Information Technology, Dr. B.R. Ambedkar National Institute of Technology Jalandhar, Punjab, 144008, India.
We introduce Timer-S1, a strong Mixture-of-Experts (MoE) time series foundation model with 8.3B total parameters, 0.75B activated parameters for each token, and a context length of 11.5K. To overcome the scalability bottleneck in existing pre-trained time series foundation models, we perform Serial Scaling in three dimensions: model architecture, dataset, and training pipeline. Timer-S1 integrates sparse TimeMoE blocks and generic TimeSTP blocks for Serial-Token Prediction (STP), a generic training objective that adheres to the serial nature of forecasting. The proposed paradigm introduces serial computations to improve long-term predictions while avoiding costly rolling-style inference and pronounced error accumulation in the standard next-token prediction. Pursuing a high-quality and unbiased training dataset, we curate TimeBench, a corpus with one trillion time points, and apply meticulous data augmentation to mitigate predictive bias. We further pioneer a post-training stage, including continued pre-training and long-context extension, to enhance short-term and long-context performance. Evaluated on the large-scale GIFT-Eval leaderboard, Timer-S1 achieves state-of-the-art forecasting performance, attaining the best MASE and CRPS scores as a pre-trained model. Timer-S1 is released to facilitate further research.
Time Series Foundation Models (TSFMs) leverage extensive pretraining to accurately predict unseen time series during inference, without the need for task-specific fine-tuning. Through large-scale evaluations on standard benchmarks, we find that leading transformer-based TSFMs exhibit redundant components in their intermediate layers. We introduce a set of tools for mechanistic interpretability of TSFMs, including ablations of specific components and direct logit attribution on the residual stream. Our findings are consistent across several leading TSFMs with diverse architectures, and across a diverse set of real-world and synthetic time-series datasets. We discover that all models in our study are robust to ablations of entire layers. Furthermore, we develop a theoretical framework framing transformers as kernel regressors, motivating a purely intrinsic strategy for ablating heads based on the stable rank of the per-head projection matrices. Using this approach, we uncover the specific heads responsible for degenerate phenomena widely observed in TSFMs, such as parroting of motifs from the context and seasonality bias. Our study sheds light on the universal properties of this emerging class of architectures for continuous-time sequence modeling.
Anthony Bao, Venkata Hasith Vattikuti, Jeffrey Lai +1
ECE Department, UT Austin · Department of Physics, UT Austin · Oden Institute, UT Austin