HART: Hybrid Attention-RWKV Time series Foundation Model

Qingwen Lin; Boyan Xu; Shifeng Xie; Zhifeng Hao; Keli Zhang; Ruichu Cai

cs.LGOct 9, 2026

HART: Hybrid Attention-RWKV Time series Foundation Model

Authors: Qingwen Lin, Boyan Xu, Shifeng Xie, Zhifeng Hao, Keli Zhang, Ruichu Cai

Organizations: School of Computer Science, Guangdong University of Technology · Huawei Noah’s Ark Lab · LIPADE, Université Paris Cité, Paris, France · College of Science, Shantou University ·

Abstract

Non-autoregressive multi-step forecasting (NAF) enables time series foundation models to predict an entire future horizon in a single forward pass. Despite its strong sequence modeling capabilities, RWKV-7 exhibits long-horizon degradation in our NAF experiments. In this setup, future patches are represented as missing inputs, requiring the recurrent state to evolve across the prediction horizon without new observations. Our analysis of RWKV-7's generalized delta rule suggests a mechanism we term Placeholder Idling Decay (PID): repeated state transitions can attenuate historical information when placeholder-driven writes provide insufficient compensation. Layer-wise analysis reveals reduced state-injection norms in most layers, while transition-matrix norms remain similar between historical and prediction regions, consistent with PID. Motivated by this analysis, we propose HART, a Hybrid Attention-RWKV Time-series Foundation Model that alternates attention and RWKV blocks, combining direct access to historical tokens with recurrent temporal modeling. Controlled experiments on GIFT-Eval and TIME-Benchmark show that HART improves over pure RWKV at medium and long horizons while maintaining comparable short-horizon performance, and achieves better aggregate performance than a comparably sized pure Transformer. With 36M parameters, HART achieves long-horizon MASE comparable to Toto-2.0-1B on GIFT-Eval (0.752 vs. 0.754), using approximately 1/28 of its parameters.

Figures & tables

Figure 1: Qualitative NAF predictions on a synthetic signal. Gray: history and ground truth; colored: median predictions with quantile.

Figure 2: Architecture of the proposed hybrid backbone. (a) Overall pipeline: time-patch embedding feeds into an alternating stack of Attention and RWKV blocks (

L

times), followed by the output head. (b) Attention block: RMSNorm, gated self-attention, and feed-forward with residual connections. (c) RWKV block: LayerNorm, RWKV-TimeMIX (generalized delta rule), and feed-forward with residual connections.

	$K_{t}$			$M_{t}$
Layer	Hist.	Pred.	Ratio	Hist.	Pred.	Ratio
1	8.78	6.91	0.79	6.77	6.98	1.03
2	4.72	3.10	0.66	7.20	7.29	1.01
3	4.72	2.93	0.62	7.26	7.39	1.02
4	6.00	5.25	0.87	7.24	7.32	1.01
5	7.15	4.29	0.60	7.17	7.27	1.01

Table 1: Layer-wise mean Frobenius norms of

K_{t}

and

M_{t}

in the historical and prediction regions, averaged over 4,096 sequences. Lower ratios indicate smaller prediction-region norms relative to the historical region.

Model	GIFT-Eval				TIME-Benchmark
Model	Short	Medium	Long	All	Short	Medium	Long	All
RWKV-36M	0.689	0.749 †	0.770 †	0.719 †	0.620	0.727 †	0.755 †	0.676 †
Transformer-35M	0.695	0.747	0.763 †	0.721 †	0.618	0.725 †	0.735	0.670 †
HART-36M	0.689	0.727	0.752	0.710	0.618	0.716	0.723	0.666

Table 2: Self-developed architectures on GIFT-Eval and TIME-Benchmark (MASE, geometric mean). Lower is better; bold marks the lowest displayed value in each column, including ties. †indicates that the paired median difference between HART and the baseline has a 95% bootstrap confidence interval excluding zero.

Model	GIFT-Eval				TIME-Benchmark
Model	Short	Medium	Long	All	Short	Medium	Long	All
YingLong-110M Wang et al. (2025)	0.766	0.856	0.885	0.809	–	–	–	–
YingLong-300M Wang et al. (2025)	0.758	0.841	0.867	0.798	–	–	–	–
TimesFM-2.0-500M Das et al. (2023)	0.704	0.810	0.860	0.758	0.652	0.791	0.797	0.718
Sundial-128M Liu et al. (2025c)	0.732	0.764	0.785	0.750	0.700	0.813	0.831	0.757
Xihe-1.5B Sun et al. (2025)	0.689	0.718	0.763	0.711	–	–	–	–

Table 3: External baselines on GIFT-Eval and TIME-Benchmark (MASE, geometric mean). Lower is better; bold marks the lowest displayed value in each column, including ties.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Model	bitbrains_fast_storage/5T	bitbrains_rnd/5T	bizitobs_application/10S	bizitobs_l2c/5T	bizitobs_l2c/H	bizitobs_service/10S	electricity/15T	electricity/H	ett1/15T	ett1/H	ett2/15T	ett2/H	jena_weather/10T	jena_weather/H	kdd_cup_2018/H	loop_seattle/5T	loop_seattle/H	m_dense/H	solar/10T	solar/H	sz_taxi/15T
HART-36M	0.784	0.947	0.969	0.531	0.394	0.982	0.742	0.741	0.853	1.049	0.889	0.970	1.025	0.618	0.709	0.604	0.562	0.451	0.909	0.814	0.784
RWKV-36M	0.797	0.951	1.031	0.471	0.420	0.959	0.785	0.766	0.933	1.062	0.945	0.937	1.011	0.619	0.714	0.691	0.582	0.486	0.924	0.834	0.785
Transformer-35M	0.783	0.947	0.977	0.636	0.397	1.090	0.755	0.744	0.847	1.029	0.902	0.883	0.979	0.626	0.736	0.621	0.567	0.454	0.935	0.820	0.784
Kairos-23M	0.859	0.976	0.947	0.764	0.390	1.015	0.818	0.845	0.849	0.903	0.896	0.894	0.863	0.842	0.765	0.701	0.611	0.568	0.926	0.857	0.768
TiRex-35M	0.805	0.957	1.059	0.417	0.410	1.093	0.801	0.810	0.857	0.893	0.884	1.050	0.852	0.777	0.756	0.761	0.566	0.470	0.954	0.771	0.737
Toto-2.0-22M	0.736	0.934	1.037	0.693	0.402	0.967	0.787	0.821	0.838	0.914	0.875	0.925	0.852	0.837	0.762	0.714	0.583	0.523	0.954	0.859	0.741

Appendix

Table 4: Per-dataset MASE on GIFT-Eval long-term datasets (21 datasets).

Model	bitbrains_fast_storage/5T	bitbrains_rnd/5T	bizitobs_application/10S	bizitobs_l2c/5T	bizitobs_l2c/H	bizitobs_service/10S	electricity/15T	electricity/H	ett1/15T	ett1/H	ett2/15T	ett2/H	jena_weather/10T	jena_weather/H	kdd_cup_2018/H	loop_seattle/5T	loop_seattle/H	m_dense/H	solar/10T	solar/H	sz_taxi/15T
HART-36M	0.805	0.965	0.765	0.463	0.331	0.817	0.717	0.745	0.845	0.889	0.830	0.898	0.932	0.840	0.691	0.619	0.620	0.424	0.903	0.890	0.774
RWKV-36M	0.815	0.968	0.906	0.484	0.340	0.867	0.737	0.756	0.895	0.879	0.883	0.857	0.947	0.854	0.690	0.700	0.632	0.452	0.880	0.902	0.773
Transformer-35M	0.804	0.967	0.962	0.595	0.341	0.931	0.724	0.753	0.834	0.904	0.847	0.801	0.919	0.861	0.696	0.629	0.613	0.434	0.870	0.903	0.771
Kairos-23M	0.886	0.989	0.936	0.605	0.325	0.906	0.788	0.818	0.854	0.840	0.865	0.814	0.882	0.944	0.726	0.702	0.661	0.511	0.812	0.900	0.781
TiRex-35M	0.822	0.972	0.982	0.498	0.344	0.889	0.771	0.782	0.854	0.788	0.833	0.916	0.846	0.853	0.725	0.763	0.620	0.445	0.924	0.838	0.753
Toto-2.0-22M	0.763	0.955	0.919	0.512	0.331	0.830	0.764	0.799	0.837	0.807	0.842	0.842	0.792	0.942	0.733	0.722	0.634	0.470	0.938	0.932	0.756

Appendix

Table 5: Per-dataset MASE on GIFT-Eval medium-term datasets (21 datasets).

Model	bizitobs_application	bizitobs_l2c/5T	bizitobs_l2c/H	bizitobs_service	car_parts/M	covid_deaths/D	electricity/15T	electricity/D	electricity/H	electricity/W	ett1/15T	ett1/D	ett1/H	ett1/W	ett2/15T	ett2/D	ett2/H	ett2/W	hierarchical_sales/D	hierarchical_sales/W	hospital/M	jena_weather/10T	jena_weather/D	jena_weather/H	kdd_cup_2018/D	kdd_cup_2018/H	loop_seattle/5T	loop_seattle/D
HART-36M	0.372	0.265	0.366	0.585	0.711	0.960	0.584	0.714	0.661	0.694	0.707	0.976	0.846	0.938	0.675	0.940	0.772	0.976	0.664	0.700	0.908	0.356	0.649	0.700	0.790	0.638	0.728	0.522
RWKV-36M	0.422	0.238	0.365	0.627	0.710	0.955	0.591	0.717	0.678	0.691	0.728	0.971	0.862	0.909	0.676	0.918	0.776	0.927	0.660	0.702	0.907	0.367	0.620	0.706	0.793	0.638	0.761	0.518
Transformer-35M	0.488	0.270	0.361	0.635	0.705	0.972	0.569	0.713	0.668	0.704	0.710	0.972	0.853	0.938	0.669	0.898	0.774	0.978	0.657	0.705	0.907	0.373	0.638	0.707	0.784	0.627	0.749	0.525
Kairos-23M	0.573	0.288	0.396	0.685	0.819	0.874	0.649	0.730	0.732	0.678	0.761	0.921	0.873	0.806	0.749	0.935	0.809	1.252	0.663	0.721	0.846	0.387	0.758	0.733	0.791	0.706	0.741	0.508
TiRex-35M	0.537	0.283	0.407	0.674	0.705	0.727	0.617	0.710	0.670	0.690	0.759	0.944	0.836	0.904	0.706	0.943	0.784	0.983	0.657	0.704	0.821	0.377	0.645	0.724	0.802	0.701	0.735	0.505
Toto-2.0-22M	0.542	0.254	0.340	0.631	0.735	0.632	0.596	0.737	0.779	0.773	0.713	0.930	0.871	0.857	0.669	0.926	0.802	1.054	0.661	0.711	0.823	0.351	0.711	0.729	0.801	0.706	0.730	0.514

Appendix

Table 6: Per-dataset MASE on GIFT-Eval short-term datasets (Part 1: A to M, 28 datasets).

Model	loop_seattle/H	m4_daily/D	m4_hourly/H	m4_monthly/M	m4_quarterly/Q	m4_weekly/W	m4_yearly/A	m_dense/D	m_dense/H	restaurant/D	saugeen/D	saugeen/M	saugeen/W	solar/10T	solar/D	solar/H	solar/W	sz_taxi/15T	sz_taxi/H	temperature_rain/D	us_births/D	us_births/M	us_births/W	bitbrains_fast_storage/5T	bitbrains_fast_storage/H	bitbrains_rnd/5T	bitbrains_rnd/H
HART-36M	0.654	1.005	0.558	0.834	0.874	0.703	1.100	0.407	0.520	0.671	0.824	0.797	0.566	0.991	0.841	0.867	0.675	0.717	0.770	0.656	0.193	1.243	0.731	0.589	0.827	0.828	0.970
RWKV-36M	0.665	0.996	0.550	0.824	0.865	0.698	1.112	0.412	0.512	0.672	0.789	0.778	0.562	0.797	0.844	0.878	0.670	0.717	0.770	0.710	0.202	1.541	0.652	0.591	0.830	0.829	0.969
Transformer-35M	0.664	0.998	0.554	0.826	0.860	0.706	1.097	0.396	0.535	0.674	0.846	0.808	0.593	0.913	0.838	0.884	0.761	0.718	0.773	0.656	0.183	1.458	0.676	0.595	0.826	0.833	0.970
Kairos-23M	0.667	1.011	0.882	0.771	0.743	0.892	0.752	0.407	0.571	0.692	0.895	0.715	0.623	1.071	0.859	0.912	0.715	0.727	0.773	0.699	0.234	1.100	0.794	0.711	0.854	0.889	0.980
TiRex-35M	0.638	0.986	0.605	0.731	0.715	0.717	0.848	0.405	0.526	0.674	0.879	0.767	0.588	0.984	0.827	0.782	0.707	0.710	0.763	0.660	0.237	0.791	0.669	0.608	0.785	0.840	0.963
Toto-2.0-22M	0.680	0.978	0.691	0.728	0.730	0.728	0.826	0.415	0.539	0.700	0.878	0.732	0.630	0.822	0.842	0.912	0.827	0.715	0.765	0.682	0.208	0.927	0.695	0.557	0.707	0.817	0.959

Appendix

Table 7: Per-dataset MASE on GIFT-Eval short-term datasets (Part 2: M–Z, 27 datasets).

Model	Australia_Solar/H	CPHL/15T	CPHL/30T	CPHL/H	Coastal_T_S/15T	Coastal_T_S/20T	Coastal_T_S/5T	Coastal_T_S/H	EWELD_Load/15T	Finland_Traffic/15T	MetroPT-3/5T	NE_China_Wind/H	OpenElectricity_NEM/5T	SG_Carpark/15T	SG_PM25/H	SG_Weather/D	Smart_Manufacturing/H	Water_Quality_Darwin/15T	current_velocity/10T	current_velocity/15T	current_velocity/20T	current_velocity/5T	current_velocity/H	epf_electricity_price/H
HART-36M	0.834	0.779	0.806	0.805	0.860	0.796	0.813	0.885	0.664	0.479	0.748	0.725	0.690	0.492	0.696	0.710	0.666	0.869	0.667	0.752	0.790	0.663	0.671	0.687
RWKV-36M	0.847	0.764	0.846	0.815	0.857	0.881	0.858	0.868	0.723	0.595	0.759	0.713	0.766	0.598	0.700	0.710	0.668	0.848	0.718	0.765	0.805	0.794	0.655	0.680
Transformer-35M	0.861	0.738	0.788	0.798	0.853	0.810	0.902	0.839	0.664	0.491	0.749	0.744	0.669	0.542	0.711	0.719	0.666	0.832	0.689	0.759	0.796	0.787	0.694	0.694
TiRex-35M	0.830	0.751	0.804	0.858	0.846	0.837	0.852	0.885	0.715	0.568	0.743	0.796	0.862	0.623	0.705	0.715	0.667	0.905	0.729	0.763	0.809	0.737	0.698	0.679
Toto-2.0-22M	0.890	0.750	0.812	0.846	0.854	0.761	0.820	0.802	0.718	0.520	0.751	0.844	0.727	0.624	0.693	0.711	0.669	0.811	0.676	0.735	0.754	0.709	0.662	0.704
Toto-2.0-313M	0.835	0.730	0.792	0.826	0.846	0.662	0.811	0.771	0.663	0.467	0.801	0.803	0.701	0.567	0.684	0.705	0.669	0.787	0.619	0.723	0.726	0.603	0.633	0.681

Appendix

Table 8: Per-dataset MASE on TIME-Benchmark long-term tasks (24 tasks).

Model	Australia_Solar/H	CPHL/15T	CPHL/30T	CPHL/H	Coastal_T_S/15T	Coastal_T_S/20T	Coastal_T_S/5T	Coastal_T_S/H	EWELD_Load/15T	Finland_Traffic/15T	MetroPT-3/5T	NE_China_Wind/H	OpenElectricity_NEM/5T	SG_Carpark/15T	SG_PM25/H	SG_Weather/D	Smart_Manufacturing/H	Water_Quality_Darwin/15T	current_velocity/10T	current_velocity/15T	current_velocity/20T	current_velocity/5T	current_velocity/H	epf_electricity_price/H
HART-36M	0.773	0.758	0.813	0.808	0.843	0.780	0.774	0.799	0.666	0.682	0.755	0.750	0.532	0.534	0.758	0.729	0.683	0.834	0.667	0.725	0.740	0.646	0.624	0.642
RWKV-36M	0.787	0.765	0.824	0.800	0.847	0.780	0.776	0.825	0.685	0.706	0.760	0.725	0.610	0.588	0.762	0.731	0.686	0.804	0.698	0.740	0.739	0.646	0.607	0.639
Transformer-35M	0.787	0.733	0.820	0.792	0.841	0.739	0.799	0.814	0.674	0.707	0.754	0.768	0.530	0.568	0.770	0.735	0.684	0.824	0.680	0.731	0.744	0.702	0.641	0.662
TiRex-35M	0.805	0.758	0.834	0.826	0.851	0.837	0.805	0.857	0.694	0.811	0.743	0.789	0.689	0.703	0.766	0.735	0.685	0.876	0.725	0.736	0.748	0.750	0.636	0.629
Toto-2.0-22M	0.826	0.744	0.804	0.847	0.848	0.694	0.784	0.743	0.681	0.762	0.749	0.811	0.583	0.718	0.756	0.731	0.687	0.762	0.681	0.724	0.715	0.663	0.616	0.655
Toto-2.0-313M	0.771	0.751	0.802	0.832	0.844	0.659	0.772	0.719	0.651	0.646	0.778	0.794	0.523	0.564	0.742	0.726	0.684	0.734	0.645	0.728	0.695	0.639	0.593	0.632

Appendix

Table 9: Per-dataset MASE on TIME-Benchmark medium-term tasks (24 tasks).

Model	Australia_Solar/H	Auto_Production_SF/M	CPHL/15T	CPHL/30T	CPHL/H	Coastal_T_S/15T	Coastal_T_S/20T	Coastal_T_S/5T	Coastal_T_S/H	Commodity_Import/M	Commodity_Production/M	Crypto/D	ECDC_COVID/D	ECDC_COVID/W	EWELD_Load/15T	Finland_Traffic/15T	Global_Influenza/W	Global_Price/Q	Housing_Inventory/M	JOLTS/M	Job_Claims/W	MetroPT-3/5T	NE_China_Wind/H	Oil_Price/B	Online_Retail_2_UCI/D
HART-36M	0.811	0.797	0.457	0.710	0.796	0.758	0.332	0.619	0.722	0.831	0.787	1.016	0.975	0.882	0.548	0.497	0.601	0.804	0.934	0.653	1.070	0.708	0.559	0.887	0.271
RWKV-36M	0.815	0.719	0.466	0.717	0.798	0.760	0.328	0.617	0.724	0.818	0.768	1.003	0.983	0.904	0.554	0.475	0.598	0.806	0.957	0.657	1.044	0.713	0.572	0.902	0.291
Transformer-35M	0.797	0.841	0.464	0.712	0.805	0.760	0.324	0.621	0.726	0.802	0.781	1.019	0.893	0.955	0.553	0.469	0.615	0.801	0.909	0.627	1.015	0.701	0.574	0.870	0.271
TiRex-35M	0.839	0.763	0.470	0.727	0.804	0.762	0.342	0.626	0.729	0.802	0.733	0.985	0.815	0.808	0.578	0.586	0.649	0.820	0.672	0.609	0.854	0.702	0.576	0.889	0.284
Toto-2.0-22M	0.892	0.698	0.468	0.704	0.790	0.760	0.319	0.624	0.719	0.802	0.742	1.052	0.847	0.996	0.566	0.532	0.679	0.771	0.707	0.571	0.873	0.735	0.600	0.877	0.311
Toto-2.0-313M	0.781	0.600	0.463	0.700	0.792	0.757	0.292	0.616	0.704	0.783	0.706	1.143	0.807	0.985	0.543	0.440	0.650	0.780	0.591	0.517	0.862	0.723	0.583	0.865	0.292

Appendix

Table 10: Per-dataset MASE on TIME-Benchmark short-term tasks (Part 1: A to M, 25 tasks).

Model	OpenElectricity_NEM/5T	Port_Activity/D	Port_Activity/W	SG_Carpark/15T	SG_PM25/H	SG_Weather/D	Smart_Manufacturing/H	Supply_Chain_Customer/D	Supply_Chain_Location/D	US_Labor/M	US_Term_Structure/B	Uncertainty_1M/M	Vehicle_Sales/M	Vehicle_Supply/M	WUI_Global/Q	Water_Quality_Darwin/15T	azure2019_D/5T	azure2019_I/5T	azure2019_U/5T	current_velocity/10T	current_velocity/15T	current_velocity/20T	current_velocity/5T	current_velocity/H	epf_electricity_price/H
HART-36M	0.364	0.578	0.766	0.372	0.736	0.779	0.710	0.340	0.393	0.414	0.872	0.318	0.581	0.794	0.720	0.559	0.686	0.688	0.601	0.494	0.499	0.464	0.396	0.663	0.669
RWKV-36M	0.385	0.577	0.769	0.389	0.739	0.779	0.714	0.343	0.397	0.438	0.846	0.338	0.588	0.798	0.716	0.545	0.669	0.680	0.595	0.513	0.501	0.463	0.391	0.651	0.665
Transformer-35M	0.361	0.576	0.769	0.380	0.742	0.786	0.712	0.338	0.389	0.416	0.863	0.319	0.576	0.797	0.721	0.556	0.689	0.696	0.604	0.500	0.501	0.464	0.423	0.665	0.687
TiRex-35M	0.414	0.576	0.753	0.434	0.743	0.780	0.713	0.356	0.406	0.443	0.849	0.351	0.570	0.714	0.721	0.567	0.697	0.719	0.581	0.512	0.500	0.467	0.408	0.654	0.658
Toto-2.0-22M	0.374	0.574	0.756	0.432	0.732	0.776	0.728	0.337	0.389	0.441	0.935	0.358	0.536	0.704	0.715	0.517	0.658	0.670	0.571	0.515	0.501	0.458	0.406	0.665	0.676
Toto-2.0-313M	0.354	0.573	0.754	0.379	0.725	0.775	0.722	0.324	0.379	0.451	0.869	0.370	0.516	0.690	0.713	0.493	0.619	0.623	0.549	0.505	0.495	0.446	0.423	0.642	0.659

Appendix

Table 11: Per-dataset MASE on TIME-Benchmark short-term tasks (Part 2: O–Z, 25 tasks).

Figure 3: Temporal evolution of

\|K_{t}\|_{F}

across all 12 layers. Blue: historical region; orange: prediction region; red dashed line marks the boundary. Prediction-region averages are lower in Layers 1 to 11, with transient increases in some layers; Layer 12 has a higher prediction-region average.

Figure 4: Temporal evolution of

\|M_{t}\|_{F}

across all 12 layers. Blue: historical region; orange: prediction region; red dashed line marks the boundary. The

M_{t}

norms remain similar across the boundary and throughout the prediction region, with small increases in some layers.

Experiment	$H$	RWKV-AR	RWKV-NAF	HART-AR	HART-NAF	Transformer-AR	Transformer-NAF
SE1 Regime Switch	S	0.162	0.163	0.161	0.161	0.162	0.162
	M	0.126	0.140	0.125	0.126	0.126	0.127
	L	0.131	0.191	0.130	0.131	0.131	0.133
SE2 Ultra-Long Cycle	S	0.488	0.490	0.481	0.483	0.482	0.483
	M	0.205	0.246	0.200	0.226	0.199	0.215
	L	0.201	0.296	0.198	0.228	0.196	0.212

Appendix

Table 12: NAF versus teacher-forced AR at 16k context length (MASE). AR uses real future inputs as a diagnostic control. Lower is better.

Time Series Forecasting

Time series forecasting predicts future values from ordered observations using statistical, neural, and probabilistic models to support decisions under uncertainty.

See all 839 papers

Retrieval Mechanisms Surpass Long-Context Scaling in Time Series Forecasting
May 6, 2026Rishi Ahuja, Kumar Prateek, Simranjit Singh +1Time Series Forecasting Long-Context Modeling
Department of Information Technology, Dr. B.R. Ambedkar National Institute of Technology Jalandhar, Punjab, 144008, India.
Timer-S1: A Billion-Scale Time Series Foundation Model with Serial Scaling
Mar 5, 2026Yong Liu, Xingjian Su, Shiyu Wang +7Time Series Forecasting Time-Series Foundation Models
Tsinghua University · ByteDance
Universal Redundancies in Time Series Foundation Models
Feb 2, 2026Anthony Bao, Venkata Hasith Vattikuti, Jeffrey Lai +1Transformer Interpretability Kernel Regression
ECE Department, UT Austin · Department of Physics, UT Austin · Oden Institute, UT Austin

May 6, 2026J/K move · Enter open · S save

Retrieval Mechanisms Surpass Long-Context Scaling in Time Series Forecasting

Rishi Ahuja, Kumar Prateek, Simranjit Singh, Vijay Kumar

Department of Information Technology, Dr. B.R. Ambedkar National Institute of Technology Jalandhar, Punjab, 144008, India.

Time Series Forecasting Long-Context Modeling Retrieval-Augmented Time Series Forecasting Time-Series Foundation Models

Open paper AskPDF arXiv

Time Series Foundation Models (TSFMs) have borrowed the long context paradigm from natural language processing under the premise that feeding more history into the model improves forecast quality. But in stochastic domains, distant history is often just high-frequency noise, not signal. Hence, the proposed work tests whether this premise actually holds by running continuous context architectures (PatchTST included) through the ETTh1 benchmark. The obtained results contradict the premise: an inverse scaling law shows up clearly, with forecasting error rising as context gets longer. A 3,000-step window causes performance to drop by over 68%, evidence that attention mechanisms are poor at ignoring irrelevant historical volatility. Retrieval-Augmented Forecasting (RAFT) is evaluated as an alternative. RAFT achieves a mean squared error (MSE) of 0.379 with a fixed 720-step window and selective retrieval, outperforming both long-context configurations and zero-shot foundation models (Chronos, Moirai) despite requiring far less computation. In addition, the retrieval step injects only the most relevant historical segments as dynamic exogenous variables, which gives the model a context-informed inductive bias it cannot build on its own from raw sequences. Therefore, foundation models going forward need to shift architecturally toward selective retrieval.

Abstract

Time Series Forecasting

Explore similar work

Retrieval Mechanisms Surpass Long-Context Scaling in Time Series Forecasting

Timer-S1: A Billion-Scale Time Series Foundation Model with Serial Scaling

Universal Redundancies in Time Series Foundation Models