Bi-level Heterogeneous Learning for Time Series Foundation Models: A Federated Learning Approach

Shengchao Chen; Guodong Long; Dikai Liu; Jing Jiang

cs.LGApr 8, 2026

Bi-level Heterogeneous Learning for Time Series Foundation Models: A Federated Learning Approach

Authors: Shengchao Chen, Guodong Long, Dikai Liu, Jing Jiang

Organizations: Australian AI Institute, University of Technology Sydney · Robotics Institute, University of Technology Sydney

Abstract

Heterogeneity in time series data is more pronounced than in vision or language, as temporal dynamics vary substantially across domains and tasks. Existing efforts on training time series foundation models (TSFMs) from scratch are often trained with mixed-batch strategies that merge large-scale datasets, which can cause gradient conflicts and degrade representation quality. To address this, we propose a fine-grained learning method that distills invariant knowledge from heterogeneous series while reducing cross-domain interference. We characterize heterogeneity at two levels: inter-domain and intra-domain. To tackle this bi-level heterogeneity, we design a federated learning method that mitigates intra-domain conflicts by enforcing domain-invariant and semantically consistent representations through local regularization, and addresses inter-domain discrepancies by enhancing cross-domain collaboration via domain-aware aggregation. Experiments across diverse benchmarks show that TSFMs trained with our method consistently outperform both centralized and federated TSFM baselines in point and probabilistic forecasting, while also achieving competitive zero-shot performance at scale, offering a flexible pathway for training TSFMs from scratch in heterogeneous environments.

Figures & tables

Figure 1: FedTRL outperforms centralized TSFMs in zero-shot point and probabilistic forecasting . Details are provided in Sec. 4

Figure 2: Structure of FedTRL (single client example). Each client performs unsupervised diffusion-based reconstruction and uploads only the encoder $\theta_{E}$ and prototypes $\bar{\mathbf{p}}$ per round; the server applies domain-aware aggregation (DaG) to produce a unified encoder.

	Ours		Federated Self-supervised †								Federated FMs				Centralized
Methods	FedTRL		SimMTM		PatchTST		TimeMAE		CoST		FFTS		FedAvg		All Mixed
Metric	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
ETTh1	0.448	0.472	0.495	0.512	0.478	0.495	0.502	0.518	0.521	0.540	0.463	0.485	0.476	0.495	0.458	0.488
ETTh2	0.382	0.414	0.442	0.468	0.410	0.432	0.455	0.474	0.471	0.490	0.395	0.419	0.410	0.433	0.399	0.424
ETTm1	0.375	0.402	0.415	0.439	0.386	0.408	0.421	0.444	0.436	0.455	0.380	0.406	0.399	0.406	0.372	0.398
ETTm2	0.278	0.330	0.324	0.352	0.296	0.340	0.330	0.360	0.345	0.369	0.290	0.338	0.299	0.341	0.286	0.336

Table 1: In-domain forecasting results (averaged across forecasting horizons

\{96,192,336,720\}

). Bold : the best; Underline : the second best. The † symbol denotes that based on the FedAvg aggregation protocol. Full results are provided in Table 12 .

Models	FedTRL		TimeMixer		TimeXer		iTransformer		PatchTST		TimesNet		DLinear		Autoformer		Stationary		LightTS
Metrics	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
ETTh1	0.372	0.399	0.448	0.443	0.491	0.489	0.474	0.476	0.413	0.430	0.458	0.450	0.422	0.437	0.496	0.487	0.570	0.537	0.491	0.479
ETTh2	0.330	0.383	0.364	0.394	0.356	0.398	0.372	0.411	0.330	0.379	0.414	0.427	0.431	0.446	0.450	0.459	0.526	0.516	0.602	0.543
ETTm1	0.316	0.369	0.381	0.395	0.381	0.402	0.368	0.396	0.351	0.380	0.400	0.406	0.357	0.378	0.588	0.517	0.481	0.456	0.435	0.437
ETTm2	0.257	0.313	0.275	0.323	0.275	0.329	0.271	0.331	0.255	0.315	0.291	0.333	0.267	0.333	0.327	0.371	0.306	0.347	0.409	0.436
Electricity	0.143	0.247	0.175	0.272	0.171	0.273	0.161	0.256	0.161	0.252	0.192	0.295	0.188	0.295	0.227	0.338	0.193	0.296	0.229	0.329

Table 2: Full-shot point forecasting results (averaged across

\{96,192,336,720\}

). Bold : the best; Underline : the second best. Note that FedTRL-trained models are adapted to the target dataset with only one epoch.

Models	FedTRL		Time-MoE b		Time-MoE l		Time-MoE u		Moirai s		Moirai b		Moirai l		TimesFM		Moment		Chronos s		Chronos b		Chronos l
Metrics	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
ETTh1	0.399	0.420	0.400	0.424	0.394	0.419	0.412	0.426	0.428	0.427	0.417	0.419	0.480	0.439	0.473	0.443	0.683	0.566	0.545	0.472	0.591	0.468	0.588	0.466
ETTh2	0.350	0.380	0.366	0.404	0.405	0.415	0.371	0.399	0.361	0.384	0.362	0.382	0.367	0.377	0.392	0.406	0.361	0.409	0.424	0.430	0.405	0.410	0.455	0.427
ETTm1	0.349	0.383	0.394	0.415	0.376	0.405	0.356	0.391	0.436	0.410	0.406	0.385	0.422	0.391	0.433	0.418	0.670	0.536	0.640	0.499	0.645	0.500	0.555	0.465
ETTm2	0.282	0.330	0.317	0.365	0.316	0.361	0.288	0.344	0.307	0.347	0.311	0.337	0.329	0.343	0.328	0.346	0.316	0.365	0.349	0.380	0.310	0.350	0.295	0.338
Weather	0.238	0.279	0.265	0.297	0.270	0.300	0.270	0.300	0.275	0.286	0.287	0.281	0.264	0.273	–	–	0.294	0.326	0.300	0.318	0.292	0.315	0.279	0.306

Table 3: Zero-shot point forecasting results, averaged over horizons

{96,192,336,720}

with look-back lengths

{512,1024,2048,3072}

. Bold and underline indicate the best and second-best results; “–” denotes not evaluable. TSLib results are from ( Shi et al., 2024 ) ; full RW-Bench results are in Table 14 , with VisionTS ( Chen et al., 2024a ) and Sundial ( Liu et al., 2025 ) comparisons in Table 15 .

Type	Statistical Methods				Supervised Task-Specific Models					Zero-Shot Models				Zero-Shot Models (FL)
Model	Naïve	Seasonal	Auto	Auto	DeepAR	TiDE	N-BEATS	PTST.	iTrans.	TimesFM	TabPFN	Chronos	Moirai	FedAvg	FFTS	FedTRL
Model	Naïve	Naïve	ARIMA	Theta	DeepAR	TiDE	N-BEATS	PTST.	iTrans.	TimesFM	TabPFN	Chronos	Moirai	FedAvg	FFTS	(Ours)
MASE	1.260	1.000	0.964	0.978	1.206	0.980	0.842	0.762	0.802	0.680	0.748	0.786	0.809	0.872	0.766	0.675
CRPS	1.383	1.000	0.770	1.051	0.721	0.652	0.689	0.496	0.524	0.465	0.480	0.551	0.515	0.700	0.521	0.460
Rank	28.072	26.175	21.515	24.031	18.938	18.557	21.381	10.052	11.320	8.237	8.268	14.309	10.175	18.231	9.787	8.676

Table 4: Zero-shot results on GIFT-eval, which contains 32 datasets characterized by a variety of frequencies/variate numbers/prediction lengths. We evaluate performance using

100

generated series, being consistent with ( Woo et al., 2024 ) . Lower MASE/CRPS indicates a better performance. Rank assigns a numerical ranking of all

97

configurations. Baseline results are officially from ( Aksu et al., 2024 ) .

Figure 3: Results on FEV leaderboard. Baseline includes statistical methods, task-specific deep models trained on each dataset, and pre-trained foundation models. Pre-trained Models that have seen several datasets during pre-training are denoted as Pre-trained Models (Other). Lower MASE/WQL is better. FedTRL reports probabilistic forecasts using 20 generated series, following ( Ansari et al., 2024 ) .

Tasks		In-domain		Full-shot		Zero-shot
Variants		MSE	MAE	MSE	MAE	MSE	MAE
ETTh1	FedTRL	0.448	0.472	0.372	0.399	0.399	0.420
	w/o GRL	0.464	0.486	0.377	0.400	0.407	0.435
	w/o Proto	0.458	0.482	0.382	0.410	0.417	0.432
	w/o DaG	0.472	0.494	0.394	0.421	0.436	0.459
ETTm1	FedTRL	0.375	0.402	0.316	0.369	0.350	0.380

Table 5: Ablation study on point forecasting tasks.

Figure 4: Relative performance drop of FedTRL ablations in zero-shot probabilistic forecasting on GIFT-eval and FEV leaderboard.

Model	ETTh1		ETTm1		Weather		Traffic
Model	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
SimMTM †	0.495	0.512	0.415	0.439	0.285	0.310	0.472	0.315
+ FedTRL	0.476	0.501	0.401	0.430	0.263	0.301	0.459	0.305
Improvement (%)	3.84	2.15	3.37	2.05	7.88	2.48	2.82	3.13
PatchTST †	0.478	0.495	0.386	0.408	0.260	0.295	0.438	0.295
+ FedTRL	0.461	0.487	0.378	0.404	0.252	0.288	0.430	0.291

Table 6: Results on ETT, Weather and Traffic (averaging across four horizons). Improvement (%) is relative to the baseline.

Figure 5: Model scalability across zero-shot both point forecasting and probabilistic forecasting benchmarks.

Figure 6: Hyperparameter sensitivity results (Avg. MSE). Red Line : TSLib results; Blue Line : RW-Bench results.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Domain	# Channels	Frequency	Size	Forecast Length	SE †	ACF †	Proxy †
ETTh1	Power	7	Hourly	(8545, 2881, 2881)	$\{96,192,336,720\}$	0.523	20.372	0.857
ETTh2	Power	7	Hourly	(8545, 2881, 2881)	$\{96,192,336,720\}$	0.652	35.940	0.943
ETTm1	Power	7	15 Minute	(34465, 11521, 11521)	$\{96,192,336,720\}$	0.580	24.335	0.949
ETTm2	Power	7	15 Minute	(34465, 11521, 11521)	$\{96,192,336,720\}$	0.696	40.276	0.978
Electricity	Energy	321	Hourly	(17805, 2537, 5166)	$\{96,192,336,720\}$	0.706	12.137	0.814
Traffic	Transportation	862	Hourly	(11673, 1661, 3413)	$\{96,192,336,720\}$	0.567	6.977	0.681

Appendix

Table 7: Dataset statistics about TSLib ( Wu et al., 2022 ) . Channels indicates the number of time series (i.e., variables), and the size is organized in (training, validation, testing). † SE: Spectral Entropy; ACF: Autocorrelation-based; Proxy: Error-based. All three quantify time series forecastability, higher scores indicate greater predictability (i.e., lower task difficulty).

Figure 7: Visualization of RW-Bench samples: red denotes temperature, blue denotes precipitation, and green denotes humidity. Variables are arranged following the order in Table 8 . RW-Bench will be open-sourced once the required permissions are obtained.

Dataset	Region	# Channels	Frequency	Size	Forecast Length	SE †	ACF †	Proxy †
RW-1	Yuzhno-Sakhalinsk	10	Hourly	January 2022 - August 2025	$\{96,192,336,720\}$	0.438	17.089	0.682
RW-2	Tokyo	10	Hourly	January 2022 - August 2025	$\{96,192,336,720\}$	0.438	17.090	0.682
RW-3	Beijing	10	Hourly	January 2022 - August 2025	$\{96,192,336,720\}$	0.431	16.660	0.711
RW-4	Mumbai/Santacruz	10	Hourly	January 2022 - August 2025	$\{96,192,336,720\}$	0.398	16.474	0.706
RW-5	Ho Chi Minh City	10	Hourly	January 2022 - August 2025	$\{96,192,336,720\}$	0.462	17.292	0.768
RW-6	Cairo	10	Hourly	January 2022 - August 2025	$\{96,192,336,720\}$	0.456	16.912	0.734

Appendix

Table 8: Dataset statistics about real-world weather benchmark (RW-Bench). Channels indicates the number of time series (i.e., variables), and the size is organized in (training, validation, testing). † SE: Spectral Entropy; ACF: Autocorrelation-based; Proxy: Error-based. All three quantify time series forecastability, higher scores indicate greater predictability (i.e., lower task difficulty).

Category	Configuration	Configuration
Category	(In-domain Forecasting)	(Large Foundation Model)
Optimizer	Adam	Adam
Batch Size	128	2048
Local Epochs	10	100
Global Rounds	200	10,000
Learning Rate (Local Updating)	0.0001	0.0005

Appendix

Table 9: Detailed model architecture and training configuration.

Method	FedTRL	Time-MoE	Moirai	MOMENT	LLMTime	Chronos	Lag-Llama	TimesFM
Method	(Ours)	( 2024 )	( 2024 )	( 2024 )	( 2024 )	( 2024 )	( 2023 )	( 2023b )
Architecture	Encoder	Decoder	Encoder	Encoder	Decoder	EncDec	Decoder	Decoder
Model Size		113M	14M	40M		46M		17M
	302M	453M	91M	125M	–	200M	200M	70M
		2.4B	311M	385M		710M		200M
Pre-training Scale	300B	300B	231B	1.13B	–	84B	0.36B	100B

Appendix

Table 10: Comparison of time series foundation models. Architecture indicates the Transformer variant. Model Size reports parameter counts across scales. Pre-training Scale refers to the number of time points in pre-training datasets. Token Level specifies the granularity of time-series tokens. Tokenization describes which values are embedded from the series. Context Length denotes the maximum supported input length. Probabilistic indicates the ability to generate multiple possible predictions, in contrast to deterministic forecasters. Training refers to the training scenarios for this model, including centralized and federated (decentralized).

Dataset	FedTRL	FFTS	FedAvg	FedProx	FedPer	FedRep	pFedMe
ETT-h1	0.448	0.463	0.476	0.477	0.492	0.482	0.499
ETT-m1	0.375	0.380	0.399	0.395	0.405	0.401	0.414
Weather	0.241	0.252	0.264	0.262	0.281	0.280	0.287

Appendix

Table 11: In-domain forecasting results of FL baselines. Bold : the best.

		Ours		Federated Unsupervised †								Federated FMs				Centralized
Methods		FedTRL		SimMTM		PatchTST		TimeMAE		CoST		FFTS		FedAvg		All Mixed
Metric		MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
ETTh1	96	0.367	0.441	0.399	0.477	0.391	0.462	0.410	0.485	0.414	0.498	0.387	0.456	0.393	0.462	0.372	0.454
	192	0.433	0.464	0.466	0.503	0.449	0.482	0.476	0.507	0.500	0.531	0.444	0.481	0.454	0.490	0.437	0.478
	336	0.467	0.482	0.519	0.523	0.504	0.508	0.531	0.530	0.550	0.555	0.479	0.491	0.495	0.502	0.485	0.500
	720	0.525	0.501	0.595	0.546	0.569	0.528	0.591	0.549	0.620	0.576	0.542	0.511	0.562	0.527	0.539	0.520

Appendix

Table 12: Full results of in-domain forecasting (for four different forecasting horizons

\{96,192,336,720\}

). Bold : the best; Underline : the second best. The † symbol denotes that the federated unsupervised methods are based on the standard FedAvg ( McMahan et al., 2017 ) aggregation protocol.

Models		Ours		Pretrained Time Series Foundation Models (Zero-shot)
Models		FedTRL		Time-MoE base		Time-MoE large		Time-MoE ultra		Moirai small		Moirai base		Moirai large		TimesFM		Moment		Chronos small		Chronos base		Chronos large
Metrics		MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
ETTh1	96	0.346	0.381	0.357	0.381	0.350	0.382	0.349	0.379	0.401	0.402	0.376	0.392	0.349	0.379	0.414	0.404	0.688	0.557	0.466	0.409	0.440	0.393	0.441	0.390
	192	0.386	0.407	0.384	0.404	0.388	0.412	0.395	0.413	0.388	0.412	0.412	0.413	0.434	0.415	0.465	0.434	0.688	0.560	0.530	0.450	0.492	0.426	0.502	0.424
	336	0.420	0.443	0.411	0.434	0.411	0.430	0.447	0.453	0.433	0.428	0.433	0.428	0.495	0.445	0.503	0.456	0.675	0.563	0.570	0.486	0.550	0.462	0.576	0.467
	720	0.444	0.449	0.449	0.477	0.427	0.455	0.457	0.462	0.439	0.454	0.447	0.444	0.611	0.510	0.511	0.481	0.683	0.585	0.615	0.543	0.882	0.591	0.835	0.583

Appendix

Table 13: Full results of zero-shot forecasting experiments. A lower MSE or MAE indicates a better prediction. TimesFM, due to its use of Weather datasets in pretraining, is not evaluated on this dataset and is denoted by a dash (

-

). Bold : the best, Underline : the second best.

Models		Ours		Pretrained Time Series Foundation Models (Zero-shot)
Models		FedTRL		Time-MoE base		Time-MoE large		Moirai small		Moirai base		Moirai large		TimesFM		Moment		Chronos small		Chronos base		Chronos large
Metrics		MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
RW1	96	0.508	0.453	0.510	0.460	0.505	0.455	0.885	0.547	0.879	0.529	0.986	0.530	0.883	0.703	0.674	0.594	0.850	0.581	0.879	0.589	0.873	0.583
	192	0.528	0.473	0.540	0.492	0.532	0.485	0.950	0.569	0.896	0.546	1.011	0.548	0.903	0.711	0.717	0.600	0.917	0.620	0.979	0.634	0.950	0.623
	336	0.546	0.488	0.566	0.519	0.556	0.505	1.148	0.588	0.813	0.552	0.868	0.560	0.889	0.708	0.913	0.700	0.957	0.649	1.047	0.667	0.973	0.646
	720	0.552	0.497	0.643	0.580	0.568	0.519	1.869	0.667	0.958	0.602	1.259	0.625	0.877	0.705	1.740	1.019	1.089	0.708	1.142	0.715	1.012	0.687

Appendix

Table 14: Full results of real-world weather benchmark (RW-bench).

Dataset/Model	FedTRL (Ours)	VisionTS	Sundial (94B)	Sundial (230B)	Sundial (1032B)
ETTh1	0.399	0.412	0.402	0.403	0.411
ETTh2	0.350	0.365	0.377	0.364	0.333
ETTm1	0.349	0.391	0.367	0.352	0.336
ETTm2	0.282	0.296	0.280	0.273	0.258
Weather	0.238	0.303	0.254	0.252	0.234
$1^{st}$ Count	5	0	0	0	0

Appendix

Table 15: Additional zero-shot forecasting results averaged over four horizons (96, 192, 336, 720). The results from Sundial are from the original paper ( Liu et al., 2025 ) because its evaluation protocol is consistent with ours.

Patch Size	Context Length	Prediction Length	Layers	Dimension	Heads	Total Parameters
$(P)$	$(T)$	$(F)$	$(L)$	$(D,D_{\text{ff}})$	$H$	$\#\text{Count}$
$16$	$3072$	$\{16,720\}$	12	$(512,2048)$	$8$	$38$ M
$16$	$3072$	$\{16,720\}$	16	$(768,3072)$	$12$	$114$ M
$16$	$3072$	$\{16,720\}$	24	$(1024,4096)$	$16$	$302$ M

Appendix

Table 16: Model configurations of scalability exploration, where gray shading is the default setting.

Full-shot Forecasting
Task / Report	Datasets / Metric	Pretraining Datasets Scale (Billion)
Task / Report	Datasets / Metric	90B	120B	180B	300B (Original)	450B	540B
MSE Report	ETTh1	0.401	0.387	0.372	0.371	0.363	0.363
	ETTm1	0.343	0.336	0.328	0.316	0.312	0.310
	Weather	0.232	0.231	0.227	0.214	0.204	0.202
Zero-shot Point Forecasting

Appendix

Table 17: Full/Zero-shot point forecasting (MSE report) and zero-shot probabilistic forecasting (Avg. MASE report) results across different pretraining datasets scales.

Method	ETTh1	ETTm1	Traffic	Weather
FedAvg	0.511	0.433	0.488	0.295
FFTS	0.492	0.421	0.479	0.290
Centralized All Mixed	0.498	0.427	0.483	0.290
FedTRL (Original)	0.448	0.375	0.426	0.241
FedTRL (Ours)	0.472	0.411	0.477	0.282

Appendix

Table 18: In-domain forecasting results (MSE averaged across

\{96,192,336,720\}

). Corresponding domain dataset has been excluded from the pretraining dataset (energy, traffic, respectively).

Method	ETTh1	ETTm1	Weather
FedAvg	0.438	0.384	0.292
FFTS	0.427	0.380	0.282
Centralized All Mixed	0.431	0.382	0.288
FedTRL (Original)	0.399	0.349	0.238
FedTRL	0.418 ( $\downarrow$ 4.76%)	0.366 ( $\downarrow$ 4.87%)	0.269 ( $\downarrow$ 13.03%)

Appendix

Table 19: Zero-shot forecasting results (MSE averaged across

\{96,192,336,720\}

). Corresponding domain dataset has been excluded from the pretraining dataset (energy, traffic, respectively).

Method	Inter-domain Centroid Distance ( $\uparrow$ )	Within-domain Variance ( $\downarrow$ )
All Mixed (Centralized)	1.08	0.91
FedAvg	1.06	0.85
FFTS	1.34	0.69
FedTRL (Ours)	1.27	0.53

Appendix

Table 20: Representation-level Domain Analysis. We report the average inter-domain centroid distance (higher is better separation) and average within-domain variance (lower is more compact).

Method	Avg. Gradient Cosine Similarity ( $\uparrow$ )	Gradient Norm Variance ( $\downarrow$ )
All Mixed (Centralized)	0.23	0.021
FedAvg	0.33	0.015
FFTS	0.37	0.013
FedTRL (Ours)	0.43	0.010

Appendix

Table 21: Gradient Conflict Analysis Across Domains. We report the average pairwise cosine similarity between domain-wise gradients (higher means less conflict) and the variance of gradient norms across domains (lower means more balanced contributions).

Task Type / Dataset (Metric)	Method	Inter-domain Imbalance Level
Task Type / Dataset (Metric)	Method	Balanced (Simulated)	Mildly Imbalanced (Simulated)	Real Skew (Original Pretraining)
In-domain Forecasting (MSE on Weather)	All Mixed (Centralized)	0.235	0.248	0.254
	FedAvg	0.231	0.248	0.264
	FFTS	0.224	0.235	0.252
	FedTRL (Ours)	0.216	0.230	0.241
Full-shot Forecasting (MSE on Weather)	All Mixed (Centralized)	0.227	0.235	0.238

Appendix

Table 22: Forecasting results under different inter-domain imbalance levels.

Figure 8: Showcase of diffusion-based reconstruction during pretraining.

Figure 9: Zero-shot point forecasting results on our RW-Bench. From top to bottom, the prediction horizons are 96, 192, 336, and 720, corresponding to lookback window lengths of 512, 1024, 2048, and 3072, respectively. The visual samples are randomly selected from RW-Bench.

Figure 10: Zero-shot probabilistic forecasting results on FEV-leaderboard datasets, with visual samples randomly drawn from the benchmark.

Time-Series Foundation Models

Time-series foundation models are pretrained sequence models designed to transfer across forecasting, classification, anomaly detection, and other temporal tasks.

See all 298 papers

Apr 6, 2026

Discrete Prototypical Memories for Federated Time Series Foundation Models

Leveraging Large Language Models (LLMs) as federated learning (FL)-based time series foundation models offers a promising way to transfer the generalization capabilities of LLMs to time series data while preserving access to private data. However, the semantic misalignment between time-series data and the text-centric latent space of existing LLMs often leads to degraded performance. Meanwhile, the parameter-sharing mechanism in existing FL methods model heterogeneous cross-domain time-series data into a unified continuous latent space, which contradicts the fact that time-series semantics frequently manifest as discrete and recurring regimes. To address these limitations, we propose FeDPM, a federated framework for time-series foundation models based on discrete prototypical memories. Specifically, we learn local prototypical memory priors for intra-domain time-series data. We then align cross-domain memories to promote a unified discrete latent space and introduce a domain-specific memory update mechanism to balance shared and personalized prototypical knowledge. Extensive experiments demonstrate the efficiency and effectiveness of FeDPM. The code is publicly available at https://anonymous.4open.science/r/FedUnit-64D1.

Liwei Deng, Qingxiang Liu, Xinhe Niu +5

Australian Artificial Intelligence Institute, University of Technology Sydney · The Hong Kong University of Science and Technology (Guangzhou) · Shenzhen University +2

Time-Series Foundation Models Federated Learning Heterogeneous Federated Learning

Ask PDF

Mar 3, 2026

Adapting Time Series Foundation Models through Data Mixtures

Time series foundation models (TSFMs) have become increasingly popular for zero-shot forecasting. However, for a new time series domain not fully covered by the pretraining set, performance can suffer. Therefore, when a practitioner cares about a new domain and has access to a set of related datasets, the question arises: how best to fine-tune a TSFM to improve zero-shot forecasting? A typical approach to this type of problem is to fine-tune a LoRA module on all datasets or separately on each dataset. Tuning a separate module on each dataset allows for the specialisation of the TSFM to different types of data distribution, by selecting differing combinations of per-dataset modules for different time series contexts. However, we find that, using per-dataset modules might not be optimal, since a time series dataset can contain data from several types of distributions, i.e. sub-domains. This can be due to the distribution shifting or having differing distributions for different dimensions of the time series. Hence, we propose MixFT which re-divides the data using Bayesian mixtures into sets that best represent the sub-domains present in the data, and fine-tunes separately on each of these sets. This re-division of the data ensures that each set is more homogeneous, leading to fine-tuned modules focused on specific sub-domains. Our experiments show that MixFT performs better than per-dataset methods and when fine-tuning a single module on all the data. This suggests that by re-partitioning the data to represent sub-domains we can better specialise TSFMs to improve zero-shot forecasting.

Thomas L. Lee, Edoardo M. Ponti, Amos Storkey

School of Informatics, University of Edinburgh.

Time-Series Foundation Models Zero-Shot Time Series Forecasting Supervised Fine-Tuning Multi-Source Domain Adaptation

Ask PDF

May 26, 2026

Falcon-X: A Time Series Foundation Model for Heterogeneous Multivariate Modeling

Time series foundation models (TSFMs) are transforming the forecasting paradigm through large-scale cross-domain pretraining. However, most existing TSFMs remain univariate, and recent efforts to enable cross-variate modeling still operate directly within the raw variate space. This design introduces fundamental limitations in semantic alignment and relational expressivity. Specifically, raw-space group mixing lacks a dedicated mechanism to align heterogeneous physical quantities, while standard non-negative attention fails to capture the complex synergistic and antagonistic interactions ubiquitous in real-world systems. To address these challenges, we propose Falcon-X, decouples variates from the raw space and maps them into a unified latent prototype space. Falcon-X employs a Unified Prototype Diff-Attention mechanism that explicitly evaluates both positive and negative semantic affinities to explicitly align heterogeneous variates. Cross-variate interactions are then efficiently performed within this shared space via Latent Entity Attention, naturally facilitating zero-shot structural transfer. Finally, a Variate Reassembly Router robustly reconstructs variate-specific trajectories via a request-and-dispatch mechanism. Extensive evaluations on the GIFT-Eval and fev-bench benchmarks demonstrate that Falcon-X achieves excellent forecasting performance, offering a principled and scalable paradigm for complex multivariate environments. Falcon-X is publicly released to support future research.

Yiding Liu, Yifan Hu, Hongjie Xia +5

Multivariate Time Series Forecasting Zero-Shot Time Series Forecasting Differential Attention Time-Series Foundation Models

Ask PDF

Abstract

Time-Series Foundation Models

Explore similar work

Discrete Prototypical Memories for Federated Time Series Foundation Models

Adapting Time Series Foundation Models through Data Mixtures

Falcon-X: A Time Series Foundation Model for Heterogeneous Multivariate Modeling