TiMi: Empower Time Series Transformers with Multimodal Mixture of Experts

Jiafeng Lin; Yuxuan Wang; Huakun Luo; Jianmin Wang; Zhongyi Pei

cs.LGFeb 25, 2026

TiMi: Empower Time Series Transformers with Multimodal Mixture of Experts

Authors: Jiafeng Lin, Yuxuan Wang, Huakun Luo, Jianmin Wang, Zhongyi Pei

Organizations: School of Software, BNRist, Tsinghua University.

Abstract

Multimodal time series forecasting has garnered significant attention for its potential to provide more accurate predictions than traditional single-modality models by leveraging rich information inherent in other modalities. However, due to fundamental challenges in modality alignment, existing methods often struggle to effectively incorporate multimodal data into predictions, particularly textual information that has a causal influence on time series fluctuations, such as emergency reports and policy announcements. In this paper, we reflect on the role of textual information in numerical forecasting and propose Time series transformers with Multimodal Mixture-of-Experts, TiMi, to unleash the causal reasoning capabilities of LLMs. Concretely, TiMi utilizes LLMs to generate inferences on future developments, which serve as guidance for time series forecasting. To seamlessly integrate both exogenous factors and time series into predictions, we introduce a Multimodal Mixture-of-Experts (MMoE) module as a lightweight plug-in to empower Transformer-based time series models for multimodal forecasting, eliminating the need for explicit representation-level alignment. Experimentally, our proposed TiMi demonstrates consistent state-of-the-art performance on sixteen real-world multimodal forecasting benchmarks, outperforming advanced baselines while offering both strong adaptability and interpretability.

Figures & tables

Figure 1: Two types of multimodal time series data exist in real-world forecasting scenarios. Compared to series-metadata data, which is semantically aligned like vision-language data, series-textual data lacks this direct alignment but often offers richer and more informative insights for prediction.

Figure 2: Three categories of multimodal time series forecasting models distinguished through fusion.

Figure 3: Overall design of TiMi, a time series-centric Transformer model. Textual content is encoded by LLMs to infer future trends of predictions. The MMoE module integrates a global view of both historical input and causal knowledge derived from text, thereby enhancing the time series modeling.

Models	TiMi		Time-MMD		AutoTimes		Time-LLM		GPT4TS		TimeXer		iTransformer		PatchTST		TimesNet		DLinear		Autoformer
Models	(Ours)		( 2024a )		( 2024d )		( 2023 )		( 2023 )		( 2024 )		( 2024c )		( 2022 )		( 2023a )		( 2023 )		( 2021 )
Metric	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
Agriculture	0.193	0.299	0.213	0.301	1.032	0.716	0.262	0.318	0.224	0.306	0.230	0.306	0.227	0.306	0.242	0.312	0.217	0.316	0.511	0.514	0.282	0.350
Climate	0.872	0.737	0.968	0.783	0.944	0.758	0.915	0.750	0.928	0.756	0.950	0.767	0.970	0.779	0.977	0.779	0.930	0.767	0.915	0.750	0.943	0.763
Economy	0.012	0.086	0.022	0.123	0.736	0.729	0.018	0.110	0.014	0.096	0.013	0.091	0.012	0.087	0.013	0.093	0.014	0.096	0.086	0.237	0.061	0.194
Energy	0.229	0.344	0.254	0.365	0.335	0.432	0.266	0.380	0.280	0.397	0.247	0.361	0.253	0.364	0.288	0.392	0.271	0.383	0.310	0.418	0.306	0.418

Table 1: Multimodal time series forecasting results on Time-MMD datasets with the best in red and the second underlined . Based on the varying sampling frequencies of each dataset, there are three fixed input lengths of

8

24

, and

96

, respectively corresponding to three sets of prediction lengths

\{6,8,10,12\}

\{12,24,36,48\}

, and

\{48,96,192,336\}

. The table reports the average results across all prediction lengths.

Figure 4: Irregular multimodal time series forecasting results on Time-IMM datasets. The metric for comparison is MSE. We compared four methods using PatchTST as the temporal backbone, among which the baseline results of IMM-TSF and PatchTST were derived from Time-IMM ( Chang et al., 2025 ) .

Figure 5: Multimodal time series forecasting results of replacing the LLM backbone. The time series backbone is fixed as PatchTST. The result is the average performance across four dataset-specific prediction horizons. For detailed results, please refer to the Appendix H.3 .

Datasets		Agriculture		Climate		Energy		Security		SocialGood		Traffic
Metric		MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
TimeXer	Original	0.230	0.306	0.950	0.767	0.247	0.361	78.924	4.305	1.302	0.514	0.176	0.220
	+MMoE	0.203	0.304	0.878	0.738	0.226	0.342	71.635	3.929	0.962	0.512	0.155	0.219
	Promotion	11.7%	0.7%	7.6%	3.8%	8.5%	5.3%	9.2%	8.7%	26.1%	0.4%	11.9%	0.5%
PatchTST	Original	0.242	0.312	0.977	0.779	0.288	0.392	84.305	4.560	1.293	0.523	0.173	0.219
	+MMoE	0.193	0.299	0.872	0.737	0.229	0.344	71.519	3.896	0.886	0.449	0.154	0.218

Table 2: Performance improvement achieved by applying the MMoE module. Each result represents the average across the four prediction lengths, with ”Promotion” indicating the percentage enhancement relative to the original model. The results fully demonstrate the versatility of the TiMi framework. Complete results are provided in the Appendix H.1 .

Design	Textual	Temporal	Agriculture		Energy		Health(US)		SocialGood		Traffic
Design	Textual	Temporal	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
TiMi	TMoE	SMoE	0.193	0.299	0.229	0.344	1.194	0.745	0.886	0.449	0.154	0.218
w/o	w/o	SMoE	0.204	0.301	0.231	0.346	1.267	0.759	0.988	0.463	0.154	0.226
w/o	TMoE	w/o	0.200	0.300	0.251	0.366	1.255	0.764	0.892	0.473	0.167	0.214
Replace	MoE	MoE	0.231	0.311	0.264	0.373	1.268	0.763	0.958	0.471	0.162	0.222
Replace	Cross	SMoE	0.222	0.304	0.272	0.373	1.268	0.764	0.904	0.453	0.160	0.234

Table 3: Ablations results on TiMi. The experiments includes the removal of specific components, the replacement of components, and the random initialization of the LLM. Each result reports the average performance across the dataset’s four prediction lengths.

Figure 6: Visualization of the role of Series-aware MoE. Time series are grouped into four trend regimes based on MK test, represented by distinct colors. Series with different trend are consistently routed to different experts.

Figure 7: Visualization results on the Health(US) dataset.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Agriculture		Climate		Economy
Horizon	MSE	MAE	MSE	MAE	MSE	MAE
$6/6/6$	0.125 $\pm$ 0.002	0.244 $\pm$ 0.001	0.860 $\pm$ 0.003	0.733 $\pm$ 0.002	0.011 $\pm$ 0.001	0.084 $\pm$ 0.001
$8/8/8$	0.174 $\pm$ 0.003	0.285 $\pm$ 0.002	0.865 $\pm$ 0.001	0.734 $\pm$ 0.001	0.011 $\pm$ 0.001	0.085 $\pm$ 0.001
$10/10/10$	0.220 $\pm$ 0.004	0.315 $\pm$ 0.003	0.879 $\pm$ 0.004	0.739 $\pm$ 0.002	0.012 $\pm$ 0.001	0.088 $\pm$ 0.001
$12/12/12$	0.254 $\pm$ 0.010	0.350 $\pm$ 0.008	0.885 $\pm$ 0.005	0.741 $\pm$ 0.005	0.012 $\pm$ 0.001	0.088 $\pm$ 0.001
Dataset	Energy		Environment		Health(US)

Appendix

Table 4: Robustness of TiMi performance on the Time-MMD datasets. The results are obtained from three random seeds. The three values of horizon in each row of the table correspond respectively to the three datasets on the right.

Figure 8: General multi-dimensional time series reasoning prompts. The components include domain-specific prompts, textual information, system prompts, question and option prompts in three time series dimensions (trend, frequency, and noise).

Figure 9: A reasoning case on the Energy dataset. Prompts exhibit strong adaptability to variations in text quality, effectively extracting information about key temporal attributes.

Figure 10: Visualization of Series-aware analysis. The left and right figures are respectively conducted on dataset Climate and dataset Traffic. The strong downward, weak downward, weak upward, and strong upward correspond respectively to

Z\in(-\infty,-1)

Z\in[-1,0)

Z\in[0,1)

Z\in[1,+\infty)

Interference	Ratio	Agriculture		Energy		Health(US)		SocialGood		Traffic
Interference	Ratio	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
TiMi	-	0.193	0.299	0.229	0.344	1.194	0.745	0.886	0.449	0.154	0.218
Noisy	5%	0.198	0.300	0.230	0.346	1.236	0.753	0.886	0.449	0.154	0.219
Noisy	20%	0.200	0.301	0.234	0.348	1.240	0.755	0.887	0.450	0.155	0.219
Synonymous	20%	0.205	0.304	0.252	0.365	1.264	0.882	1.025	0.470	0.159	0.221
Irrelevant	20%	0.201	0.301	0.232	0.347	1.245	0.758	0.887	0.450	0.156	0.220

Appendix

Table 5: Results of stress-test across multiple textual information distortion scenarios.

Figure 11: Visualization results on the Energy and Health(US) datasets configured with input-24-predict-12 setting. We make comparisons among the three models, TiMi, Time-MMD and PatchTST.

Models	Agriculture		Climate		Economy		Energy		Health(US)		Security		SocialGood		Traffic
Metric	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
TiMi	0.123	0.234	0.845	0.707	0.133	0.250	0.144	0.247	0.611	0.472	14.031	1.305	0.378	0.352	0.035	0.094
Time-MMD	0.127	0.242	0.889	0.731	0.148	0.276	0.145	0.248	0.716	0.564	15.567	1.419	0.417	0.390	0.071	0.152
PatchTST	0.128	0.242	0.849	0.710	0.140	0.257	0.155	0.257	0.625	0.484	15.198	1.382	0.394	0.372	0.041	0.110

Appendix

Table 6: Performance comparison on the temporal held-out test set (July 2023 – May 2024) to validate robustness against potential pre-training data leakage.

Models		TimeXer		TimeXer		PatchTST		PatchTST		Autoformer		Autoformer
Models		( 2024 )		+MMoE		( 2022 )		+MMoE		( 2021 )		+MMoE
Metric		MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
Agriculture	6	0.143	0.248	0.132	0.254	0.174	0.262	0.125	0.244	0.220	0.317	0.172	0.282
	8	0.218	0.299	0.170	0.289	0.216	0.302	0.174	0.285	0.257	0.336	0.190	0.290
	10	0.240	0.316	0.228	0.321	0.262	0.327	0.220	0.315	0.304	0.358	0.265	0.367
	12	0.320	0.361	0.281	0.353	0.315	0.357	0.254	0.350	0.349	0.391	0.305	0.397

Appendix

Table 7: Detailed results of the multimodal time series forecasting performance improvement achieved by applying the Multimodal Mixture of Experts (MMoE) module . Avg means the average results from all four prediction lengths.

Datasets	GDELT		RepoHealth		FNSPID		ClusterTrace		ILINet		CESNET		EPA-Air
Metric	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
TiMi (Ours)	1.0293	0.6889	0.5323	0.4161	0.1269	0.2217	0.6890	0.6756	0.9620	0.6651	0.8957	0.7271	0.5987	0.5792
IMM-TSF ( 2025 )	1.0511	0.6944	0.5520	0.4180	0.1277	0.2212	1.0370	0.8389	1.1727	0.7373	1.1242	0.8192	0.6204	0.5992
Time-MMD ( 2024a )	1.2352	0.7731	0.5786	0.4531	0.1413	0.2451	0.8304	0.7447	1.0404	0.6753	0.9706	0.7614	0.6506	0.5975
PatchTST ( 2022 )	1.0670	0.7051	0.5727	0.4185	0.1301	0.2240	2.0366	0.9896	1.1606	0.7532	1.3177	0.8797	0.6196	0.5947

Appendix

Table 8: Full results of irregular multimodal time series forecasting on Time-IMM datasets. The complete MSE and MAE indicators are presented.

Datasets	Agriculture		Climate		Economy		Energy		Health(US)		Security		SocialGood		Traffic
Metric	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
Qwen2.5-7B	0.193	0.299	0.872	0.737	0.012	0.086	0.229	0.344	1.194	0.745	71.519	3.896	0.886	0.449	0.153	0.220
Qwen2.5-0.5B	0.223	0.309	0.892	0.774	0.012	0.089	0.255	0.364	1.270	0.767	73.520	4.026	1.002	0.473	0.159	0.217
GPT2	0.241	0.318	0.889	0.743	0.013	0.091	0.253	0.367	1.270	0.770	73.127	3.970	0.939	0.462	0.162	0.218
w/o LLM	0.242	0.312	0.977	0.779	0.013	0.093	0.288	0.392	1.252	0.759	84.305	4.560	1.293	0.523	0.173	0.219

Appendix

Table 9: Full results of replacing LLM backbone for time series forecasting. Each result is the average of the four predicted lengths of this dataset. w/o represents the baseline result of the single-modality.

Datasets		Agriculture		Climate		Economy		Energy		Environment		Health(US)		Security		SocialGood		Traffic
Metric		MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
	6/12/48	0.125	0.244	0.860	0.733	0.011	0.084	0.093	0.220	0.404	0.460	0.938	0.665	67.207	3.766	0.782	0.378	0.135	0.195
TiMi	8/24/96	0.174	0.285	0.865	0.734	0.011	0.085	0.196	0.322	0.408	0.464	1.163	0.732	68.888	3.847	0.820	0.408	0.152	0.221
(Ours)	10/36/192	0.220	0.315	0.879	0.739	0.012	0.088	0.271	0.389	0.408	0.463	1.300	0.787	74.324	3.930	0.925	0.465	0.158	0.221
	12/48/336	0.254	0.350	0.885	0.741	0.012	0.088	0.354	0.446	0.411	0.462	1.374	0.796	75.658	4.040	1.016	0.545	0.170	0.234
	Avg	0.193	0.299	0.872	0.737	0.012	0.086	0.229	0.344	0.408	0.462	1.194	0.745	71.519	3.896	0.886	0.449	0.154	0.218

Appendix

Table 10: Full results compared with the latest multimodal time series forecasting models with the best in bold . For monthly-sampled datasets such as Agriculture, Climate, Economy, Security, SocialGood, and Traffic, the prediction horizons are

H\in\{6,8,10,12\}

. For weekly-sampled datasets such as Energy and Health(US),

H\in\{12,24,36,48\}

. For the daily-sampled Environment dataset,

H\in\{48,96,192,336\}

. Avg means the average results from all four prediction lengths.

Models		TiMi		Time-MMD		AutoTimes		Time-LLM		GPT4TS		TimeXer		iTransformer		PatchTST		TimesNet		DLinear		Autoformer
Models		(Ours)		( 2024a )		( 2024d )		( 2023 )		( 2023 )		( 2024 )		( 2024c )		( 2022 )		( 2023a )		( 2023 )		( 2021 )
Metric		MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
Agriculture	6	0.125	0.244	0.145	0.251	0.473	0.487	0.159	0.245	0.134	0.241	0.143	0.248	0.149	0.254	0.174	0.262	0.159	0.267	0.304	0.402	0.220	0.317
	8	0.174	0.285	0.181	0.287	0.583	0.501	0.268	0.323	0.211	0.310	0.218	0.299	0.188	0.285	0.216	0.302	0.196	0.302	0.692	0.637	0.257	0.336
	10	0.220	0.315	0.254	0.318	1.294	0.833	0.272	0.323	0.241	0.317	0.240	0.316	0.250	0.322	0.262	0.327	0.239	0.331	0.484	0.482	0.304	0.358
	12	0.254	0.350	0.270	0.348	1.777	1.042	0.349	0.381	0.310	0.357	0.320	0.361	0.322	0.362	0.315	0.357	0.276	0.365	0.566	0.535	0.349	0.391

Appendix

Table 11: Full results of multimodal time series forecasting . There are three fixed input lengths of

8

24

, and

96

, each corresponding to a specific set of prediction lengths

\{6,8,10,12\}

\{12,24,36,48\}

, and

\{48,96,192,336\}

, respectively, following the setting of Time-MMD ( Liu et al., 2024a ) . Avg means the average results from all four prediction lengths.

Time Series Transformers

Time Series Transformers use attention-based sequence models to forecast, classify, and detect patterns in temporal data.

See all 138 papers

Multi-Modal Time Series Prediction via Mixture of Modulated Experts
Jan 29, 2026Lige Zhang, Ali Maatouk, Jialin Chen +3Multivariate Time Series Forecasting Time Series Forecasting
Emory University · Yale University · Amazon
TAC-Time: Texts as Channels For Multimodal Time Series Forecasting
Sep 21, 2026Jiayi Liang, Xiaotian Gu, Xinyu Xie +2Multivariate Time Series Forecasting Explainable Time Series Forecasting
East China Normal University Shanghai, China
Spectral Text Fusion: A Frequency-Aware Approach to Multimodal Time-Series Forecasting
Feb 2, 2026Huu Hiep Nguyen, Minh Hoang Nguyen, Dung Nguyen +1Multivariate Time Series Forecasting Frequency-Domain Time Series Modeling
Applied Artificial Intelligence Initiative Deakin University Geelong, Australia

Jan 29, 2026J/K move · Enter open · S save

Multi-Modal Time Series Prediction via Mixture of Modulated Experts

Lige Zhang, Ali Maatouk, Jialin Chen, Karthik Charan Konduri+2

Emory University · Yale University · Amazon

Multivariate Time Series Forecasting Time Series Forecasting Mixture-of-Experts Models Multimodal Fusion

Open paper AskPDF arXiv

Real-world time series exhibit complex and evolving dynamics, making accurate forecasting extremely challenging. Recent multi-modal forecasting methods leverage textual information such as news reports to improve prediction, but most rely on token-level fusion that mixes temporal patches with language tokens in a shared embedding space. However, such fusion can be ill-suited when high-quality time-text pairs are scarce and when time series exhibit substantial variation in characteristics, thus complicating cross-modal alignment. In parallel, mixture-of-experts (MoE) architectures have proven effective for both time series modeling and multi-modal learning, yet many existing MoE-based modality integration methods still depend on token-level fusion. To address this, we propose Expert Modulation, a new mechanism for multi-modal time series prediction that conditions both routing and expert computation on textual signals, enabling direct and efficient cross-modal control over expert behavior. Through theoretical analysis and experiments, our proposed method demonstrates strong improvements in multi-modal time series prediction. The current code implementation is available at https://github.com/BruceZhangReve/MoME

Abstract

Time Series Transformers

Explore similar work

Multi-Modal Time Series Prediction via Mixture of Modulated Experts

TAC-Time: Texts as Channels For Multimodal Time Series Forecasting

Spectral Text Fusion: A Frequency-Aware Approach to Multimodal Time-Series Forecasting