QiYao-M: Multimodal Time Series Foundation Model with Role-Aware Modeling of Endogenous and Exogenous Modalities

Hanyin Cheng; Linfeng Wang; Zhengbo Qu; Yang Shu; Zhongwen Rao; Meng Wang; Yijie Li; Xin Jiang

cs.LGSep 28, 2026

QiYao-M: Multimodal Time Series Foundation Model with Role-Aware Modeling of Endogenous and Exogenous Modalities

Authors: Hanyin Cheng, Linfeng Wang, Zhengbo Qu, Yang Shu, Zhongwen Rao, Meng Wang, Yijie Li, Xin Jiang, +2 more

Organizations: East China Normal University · Huawei Technologies Co., Ltd.

Abstract

Existing multimodal time series foundation models (TSFMs) typically model heterogeneous modalities through largely shared mechanisms, overlooking the distinct forecasting roles of endogenous and exogenous modalities. In this work, we propose QiYao-M, a role-aware multimodal TSFM that models the two types of modalities separately. For endogenous modalities, to capture how they evolve along with the underlying temporal dynamics, we introduce an Endo-Multimodal Predictor and Endo-Multimodal Supervision to explicitly learn their evolution from history to the future. For exogenous modalities, to generalize across domains and across various modality types and numbers under the scarcity of exo-multimodal pretraining data, we propose an Exo-Multimodal Retrieval Enhancer that enables rapid downstream adaptation without updating the TSFM parameters. We further introduce Endo-Modality Proxy Training to train this retrieval module without exogenous multimodal pretraining data. Extensive experiments across unimodal and multimodal benchmarks demonstrate strong forecasting performance in scenarios both with and without exogenous modalities.

Figures & tables

Figure 1: Illustration of different modeling paradigms for TSFMs. (a) Unimodal TSFMs on numerical time series. (b) Role-agnostic multimodal TSFMs model endogenous (endo-) and exogenous (exo-) modalities through largely shared mechanisms. (c) Our role-aware multimodal TSFM separately models endo- and exo- modalities according to their distinct characteristics.

Figure 2: The Multimodal Modeling and Training Strategy of QiYao-M.

Figure 3: Key steps of the Exo-Multimodal Retrieval Enhancer .

Figure 4: Evaluation summary.

Panel of Figure 5 — Figure 5: Performance of QiYao-M on unimodal benchmarks.

Modal Type	TS + Text										TS + Text + Image				TS + Text + Image + Table
Methods	Agriculture		Climate		Energy		Health		Social Good		TAOBAO		Tianchi		HS300		SP500
Methods	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
Multimodal End-to-End Models
GPT4MTS	0.225	0.298	1.182	0.889	0.262	0.380	1.464	0.799	0.920	0.450	0.534	0.257	1.320	0.147	0.780	0.535	0.767	0.642
CALF	0.250	0.315	1.286	0.922	0.244	0.365	1.491	0.775	0.906	0.401	0.512	0.250	1.320	0.119	0.674	0.502	0.743	0.617
Time-VLM	0.237	0.302	1.195	0.899	0.260	0.374	1.565	0.860	0.868	0.444	0.528	0.271	1.368	0.137	0.662	0.499	0.756	0.618

Table 1: Average forecasting results on multimodal datasets. The best and second-best results are highlighted in purple and blue , respectively. Full results are listed in Section D.2 of Appendix D .

Figure 7

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Deterministic patch-level text construction for a real 32-point patch from Taobao-Fashion item 948150 . The template, numerical patch, and generated description are shown together to make the supervision auditable.

Figure 8: Actual $224\times 224$ patch-level RGB image generated from the listed real 32-point Taobao-Fashion patch. The three colored curves encode the value, first-difference, and second-difference views, respectively.

Panel of Figure 9 — Figure 9: Deterministic sequence-level text construction from a real 256-point Taobao-Fashion input window. The item identifier and date range are retained to make the example traceable to the source data.

Panel of Figure 10 — Figure 10: The three sequence-level image views generated deterministically from the same real 256-point Taobao-Fashion historical sequence.

Dataset	Domain	Frequency	Variates	Prediction Length
Jena Weather	Nature	10T / H / D	21	$\{48,480,720\}$ (10T/H); 30 (D)
BizITObs–Application	Web/CloudOps	10S	2	$\{60,600,900\}$
BizITObs–Service	Web/CloudOps	10S	2	$\{60,600,900\}$
BizITObs–L2C	Web/CloudOps	5T / H	7	$\{48,480,720\}$
Bitbrains–Fast Storage	Web/CloudOps	5T / H	2	$\{48,480,720\}$ (5T); 48 (H)
Bitbrains–rnd	Web/CloudOps	5T / H	2	$\{48,480,720\}$ (5T); 48 (H)

Appendix

Table 3: Dataset statistics and forecasting configurations of GIFT-Eval. Multiple frequencies of the same underlying dataset are grouped into one row.

Benchmark	Dataset	Modalities	Frequency	Scale	Prediction Length
TimeMMD	Agriculture	TS + Text	Monthly	$496\times 1$	$\{6,8,10,12\}$
	Climate	TS + Text	Monthly	$496\times 5$	$\{6,8,10,12\}$
	Energy	TS + Text	Weekly	$1{,}479\times 9$	$\{12,24,36,48\}$
	Health	TS + Text	Weekly	$1{,}389\times 11$	$\{12,24,36,48\}$
	Social Good	TS + Text	Monthly	$900\times 1$	$\{6,8,10,12\}$
MoTIME	TaobaoFashion	TS + Text + Image	Daily	$365\times 890$	$\{1,7,14,21,28\}$

Appendix

Table 5: Statistics of the exogenous multimodal forecasting datasets used in our experiments. The modality column reports the external information available to each dataset.

Model Type	Model	Code / Model Repository
Multimodal End-to-End	GPT4MTS	https://github.com/Flora-jia-jfr/ GPT4MTS-Prompt-based-Large-Language-Model- for-Multimodal-Time-series-Forecasting
	CALF	https://github.com/Hank0626/CALF
	Time-VLM	https://github.com/CityMind-Lab/ICML25-TimeVLM
	TATS	https://github.com/iDEA-iSAIL-Lab-UIUC/TaTS
Unimodal Foundation	Zeus	https://github.com/GestaltCogTeam/Zeus
	Chronos-2	https://github.com/amazon-science/chronos-forecasting

Appendix

Table 6: Code and model repositories of the baseline methods used in our experiments.

Model	Backbone Layers	Fusion After	Model Dim.	FFN Dim.	Heads	Patch Size	CLIP Dim.	Retrieval Layers	Main Parameters
QiYao-M	24	16	1024	2736	16	32	512	5	375.63M

Appendix

Table 7: Architecture and parameter counts of QiYao-M. The main-model count excludes the frozen CLIP feature encoder and does not double-count the shared forecasting head.

Figure 11: Sensitivity analysis of the retrieval module.

Methods	H=6		H=8		H=10		H=12		Avg.
	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
Multimodal End-to-End Models
GPT4MTS	0.161	0.257	0.207	0.288	0.230	0.305	0.301	0.342	0.225	0.298
CALF	0.142	0.250	0.195	0.285	0.350	0.370	0.314	0.355	0.250	0.315
Time-VLM	0.143	0.245	0.215	0.287	0.271	0.320	0.322	0.359	0.237	0.302
TATS	0.140	0.251	0.187	0.282	0.244	0.320	0.290	0.350	0.215	0.301

Appendix

Table 8: Step-wise forecasting results on Agriculture . Lower MSE and MAE indicate better performance. The best and second-best results are highlighted in purple and blue , respectively.

Horizons	H=6		H=8		H=10		H=12		Avg.
Metrics	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
Multimodal End-to-End Models
GPT4MTS	1.199	0.895	1.205	0.899	1.173	0.885	1.152	0.876	1.182	0.889
CALF	1.231	0.910	1.227	0.911	1.508	0.989	1.177	0.883	1.286	0.922
Time-VLM	1.218	0.907	1.181	0.914	1.179	0.880	1.203	0.896	1.195	0.899
TATS	1.194	0.897	1.178	0.886	1.170	0.881	1.179	0.885	1.180	0.887

Appendix

Table 9: Step-wise forecasting results on Climate . Lower MSE and MAE indicate better performance. The best and second-best results are highlighted in purple and blue , respectively.

Horizons	H=12		H=24		H=36		H=48		Avg.
Metrics	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
Multimodal End-to-End Models
GPT4MTS	0.111	0.244	0.232	0.362	0.308	0.418	0.398	0.496	0.262	0.380
CALF	0.102	0.224	0.210	0.346	0.300	0.420	0.365	0.470	0.244	0.365
Time-VLM	0.114	0.253	0.227	0.359	0.309	0.410	0.390	0.475	0.260	0.374
TATS	0.105	0.232	0.216	0.344	0.309	0.418	0.391	0.480	0.255	0.368

Appendix

Table 10: Step-wise forecasting results on Energy . Lower MSE and MAE indicate better performance. The best and second-best results are highlighted in purple and blue , respectively.

Horizons	H=12		H=24		H=36		H=48		Avg.
Metrics	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
Multimodal End-to-End Models
GPT4MTS	0.985	0.658	1.513	0.802	1.601	0.846	1.757	0.889	1.464	0.799
CALF	0.964	0.609	1.451	0.749	1.713	0.851	1.836	0.889	1.491	0.775
Time-VLM	1.198	0.727	1.491	0.839	1.867	0.967	1.702	0.907	1.565	0.860
TATS	0.899	0.612	1.307	0.759	1.523	0.827	1.693	0.872	1.356	0.767

Appendix

Table 11: Step-wise forecasting results on Health . Lower MSE and MAE indicate better performance. The best and second-best results are highlighted in purple and blue , respectively.

Horizons	H=6		H=8		H=10		H=12		Avg.
Metrics	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
Multimodal End-to-End Models
GPT4MTS	0.718	0.378	0.942	0.505	0.929	0.446	1.093	0.470	0.920	0.450
CALF	0.782	0.360	0.874	0.386	0.976	0.420	0.991	0.439	0.906	0.401
Time-VLM	0.732	0.379	0.822	0.427	0.916	0.465	1.005	0.505	0.868	0.444
TATS	0.753	0.370	0.875	0.409	0.991	0.459	1.053	0.474	0.918	0.428

Appendix

Table 12: Step-wise forecasting results on Social Good . Lower MSE and MAE indicate better performance. The best and second-best results are highlighted in purple and blue , respectively.

Horizons	H=1		H=7		H=14		H=21		H=28		Avg.
Metrics	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
Multimodal End-to-End Models
GPT4MTS	0.384	0.189	0.473	0.232	0.613	0.297	0.640	0.293	0.559	0.272	0.534	0.257
CALF	0.420	0.230	0.488	0.240	0.544	0.263	0.536	0.254	0.570	0.263	0.512	0.250
Time-VLM	0.419	0.226	0.489	0.257	0.555	0.290	0.568	0.290	0.609	0.289	0.528	0.271
TATS	0.443	0.242	0.515	0.258	0.538	0.273	0.548	0.267	0.579	0.274	0.525	0.263

Appendix

Table 13: Step-wise forecasting results on TAOBAO-fashion . Lower MSE and MAE indicate better performance. The best and second-best results are highlighted in purple and blue , respectively.

Horizons	H=1		H=7		H=14		H=21		H=28		Avg.
Metrics	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
Multimodal End-to-End Models
GPT4MTS	1.931	0.156	1.384	0.138	1.112	0.154	1.013	0.141	1.162	0.145	1.320	0.147
CALF	2.017	0.128	1.346	0.108	1.065	0.111	0.996	0.114	1.176	0.134	1.320	0.119
Time-VLM	1.982	0.124	1.416	0.135	1.217	0.155	1.027	0.129	1.200	0.140	1.368	0.137
TATS	2.035	0.135	1.411	0.134	1.165	0.138	1.052	0.133	1.206	0.139	1.374	0.136

Appendix

Table 14: Step-wise forecasting results on Tianchi . Lower MSE and MAE indicate better performance. The best and second-best results are highlighted in purple and blue , respectively.

Horizons	H=24		H=48		H=96		Avg.
Metrics	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
Multimodal End-to-End Models
GPT4MTS	0.321	0.352	0.626	0.493	1.392	0.759	0.780	0.535
CALF	0.283	0.333	0.574	0.475	1.165	0.696	0.674	0.502
Time-VLM	0.287	0.347	0.500	0.456	1.200	0.693	0.662	0.499
TATS	0.281	0.330	0.561	0.461	1.309	0.737	0.717	0.509

Appendix

Table 15: Step-wise forecasting results on HS300 . Lower MSE and MAE indicate better performance. The best and second-best results are highlighted in purple and blue , respectively.

Horizons	H=24		H=48		H=96		Avg.
Metrics	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
Multimodal End-to-End Models
GPT4MTS	0.484	0.503	0.723	0.635	1.095	0.786	0.767	0.642
CALF	0.368	0.434	0.598	0.564	1.262	0.853	0.743	0.617
Time-VLM	0.395	0.449	0.760	0.629	1.114	0.776	0.756	0.618
TATS	0.365	0.430	0.617	0.570	1.014	0.742	0.665	0.581

Appendix

Table 16: Step-wise forecasting results on SP500 . Lower MSE and MAE indicate better performance. The best and second-best results are highlighted in purple and blue , respectively.

Explore similar work

Rethinking Post-Training Recipes for Multimodal Time-Series Forecasting
May 28, 2026Haoxin Liu, Yichen Zhou, Rajat Sen +2Zero-Shot Forecasting Time Series Foundation Models
Georgia Institute of Technology · Google Research
Chronicle: A Multimodal Foundation Model for Joint Language and Time Series Understanding
May 18, 2026Paul Quinlan, Jeremy Levasseur, Qingguo Li +1Time Series Foundation Models Time Series
1InertialAI · Department of Electrical and Computer Engineering, Queen’s University · Department of Mechanical and Materials Engineering, Queen’s University
Rethinking Multimodal Fusion for Time Series: Text Modalities Need Constrained Fusion
Mar 23, 2026Seunghan Lee, Jun Seo, Jaehoon Lee +7Multimodal Fusion Multi-Stream Temporal Fusion
LG AI Research Seoul, South Korea

May 28, 2026, cs.LGJ/K move · Enter open · S save

Rethinking Post-Training Recipes for Multimodal Time-Series Forecasting

Haoxin Liu, Yichen Zhou, Rajat Sen, B. Aditya Prakash+1

Georgia Institute of Technology · Google Research

Zero-Shot Forecasting Time Series Foundation Models Post-Training

Open paper AskPDF arXiv

Time-Series Foundation Models (TSFMs) excel at zero-shot unimodal forecasting using numerical data, but unlike LLMs they cannot consume multimodal, non-numerical context that often shape real-world trajectories. In this work, we bridge this gap and argue for a multimodal time-series forecasting approach that post-trains LLMs to act as context-guided revisors over strong numerical TSFM priors. We introduce PostTime, a post-training recipe combining Supervised Fine-Tuning (SFT) and Reinforcement Learning with Verifiable Rewards (RLVR), along with a methodology to generate automated reasoning traces for forecast revisions. PostTime teaches an LLM to generate context-conditioned forecast interventions -- decisions to revise, preserve, or ignore the TSFM prior based on the multimodal context. We evaluate this approach on the TimesX multimodal forecasting benchmark using a Gemma-3-4B LLM and TimesFM-2.5 TSFM, and show that it significantly outperforms standalone TSFMs, LLM-only baselines, and existing multimodal forecasting approaches.

Abstract

Explore similar work

Rethinking Post-Training Recipes for Multimodal Time-Series Forecasting

Chronicle: A Multimodal Foundation Model for Joint Language and Time Series Understanding

Rethinking Multimodal Fusion for Time Series: Text Modalities Need Constrained Fusion