MM-ISTS: Cooperating Irregularly Sampled Time Series Forecasting with Multimodal Vision-Text LLMs

Zhi Lei; Chenxi Liu; Hao Miao; Wanghui Qiu; Bin Yang; Chenjuan Guo

cs.CVMar 6, 2026

MM-ISTS: Cooperating Irregularly Sampled Time Series Forecasting with Multimodal Vision-Text LLMs

Authors: Zhi Lei, Chenxi Liu, Hao Miao, Wanghui Qiu, Bin Yang, Chenjuan Guo

Organizations: East China Normal University, Shanghai, China · Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences, Hong Kong, China · Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China

Abstract

Irregularly sampled time series (ISTS) are widespread in real-world scenarios, exhibiting asynchronous observations on uneven time intervals across diverse variables. Existing ISTS forecasting methods often solely utilize historical observations to predict future ones while falling short in learning contextual semantics and fine-grained temporal patterns. To address these problems, we propose MM-ISTS, a multimodal ISTS forecasting framework augmented by vision-text large language models, which bridges temporal, visual, and textual modalities. MM-ISTS encompasses a two-stage encoding mechanism. In particular, a Cross-Modal Vision-Text Encoding module is proposed to automatically generate informative visual images and textual data, enabling the capture of intricate temporal patterns and comprehensive contextual understanding, in collaboration with multimodal LLMs (MLLMs). In parallel, ISTS encoding extracts complementary yet enriched temporal features from historical ISTS observations, including multi-view embedding fusion and a Temporal-Variable Encoder. Further, we propose an Adaptive Query-Based Feature Extractor to compress MLLM token embeddings while preserving useful knowledge, which in turn reduces computational costs. In addition, a Multimodal Alignment module with Modality-Aware Gating is designed to alleviate the modality gaps. Extensive experiments on real data offer insight into the effectiveness of the proposed solutions.

Figures & tables

Figure 1. Overview of the proposed MM-ISTS framework.

Dataset	PhysioNet		MIMIC		Human Activity		USHCN
Metric	MSE $\times 10^{-3}$	MAE $\times 10^{-2}$	MSE $\times 10^{-2}$	MAE $\times 10^{-2}$	MSE $\times 10^{-3}$	MAE $\times 10^{-2}$	MSE $\times 10^{-1}$	MAE $\times 10^{-1}$
DLinear	41.86 $\pm$ 0.05	15.52 $\pm$ 0.03	4.90 $\pm$ 0.00	16.29 $\pm$ 0.05	4.03 $\pm$ 0.01	4.21 $\pm$ 0.01	6.21 $\pm$ 0.00	3.88 $\pm$ 0.02
TimesNet	16.48 $\pm$ 0.11	6.14 $\pm$ 0.03	5.88 $\pm$ 0.08	13.62 $\pm$ 0.07	3.12 $\pm$ 0.01	3.56 $\pm$ 0.02	5.58 $\pm$ 0.05	3.60 $\pm$ 0.04
PatchTST	12.00 $\pm$ 0.23	6.02 $\pm$ 0.14	3.78 $\pm$ 0.03	12.43 $\pm$ 0.10	4.29 $\pm$ 0.14	4.80 $\pm$ 0.09	5.75 $\pm$ 0.01	3.57 $\pm$ 0.02
Crossformer	6.66 $\pm$ 0.11	4.81 $\pm$ 0.11	2.65 $\pm$ 0.10	9.56 $\pm$ 0.29	4.29 $\pm$ 0.20	4.89 $\pm$ 0.17	5.25 $\pm$ 0.04	3.27 $\pm$ 0.09
Graph WaveNet	6.04 $\pm$ 0.28	4.41 $\pm$ 0.11	2.93 $\pm$ 0.09	10.50 $\pm$ 0.15	2.89 $\pm$ 0.03	3.40 $\pm$ 0.05	5.29 $\pm$ 0.04	3.16 $\pm$ 0.09

Table 1. Overall performance comparison on four datasets. The best results are highlighted in bold, and the second-best are underlined.

Panel of Figure 2 — Figure 2. Ablation study.

Panel of Figure 3 — Figure 3. Parameter sensitivity and efficiency analysis.

Panel of Figure 4 — Figure 4. Text source comparison, cross-attention patterns, and adaptive gating behavior.

Panel of Figure 5 — Figure 5. Effect of image construction variants and different prompt components.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	#Samp.	#Var.	Miss.%
Human Activity	5,400	12	75.0
USHCN	26,736	5	77.9
PhysioNet	12,000	41	88.4
MIMIC-III	23,457	96	96.7

Appendix

Table A1 . Statistics of the main ISTS benchmark datasets.

Group	Dataset	Vars. ( $N$ )	Max Hist. ( $L$ )	Pixels/Ch. ( $N\times L$ )
Main benchmarks	Human Activity	12	98	1,176
	USHCN	5	205	1,025
	PhysioNet	41	128	5,248
	MIMIC-III	96	280	26,880
Additional datasets with thier own texts	FNSPID	6	23	138
	ClusterTrace	11	228	2,508

Appendix

Table A2 . Constructed image sizes for each dataset.

Panel of Figure 9 — Figure A1 . Effect of different numbers of variables on PhysioNet.

Layer Position	MSE $\times 10^{-3}$	MAE $\times 10^{-2}$
Last	$2.49\pm 0.03$	$3.08\pm 0.03$
3rd-to-last	$\mathbf{2.47}\pm\mathbf{0.01}$	$\mathbf{3.06}\pm\mathbf{0.02}$
5th-to-last	$2.48\pm 0.02$	$3.06\pm 0.02$
7th-to-last	$2.49\pm 0.02$	$3.07\pm 0.03$

Appendix

Table A3 . Effect of different MLLM hidden layers.

Panel of Figure 11 — Figure A2 . PhysioNet hyperparameter sensitivity.

Panel of Figure 12 — Figure A3 . Human Activity hyperparameter sensitivity.

Figure A4 . Efficiency analysis on PhysioNet.

Panel of Figure 14 — Figure A5 . ISTS Encoding branch component ablation study.

Figure A6 . T-SNE visualization of representations from four model stages.

Irregular Time-Series Modeling

Irregular time-series modeling learns from observations that arrive at uneven times, across variables, or with missing values.

See all 103 papers

TAC-Time: Texts as Channels For Multimodal Time Series Forecasting
Sep 21, 2026Jiayi Liang, Xiaotian Gu, Xinyu Xie +2Multivariate Time Series Forecasting Explainable Time Series Forecasting
East China Normal University Shanghai, China
STaT: Resolving Shape Distortion in Non-Stationary Time Series via Tri-Modal Synergy
May 25, 2026Hui Cheng, Jinsheng Guo, Zhenhao Weng +2Non-Stationary Time Series Forecasting Multimodal Fusion
Hefei University of Technology Hefei, China
Learning Recursive Multi-Scale Representations for Irregular Multivariate Time Series Forecasting
Feb 25, 2026Boyuan Li, Zhen Liu, Yicheng Luo +1Irregular Time-Series Modeling Multivariate Time Series Forecasting
School of Computer Science and Engineering, South China University of Technology

Sep 21, 2026J/K move · Enter open · S save

TAC-Time: Texts as Channels For Multimodal Time Series Forecasting

Jiayi Liang, Xiaotian Gu, Xinyu Xie, Yuanbin Wu+1

East China Normal University Shanghai, China

Multivariate Time Series Forecasting Explainable Time Series Forecasting Multimodal Fusion

Open paper AskPDF arXiv

Most existing time series forecasting methods rely solely on numerical observations, overlooking rich contextual information from auxiliary texts. Recent multimodal approaches attempt to incorporate textual signals, but they often treat text as static features or use large language models as forecasting backbones, limiting their ability to capture temporal dynamics and increasing computational cost. To address these challenges, we propose TAC-Time, a unified framework that transforms textual information into additional temporal channels. By modeling text features jointly with numerical sequences in a shared temporal backbone, TAC-Time preserves temporal continuity and periodic structures while remaining efficient and scalable. This formulation also enables systematic interpretability analyses. We show strong cross-modal dependencies through attention and frequency-domain analyses, and identify predictive textual signals whose correlation-aware alignment yields partial forecasting improvements. Extensive experiments on real-world multimodal benchmarks demonstrate that TAC-Time outperforms prior methods.

Abstract

Irregular Time-Series Modeling

Explore similar work

TAC-Time: Texts as Channels For Multimodal Time Series Forecasting

STaT: Resolving Shape Distortion in Non-Stationary Time Series via Tri-Modal Synergy

Learning Recursive Multi-Scale Representations for Irregular Multivariate Time Series Forecasting