Organizations: East China Normal University, Shanghai, China · Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences, Hong Kong, China · Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China
Irregularly sampled time series (ISTS) are widespread in real-world scenarios, exhibiting asynchronous observations on uneven time intervals across diverse variables. Existing ISTS forecasting methods often solely utilize historical observations to predict future ones while falling short in learning contextual semantics and fine-grained temporal patterns. To address these problems, we propose MM-ISTS, a multimodal ISTS forecasting framework augmented by vision-text large language models, which bridges temporal, visual, and textual modalities. MM-ISTS encompasses a two-stage encoding mechanism. In particular, a Cross-Modal Vision-Text Encoding module is proposed to automatically generate informative visual images and textual data, enabling the capture of intricate temporal patterns and comprehensive contextual understanding, in collaboration with multimodal LLMs (MLLMs). In parallel, ISTS encoding extracts complementary yet enriched temporal features from historical ISTS observations, including multi-view embedding fusion and a Temporal-Variable Encoder. Further, we propose an Adaptive Query-Based Feature Extractor to compress MLLM token embeddings while preserving useful knowledge, which in turn reduces computational costs. In addition, a Multimodal Alignment module with Modality-Aware Gating is designed to alleviate the modality gaps. Extensive experiments on real data offer insight into the effectiveness of the proposed solutions.
Figures & tables
Figure 1. Overview of the proposed MM-ISTS framework.
Dataset
PhysioNet
MIMIC
Human Activity
USHCN
Metric
MSE ×10−3
MAE ×10−2
MSE ×10−2
MAE ×10−2
MSE ×10−3
MAE ×10−2
MSE ×10−1
MAE ×10−1
DLinear
41.86 ± 0.05
15.52 ± 0.03
4.90 ± 0.00
16.29 ± 0.05
4.03 ± 0.01
4.21 ± 0.01
6.21 ± 0.00
3.88 ± 0.02
TimesNet
16.48 ± 0.11
6.14 ± 0.03
5.88 ± 0.08
13.62 ± 0.07
3.12 ± 0.01
3.56 ± 0.02
5.58 ± 0.05
3.60 ± 0.04
PatchTST
12.00 ± 0.23
6.02 ± 0.14
3.78 ± 0.03
12.43 ± 0.10
4.29 ± 0.14
4.80 ± 0.09
5.75 ± 0.01
3.57 ± 0.02
Crossformer
6.66 ± 0.11
4.81 ± 0.11
2.65 ± 0.10
9.56 ± 0.29
4.29 ± 0.20
4.89 ± 0.17
5.25 ± 0.04
3.27 ± 0.09
Graph WaveNet
6.04 ± 0.28
4.41 ± 0.11
2.93 ± 0.09
10.50 ± 0.15
2.89 ± 0.03
3.40 ± 0.05
5.29 ± 0.04
3.16 ± 0.09
Table 1. Overall performance comparison on four datasets. The best results are highlighted in bold, and the second-best are underlined.
Figure 2. Ablation study.
Figure 3. Parameter sensitivity and efficiency analysis.
Figure 4. Text source comparison, cross-attention patterns, and adaptive gating behavior.
Figure 5. Effect of image construction variants and different prompt components.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
#Samp.
#Var.
Miss.%
Human Activity
5,400
12
75.0
USHCN
26,736
5
77.9
PhysioNet
12,000
41
88.4
MIMIC-III
23,457
96
96.7
Appendix
Table A1 . Statistics of the main ISTS benchmark datasets.
Group
Dataset
Vars. ( N )
Max Hist. ( L )
Pixels/Ch. ( N×L )
Main benchmarks
Human Activity
12
98
1,176
USHCN
5
205
1,025
PhysioNet
41
128
5,248
MIMIC-III
96
280
26,880
Additional datasets with thier own texts
FNSPID
6
23
138
ClusterTrace
11
228
2,508
Appendix
Table A2 . Constructed image sizes for each dataset.
Figure A1 . Effect of different numbers of variables on PhysioNet.
Layer Position
MSE ×10−3
MAE ×10−2
Last
2.49±0.03
3.08±0.03
3rd-to-last
2.47±0.01
3.06±0.02
5th-to-last
2.48±0.02
3.06±0.02
7th-to-last
2.49±0.02
3.07±0.03
Appendix
Table A3 . Effect of different MLLM hidden layers.
Figure A2 . PhysioNet hyperparameter sensitivity.
Figure A3 . Human Activity hyperparameter sensitivity.
Most existing time series forecasting methods rely solely on numerical observations, overlooking rich contextual information from auxiliary texts. Recent multimodal approaches attempt to incorporate textual signals, but they often treat text as static features or use large language models as forecasting backbones, limiting their ability to capture temporal dynamics and increasing computational cost. To address these challenges, we propose TAC-Time, a unified framework that transforms textual information into additional temporal channels. By modeling text features jointly with numerical sequences in a shared temporal backbone, TAC-Time preserves temporal continuity and periodic structures while remaining efficient and scalable. This formulation also enables systematic interpretability analyses. We show strong cross-modal dependencies through attention and frequency-domain analyses, and identify predictive textual signals whose correlation-aware alignment yields partial forecasting improvements. Extensive experiments on real-world multimodal benchmarks demonstrate that TAC-Time outperforms prior methods.
Recent research in time series forecasting frequently investigates the integration of textual and visual modalities with numerical models to better navigate non-stationary environments. Despite delivering solid numerical results, existing multi-modal approaches usually encounter a dilemma: prioritizing the minimization of average errors can result in excessively smooth forecasts that overlook essential fluctuations. To resolve this limitation, we introduce STaT, an innovative multimodal architecture for Symbolic-Temporal-Textual Alignment, which seamlessly unites three synergistic modalities. Specifically, the symbolic modality converts continuous time series into discrete tokens, facilitating the accurate identification of structural patterns and turning points; the temporal modality extracts inherent sequential dependencies; and the textual modality leverages domain semantics to steer the macroscopic forecasting trends. Comprehensive evaluations on eight real-world benchmarks indicate that STaT delivers exceptional performance, enhancing conventional magnitude indicators by up to 8.9% while simultaneously decreasing shape distortion by up to 8.5%.
Irregular Multivariate Time Series (IMTS) are characterized by uneven intervals between consecutive timestamps, which carry sampling pattern information valuable and informative for learning temporal and variable dependencies. In addition, IMTS often exhibit diverse dependencies across multiple time scales. However, many existing multi-scale IMTS methods use resampling to obtain the coarse series, which can alter the original timestamps and disrupt the sampling pattern information. To address the challenge, we propose ReIMTS, a Recursive multi-scale modeling approach for Irregular Multivariate Time Series forecasting. Instead of resampling, ReIMTS keeps timestamps unchanged and recursively splits each sample into subsamples with progressively shorter time periods. Based on the original sampling timestamps in these long-to-short subsamples, an irregularity-aware representation fusion mechanism is proposed to capture global-to-local dependencies for accurate forecasting. Extensive experiments demonstrate an average performance improvement of 27.1% in the forecasting task across different models and real-world datasets. Our code is available at https://github.com/Ladbaby/PyOmniTS.
Boyuan Li, Zhen Liu, Yicheng Luo +1
School of Computer Science and Engineering, South China University of Technology