cs.LGOct 18, 2026

CVTA: Cross-Variable Temporal Attention for Multivariate Irregular Time Series Prediction

Authors: Ankitkumar Joshi, Milos Hauskrecht

Organizations: Department of Computer Science, University of Pittsburgh, Pittsburgh, PA 15260

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Irregularly sampled multivariate time series arise in many real-world domains such as healthcare, climate science, and large-scale monitoring systems. Characterized by asynchronous observations and heterogeneity across variables, such time series pose significant challenges for forecasting. In this work, we introduce Cross-Variable Temporal Attention (CVTA), a short-term, continuous-time forecasting framework that learns prediction functions from its compact, interpretable state summarizing recent temporal dynamics across variables. CVTA uses Variable-Conditioned Embeddings to project heterogeneous variables into variable-specific latent spaces, and applies temporal attention to capture cross-variable dependencies from this state. Furthermore, we demonstrate that CVTA can be integrated as an auxiliary module into existing state-of-the-art prediction models through a dynamic meta-decision model. Extensive experiments on multiple real-world benchmarks show that CVTA achieves strong short-term predictive accuracy, supports efficient online inference, improves existing prediction models when used as an auxiliary module, and learns attention patterns that align well with domain knowledge.

Figures & tables

 Regular Time SeriesIrregular Time SeriesEvent Time Series
MarkingContinuousContinuousDiscrete
SamplingUniformNon-UniformNon-Uniform
Table 1: Characterization of time series types by marking and sampling.
Figure 1: Overview of attention-based architectures for irregular multivariate time series prediction. Irregularly sampled multivariate observations ( top left ) can be modeled using different attention mechanisms: point-wise attention ( bottom left ) applies attention directly to individual observations; patch-based attention ( bottom right ) aggregates observations into temporal patches before applying attention; and structured message passing ( top right ) represents time points and variables as nodes in graphs or hypergraphs and learns attention via message passing.
Figure 3
Figure 3: Cross-Variable Temporal Attention (CVTA) architectures. ( Left ) CVTA model performs attention over recent observations and local dynamics using Variable-Conditioned Embeddings (VCEs), enabling robust forecasting under irregular sampling and missing data. ( Right ) CVTA integrated as an auxiliary module with a base model and dynamic meta-decision.
 USHCNPhysionetMIMIC-IIIMIMIC-IVMIMIC-HiRes
Missingness typeMCARNMARNMARNMARNMAR
# Vars.53796100393
Mean len730.072.1630.23154.34244.30
Max len730185969204299
Interarrival Variability0.8940.6091.3851.6091.015
Missingness (%)22.3683.8595.5997.8398.74
Table 2: Dataset statistics and sampling characteristics.
ModelUSHCNPhysionetMIMIC-IIIMIMIC-IVMIMIC-HiRes
MSE ×10−2\times 10^{-2}MAE ×10−2\times 10^{-2}MSE ×10−2\times 10^{-2}MAE ×10−2\times 10^{-2}MSE ×10−1\times 10^{-1}MAE ×10−1\times 10^{-1}MSE ×10−1\times 10^{-1}MAE ×10−1\times 10^{-1}MSE ×10−2\times 10^{-2}MAE ×10−2\times 10^{-2}
CRU0.030\scalebox{0.75}{\pm 0.019}0.519\scalebox{0.75}{\pm 0.166}0.807\scalebox{0.75}{\pm 0.035}5.233\scalebox{0.75}{\pm 0.242}6.913\scalebox{0.75}{\pm \text{0.040}}5.797\scalebox{0.75}{\pm \text{0.103}}4.013\scalebox{0.75}{\pm \text{0.022}}4.128\scalebox{0.75}{\pm \text{0.141}}1.236\scalebox{0.75}{\pm 0.035}6.735\scalebox{0.75}{\pm 0.139}
f-CRU0.020\scalebox{0.75}{\pm 0.007}0.455\scalebox{0.75}{\pm 0.077}1.095\scalebox{0.75}{\pm 0.069}5.295\scalebox{0.75}{\pm 0.132}7.021\scalebox{0.75}{\pm \text{0.030}}6.134\scalebox{0.75}{\pm \text{0.052}}4.142\scalebox{0.75}{\pm \text{0.021}}4.290\scalebox{0.75}{\pm \text{0.076}}1.191\scalebox{0.75}{\pm 0.043}6.535\scalebox{0.75}{\pm 0.214}
RKN- Δt\Delta_{t}0.015\scalebox{0.75}{\pm 0.016}0.367\scalebox{0.75}{\pm 0.178}0.680\scalebox{0.75}{\pm 0.042}4.854\scalebox{0.75}{\pm 0.197}7.784\scalebox{0.75}{\pm 0.086}6.102\scalebox{0.75}{\pm 0.038}5.707\scalebox{0.75}{\pm \text{0.031}}5.961\scalebox{0.75}{\pm \text{0.029}}1.292\scalebox{0.75}{\pm 0.042}6.820\scalebox{0.75}{\pm 0.081}
GRU- Δt\Delta_{t}0.035\scalebox{0.75}{\pm 0.001}0.717\scalebox{0.75}{\pm 0.018}0.449\scalebox{0.75}{\pm 0.018}4.160\scalebox{0.75}{\pm 0.154}5.731\scalebox{0.75}{\pm 0.009}4.946\scalebox{0.75}{\pm 0.021}4.359\scalebox{0.75}{\pm \text{0.005}}4.729\scalebox{0.75}{\pm \text{0.008}}1.414\scalebox{0.75}{\pm 0.051}7.226\scalebox{0.75}{\pm 0.086}
GRU-D0.015\scalebox{0.75}{\pm 0.009}0.386\scalebox{0.75}{\pm 0.110}0.672\scalebox{0.75}{\pm 0.026}5.459\scalebox{0.75}{\pm 0.094}5.906\scalebox{0.75}{\pm \text{0.028}}5.362\scalebox{0.75}{\pm \text{0.046}}4.741\scalebox{0.75}{\pm \text{0.011}}4.867\scalebox{0.75}{\pm \text{0.038}}0.984\scalebox{0.75}{\pm 0.028}5.853\scalebox{0.75}{\pm 0.102}
Table 3: Next observation prediction benchmark. We report the mean and standard deviation of the evaluation metrics, MSE and MAE (lower is better), over five independent random seeds. "OOM" refers to out-of-memory GPU error encountered during training. For better readability, we annotate the best and the second-best models.
ModelUSHCN↓\downarrow %Physionet↓\downarrow %MIMIC-III↓\downarrow %MIMIC-IV↓\downarrow %MIMIC-HiRes↓\downarrow %Avg. ↓\downarrow %
CVTA0.003 ±\pm 0.002–0.226 ±\pm 0.009–3.957 ±\pm 0.009–2.026 ±\pm 0.006–0.513 ±\pm 0.018––
w/o VCE0.013\scalebox{0.75}{\pm 0.003}333.30.494\scalebox{0.75}{\pm 0.008}118.65.236\scalebox{0.75}{\pm 0.008}32.32.932\scalebox{0.75}{\pm 0.006}44.81.016\scalebox{0.75}{\pm 0.036}98.1125.4
w/o local slope0.006\scalebox{0.75}{\pm 0.004}100.00.241\scalebox{0.75}{\pm 0.009}6.64.005\scalebox{0.75}{\pm 0.009}1.22.129\scalebox{0.75}{\pm 0.009}5.10.903\scalebox{0.75}{\pm 0.018}76.037.8
w/o key mask0.004\scalebox{0.75}{\pm 0.003}33.30.227\scalebox{0.75}{\pm 0.010}0.43.960\scalebox{0.75}{\pm 0.009}0.12.080\scalebox{0.75}{\pm 0.006}2.70.809\scalebox{0.75}{\pm 0.016}57.718.8
w/o value mask0.010\scalebox{0.75}{\pm 0.001}233.30.237\scalebox{0.75}{\pm 0.007}4.94.110\scalebox{0.75}{\pm 0.007}3.92.117\scalebox{0.75}{\pm 0.006}4.50.832\scalebox{0.75}{\pm 0.018}62.261.8
w/o elapsed time0.004\scalebox{0.75}{\pm 0.003}33.30.241\scalebox{0.75}{\pm 0.004}6.64.112\scalebox{0.75}{\pm 0.008}3.92.168\scalebox{0.75}{\pm 0.005}7.00.869\scalebox{0.75}{\pm 0.019}69.424.0
Table 4: CVTA architectural ablation study. We report MSE (mean ±\pm std) and relative MSE performance drop in percentage ( ↓\downarrow %), relative to CVTA’s. Avg. ↓\downarrow % is the average MSE degradation across all the datasets.
MethodUSHCNPhysionetMIMIC-IIIMIMIC-IVMIMIC-HiRes
 H=1H=3H=5H=7H=1H=3H=5H=7H=1H=3H=5H=7H=1H=3H=5H=7H=1H=3H=5H=7
CVTA0.0030.0100.0130.0150.2260.2450.2540.2743.9574.1554.2824.4352.0262.2592.3822.4520.5130.8320.9131.041
GRUwE (PM)0.0110.0330.1180.1840.2340.2500.2620.2794.0104.1134.2164.3222.1102.2872.3762.4610.8091.2241.3261.329
GRUwE+CVTA0.0040.0170.0240.0280.2110.2300.2390.2583.7833.9254.0394.1512.0152.1912.2852.3340.6040.9310.9590.989
Improv. ( ↑\uparrow %)+63.6+48.5+79.7+84.8+9.8+8.0+8.8+7.5+5.7+4.6+4.2+4.0+4.5+4.2+3.8+5.2+25.3+23.9+27.7+25.6
GraFITi (PM)0.0720.0810.0990.1130.2340.2430.2470.2564.0594.0834.2164.2642.1462.2772.3702.3711.4091.4451.4621.703
Table 5: Multi - step prediction performance in MSE (we report mean MSE over five random seeds) across all datasets for prediction horizons H={1,3,5,7}\mathrm{H}=\{1,3,5,7\} steps. ’–’ indicates GPU memory error encountered during training.
Figure 4: (a) Trade-off between training efficiency, predictive performance, and peak memory usage during batched training. (b) Efficiency profile of online inference across top-performing models. (c) CVTA’s skill deterioration horizon by variable ( ΔTmin⁡∗\Delta T^{*}_{\min} ). (d) Top-attended variables and average attention scores.
Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: MSE profiles for CVTA, GraFITi and HyperIMTS for representative variables on the Physionet benchmark.
VariableΔTGraFITi⋆\Delta T^{\star}_{\mathrm{GraFITi}}ΔTGRUwE⋆\Delta T^{\star}_{\mathrm{GRUwE}}ΔTHyperIMTS⋆\Delta T^{\star}_{\mathrm{HyperIMTS}}ΔTmin⁡⋆\Delta T^{\star}_{\min}
 (hours)(hours)(hours)(hours)
BUN5.255.255.755.753.753.753.753.75
Creatinine0.750.75>6.0>6.00.750.750.750.75
GCS2.252.251.251.251.251.251.251.25
HCT1.251.251.251.251.251.251.251.25
Na>6.0>6.0>6.0>6.0>6.0>6.0>6.0>6.0
Appendix
Table 6: Skill deterioration horizons of CVTA relative to GraFITi, GRUwE, and HyperIMTS models for representative variables on the Physionet benchmark. ΔTmin⁡⋆\Delta T^{\star}_{\min} denotes the earliest deterioration horizon across the three reference models and provides a conservative estimate of CVTA’s competitive prediction horizon.
Figure 6: Efficiency of online inference. Scatter plot of inference latency versus GPU memory usage. Models without compact fixed-size state (top - right) require buffering past observations, leading to higher computational costs, while models that support compact fixed-size states (bottom - left) maintain constant, low overhead.
ModelLatent Dim.# Layers# HeadsEmbed. Dim.Refs / Patches
mTAND16, 32, 64, 12811, 2, 4Time: 16, 32, 64Ref pts: 32, 64
GRU-D16, 32, 64, 1281–––
CRU16, 32, 64, 1281––Basis: 10, 20
f-CRU16, 32, 64, 1281––Basis: 10, 20
Latent ODE16, 32, 64, 128Gen/Rec: 3–Rec dim: 20, 32, 64GRU units: 20
ODE-RNN16, 32, 64, 1281–––
Appendix
Table 7: Hyperparameter values for the grid search for all the methods.
ModelUSHCNPhysionetMIMIC-III
MSE ×10−2\times 10^{-2}MAE ×10−2\times 10^{-2}MSE ×10−2\times 10^{-2}MAE ×10−2\times 10^{-2}MSE ×10−1\times 10^{-1}MAE ×10−1\times 10^{-1}
FFN( D =2, lsd =256, residual =F)0.0320.3980.3373.6544.9954.389
FFN( D =2, lsd =64, residual =T)0.0330.3590.2952.9524.7524.169
FFN( D =2, lsd =128, residual =T)0.0240.2230.2712.9114.6314.050
FFN( D =2, lsd =256, residual =T)0.0230.1830.2692.9244.6264.032
CVTA0.0030.1120.2262.5433.9573.778
Appendix
Table 8: FFN ablation using the same state summary as CVTA. We report test MSE and MAE averaged over five random seeds, using the same metric scaling as in the main benchmark table.
ModelUSHCNPhysionetMIMIC-IIIMIMIC-IVMIMIC-HiRes
 MSE ×10−2\times 10^{-2}MAE ×10−2\times 10^{-2}MSE ×10−2\times 10^{-2}MAE ×10−2\times 10^{-2}MSE ×10−1\times 10^{-1}MAE ×10−1\times 10^{-1}MSE ×10−1\times 10^{-1}MAE ×10−1\times 10^{-1}MSE ×10−2\times 10^{-2}MAE ×10−2\times 10^{-2}
CVTA0.003 ±\pm 0.0020.112 ±\pm 0.0330.226 ±\pm 0.0092.543 ±\pm 0.0243.957 ±\pm 0.0093.778 ±\pm 0.0162.026 ±\pm 0.0062.811 ±\pm 0.0160.513 ±\pm 0.0183.941 ±\pm 0.034
w/o VCE0.013\scalebox{0.75}{\pm 0.003}0.191\scalebox{0.75}{\pm 0.022}0.494\scalebox{0.75}{\pm 0.008}3.259\scalebox{0.75}{\pm 0.036}5.236\scalebox{0.75}{\pm 0.008}4.224\scalebox{0.75}{\pm 0.023}2.932\scalebox{0.75}{\pm 0.006}3.284\scalebox{0.75}{\pm 0.019}1.016\scalebox{0.75}{\pm 0.036}6.132\scalebox{0.75}{\pm 0.061}
w/o local slope0.006\scalebox{0.75}{\pm 0.004}0.124\scalebox{0.75}{\pm 0.031}0.241\scalebox{0.75}{\pm 0.009}2.626\scalebox{0.75}{\pm 0.032}4.005\scalebox{0.75}{\pm 0.009}3.829\scalebox{0.75}{\pm 0.021}2.129\scalebox{0.75}{\pm 0.009}2.886\scalebox{0.75}{\pm 0.021}0.903\scalebox{0.75}{\pm 0.018}4.606\scalebox{0.75}{\pm 0.031}
w/o key mask0.004\scalebox{0.75}{\pm 0.003}0.114\scalebox{0.75}{\pm 0.032}0.227\scalebox{0.75}{\pm 0.010}2.563\scalebox{0.75}{\pm 0.034}3.960\scalebox{0.75}{\pm 0.009}3.791\scalebox{0.75}{\pm 0.018}2.080\scalebox{0.75}{\pm 0.006}2.838\scalebox{0.75}{\pm 0.015}0.809\scalebox{0.75}{\pm 0.016}4.541\scalebox{0.75}{\pm 0.032}
w/o value mask0.010\scalebox{0.75}{\pm 0.001}0.213\scalebox{0.75}{\pm 0.031}0.237\scalebox{0.75}{\pm 0.007}2.595\scalebox{0.75}{\pm 0.021}4.110\scalebox{0.75}{\pm 0.007}3.875\scalebox{0.75}{\pm 0.023}2.117\scalebox{0.75}{\pm 0.006}2.888\scalebox{0.75}{\pm 0.014}0.832\scalebox{0.75}{\pm 0.018}4.688\scalebox{0.75}{\pm 0.030}
Appendix
Table 9: CVTA Architectural Ablation study results (MSE / MAE) across datasets.
ModelMetricUSHCNPhysionetMIMIC-IIIMIMIC-IVMIMIC-HiRes
  H=1H=3H=5H=7H=1H=3H=5H=7H=1H=3H=5H=7H=1H=3H=5H=7H=1H=3H=5H=7
CVTAMean MSE0.0030.0100.0130.0150.2260.2450.2540.2743.9574.1554.2824.4352.0262.2592.3822.4520.5130.8320.9131.041
 Std MSE0.0020.0020.0040.0070.0090.0140.0120.0210.0090.0080.0140.0130.0060.0070.0110.0230.0180.0200.0210.034
GRUwE (PM)Mean MSE0.0110.0330.1180.1840.2340.2500.2620.2794.0104.1134.2164.3222.1102.2872.3762.4610.8091.2241.3261.329
 Std MSE0.0140.0150.0160.0210.0070.0040.0080.0150.0080.0080.0130.0120.0080.0130.0110.0160.0220.0210.0320.030
GRUwE+CVTAMean MSE0.0040.0170.0240.0280.2110.2300.2390.2583.7833.9254.0394.1512.0152.1912.2852.3340.6040.9310.9590.989
Appendix
Table 10: Multi-step prediction performance in MSE and MAE across all datasets for horizons H={1,3,5,7}H=\{1,3,5,7\} . Results report mean and standard deviation over five seeds. ‘–’ indicates GPU out-of-memory error during training.
TargetTop-8 attended variables with attention scores
LactateHCO 3 (0.2208), Lactate (0.1651), RespRate (0.1095), TroponinT (0.0771), Platelets (0.0636), ALT (0.0589), PaCO 2 (0.0376), FiO 2 (0.0357)
CreatinineCreatinine (0.8387), TroponinT (0.0392), Urine (0.0317), BUN (0.0146), Cholesterol (0.0133), Bilirubin (0.0112), HCO 3 (0.0110), ALP (0.0076)
Appendix
Table 11: Top-8 attended variables for high-lactate and high-creatinine clinical scenarios.
Figure 7: Visualizing the learned UMAP projections pertaining to the variable embeddings in MIMIC-III dataset
Figure 8: Visualizing the learned UMAP projections pertaining to the variable embeddings in MIMIC-IV dataset
VariableBaseline MSE (×10−3)(\times 10^{-3})Masked Top-3 MSE (×10−3)(\times 10^{-3})Masked Bottom-3 MSE (×10−3)(\times 10^{-3})
Albumin19.6719.6774.4274.4219.9319.93
ALT2.482.483.553.552.482.48
Bilirubin7.877.8713.1113.117.877.87
BUN6.766.7612.6112.616.756.75
Creatinine2.022.029.069.062.022.02
FiO2\mathrm{FiO}_{2}24.4524.4586.9186.9124.4424.44
Appendix
Table 12: Effect of masking top- and bottom-attended variables.
More in this topic

Multivariate Time Series Forecasting

Multivariate time series forecasting predicts future values across related variables by learning their temporal and cross-variable dependencies.

See all 289 papers

Explore similar work

CardsList
Oct 14, 2026

CTIA-ODE: Continuous-Time Inter-Variable Attention for Long-Term Time Series Forecasting

Although multivariate time series forecasting has broad application prospects, effectively incorporating exogenous variables remains challenging. Prevailing discrete-time architectures often impose rigid temporal alignment across heterogeneous variables, which may limit their ability to capture continuously evolving external effects. To address this limitation, we introduce CTIA-ODE, a continuous-time forecasting framework centered on Continuous-Time Inter-Variable Attention (CTIA). Specifically, CTIA is embedded within an ordinary differential equation (ODE) solver, enabling the continuously evolving endogenous trajectory to dynamically query a static exogenous context along the integration path without requiring explicit temporal alignment. To improve the stability of the continuous latent dynamics, we further introduce a dual-stabilization strategy: architecturally, a Linear Residual Augmentation (LRA) module provides a macroscopic trend anchor, while an Outlier-Robust Hybrid Loss (ORHL) mitigates the influence of heavy-tailed anomalies during optimization. Extensive experiments across eight real-world datasets demonstrate that CTIA-ODE achieves state-of-the-art forecasting performance across diverse datasets and prediction horizons, while maintaining a favorable accuracy-efficiency trade-off with a compact memory footprint and low inference latency, particularly under long-horizon forecasting settings. Anonymous code is available at https://anonymous.4open.science/r/CTIA-ODE-15A6/.

Chengyi Pan, Huangqimei Zheng, Hao Xue

UGOD, HKUST-GZ · School of Software, Yunnan University · UGOD, The Hong Kong University of Science and Technology (Guangzhou)

Time Series ModelingExogenous Time Series ForecastingTime Series ForecastingMultivariate Time Series ForecastingLong-Term Time Series ForecastingNeural ODEs
AskPDF
Mar 6, 2026

UniMamba: A Unified Spatial-Temporal Modeling Framework with State-Space and Attention Integration

Multivariate time series forecasting is fundamental to numerous domains such as energy, finance, and environmental monitoring, where complex temporal dependencies and cross-variable interactions pose enduring challenges. Existing Transformer-based methods capture temporal correlations through attention mechanisms but suffer from quadratic computational cost, while state-space models like Mamba achieve efficient long-context modeling yet lack explicit temporal pattern recognition. Therefore we introduce UniMamba, a unified spatial-temporal forecasting framework that integrates efficient state-space dynamics with attention-based dependency learning. UniMamba employs a Mamba Variate-Channel Encoding Layer enhanced with FFT-Laplace Transform and TCN to capture global temporal dependencies, and a Spatial Temporal Attention Layer to jointly model inter-variate correlations and temporal evolution. A Feedforward Temporal Dynamics Layer further fuses continuous and discrete contexts for accurate forecasting. Comprehensive experiments on eight public benchmark datasets demonstrate that UniMamba consistently outperforms state-of-the-art forecasting models in both forecasting accuracy and computational efficiency, establishing a scalable and robust solution for long-sequence multivariate time-series prediction.

Xingsheng Chen, Xianpei Mu, Deyu Yi +6

School of Computing and Data Science, The University of Hong Kong, Hong Kong, China · School of Information Engineering, Beijing Institute of Graphic Communication, Beijing, China · Innovation Engineering College, Macau University of Science and Technology, Macau, China +1

Multivariate Time Series ForecastingLong-Term Time Series ForecastingSelective SSMsSpatiotemporal Forecasting
AskPDF
Oct 3, 2026

QiYao-I: A Manifold Based Foundation Model for Irregular Multivariate Time Series Forecasting

Irregular multivariate time series forecasting is a challenging yet important problem in real-world applications, where observations are often irregularly sampled and asynchronously recorded across variables. Existing time series foundation models are mostly built on regularly sampled sequences, making them difficult to generalize to irregular time intervals and asynchronous cross-variable dependencies. To address these challenges, we propose QiYao-I, a manifold based foundation model for irregular multivariate time series forecasting. Specifically, we introduce a novel sampling-conditioned temporal manifold attention mechanism that maps real timestamps into a learnable temporal manifold feature space and injects temporal manifold biases into attention layers, enabling the model to capture both irregular time intervals and local sampling structures. Further, we propose a dynamic variable interaction mechanism with frequency awareness. It selectively performs cross-variable message passing under asynchronous observations. Extensive experiments on real-world irregular multivariate forecasting benchmarks demonstrate that QiYao-I achieves superior performance compared with both time series foundation models and end-to-end irregular forecasting models, showing strong generalization ability in zero-shot and few-shot settings.

Linfeng Wang, Ruitong Zhang, Kai Zhao +6

East China Normal University · Huawei Technologies Ltd.

Multivariate Time Series ForecastingIrregular Time-Series ModelingTime-Series Foundation Models
AskPDF