cs.LGOct 18, 2026

CVTA: Cross-Variable Temporal Attention for Multivariate Irregular Time Series Prediction

Authors: Ankitkumar Joshi, Milos Hauskrecht

Organizations: Department of Computer Science, University of Pittsburgh, Pittsburgh, PA 15260

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Irregularly sampled multivariate time series arise in many real-world domains such as healthcare, climate science, and large-scale monitoring systems. Characterized by asynchronous observations and heterogeneity across variables, such time series pose significant challenges for forecasting. In this work, we introduce Cross-Variable Temporal Attention (CVTA), a short-term, continuous-time forecasting framework that learns prediction functions from its compact, interpretable state summarizing recent temporal dynamics across variables. CVTA uses Variable-Conditioned Embeddings to project heterogeneous variables into variable-specific latent spaces, and applies temporal attention to capture cross-variable dependencies from this state. Furthermore, we demonstrate that CVTA can be integrated as an auxiliary module into existing state-of-the-art prediction models through a dynamic meta-decision model. Extensive experiments on multiple real-world benchmarks show that CVTA achieves strong short-term predictive accuracy, supports efficient online inference, improves existing prediction models when used as an auxiliary module, and learns attention patterns that align well with domain knowledge.

Figures & tables

 Regular Time SeriesIrregular Time SeriesEvent Time Series
MarkingContinuousContinuousDiscrete
SamplingUniformNon-UniformNon-Uniform
Table 1: Characterization of time series types by marking and sampling.
Figure 1: Overview of attention-based architectures for irregular multivariate time series prediction. Irregularly sampled multivariate observations ( top left ) can be modeled using different attention mechanisms: point-wise attention ( bottom left ) applies attention directly to individual observations; patch-based attention ( bottom right ) aggregates observations into temporal patches before applying attention; and structured message passing ( top right ) represents time points and variables as nodes in graphs or hypergraphs and learns attention via message passing.
Figure 3
Figure 3: Cross-Variable Temporal Attention (CVTA) architectures. ( Left ) CVTA model performs attention over recent observations and local dynamics using Variable-Conditioned Embeddings (VCEs), enabling robust forecasting under irregular sampling and missing data. ( Right ) CVTA integrated as an auxiliary module with a base model and dynamic meta-decision.
 USHCNPhysionetMIMIC-IIIMIMIC-IVMIMIC-HiRes
Missingness typeMCARNMARNMARNMARNMAR
# Vars.53796100393
Mean len730.072.1630.23154.34244.30
Max len730185969204299
Interarrival Variability0.8940.6091.3851.6091.015
Missingness (%)22.3683.8595.5997.8398.74
Table 2: Dataset statistics and sampling characteristics.
ModelUSHCNPhysionetMIMIC-IIIMIMIC-IVMIMIC-HiRes
MSE ×10−2\times 10^{-2}MAE ×10−2\times 10^{-2}MSE ×10−2\times 10^{-2}MAE ×10−2\times 10^{-2}MSE ×10−1\times 10^{-1}MAE ×10−1\times 10^{-1}MSE ×10−1\times 10^{-1}MAE ×10−1\times 10^{-1}MSE ×10−2\times 10^{-2}MAE ×10−2\times 10^{-2}
CRU0.030\scalebox{0.75}{\pm 0.019}0.519\scalebox{0.75}{\pm 0.166}0.807\scalebox{0.75}{\pm 0.035}5.233\scalebox{0.75}{\pm 0.242}6.913\scalebox{0.75}{\pm \text{0.040}}5.797\scalebox{0.75}{\pm \text{0.103}}4.013\scalebox{0.75}{\pm \text{0.022}}4.128\scalebox{0.75}{\pm \text{0.141}}1.236\scalebox{0.75}{\pm 0.035}6.735\scalebox{0.75}{\pm 0.139}
f-CRU0.020\scalebox{0.75}{\pm 0.007}0.455\scalebox{0.75}{\pm 0.077}1.095\scalebox{0.75}{\pm 0.069}5.295\scalebox{0.75}{\pm 0.132}7.021\scalebox{0.75}{\pm \text{0.030}}6.134\scalebox{0.75}{\pm \text{0.052}}4.142\scalebox{0.75}{\pm \text{0.021}}4.290\scalebox{0.75}{\pm \text{0.076}}1.191\scalebox{0.75}{\pm 0.043}6.535\scalebox{0.75}{\pm 0.214}
RKN- Δt\Delta_{t}0.015\scalebox{0.75}{\pm 0.016}0.367\scalebox{0.75}{\pm 0.178}0.680\scalebox{0.75}{\pm 0.042}4.854\scalebox{0.75}{\pm 0.197}7.784\scalebox{0.75}{\pm 0.086}6.102\scalebox{0.75}{\pm 0.038}5.707\scalebox{0.75}{\pm \text{0.031}}5.961\scalebox{0.75}{\pm \text{0.029}}1.292\scalebox{0.75}{\pm 0.042}6.820\scalebox{0.75}{\pm 0.081}
GRU- Δt\Delta_{t}0.035\scalebox{0.75}{\pm 0.001}0.717\scalebox{0.75}{\pm 0.018}0.449\scalebox{0.75}{\pm 0.018}4.160\scalebox{0.75}{\pm 0.154}5.731\scalebox{0.75}{\pm 0.009}4.946\scalebox{0.75}{\pm 0.021}4.359\scalebox{0.75}{\pm \text{0.005}}4.729\scalebox{0.75}{\pm \text{0.008}}1.414\scalebox{0.75}{\pm 0.051}7.226\scalebox{0.75}{\pm 0.086}
GRU-D0.015\scalebox{0.75}{\pm 0.009}0.386\scalebox{0.75}{\pm 0.110}0.672\scalebox{0.75}{\pm 0.026}5.459\scalebox{0.75}{\pm 0.094}5.906\scalebox{0.75}{\pm \text{0.028}}5.362\scalebox{0.75}{\pm \text{0.046}}4.741\scalebox{0.75}{\pm \text{0.011}}4.867\scalebox{0.75}{\pm \text{0.038}}0.984\scalebox{0.75}{\pm 0.028}5.853\scalebox{0.75}{\pm 0.102}
Table 3: Next observation prediction benchmark. We report the mean and standard deviation of the evaluation metrics, MSE and MAE (lower is better), over five independent random seeds. "OOM" refers to out-of-memory GPU error encountered during training. For better readability, we annotate the best and the second-best models.
ModelUSHCN↓\downarrow %Physionet↓\downarrow %MIMIC-III↓\downarrow %MIMIC-IV↓\downarrow %MIMIC-HiRes↓\downarrow %Avg. ↓\downarrow %
CVTA0.003 ±\pm 0.002–0.226 ±\pm 0.009–3.957 ±\pm 0.009–2.026 ±\pm 0.006–0.513 ±\pm 0.018––
w/o VCE0.013\scalebox{0.75}{\pm 0.003}333.30.494\scalebox{0.75}{\pm 0.008}118.65.236\scalebox{0.75}{\pm 0.008}32.32.932\scalebox{0.75}{\pm 0.006}44.81.016\scalebox{0.75}{\pm 0.036}98.1125.4
w/o local slope0.006\scalebox{0.75}{\pm 0.004}100.00.241\scalebox{0.75}{\pm 0.009}6.64.005\scalebox{0.75}{\pm 0.009}1.22.129\scalebox{0.75}{\pm 0.009}5.10.903\scalebox{0.75}{\pm 0.018}76.037.8
w/o key mask0.004\scalebox{0.75}{\pm 0.003}33.30.227\scalebox{0.75}{\pm 0.010}0.43.960\scalebox{0.75}{\pm 0.009}0.12.080\scalebox{0.75}{\pm 0.006}2.70.809\scalebox{0.75}{\pm 0.016}57.718.8
w/o value mask0.010\scalebox{0.75}{\pm 0.001}233.30.237\scalebox{0.75}{\pm 0.007}4.94.110\scalebox{0.75}{\pm 0.007}3.92.117\scalebox{0.75}{\pm 0.006}4.50.832\scalebox{0.75}{\pm 0.018}62.261.8
w/o elapsed time0.004\scalebox{0.75}{\pm 0.003}33.30.241\scalebox{0.75}{\pm 0.004}6.64.112\scalebox{0.75}{\pm 0.008}3.92.168\scalebox{0.75}{\pm 0.005}7.00.869\scalebox{0.75}{\pm 0.019}69.424.0
Table 4: CVTA architectural ablation study. We report MSE (mean ±\pm std) and relative MSE performance drop in percentage ( ↓\downarrow %), relative to CVTA’s. Avg. ↓\downarrow % is the average MSE degradation across all the datasets.
MethodUSHCNPhysionetMIMIC-IIIMIMIC-IVMIMIC-HiRes
 H=1H=3H=5H=7H=1H=3H=5H=7H=1H=3H=5H=7H=1H=3H=5H=7H=1H=3H=5H=7
CVTA0.0030.0100.0130.0150.2260.2450.2540.2743.9574.1554.2824.4352.0262.2592.3822.4520.5130.8320.9131.041
GRUwE (PM)0.0110.0330.1180.1840.2340.2500.2620.2794.0104.1134.2164.3222.1102.2872.3762.4610.8091.2241.3261.329
GRUwE+CVTA0.0040.0170.0240.0280.2110.2300.2390.2583.7833.9254.0394.1512.0152.1912.2852.3340.6040.9310.9590.989
Improv. ( ↑\uparrow %)+63.6+48.5+79.7+84.8+9.8+8.0+8.8+7.5+5.7+4.6+4.2+4.0+4.5+4.2+3.8+5.2+25.3+23.9+27.7+25.6
GraFITi (PM)0.0720.0810.0990.1130.2340.2430.2470.2564.0594.0834.2164.2642.1462.2772.3702.3711.4091.4451.4621.703
Table 5: Multi - step prediction performance in MSE (we report mean MSE over five random seeds) across all datasets for prediction horizons H={1,3,5,7}\mathrm{H}=\{1,3,5,7\} steps. ’–’ indicates GPU memory error encountered during training.
Figure 4: (a) Trade-off between training efficiency, predictive performance, and peak memory usage during batched training. (b) Efficiency profile of online inference across top-performing models. (c) CVTA’s skill deterioration horizon by variable ( ΔTmin⁡∗\Delta T^{*}_{\min} ). (d) Top-attended variables and average attention scores.
Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: MSE profiles for CVTA, GraFITi and HyperIMTS for representative variables on the Physionet benchmark.
VariableΔTGraFITi⋆\Delta T^{\star}_{\mathrm{GraFITi}}ΔTGRUwE⋆\Delta T^{\star}_{\mathrm{GRUwE}}ΔTHyperIMTS⋆\Delta T^{\star}_{\mathrm{HyperIMTS}}ΔTmin⁡⋆\Delta T^{\star}_{\min}
 (hours)(hours)(hours)(hours)
BUN5.255.255.755.753.753.753.753.75
Creatinine0.750.75>6.0>6.00.750.750.750.75
GCS2.252.251.251.251.251.251.251.25
HCT1.251.251.251.251.251.251.251.25
Na>6.0>6.0>6.0>6.0>6.0>6.0>6.0>6.0
Appendix
Table 6: Skill deterioration horizons of CVTA relative to GraFITi, GRUwE, and HyperIMTS models for representative variables on the Physionet benchmark. ΔTmin⁡⋆\Delta T^{\star}_{\min} denotes the earliest deterioration horizon across the three reference models and provides a conservative estimate of CVTA’s competitive prediction horizon.
Figure 6: Efficiency of online inference. Scatter plot of inference latency versus GPU memory usage. Models without compact fixed-size state (top - right) require buffering past observations, leading to higher computational costs, while models that support compact fixed-size states (bottom - left) maintain constant, low overhead.
ModelLatent Dim.# Layers# HeadsEmbed. Dim.Refs / Patches
mTAND16, 32, 64, 12811, 2, 4Time: 16, 32, 64Ref pts: 32, 64
GRU-D16, 32, 64, 1281–––
CRU16, 32, 64, 1281––Basis: 10, 20
f-CRU16, 32, 64, 1281––Basis: 10, 20
Latent ODE16, 32, 64, 128Gen/Rec: 3–Rec dim: 20, 32, 64GRU units: 20
ODE-RNN16, 32, 64, 1281–––
Appendix
Table 7: Hyperparameter values for the grid search for all the methods.
ModelUSHCNPhysionetMIMIC-III
MSE ×10−2\times 10^{-2}MAE ×10−2\times 10^{-2}MSE ×10−2\times 10^{-2}MAE ×10−2\times 10^{-2}MSE ×10−1\times 10^{-1}MAE ×10−1\times 10^{-1}
FFN( D =2, lsd =256, residual =F)0.0320.3980.3373.6544.9954.389
FFN( D =2, lsd =64, residual =T)0.0330.3590.2952.9524.7524.169
FFN( D =2, lsd =128, residual =T)0.0240.2230.2712.9114.6314.050
FFN( D =2, lsd =256, residual =T)0.0230.1830.2692.9244.6264.032
CVTA0.0030.1120.2262.5433.9573.778
Appendix
Table 8: FFN ablation using the same state summary as CVTA. We report test MSE and MAE averaged over five random seeds, using the same metric scaling as in the main benchmark table.
ModelUSHCNPhysionetMIMIC-IIIMIMIC-IVMIMIC-HiRes
 MSE ×10−2\times 10^{-2}MAE ×10−2\times 10^{-2}MSE ×10−2\times 10^{-2}MAE ×10−2\times 10^{-2}MSE ×10−1\times 10^{-1}MAE ×10−1\times 10^{-1}MSE ×10−1\times 10^{-1}MAE ×10−1\times 10^{-1}MSE ×10−2\times 10^{-2}MAE ×10−2\times 10^{-2}
CVTA0.003 ±\pm 0.0020.112 ±\pm 0.0330.226 ±\pm 0.0092.543 ±\pm 0.0243.957 ±\pm 0.0093.778 ±\pm 0.0162.026 ±\pm 0.0062.811 ±\pm 0.0160.513 ±\pm 0.0183.941 ±\pm 0.034
w/o VCE0.013\scalebox{0.75}{\pm 0.003}0.191\scalebox{0.75}{\pm 0.022}0.494\scalebox{0.75}{\pm 0.008}3.259\scalebox{0.75}{\pm 0.036}5.236\scalebox{0.75}{\pm 0.008}4.224\scalebox{0.75}{\pm 0.023}2.932\scalebox{0.75}{\pm 0.006}3.284\scalebox{0.75}{\pm 0.019}1.016\scalebox{0.75}{\pm 0.036}6.132\scalebox{0.75}{\pm 0.061}
w/o local slope0.006\scalebox{0.75}{\pm 0.004}0.124\scalebox{0.75}{\pm 0.031}0.241\scalebox{0.75}{\pm 0.009}2.626\scalebox{0.75}{\pm 0.032}4.005\scalebox{0.75}{\pm 0.009}3.829\scalebox{0.75}{\pm 0.021}2.129\scalebox{0.75}{\pm 0.009}2.886\scalebox{0.75}{\pm 0.021}0.903\scalebox{0.75}{\pm 0.018}4.606\scalebox{0.75}{\pm 0.031}
w/o key mask0.004\scalebox{0.75}{\pm 0.003}0.114\scalebox{0.75}{\pm 0.032}0.227\scalebox{0.75}{\pm 0.010}2.563\scalebox{0.75}{\pm 0.034}3.960\scalebox{0.75}{\pm 0.009}3.791\scalebox{0.75}{\pm 0.018}2.080\scalebox{0.75}{\pm 0.006}2.838\scalebox{0.75}{\pm 0.015}0.809\scalebox{0.75}{\pm 0.016}4.541\scalebox{0.75}{\pm 0.032}
w/o value mask0.010\scalebox{0.75}{\pm 0.001}0.213\scalebox{0.75}{\pm 0.031}0.237\scalebox{0.75}{\pm 0.007}2.595\scalebox{0.75}{\pm 0.021}4.110\scalebox{0.75}{\pm 0.007}3.875\scalebox{0.75}{\pm 0.023}2.117\scalebox{0.75}{\pm 0.006}2.888\scalebox{0.75}{\pm 0.014}0.832\scalebox{0.75}{\pm 0.018}4.688\scalebox{0.75}{\pm 0.030}
Appendix
Table 9: CVTA Architectural Ablation study results (MSE / MAE) across datasets.
ModelMetricUSHCNPhysionetMIMIC-IIIMIMIC-IVMIMIC-HiRes
  H=1H=3H=5H=7H=1H=3H=5H=7H=1H=3H=5H=7H=1H=3H=5H=7H=1H=3H=5H=7
CVTAMean MSE0.0030.0100.0130.0150.2260.2450.2540.2743.9574.1554.2824.4352.0262.2592.3822.4520.5130.8320.9131.041
 Std MSE0.0020.0020.0040.0070.0090.0140.0120.0210.0090.0080.0140.0130.0060.0070.0110.0230.0180.0200.0210.034
GRUwE (PM)Mean MSE0.0110.0330.1180.1840.2340.2500.2620.2794.0104.1134.2164.3222.1102.2872.3762.4610.8091.2241.3261.329
 Std MSE0.0140.0150.0160.0210.0070.0040.0080.0150.0080.0080.0130.0120.0080.0130.0110.0160.0220.0210.0320.030
GRUwE+CVTAMean MSE0.0040.0170.0240.0280.2110.2300.2390.2583.7833.9254.0394.1512.0152.1912.2852.3340.6040.9310.9590.989
Appendix
Table 10: Multi-step prediction performance in MSE and MAE across all datasets for horizons H={1,3,5,7}H=\{1,3,5,7\} . Results report mean and standard deviation over five seeds. ‘–’ indicates GPU out-of-memory error during training.
TargetTop-8 attended variables with attention scores
LactateHCO 3 (0.2208), Lactate (0.1651), RespRate (0.1095), TroponinT (0.0771), Platelets (0.0636), ALT (0.0589), PaCO 2 (0.0376), FiO 2 (0.0357)
CreatinineCreatinine (0.8387), TroponinT (0.0392), Urine (0.0317), BUN (0.0146), Cholesterol (0.0133), Bilirubin (0.0112), HCO 3 (0.0110), ALP (0.0076)
Appendix
Table 11: Top-8 attended variables for high-lactate and high-creatinine clinical scenarios.
Figure 7: Visualizing the learned UMAP projections pertaining to the variable embeddings in MIMIC-III dataset
Figure 8: Visualizing the learned UMAP projections pertaining to the variable embeddings in MIMIC-IV dataset
VariableBaseline MSE (×10−3)(\times 10^{-3})Masked Top-3 MSE (×10−3)(\times 10^{-3})Masked Bottom-3 MSE (×10−3)(\times 10^{-3})
Albumin19.6719.6774.4274.4219.9319.93
ALT2.482.483.553.552.482.48
Bilirubin7.877.8713.1113.117.877.87
BUN6.766.7612.6112.616.756.75
Creatinine2.022.029.069.062.022.02
FiO2\mathrm{FiO}_{2}24.4524.4586.9186.9124.4424.44
Appendix
Table 12: Effect of masking top- and bottom-attended variables.
More in this topic

Multivariate Time Series Forecasting

Multivariate time series forecasting predicts future values across related variables by learning their temporal and cross-variable dependencies.

See all 289 papers

Explore similar work

CardsList

  1. CTIA-ODE: Continuous-Time Inter-Variable Attention for Long-Term Time Series Forecasting

    Oct 14, 2026Chengyi Pan, Huangqimei Zheng, Hao XueTime Series ModelingExogenous Time Series Forecasting

    UGOD, HKUST-GZ · School of Software, Yunnan University · UGOD, The Hong Kong University of Science and Technology (Guangzhou)

  2. UniMamba: A Unified Spatial-Temporal Modeling Framework with State-Space and Attention Integration

    Mar 6, 2026Xingsheng Chen, Xianpei Mu, Deyu Yi +6Multivariate Time Series ForecastingLong-Term Time Series Forecasting

    School of Computing and Data Science, The University of Hong Kong, Hong Kong, China · School of Information Engineering, Beijing Institute of Graphic Communication, Beijing, China · Innovation Engineering College, Macau University of Science and Technology, Macau, China +1

  3. QiYao-I: A Manifold Based Foundation Model for Irregular Multivariate Time Series Forecasting

    Oct 3, 2026Linfeng Wang, Ruitong Zhang, Kai Zhao +6Multivariate Time Series ForecastingIrregular Time-Series Modeling

    East China Normal University · Huawei Technologies Ltd.