Which Model, For Which Variable? Per-Variable Forecast Combination in Sparse ICU Time Series
Organizations: Department of Computational Linguistics, Heidelberg University, Germany · HIDSS4Health — Helmholtz Information & Data Science School for Health, Karlsruhe/Heidelberg, Germany · Department of Computational Linguistics & Interdisciplinary Center for Scientific Computing (IWR), Heidelberg University, Germany
Abstract
Clinical forecasting papers typically report a single best architecture. We show that on multivariate ICU physiology this question conceals variable-specific differences: no architecture is best on every variable, and which model wins changes from variable to variable. Across five ICU databases (MIMIC-III, MIMIC-IV, eICU, HiRID, and UMM (University Medicine Mannheim), a German single-center ICU) we forecast 94 channels over a 24-hour horizon with a pool of 14 models, and combine them with weights fitted per variable. On test, the combination beats the best single model on all five databases by 6.1-11.0% mean squared error (mean 8.7%). The gain is concentrated where clinical data is thinnest: the sparsest third of channels improves by 11.9% and laboratory values by 7.5%, against 3.1% for the densest third. Rank-based harmonic weighting of the best five models per variable, with no fitted weights at all, recovers about four-fifths of the gain. Extending the pool afterwards with a per-channel gradient-boosted tree, ARIMA and ETS makes the tree the validation-selected best single model on all five databases; per-variable combination still improves on it by 2.9-5.9% (significant on four of five), while equal weighting of the same members does not. As on univariate benchmarks, learned combination beats simple averaging - but only once the weights are constrained. We also report which model to prefer when, and show that the ordering changes with database size.
Figures & tables
| stays | patients | windows | density | charted | |
|---|---|---|---|---|---|
| MIMIC-III | 28 514 | 22 338 | 431 111 | 11.0% | 85 |
| MIMIC-IV | 46 220 | 35 706 | 693 548 | 11.4% | 89 |
| eICU | 77 115 | 63 859 | 1 065 028 | 8.8% | 79 |
| HiRID | 8 621 | 8 621 | 136 683 | 13.3% | 66 |
| UMM | 4 490 | 4 228 | 97 061 | 18.6% | 80 |
| best | select | harm-5 | ens. | |
|---|---|---|---|---|
| UMM | 0.6036 | 0.5740 | 0.5505 | 0.5375 |
| — | ||||
| HiRID | 0.4865 | 0.4509 | 0.4450 | 0.4378 |
| — | ||||
| MIMIC-III | 0.4638 | 0.4460 | 0.4244 | 0.4184 |
| — |
| stacker | harm-15 | equal | |
|---|---|---|---|
| UMM | |||
| HiRID | |||
| MIMIC-III | |||
| MIMIC-IV | |||
| eICU |
| family | UMM | HiRID | M-III | M-IV | eICU |
|---|---|---|---|---|---|
| 2.9k | 5.5k | 18k | 30k | 50k | |
| STraTS | 12.2 | 24.4 | 30.0 | 40.0 | 35.5 |
| DenseAR | 22.8 | 21.1 | 34.2 | 20.6 | 35.8 |
| Informer | 21.2 | 9.3 | 7.1 | 7.5 | 2.9 |
| GRU | 6.2 | 11.1 | 2.6 | 14.1 | 3.9 |
| Dense DMS | 9.1 | 8.7 | 7.9 | 3.0 | 7.3 |
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
| train | val | test | obs./ | ||||
|---|---|---|---|---|---|---|---|
| stays | windows | stays | windows | stays | windows | window | |
| MIMIC-III | 18 242 | 274 195 | 4 568 | 69 351 | 5 704 | 87 565 | 249 |
| MIMIC-IV | 29 604 | 444 870 | 7 382 | 111 567 | 9 234 | 137 111 | 257 |
| eICU | 49 533 | 685 134 | 12 197 | 168 643 | 15 385 | 211 251 | 200 |
| HiRID | 5 543 | 88 407 | 1 398 | 22 143 | 1 680 | 26 133 | 304 |
| UMM | 2 889 | 62 466 | 710 | 15 054 | 891 | 19 541 | 425 |
| channel | UMM | HiRID | M-III | M-IV | eICU | channel | UMM | HiRID | M-III | M-IV | eICU |
|---|---|---|---|---|---|---|---|---|---|---|---|
| hr | 99.9 | 100.0 | 99.4 | 99.9 | 98.7 | peep | 74.2 | 66.3 | 44.7 | 39.8 | 31.3 |
| spo2 | 99.8 | 100.0 | 99.3 | 99.9 | 97.7 | alp | 82.6 | 36.9 | 36.8 | 46.2 | 45.1 |
| glucose | 99.7 | 99.3 | 99.0 | 98.6 | 96.5 | tidal_volume | 70.9 | 66.1 | 40.7 | 40.1 | 26.2 |
| potassium | 99.8 | 98.5 | 99.1 | 98.6 | 95.3 | albumin | 83.0 | 14.8 | 32.6 | 31.7 | 50.7 |
| sodium | 99.8 | 98.4 | 99.0 | 98.6 | 94.9 | neutrophils_pct | 38.1 | 59.1 | 27.0 | 25.1 | 54.1 |
| hemoglobin | 99.8 | 97.5 | 98.6 | 98.4 | 93.4 | pt | — | — | 78.7 | 81.6 | 34.5 |
| family | UMM | HiRID | MIMIC-III | MIMIC-IV | eICU |
|---|---|---|---|---|---|
| Autoformer | ms2 | hp_d512_L2 | hp_d128_L2 | hp_d512_L1 | hp_d512_L3 |
| DLinear | ms | ms | hp_k21_individual | ms2 | ms2 |
| Dense DMS | hp_d256_L2 | ms | ms | hp_d256_L1 | ms |
| DenseAR | default | default | default | pteacherctrl | default |
| GRU | gru | gru | ms | ms | ms |
| Informer | hp_d256_L2 | hp_d512_L3 | hp_d256_L2 | hp_d128_L1 | hp_d256_L1 |
| member | run | layers | dropout | lr | seed | ckpts | |
|---|---|---|---|---|---|---|---|
| UMM | |||||||
| Autoformer | ms2_s456 | 128 | 1 | 0.1 | 0.0003 | 456 | 17 |
| DLinear | ms_s123 | 128 | 2 | 0.2 | 0.0003 | 123 | 16 |
| Dense DMS | hp_d256_L2_s123 | 256 | 2 | 0.1 | 0.0003 | 123 | 17 |
| DenseAR | hpsf_6a82f5b0 | 512 | 1/1 | 0.0 | 0.0001 | 42 | 290 |
| GRU | hps_b7eec8ac | 512 | 1 | 0.1 | 0.0003 | 42 | 45 |
| UMM | HiRID | MIMIC-III | MIMIC-IV | eICU | mean | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| combiner | dev | test | dev | test | dev | test | dev | test | dev | test | dev | test |
| greedy | -8.85 | -10.69 | -3.21 | -12.40 | -5.39 | -9.87 | -5.57 | -6.57 | -4.89 | -6.34 | -5.58 | -9.17 |
| simplex | -10.04 | -10.69 | -6.80 | -12.42 | -5.33 | -9.83 | -5.49 | -6.54 | -4.96 | -6.30 | -6.52 | -9.16 |
| shrunk100 | -10.79 | -10.94 | -7.97 | -10.29 | -5.47 | -9.83 | -5.52 | -6.54 | -4.97 | -6.10 | -6.94 | -8.74 |
| ens. | -10.67 | -10.95 | -8.00 | -10.01 | -5.61 | -9.79 | -5.57 | -6.51 | -5.00 | -6.09 | -6.97 | -8.67 |
| harm-5 | -8.02 | -8.79 | -4.01 | -8.52 | -5.12 | -8.48 | -4.70 | -4.97 | -4.12 | -5.01 | -5.19 | -7.15 |
| combiner | UMM | HiRID | MIMIC-III | MIMIC-IV | eICU |
|---|---|---|---|---|---|
| ens. | -10.9 [-13.3, -8.2] | -10.0 [-15.5, -6.9] | -9.8 [-11.5, -7.9] | -6.5 [-8.6, -4.9] | -6.1 [-6.7, -5.2] |
| simplex | -10.7 [-13.1, -8.0] | -12.4 [-16.7, -8.9] | -9.8 [-11.8, -7.6] | -6.5 [-8.6, -4.9] | -6.3 [-6.9, -5.3] |
| greedy | -10.7 [-13.1, -7.9] | -12.4 [-16.7, -8.8] | -9.9 [-11.8, -7.6] | -6.6 [-8.6, -5.0] | -6.3 [-6.9, -5.3] |
| harm-5 | -8.8 [-11.2, -6.0] | -8.5 [-14.2, -5.3] | -8.5 [-10.1, -6.8] | -5.0 [-6.3, -3.8] | -5.0 [-5.8, -3.8] |
| harm-3 | -8.1 [-10.4, -5.4] | -8.1 [-13.7, -4.8] | -7.6 [-9.2, -6.0] | -4.1 [-5.2, -3.2] | -4.5 [-5.2, -3.2] |
| select | -4.9 [-9.3, -0.6] ∗ | -7.3 [-12.3, -3.3] | -3.8 [-5.5, -2.2] | -1.5 [-2.3, -0.7] ∗ | -1.5 [-2.5, -0.0] ∗ |
| family | UMM | HiRID | MIMIC-III | MIMIC-IV | eICU | mean |
|---|---|---|---|---|---|---|
| STraTS | 12.2 | 24.4 | 30.0 | 40.0 | 35.5 | 28.4 |
| DenseAR | 22.8 | 21.1 | 34.2 | 20.6 | 35.8 | 26.9 |
| Informer | 21.2 | 9.3 | 7.1 | 7.5 | 2.9 | 9.6 |
| GRU | 6.2 | 11.1 | 2.6 | 14.1 | 3.9 | 7.6 |
| Dense DMS | 9.1 | 8.7 | 7.9 | 3.0 | 7.3 | 7.2 |
| DLinear | 13.9 | 16.5 | 1.7 | 1.1 | 0.8 | 6.8 |
| DenseAR | Informer | DenseDMS | GRU | LSTM | STraTS | Mamba | DLinear | PatchTST | Autoformer | Moirai | TiRex-2 | LOCF | null | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| DenseAR | 1.00 | 0.95 | 0.94 | 0.95 | 0.94 | 0.92 | 0.94 | 0.90 | 0.90 | 0.86 | 0.83 | 0.84 | 0.79 | 0.67 |
| Informer | 0.95 | 1.00 | 0.94 | 0.95 | 0.94 | 0.92 | 0.94 | 0.90 | 0.90 | 0.87 | 0.83 | 0.84 | 0.79 | 0.66 |
| DenseDMS | 0.94 | 0.94 | 1.00 | 0.94 | 0.94 | 0.91 | 0.94 | 0.89 | 0.89 | 0.86 | 0.82 | 0.84 | 0.78 | 0.67 |
| GRU | 0.95 | 0.95 | 0.94 | 1.00 | 0.95 | 0.92 | 0.95 | 0.90 | 0.90 | 0.86 | 0.83 | 0.84 | 0.80 | 0.67 |
| LSTM | 0.94 | 0.94 | 0.94 | 0.95 | 1.00 | 0.92 | 0.94 | 0.89 | 0.89 | 0.85 | 0.82 | 0.83 | 0.78 | 0.68 |
| STraTS | 0.92 | 0.92 | 0.91 | 0.92 | 0.92 | 1.00 | 0.92 | 0.87 | 0.87 | 0.85 | 0.81 | 0.82 | 0.77 | 0.64 |
| family | all_charted | type:lab | type:vital | type:medication | sofa_relevant | freq:low | freq:high |
|---|---|---|---|---|---|---|---|
| ENSEMBLE | 0.530 | 0.469 | 0.444 | 0.887 | 0.844 | 0.831 | 0.407 |
| greedy | 0.532 | 0.469 | 0.444 | 0.901 | 0.856 | 0.838 | 0.407 |
| harm-5 | 0.543 | 0.484 | 0.457 | 0.897 | 0.856 | 0.851 | 0.412 |
| mean | 0.549 | 0.498 | 0.455 | 0.882 | 0.870 | 0.868 | 0.414 |
| select | 0.567 | 0.505 | 0.475 | 0.945 | 0.883 | 0.898 | 0.420 |
| Informer | 0.577 | 0.501 | 0.471 | 1.021 | 0.898 | 0.923 | 0.425 |
| family | all_charted | type:lab | type:vital | type:medication | sofa_relevant | freq:low | freq:high |
|---|---|---|---|---|---|---|---|
| ENSEMBLE | 0.429 | 0.346 | 0.393 | 0.865 | 0.545 | 0.514 | 0.425 |
| greedy | 0.430 | 0.347 | 0.393 | 0.866 | 0.546 | 0.515 | 0.425 |
| harm-5 | 0.436 | 0.355 | 0.397 | 0.871 | 0.550 | 0.526 | 0.428 |
| select | 0.455 | 0.373 | 0.407 | 0.916 | 0.562 | 0.553 | 0.441 |
| mean | 0.457 | 0.366 | 0.422 | 0.937 | 0.595 | 0.570 | 0.437 |
| DenseAR | 0.487 | 0.395 | 0.417 | 1.050 | 0.657 | 0.637 | 0.445 |
| family | all_charted | type:lab | type:vital | type:medication | sofa_relevant | freq:low | freq:high |
|---|---|---|---|---|---|---|---|
| greedy | 0.403 | 0.394 | 0.451 | 0.391 | 0.352 | 0.459 | 0.449 |
| ENSEMBLE | 0.404 | 0.394 | 0.452 | 0.390 | 0.352 | 0.459 | 0.449 |
| harm-5 | 0.410 | 0.400 | 0.453 | 0.402 | 0.363 | 0.469 | 0.452 |
| mean | 0.424 | 0.421 | 0.468 | 0.397 | 0.365 | 0.492 | 0.463 |
| select | 0.429 | 0.423 | 0.459 | 0.433 | 0.385 | 0.504 | 0.458 |
| Informer | 0.444 | 0.433 | 0.466 | 0.474 | 0.406 | 0.534 | 0.465 |
| family | all_charted | type:lab | type:vital | type:medication | sofa_relevant | freq:low | freq:high |
|---|---|---|---|---|---|---|---|
| greedy | 0.414 | 0.389 | 0.507 | 0.397 | 0.292 | 0.505 | 0.429 |
| ENSEMBLE | 0.414 | 0.389 | 0.507 | 0.396 | 0.292 | 0.505 | 0.429 |
| harm-5 | 0.421 | 0.399 | 0.511 | 0.400 | 0.292 | 0.521 | 0.430 |
| select | 0.437 | 0.415 | 0.523 | 0.418 | 0.305 | 0.551 | 0.438 |
| mean | 0.438 | 0.417 | 0.519 | 0.426 | 0.301 | 0.534 | 0.443 |
| DenseAR | 0.443 | 0.422 | 0.530 | 0.425 | 0.315 | 0.553 | 0.446 |
| family | all_charted | type:lab | type:vital | type:medication | sofa_relevant | freq:low | freq:high |
|---|---|---|---|---|---|---|---|
| greedy | 0.394 | 0.365 | 0.461 | 0.421 | 0.336 | 0.433 | 0.385 |
| ENSEMBLE | 0.394 | 0.365 | 0.461 | 0.421 | 0.335 | 0.433 | 0.386 |
| harm-5 | 0.400 | 0.371 | 0.464 | 0.430 | 0.339 | 0.444 | 0.388 |
| mean | 0.410 | 0.385 | 0.471 | 0.426 | 0.346 | 0.451 | 0.400 |
| select | 0.414 | 0.381 | 0.473 | 0.467 | 0.357 | 0.471 | 0.395 |
| DenseAR | 0.418 | 0.388 | 0.473 | 0.469 | 0.367 | 0.480 | 0.396 |
| horizon | UMM | HiRID | MIMIC-III | MIMIC-IV | eICU | mean |
|---|---|---|---|---|---|---|
| 1 | -20.99 | -14.51 | -15.35 | -9.80 | -11.58 | -14.45 |
| 2 | -15.48 | -12.40 | -16.29 | -8.15 | -11.08 | -12.68 |
| 3 | -9.19 | -25.65 | -14.63 | -8.07 | -8.20 | -13.15 |
| 4 | -8.95 | -10.62 | -11.52 | -10.23 | -9.18 | -10.10 |
| 5 | -11.08 | -10.68 | -9.01 | -8.55 | -8.08 | -9.48 |
| 6 | -11.13 | -10.42 | -14.31 | -6.47 | -8.14 | -10.09 |
| database | equal-3 | harm-3 | fit-3 | equal-5 | harm-5 | fit-5 | equal-14 | harm-14 | fit-14 | ens. |
|---|---|---|---|---|---|---|---|---|---|---|
| development (half of val held-out half) | ||||||||||
| UMM | -7.93 | -7.13 | -7.76 | -8.67 | -8.02 | -8.56 | -8.10 | -10.02 | -10.35 | -10.67 |
| HiRID | -3.05 | -3.17 | -3.18 | -3.74 | -4.01 | -4.07 | -4.50 | -5.62 | -5.14 | -8.00 |
| MIMIC-III | -4.88 | -4.53 | -4.73 | -5.05 | -5.12 | -5.13 | -3.63 | -5.91 | -5.67 | -5.61 |
| MIMIC-IV | -4.37 | -4.41 | -4.53 | -4.08 | -4.70 | -4.72 | -1.20 | -4.87 | -5.14 | -5.57 |
| eICU | -3.79 | -3.87 | -3.95 | -3.75 | -4.12 | -3.92 | -3.14 | -5.06 | -4.64 | -5.00 |
| DenseAR UMM | STraTS UMM | DenseAR HiRID | STraTS HiRID | |
|---|---|---|---|---|
| free-running (absolute) | 0.4853 | 0.5253 | 0.4344 | 0.4715 |
| ensemble, masked | ||||
| ground truth, masked | ||||
| ensemble, unmasked | ||||
| per-channel best, unmasked | ||||
| live hybrid |
| UMM | HiRID | M-III | M-IV | eICU | mean | |
|---|---|---|---|---|---|---|
| stacker (shrunk simplex) | 0.515 | 0.418 | 0.397 | 0.410 | 0.390 | |
| harm-15 | 0.524 | 0.427 | 0.403 | 0.417 | 0.394 | |
| harm-5 | 0.523 | 0.423 | 0.402 | 0.417 | 0.395 | |
| per-variable select | 0.539 | 0.431 | 0.415 | 0.429 | 0.408 | |
| equal weights | 0.544 | 0.454 | 0.422 | 0.436 | 0.407 | |
| LightGBM | 0.530 | 0.450 | 0.420 | 0.436 | 0.411 |
| member | UMM | HiRID | M-III | M-IV | eICU | mean |
|---|---|---|---|---|---|---|
| LightGBM | 75.7 | 74.5 | 61.9 | 56.8 | 58.5 | 65.5 |
| DenseAR | 1.6 | 1.7 | 13.6 | 6.0 | 16.4 | 7.9 |
| STraTS | 1.4 | 6.4 | 5.4 | 13.3 | 6.5 | 6.6 |
| PatchTST | 3.2 | 2.1 | 5.4 | 4.9 | 5.2 | 4.2 |
| DLinear | 8.1 | 8.5 | 0.8 | 1.2 | 0.7 | 3.8 |
| GRU | 1.3 | 0.9 | 0.8 | 7.1 | 2.3 | 2.5 |
| UMM | HiRID | MIMIC-III | MIMIC-IV | eICU | |
|---|---|---|---|---|---|
| random convex weights | [-7.6, -5.9] | [-6.3, -2.8] | [-5.3, -3.5] | [-0.7, +2.1] | [-1.8, -0.4] |
| 4 DenseAR seeds vs. best seed | |||||
| 4 STraTS seeds vs. best seed | |||||
| diverse pool vs. DenseAR seeds | |||||
| diverse pool vs. STraTS seeds |
Multivariate Time Series Forecasting
Multivariate time series forecasting predicts future values across related variables by learning their temporal and cross-variable dependencies.
See all 322 papers