When Does Multimodal Learning Help in Healthcare? A Benchmark on EHR and Chest X-Ray Fusion
Authors: Kejing Yin, Haizhou Xu, Wenfang Yao, Chen Liu, Zijie Chen, Yui Haang Cheung, William K. Cheung, Jing Qin
Organizations: Department of Computer Science, Hong Kong Baptist University · Division of Artificial Intelligence, Lingnan University · School of Nursing, The Hong Kong Polytechnic University
Machine learning holds promise for advancing clinical decision support, yet it remains unclear when multimodal learning truly helps in practice, particularly under modality missingness and fairness constraints. In this work, we conduct a systematic benchmark of multimodal fusion between Electronic Health Records (EHR) and chest X-rays (CXR) on standardized cohorts from MIMIC-IV and MIMIC-CXR, aiming to answer four fundamental questions: when multimodal fusion improves clinical prediction, how different fusion strategies compare, how robust existing methods are to missing modalities, and whether multimodal models achieve algorithmic fairness. Our study reveals several key insights. Multimodal fusion improves performance when modalities are complete, with gains concentrating in diseases that require complementary information from both EHR and CXR. While cross-modal learning mechanisms capture clinically meaningful dependencies beyond simple concatenation, the rich temporal structure of EHR introduces strong modality imbalance that architectural complexity alone cannot overcome. Under realistic missingness, multimodal benefits rapidly degrade unless models are explicitly designed to handle incomplete inputs. Moreover, multimodal fusion does not inherently improve fairness, with subgroup disparities mainly arising from unequal sensitivity across demographic groups. To support reproducible and extensible evaluation, we further release a flexible benchmarking toolkit that enables plug-and-play integration of new models and datasets. Together, this work provides actionable guidance on when multimodal learning helps, when it fails, and why, laying the foundation for developing clinically deployable multimodal systems that are both effective and reliable. The open-source toolkit can be found at https://github.com/jakeykj/CareBench.
Figures & tables
Figure 1: Overview of CareBench pipeline.
Phenotyping
Mortality
Length of Stay
Model
AUPRC
AUROC
F1
AUPRC
AUROC
F1
Accuracy
F1
Kappa
Transformer
0.472 ± 0.0008
0.724 ± 0.0005
0.340 ± 0.0123
0.447 ± 0.0151
0.821 ± 0.0036
0.576 ± 0.0341
0.392 ± 0.0101
0.192 ± 0.0070
0.187 ± 0.0106
LSTM
0.452 ± 0.0036
0.715 ± 0.0057
0.328 ± 0.0227
0.451 ± 0.0243
0.803 ± 0.0261
0.627 ± 0.0597
0.397 ± 0.0044
0.186 ± 0.0136
0.192 ± 0.0060
ResNet
0.400 ± 0.0014
0.668 ± 0.0022
0.269 ± 0.0145
0.225 ± 0.0062
0.688 ± 0.0090
0.492 ± 0.0224
0.346 ± 0.0076
0.133 ± 0.0025
0.115 ± 0.0022
LateFusion
0.489 ± 0.0007
0.738 ± 0.0012
0.298 ± 0.0182
0.433 ± 0.0179
0.823 ± 0.0039
0.611 ± 0.0443
0.394 ± 0.0068
0.189 ± 0.0094
0.191 ± 0.0053
MedFuse
0.478 ± 0.0030
0.732 ± 0.0014
0.256 ± 0.0191
0.472 ± 0.0098
0.851 ± 0.0056
0.548 ± 0.0454
0.403 ± 0.0013
0.167 ± 0.0078
0.194 ± 0.0035
Table 1: Predictive performance over the matched subset. Pink-highlighted cells indicate the best uni-modal method for each disease. Green-highlighted cells denote results that are statistically significantly better than the best uni-modal method based on a permutation test ( p<0.05 ). Boldface values indicate the best-performing model for each evaluation metric (column).
Disease
Transformer
LSTM
ResNet
LateFusion
MedFuse
MMTM
DAFT
UTDE
ShaSpec
Flex-MoE
DrFuse
SMIL
HEALNet
M3Care
UMSE
AUG
InfoReg
CHF
0.705
0.691
0.696
0.742
0.737
0.704
0.728
0.736
0.737
0.735
0.735
0.732
0.699
0.735
0.623
0.741
0.742
CorAth/HD
0.487
0.491
0.515
0.537
0.544
0.543
0.544
0.550
0.548
0.550
0.553
0.519
0.497
0.550
0.475
0.538
0.550
Liver Dis.
0.325
0.329
0.334
0.369
0.364
0.341
0.367
0.382
0.365
0.352
0.355
0.359
0.331
0.363
0.304
0.378
0.378
COPD
0.310
0.284
0.378
0.402
0.380
0.381
0.365
0.385
0.376
0.372
0.398
0.359
0.341
0.393
0.339
0.396
0.406
HTN
0.543
0.534
0.510
0.561
0.558
0.537
0.564
0.577
0.572
0.562
0.573
0.566
0.544
0.568
0.558
0.561
0.561
AMI
0.234
0.216
0.169
0.236
0.231
0.252
0.235
0.252
0.234
0.246
0.272
0.200
0.245
0.278
0.253
0.246
0.259
Table 2: Disease-wise AUPRC for the disease phenotype classification task on the matched cohort. Pink-highlighted cells indicate the best-performing uni-modal method for each disease. Green-highlighted cells denote results that are statistically significantly better than the best uni-modal method based on a permutation test ( p<0.05 ). Boldface values indicate the best-performing model for each disease (row).
Phenotyping
Mortality
Length of Stay
Models
AUPRC
AUROC
F1
AUPRC
AUROC
F1
ACC
F1
Kappa
Transformer
0.479 ± 0.0011
0.759 ± 0.0003
0.362 ± 0.0041
0.504 ± 0.0059
0.867 ± 0.0010
0.679 ± 0.0133
0.417 ± 0.0033
0.198 ± 0.0144
0.204 ± 0.0053
LSTM
0.468 ± 0.0013
0.755 ± 0.0003
0.334 ± 0.0224
0.480 ± 0.0131
0.861 ± 0.0061
0.661 ± 0.0232
0.412 ± 0.0034
0.178 ± 0.0043
0.197 ± 0.0034
ResNet
0.280 ± 0.0021
0.565 ± 0.0001
0.115 ± 0.0104
0.134 ± 0.0086
0.565 ± 0.0143
0.476 ± 0.0009
0.354 ± 0.0005
0.106 ± 0.0008
0.041 ± 0.0018
LateFusion
0.485 ± 0.0006
0.762 ± 0.0017
0.289 ± 0.0163
0.500 ± 0.0067
0.867 ± 0.0023
0.647 ± 0.0193
0.416 ± 0.0011
0.200 ± 0.0096
0.210 ± 0.0046
MedFuse
0.469 ± 0.0016
0.756 ± 0.0009
0.220 ± 0.0003
0.508 ± 0.0069
0.874 ± 0.0025
0.636 ± 0.0263
0.418 ± 0.0028
0.203 ± 0.0085
0.213 ± 0.0028
Table 3: Predictive Performance over the base cohort. Pink-highlighted cells indicate the best uni-modal method for each disease. Green-highlighted cells denote results that are statistically significantly better than the best uni-modal method based on a permutation test ( p<0.05 ). Boldface values indicate the best-performing model for each evaluation metric (column).
Figure 2: Robustness of the top six performing methods for the overall phenotype classification task and five disease classification tasks under varying ratios of modality missingness. The shaded areas represent standard deviations.
Figure 3: Race Group fairness gaps on the phenotype matched subset.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Data exclusion criteria for constructing the base cohort and the matched subset.
Category
Feature
MIMIC-IV Source
Summary Stats
Missing %
Categorical Features
Glasgow Coma Scale
Eye Opening
gcs.gcs_eyes
4 [3,4]
70.98
Verbal Response
gcs.gcs_verbal
5 [0,5]
71.00
Motor Response
gcs.gcs_motor
6 [6,6]
71.04
Total Score
gcs.gcs_total
15 [15,15]
70.93
Cardiac Rhythm
Absence of Ectopy
mimiciv_icu.chartevents
binary presence
30.77
Appendix
Table 4: Summary of extracted features from MIMIC-IV derived. Continuous variables are reported as mean ± std [25%,75%]; categorical variables as median [25%,75%].
Benchmark
Modalities
Multimodal?
# of Models
Accuracy?
Robustness?
Fairness?
Purushotham et al. (2018)
EHR
✗
18
✔
✗
✗
Harutyunyan et al. (2019)
EHR
✗
7
✔
✗
✗
Barbieri et al. (2020)
EHR
✗
13
✔
✗
✗
MIMIC-Extract ( Wang et al., 2020 )
EHR
✗
5
✔
✗
✗
Sheikhalishahi et al. (2020)
EHR
✗
4
✔
✗
✗
FIDDLE ( Tang et al., 2020 )
EHR
✗
4
✔
✗
✗
Appendix
Table 5: Comparison of CareBench and existing benchmarks.
Electronic health record (EHR) data are inherently multimodal, and leveraging multiple modalities can improve predictive performance. However, most existing approaches rely on deep fusion, which obscures how individual modalities contribute to predictions and limits the interpretability of multimodal reasoning. We propose an explicit multimodal routing framework for clinical prediction that enables interpretable, robust, and auditable reasoning across three EHR modalities: structured longitudinal variables (L), clinical notes (N), and chest X-rays (I). Our model constructs discrete unimodal, directional bimodal, and trimodal routes to capture both individual modality signals and asymmetric cross-modal interactions. To audit multimodal reasoning and assess robustness, we introduce inference-time route masking, which simulates missing modalities and reweights the remaining routes without retraining. We analyze changes in performance and routing weights under these scenarios to understand model decision-making. We evaluate our framework on multi-label phenotype prediction (K = 25) and binary ICU mortality prediction using trimodal patient stays from MIMIC-IV, revealing systematic differences in modality reliance across clinical condition groups. Overall, our framework offers a transparent, auditable, and practical approach to multimodal clinical prediction, providing interpretability, robustness, and insights into how different data sources drive model decisions.
Accurate time-to-event (TTE) prediction from multimodal clinical data remains challenging due to modality imbalance and distribution shift. We introduce a foundation model-driven framework for cross-modal representation alignment between CT imaging and longitudinal EHR data, designed to generalize across tasks and institutions. CT and EHR modalities are encoded independently using domain-specific foundation models and aligned in a shared latent space through four principled fusion strategies: late fusion, contrastive alignment, cross-attention, and co-attention. We evaluate two clinically distinct TTE tasks: pulmonary embolism (PE) mortality and cardiovascular disease (CVD) outcomes, on large-scale multi-institutional cohorts (PE: N=3,099 train; 1,098 internal; 435 external; CVD: N=2,951 train; 837 internal; 682 external). Fusion consistently improves concordance index by 1.5-5.4% over unimodal baselines when modalities contribute comparably. Overall, contrastive multimodal fusion, particularly with CLMBR representations, provided the most consistent and statistically robust improvements, especially for PE mortality prediction. For MACE, cross-attention (one-hot) achieved the highest internal performance and image-guided co-attention achieved the best external performance. We therefore introduce a generalizable foundation model-based cross-modal alignment framework and provide the first systematic analysis of fusion behavior under modality imbalance in TTE prediction. Our results establish task-aware multimodal alignment as a necessary design principle for robust generalization and scalable clinical deployment.
Breast cancer is a leading cause of cancer-related mortality worldwide, and timely accurate diagnosis is critical to improving survival outcomes. While convolutional neural networks (CNNs) have demonstrated strong performance on histopathology image classification, and machine learning models on structured electronic health records (EHR) have shown utility for clinical risk stratification, most existing work treats these modalities in isolation. This paper presents a systematic multimodal framework that integrates patch-level histopathology features from the BreCaHAD dataset with structured clinical data from MIMIC-IV. We train and evaluate unimodal image models (a simple CNN baseline and ResNet-18 with transfer learning), unimodal tabular models (XGBoost and a multilayer perceptron), and an intermediate-fusion model that concatenates latent representations from both modalities. ResNet-18 achieves near-perfect accuracy (1.000) and AUC (1.000) on three-class patch-level classification, while XGBoost achieves 98% accuracy on the EHR prediction task. The intermediate fusion model yields a macro-average AUC of 0.997, outperforming all unimodal baselines and delivering the largest improvements on the diagnostically critical but class-imbalanced mitosis category (AUC 0.994). Grad-CAM and SHAP interpretability analyses validate that model decisions align with established pathological and clinical criteria. Our results demonstrate that multimodal integration delivers meaningful improvements in both predictive performance and clinical transparency.
Aditya Shribhagwan Khandelwal, Mohammad Samar Ansari, Asra Aslam
University of Sheffield, UK · University of Chester, UK