Multimodal Prompt Learning with Irregular EHRs for Robust Monitoring of Critical Care Patients

Yixin Yang; Yueyang Sun; Weichen Liu; Xianbing Zhao; Sicen Liu

cs.AIAug 22, 2026

Multimodal Prompt Learning with Irregular EHRs for Robust Monitoring of Critical Care Patients

Authors: Yixin Yang, Yueyang Sun, Weichen Liu, Xianbing Zhao, Sicen Liu

Organizations: Faculty of Engineering, Shenzhen MSU-BIT University · International School, Beijing University of Posts and Telecommunications · School of Computer Science and Engineering, Southeast University · School of Artificial Intelligence and Computer Science, Jiangnan University

Abstract

Accurate assessment of patients in intensive care units (ICUs) is essential for timely clinical intervention and improved patient outcomes. Multimodal electronic health records (EHRs), including structured physiological time series and longitudinal clinical notes, provide complementary information for critical care prediction. However, in real-world clinical settings, individual modalities may be partially observed or entirely unavailable, resulting in substantial performance degradation for existing multimodal models. To address this challenge, we propose a multimodal prompt-learning framework for robust clinical prediction under diverse missing-modality scenarios. The proposed framework introduces four complementary types of prompts: generative prompts, missing-signal prompts, missing-type prompts, and temporal prompts. Generative prompts construct surrogate latent representations for unavailable modalities, while missing-signal prompts distinguish observed representations from generated ones. Missing-type prompts condition the model on different modality-availability configurations, whereas temporal prompts perform condition-specific aggregation over temporally encoded clinical sequences. Together, these prompts enable the model to capture missingness-aware intramodal dependencies and cross-modal interactions within a unified architecture. Extensive experiments demonstrate that our method outperforms existing approaches across evaluation metrics on two missingness settings. Ablation and robustness analyses further verify the complementary contributions of the four prompt types and the effectiveness of the proposed framework for clinical prediction from incomplete multimodal EHR data.

Figures & tables

Figure 1: Examples of different types of missing information in EHR data include missing clinical notes, missing MISTS, and missing data within the modality. The heart rate and temperature are monitored regularly with different frequencies, and glucose is a laboratory test ordered at irregular time intervals. The clinical notes are free text.

Figure 2: Overview of the proposed prompt-based framework for robust outcome prediction from incomplete irregular multimodal EHRs, based on a Notes-only example. Given a notes-only input condition, irregular notes encoders extract the available representation, and the Missing Modality Generation Module reconstructs the absent MISTS stream. Missing-signal prompts augment each token with a provenance embedding indicating whether its modality stream is observed or prompt-generated, while missing-type prompts inject the sample-level modality-availability condition into both post-fusion streams. The resulting representations are fused and summarized by temporal prompts for patient outcome prediction.

	MISTS-only setting							Notes-only setting
Metric	Imputation	IP-Net	mTAND	GRU-D	RAINDROP	UTDE	Ours	Flat	HierTrans	T-LSTM	FT-LSTM	GRU-D	mTAND-txt	Ours
F1	0.3973	0.3722	0.4387	0.4282	0.3946	0.4526	0.4961	0.3978	0.4876	0.5032	0.4851	0.5101	0.5257	0.5284
AUPRC	0.4436	0.3936	0.4754	0.4590	0.3623	0.4964	0.5028	0.5169	0.5298	0.5257	0.5439	0.5434	0.5605	0.5853

Table 1: Main results on MIMIC-III under single-modality settings. Best results are shown in bold and second-best results are underlined.

MISTS-only setting
Metric	GRU-D	IP-Net	SeFT	MTGNN	DGM 2 -O	mTAND	RAINDROP	UTDE	Ours
AUROC	0.8293	0.8580	0.6536	0.8314	0.8394	0.8540	0.8323	0.8620	0.8586
AUPRC	0.4541	0.5154	0.2369	0.4826	0.4832	0.5025	0.4447	0.5099	0.5202
F1	0.3114	0.4044	0.0672	0.4191	0.4190	0.4168	0.3219	0.4856	0.4585

Table 2: Main results on MIMIC-IV under single-modality settings. Best results within each setting and metric are shown in bold and second-best results are underlined.

Metric	MMTM	DAFT	MedFuse	DrFuse	CTPD	MMIMIC	Ours
AUROC	0.8776	0.8488	0.8523	0.8588	0.8694	0.8826	0.8785
AUPRC	0.5761	0.4910	0.5041	0.5141	0.5384	0.5767	0.5587
F1	0.4823	0.2541	0.3614	0.4698	0.4060	0.4839	0.5285

Table 3: Full-modality results on MIMIC-IV, where both MISTS and clinical notes are available at test time. Best results are shown in bold and second-best results are underlined.

Prompt Configuration				Performance
$P_{G}$	$P_{S}$	$P_{T}$	$P_{\tau}$	AUPRC	AUROC
				0.5383	0.8759
$\checkmark$				0.5566	0.8841
	$\checkmark$			0.5572	0.8839
		$\checkmark$		0.5527	0.8822
			$\checkmark$	0.5412	0.8782

Table 4: Prompt ablation under the Notes-only setting on MIMIC-III, including individual, three-prompt, and cumulative configurations. A check mark indicates that the corresponding prompt is enabled. Best and second-best results are shown in bold and underlined, respectively.

Figure 3: Modality dropout rate ablation on MIMIC-III. Each panel reports AUPRC on the left y-axis and AUROC on the right y-axis for

\eta\in\{0.1,0.3,0.5,0.7,0.9\}

Figure 4: t-SNE visualization of classifier-level representations on MIMIC-III. Panel (a) and (b) show the MISTS-only and Notes-only settings respectively. Each panel compares the observed single-modality representation, its prompt-completed representation, and the fully observed MISTS+Notes reference.

Figure 5: Evolution of the predicted mortality risk for a representative patient in the MIMIC-III Notes-only setting as the prompt components are cumulatively activated. All-off indicates that no prompts are used. Predictions above the decision threshold are classified as mortality, whereas those below the threshold are classified as survival.

Figure 6: Changes in predicted mortality risk after removing each note for the same patient. Negative scores shown in green indicate evidence supporting survival, whereas positive scores shown in red indicate evidence supporting mortality.

Missing-Modality Learning

Missing-modality learning develops multimodal models that remain useful when input streams are absent, corrupted, or available only during training.

See all 170 papers

Multimodal Routing for Interpretable, Robust, and Auditable Clinical Prediction
Jul 10, 2026Nikkie Hooman, Zhongjie Wu, Eric C. Larson +1Multimodal Robustness Clinical Prediction
Southern Methodist University Dallas, USA
Learning Dynamic Representations and Policies from Multimodal Clinical Time-Series with Informative Missingness
Apr 23, 2026Zihan Liang, Ziwen Pan, Ruoxuan XiongClinical Outcome Prediction Offline RL
Emory University, Atlanta, USA
A Clinical Point Cloud Paradigm for In-Hospital Mortality Prediction from Multi-Level Incomplete Multimodal EHRs
Apr 6, 2026Bohao Li, Tao Zou, Junchen Ye +2Electronic Health Records Clinical Risk Prediction
Beihang University, Beijing, China · The Hong Kong Polytechnic University, Hong Kong, China

Jul 10, 2026J/K move · Enter open · S save

Multimodal Routing for Interpretable, Robust, and Auditable Clinical Prediction

Nikkie Hooman, Zhongjie Wu, Eric C. Larson, Mehak Gupta

Southern Methodist University Dallas, USA

Multimodal Robustness Clinical Prediction Multimodal Model Interpretability Longitudinal EHR Modeling Multimodal Model Routing

Open paper AskPDF arXiv

Electronic health record (EHR) data are inherently multimodal, and leveraging multiple modalities can improve predictive performance. However, most existing approaches rely on deep fusion, which obscures how individual modalities contribute to predictions and limits the interpretability of multimodal reasoning. We propose an explicit multimodal routing framework for clinical prediction that enables interpretable, robust, and auditable reasoning across three EHR modalities: structured longitudinal variables (L), clinical notes (N), and chest X-rays (I). Our model constructs discrete unimodal, directional bimodal, and trimodal routes to capture both individual modality signals and asymmetric cross-modal interactions. To audit multimodal reasoning and assess robustness, we introduce inference-time route masking, which simulates missing modalities and reweights the remaining routes without retraining. We analyze changes in performance and routing weights under these scenarios to understand model decision-making. We evaluate our framework on multi-label phenotype prediction (K = 25) and binary ICU mortality prediction using trimodal patient stays from MIMIC-IV, revealing systematic differences in modality reliance across clinical condition groups. Overall, our framework offers a transparent, auditable, and practical approach to multimodal clinical prediction, providing interpretability, robustness, and insights into how different data sources drive model decisions.

Abstract

Missing-Modality Learning

Explore similar work

Multimodal Routing for Interpretable, Robust, and Auditable Clinical Prediction

Learning Dynamic Representations and Policies from Multimodal Clinical Time-Series with Informative Missingness

A Clinical Point Cloud Paradigm for In-Hospital Mortality Prediction from Multi-Level Incomplete Multimodal EHRs