Multimodal Prompt Learning with Irregular EHRs for Robust Monitoring of Critical Care Patients

Yixin Yang; Yueyang Sun; Weichen Liu; Xianbing Zhao; Sicen Liu

cs.AIAug 22, 2026

Multimodal Prompt Learning with Irregular EHRs for Robust Monitoring of Critical Care Patients

Authors: Yixin Yang, Yueyang Sun, Weichen Liu, Xianbing Zhao, Sicen Liu

Organizations: Faculty of Engineering, Shenzhen MSU-BIT University · International School, Beijing University of Posts and Telecommunications · School of Computer Science and Engineering, Southeast University · School of Artificial Intelligence and Computer Science, Jiangnan University

Abstract

Accurate assessment of patients in intensive care units (ICUs) is essential for timely clinical intervention and improved patient outcomes. Multimodal electronic health records (EHRs), including structured physiological time series and longitudinal clinical notes, provide complementary information for critical care prediction. However, in real-world clinical settings, individual modalities may be partially observed or entirely unavailable, resulting in substantial performance degradation for existing multimodal models. To address this challenge, we propose a multimodal prompt-learning framework for robust clinical prediction under diverse missing-modality scenarios. The proposed framework introduces four complementary types of prompts: generative prompts, missing-signal prompts, missing-type prompts, and temporal prompts. Generative prompts construct surrogate latent representations for unavailable modalities, while missing-signal prompts distinguish observed representations from generated ones. Missing-type prompts condition the model on different modality-availability configurations, whereas temporal prompts perform condition-specific aggregation over temporally encoded clinical sequences. Together, these prompts enable the model to capture missingness-aware intramodal dependencies and cross-modal interactions within a unified architecture. Extensive experiments demonstrate that our method outperforms existing approaches across evaluation metrics on two missingness settings. Ablation and robustness analyses further verify the complementary contributions of the four prompt types and the effectiveness of the proposed framework for clinical prediction from incomplete multimodal EHR data.

Figures & tables

Figure 1: Examples of different types of missing information in EHR data include missing clinical notes, missing MISTS, and missing data within the modality. The heart rate and temperature are monitored regularly with different frequencies, and glucose is a laboratory test ordered at irregular time intervals. The clinical notes are free text.

Figure 2: Overview of the proposed prompt-based framework for robust outcome prediction from incomplete irregular multimodal EHRs, based on a Notes-only example. Given a notes-only input condition, irregular notes encoders extract the available representation, and the Missing Modality Generation Module reconstructs the absent MISTS stream. Missing-signal prompts augment each token with a provenance embedding indicating whether its modality stream is observed or prompt-generated, while missing-type prompts inject the sample-level modality-availability condition into both post-fusion streams. The resulting representations are fused and summarized by temporal prompts for patient outcome prediction.

	MISTS-only setting							Notes-only setting
Metric	Imputation	IP-Net	mTAND	GRU-D	RAINDROP	UTDE	Ours	Flat	HierTrans	T-LSTM	FT-LSTM	GRU-D	mTAND-txt	Ours
F1	0.3973	0.3722	0.4387	0.4282	0.3946	0.4526	0.4961	0.3978	0.4876	0.5032	0.4851	0.5101	0.5257	0.5284
AUPRC	0.4436	0.3936	0.4754	0.4590	0.3623	0.4964	0.5028	0.5169	0.5298	0.5257	0.5439	0.5434	0.5605	0.5853

Table 1: Main results on MIMIC-III under single-modality settings. Best results are shown in bold and second-best results are underlined.

MISTS-only setting
Metric	GRU-D	IP-Net	SeFT	MTGNN	DGM 2 -O	mTAND	RAINDROP	UTDE	Ours
AUROC	0.8293	0.8580	0.6536	0.8314	0.8394	0.8540	0.8323	0.8620	0.8586
AUPRC	0.4541	0.5154	0.2369	0.4826	0.4832	0.5025	0.4447	0.5099	0.5202
F1	0.3114	0.4044	0.0672	0.4191	0.4190	0.4168	0.3219	0.4856	0.4585

Table 2: Main results on MIMIC-IV under single-modality settings. Best results within each setting and metric are shown in bold and second-best results are underlined.

Metric	MMTM	DAFT	MedFuse	DrFuse	CTPD	MMIMIC	Ours
AUROC	0.8776	0.8488	0.8523	0.8588	0.8694	0.8826	0.8785
AUPRC	0.5761	0.4910	0.5041	0.5141	0.5384	0.5767	0.5587
F1	0.4823	0.2541	0.3614	0.4698	0.4060	0.4839	0.5285

Table 3: Full-modality results on MIMIC-IV, where both MISTS and clinical notes are available at test time. Best results are shown in bold and second-best results are underlined.

Prompt Configuration				Performance
$P_{G}$	$P_{S}$	$P_{T}$	$P_{\tau}$	AUPRC	AUROC
				0.5383	0.8759
$\checkmark$				0.5566	0.8841
	$\checkmark$			0.5572	0.8839
		$\checkmark$		0.5527	0.8822
			$\checkmark$	0.5412	0.8782

Table 4: Prompt ablation under the Notes-only setting on MIMIC-III, including individual, three-prompt, and cumulative configurations. A check mark indicates that the corresponding prompt is enabled. Best and second-best results are shown in bold and underlined, respectively.

Figure 3: Modality dropout rate ablation on MIMIC-III. Each panel reports AUPRC on the left y-axis and AUROC on the right y-axis for

\eta\in\{0.1,0.3,0.5,0.7,0.9\}

Figure 4: t-SNE visualization of classifier-level representations on MIMIC-III. Panel (a) and (b) show the MISTS-only and Notes-only settings respectively. Each panel compares the observed single-modality representation, its prompt-completed representation, and the fully observed MISTS+Notes reference.

Figure 5: Evolution of the predicted mortality risk for a representative patient in the MIMIC-III Notes-only setting as the prompt components are cumulatively activated. All-off indicates that no prompts are used. Predictions above the decision threshold are classified as mortality, whereas those below the threshold are classified as survival.

Figure 6: Changes in predicted mortality risk after removing each note for the same patient. Negative scores shown in green indicate evidence supporting survival, whereas positive scores shown in red indicate evidence supporting mortality.

Missing-Modality Learning

Missing-modality learning develops multimodal models that remain useful when input streams are absent, corrupted, or available only during training.

See all 170 papers

Jul 10, 2026

Multimodal Routing for Interpretable, Robust, and Auditable Clinical Prediction

Electronic health record (EHR) data are inherently multimodal, and leveraging multiple modalities can improve predictive performance. However, most existing approaches rely on deep fusion, which obscures how individual modalities contribute to predictions and limits the interpretability of multimodal reasoning. We propose an explicit multimodal routing framework for clinical prediction that enables interpretable, robust, and auditable reasoning across three EHR modalities: structured longitudinal variables (L), clinical notes (N), and chest X-rays (I). Our model constructs discrete unimodal, directional bimodal, and trimodal routes to capture both individual modality signals and asymmetric cross-modal interactions. To audit multimodal reasoning and assess robustness, we introduce inference-time route masking, which simulates missing modalities and reweights the remaining routes without retraining. We analyze changes in performance and routing weights under these scenarios to understand model decision-making. We evaluate our framework on multi-label phenotype prediction (K = 25) and binary ICU mortality prediction using trimodal patient stays from MIMIC-IV, revealing systematic differences in modality reliance across clinical condition groups. Overall, our framework offers a transparent, auditable, and practical approach to multimodal clinical prediction, providing interpretability, robustness, and insights into how different data sources drive model decisions.

Nikkie Hooman, Zhongjie Wu, Eric C. Larson +1

Southern Methodist University Dallas, USA

Multimodal Robustness Clinical Prediction Multimodal Model Interpretability Longitudinal EHR Modeling Multimodal Model Routing

Ask PDF

Apr 23, 2026

Learning Dynamic Representations and Policies from Multimodal Clinical Time-Series with Informative Missingness

Multimodal clinical records contain structured measurements and clinical notes recorded over time, offering rich temporal information about the evolution of patient health. Yet these observations are sparse, and whether they are recorded depends on the patient's latent condition. Observation patterns also differ across modalities, as structured measurements and clinical notes arise under distinct recording processes. While prior work has developed methods that accommodate missingness in clinical time series, how to extract and use the information carried by the observation process itself remains underexplored. We therefore propose a patient representation learning framework for multimodal clinical time series that explicitly leverages informative missingness. The framework combines (1) a multimodal encoder that captures signals from structured and textual data together with their observation patterns, (2) a Bayesian filtering module that updates a latent patient state over time from observed multimodal signals, and (3) downstream modules for offline treatment policy learning and patient outcome prediction based on the learned patient state. We evaluate the framework on ICU sepsis cohorts from MIMIC-III, MIMIC-IV, and eICU. It improves both offline treatment policy learning and adverse outcome prediction, achieving FQE 0.679 versus 0.528 for clinician behavior and AUROC 0.886 for post-72-hour mortality prediction on MIMIC-III.

Zihan Liang, Ziwen Pan, Ruoxuan Xiong

Emory University, Atlanta, USA

Clinical Outcome Prediction Offline RL State Representation Learning Longitudinal EHR Modeling

Ask PDF

Apr 6, 2026

A Clinical Point Cloud Paradigm for In-Hospital Mortality Prediction from Multi-Level Incomplete Multimodal EHRs

Deep learning-based modeling of multimodal Electronic Health Records (EHRs) has become an important approach for clinical diagnosis and risk prediction. However, due to diverse clinical workflows and privacy constraints, raw EHRs are inherently multi-level incomplete, including irregular sampling, missing modalities, and sparse labels. These issues cause temporal misalignment, modality imbalance, and limited supervision. Most existing multimodal methods assume relatively complete data, and even methods designed for incompleteness usually address only one or two of these issues in isolation. As a result, they often rely on rigid temporal/modal alignment or discard incomplete data, which may distort raw clinical semantics. To address this problem, we propose HealthPoint (HP), a unified clinical point cloud paradigm for multi-level incomplete EHRs. HP represents heterogeneous clinical events as points in a continuous 4D space defined by content, time, modality, and case. To model interactions between arbitrary point pairs, we introduce a Low-Rank Relational Attention mechanism that efficiently captures high-order dependencies across these four dimensions. We further develop a hierarchical interaction and sampling strategy to balance fine-grained modeling and computational efficiency. Built on this framework, HP enables flexible event-level interaction and fine-grained self-supervision, supporting robust modality recovery and effective use of unlabeled data. Experiments on large-scale EHR datasets for risk prediction show that HP consistently achieves state-of-the-art performance and strong robustness under varying degrees of incompleteness.

Bohao Li, Tao Zou, Junchen Ye +2

Beihang University, Beijing, China · The Hong Kong Polytechnic University, Hong Kong, China

Electronic Health Records Clinical Risk Prediction Longitudinal EHR Modeling Missing-Modality Learning Point Cloud Learning

Ask PDF

Abstract

Missing-Modality Learning

Explore similar work

Multimodal Routing for Interpretable, Robust, and Auditable Clinical Prediction

Learning Dynamic Representations and Policies from Multimodal Clinical Time-Series with Informative Missingness

A Clinical Point Cloud Paradigm for In-Hospital Mortality Prediction from Multi-Level Incomplete Multimodal EHRs