TRACE: Tracking Multimodal Clinical Trajectories for Inpatient Risk Prediction

Minjun Kim; Jong Hak Moon

cs.AIOct 19, 2026

TRACE: Tracking Multimodal Clinical Trajectories for Inpatient Risk Prediction

Authors: Minjun Kim, Jong Hak Moon

Organizations: Yeji X

Abstract

Accurate inpatient risk prediction requires modeling heterogeneous clinical evidence that accumulates irregularly over time. This challenge spans in-hospital mortality, prolonged length of stay, and readmission, each with task-specific observation windows. Existing multimodal approaches combine structured health record data, chest x-rays, and clinical text, but primarily focus on modality fusion or missing-modality compensation, with limited attention to temporal trajectories over an admission. We propose TRACE (TRAjectory modeling of multimodal Clinical Events), a daily-scale framework organizing clinical events into day-level trajectories, separating day-invariant global context from day-specific local events drawn from EHR data, chest x-rays, radiology reports, and discharge summaries. The trajectory is encoded using temporal attention, allowing unavailable modalities to emit no tokens. A phenotype-guided contrastive objective further organizes admissions under heterogeneous disease distributions. Across cohorts from MIMIC-IV, eICU, and MIMIC-III comprising nearly 180,000 total stays, we evaluate TRACE on mortality, length of stay, and readmission prediction. TRACE achieves the highest AUROC across all nine dataset-task settings and the highest ACC in eight, outperforming baselines. Ablation studies validate our attention masks and contrastive learning, while trajectory-level attribution localizes risk-associated hospital days and modalities.

Figures & tables

Figure 1: Overview of TRACE . Admission-level global categories and local inputs recorded within the task-specific observation window are encoded using frozen pretrained vision or language encoders, projected into a shared space, and augmented with modality and day embeddings. For visual brevity, the category-specific global tokens $\mathbf{G}_{i}$ are shown collectively as $G_{\mathrm{cls}}$ . Global tokens, day-specific local tokens $\mathbf{L}_{i}^{(k)}$ , and learnable [TASK] and [PHENO] tokens are encoded by a masked Transformer using the temporal global–local attention mask $\mathbf{A}^{\mathrm{TGL}}$ , which keeps global context accessible, preserves temporal order among local events, and allows [PHENO] to attend to [TASK] but not the reverse. The encoded [TASK] and [PHENO] states produce $\hat{p}^{(k)}$ and $\mathbf{r}^{(k)}$ , respectively, and are jointly optimized using $\mathcal{L}^{(k)}=\mathcal{L}_{\mathrm{pred}}^{(k)}+\lambda\mathcal{L}_{\mathrm{pheno}}^{(k)}$ . Snowflake and flame icons indicate frozen and trainable modules.

MIMIC-IV: Unimodal
Dataset / Input Configuration	Mortality				Long LOS				Readmission
Dataset / Input Configuration	ACC	AUROC	AUPRC	mF1	ACC	AUROC	AUPRC	mF1	ACC	AUROC	AUPRC	mF1
C	0.9099	0.8734	0.5536	0.7544	0.6489	0.6831	0.7602	0.5085	0.8179	0.7667	0.4736	0.6911
E	0.9542	0.8742	0.7698	0.7871	0.6421	0.6531	0.8105	0.5489	0.8696	0.8109	0.6414	0.7166
R	0.8681	0.8009	0.3920	0.6541	0.7164	0.6555	0.8209	0.4174	0.8225	0.7116	0.4513	0.6001
D	0.8964	0.6947	0.1758	0.4727	–	–	–	–	0.8048	0.6009	0.2473	0.4459

Table 2: Modality ablation across MIMIC-IV and MIMIC-III. C, E, R, and D denote chest x-rays, structured EHR events, radiology reports, and discharge summaries, respectively. Both MIMIC-IV and MIMIC-III include unimodal and multimodal configurations. Discharge-summary configurations are not applicable to Long LOS because the task uses only data recorded during the first 48 hours. Best results are boldfaced separately within each dataset and task.

Task	Metric	TGL	All-Causal	Full-Attention
Mortality	ACC	0.9528	0.9533	0.9369
	AUROC	0.9804	0.9778	0.9576
	AUPRC	0.8605	0.8500	0.7911
	mF1	0.8886	0.8864	0.8275
Long LOS	ACC	0.7409	0.7305	0.7193
	AUROC	0.7675	0.7583	0.7462

Table 3: Ablation of the TGL attention mask against All-Causal and Full-Attention alternatives on MIMIC-IV.

Dataset	Phenotype	$n$	Mortality				Long LOS				Readmission
			AUROC		AUPRC		AUROC		AUPRC		AUROC		AUPRC
			$L_{\mathrm{pred}}$	$\Delta_{\mathrm{pheno}}$	$L_{\mathrm{pred}}$	$\Delta_{\mathrm{pheno}}$	$L_{\mathrm{pred}}$	$\Delta_{\mathrm{pheno}}$	$L_{\mathrm{pred}}$	$\Delta_{\mathrm{pheno}}$	$L_{\mathrm{pred}}$	$\Delta_{\mathrm{pheno}}$	$L_{\mathrm{pred}}$	$\Delta_{\mathrm{pheno}}$
MIMIC-IV	Circulatory	764	.9708	+.0061	.8272	+.0131	.7899	+.0042	.8331	+.0048	.8164	$-$ .0015	.6596	$-$ .0044
	Trauma & Poisoning	112	.9374	+.0120	.7273	+.0821	.7490	+.0127	.7948	+.0143	.8920	+.0088	.6345	$-$ .0070
	Endocrine & Metabolic	206	.9805	+.0073	.8428	+.0688	.7517	+.0189	.8686	+.0130	.7788	+.0040	.6280	$-$ .0113
	Respiratory	152	.9582	+.0117	.8273	+.0517	.7492	+.0254	.8570	+.0239	.7868	+.0047	.6901	$-$ .0170

Table 4: Phenotype-stratified performance of the

L_{\mathrm{pred}}

-only baseline and the change obtained by adding

L_{\mathrm{pheno}}

across datasets.

\Delta_{\mathrm{pheno}}

denotes the performance under the joint objective minus that under the prediction-only objective; positive improvements are boldfaced. Avg. denotes the unweighted mean across phenotypes.

Panel of Figure 2 — Figure 2: t-SNE [ 30 ] visualization of task-specific admission representations learned with the joint objective $\mathcal{L}_{\mathrm{pred}}^{(k)}+\lambda\mathcal{L}_{\mathrm{pheno}}^{(k)}$ and the prediction-only objective $\mathcal{L}_{\mathrm{pred}}^{(k)}$ . Points are colored by diagnosis-derived phenotype group.

Panel of Figure 3 — Figure 3: Trajectory-level interpretability for representative patients from two diagnosis-derived phenotypes. Prefix-level risk estimates are obtained through repeated inference using global context and local clinical events observed through each hospital day. Gradient $\times$ input attribution scores are aggregated by hospital day and modality for structured EHR, chest x-rays, radiology reports, and discharge summaries.

Clinical Outcome Prediction

Clinical outcome prediction uses patient records, imaging, signals, and biomarkers to estimate risks, disease trajectories, treatment response, and survival.

See all 265 papers

Multimodal Routing for Interpretable, Robust, and Auditable Clinical Prediction
Jul 10, 2026Nikkie Hooman, Zhongjie Wu, Eric C. Larson +1Multimodal Robustness Clinical Prediction
Southern Methodist University Dallas, USA
Learning to Select Source-Traceable Evidence for Language-Model Prediction from Irregular Clinical Time Series
May 19, 2026Kwanhyung Lee, Juhwan Choi, Jongheon Kim +5Irregular Time-Series Modeling Healthcare
Kim Jaechul Graduate School of AI, KAIST · AITRICS
A Clinical Point Cloud Paradigm for In-Hospital Mortality Prediction from Multi-Level Incomplete Multimodal EHRs
Apr 6, 2026Bohao Li, Tao Zou, Junchen Ye +2Electronic Health Records Clinical Risk Prediction
Beihang University, Beijing, China · The Hong Kong Polytechnic University, Hong Kong, China

Jul 10, 2026J/K move · Enter open · S save

Multimodal Routing for Interpretable, Robust, and Auditable Clinical Prediction

Nikkie Hooman, Zhongjie Wu, Eric C. Larson, Mehak Gupta

Southern Methodist University Dallas, USA

Multimodal Robustness Clinical Prediction Multimodal Model Interpretability Longitudinal EHR Modeling Multimodal Model Routing

Open paper AskPDF arXiv

Electronic health record (EHR) data are inherently multimodal, and leveraging multiple modalities can improve predictive performance. However, most existing approaches rely on deep fusion, which obscures how individual modalities contribute to predictions and limits the interpretability of multimodal reasoning. We propose an explicit multimodal routing framework for clinical prediction that enables interpretable, robust, and auditable reasoning across three EHR modalities: structured longitudinal variables (L), clinical notes (N), and chest X-rays (I). Our model constructs discrete unimodal, directional bimodal, and trimodal routes to capture both individual modality signals and asymmetric cross-modal interactions. To audit multimodal reasoning and assess robustness, we introduce inference-time route masking, which simulates missing modalities and reweights the remaining routes without retraining. We analyze changes in performance and routing weights under these scenarios to understand model decision-making. We evaluate our framework on multi-label phenotype prediction (K = 25) and binary ICU mortality prediction using trimodal patient stays from MIMIC-IV, revealing systematic differences in modality reliance across clinical condition groups. Overall, our framework offers a transparent, auditable, and practical approach to multimodal clinical prediction, providing interpretability, robustness, and insights into how different data sources drive model decisions.

Abstract

Clinical Outcome Prediction

Explore similar work

Multimodal Routing for Interpretable, Robust, and Auditable Clinical Prediction

Learning to Select Source-Traceable Evidence for Language-Model Prediction from Irregular Clinical Time Series

A Clinical Point Cloud Paradigm for In-Hospital Mortality Prediction from Multi-Level Incomplete Multimodal EHRs