DeliMF: Gated Cross-Attention Fusion of Structured Time Series and Clinical Text for Early Intensive Care Delirium Prediction

Yibo Yang; Zerui Huang; Haomiao Yang; Zixia Wang

cs.LGOct 17, 2026

DeliMF: Gated Cross-Attention Fusion of Structured Time Series and Clinical Text for Early Intensive Care Delirium Prediction

Authors: Yibo Yang, Zerui Huang, Haomiao Yang, Zixia Wang

Organizations: Independent Researcher, China · Xinjin District People’s Hospital, China · University of Exeter, United Kingdom

Abstract

Delirium affects a large fraction of intensive care unit (ICU) patients and foreshadows longer stays, lasting cognitive decline, and higher mortality. Flagging the patients who will become delirious within the first day of admission would give clinicians a window to intervene. Yet state-of-the-art predictors rely only on the structured electronic health record (EHR) of vital signs, laboratory values, and medications, while overlooking the free-text clinical notes written in the same window, which carry information the numbers cannot encode. To this end, we present DeliMF, a multimodal framework that fuses the structured time series with clinical text. It encodes the structured signals with a temporal Transformer and the time-stamped radiology reports with a clinical text encoder, and fuses the two through a gated cross-attention module. The gate learns a per-patient weight on the text; when a patient has no note the gate closes and DeliMF reduces to its structured-only form, giving robustness to missing text by construction. On an incident-only MIMIC-IV cohort, DeliMF improves significantly over the baselines, particularly on patients who carry clinical text, while losing nothing on those without. Ablation studies confirm the contribution of multimodal fusion and gated cross-attention, and a case study further shows that fusing clinical text with the structured stream recovers delirium cases that the structured data alone misses. We hope DeliMF encourages early ICU prediction to make fuller use of the overlooked clinical text and contributes to the early identification and timely intervention of delirium.

Figures & tables

Fig. 1: Overview of DeliMF , traced through a real MIMIC-IV case (61-year-old admitted after a fall, incident-delirium label). The structured tower embeds each event from its value, feature id, and timestamp and encodes the sequence with a temporal Transformer, pooled to $h_{s}$ . The text tower encodes the first-24 h radiology report with ClinicalBERT, pooled to $h_{t}$ . In the gated cross-attention module, the structured states query the text, and a per-patient gate weights the two representations into the fused $h_{f}$ ; when no note is available the gate closes and DeliMF reduces to its structured-only form. A two-layer MLP head maps $h_{f}$ to the delirium probability. For this case the report names a subarachnoid hemorrhage with diffuse axonal injury, precipitants absent from the tabular record, and DeliMF raises the probability to 0.56 while the structured-only baseline assigns 0.21 and misses the case.

Fig. 2: Data and cohort overview. (a) Cohort construction from all brain-status-assessed ICU stays down to the incident-only cohort of 23,431; the final (red) bar is the analyzed cohort. (b) Age distribution (bars) and delirium rate (line), which rises monotonically with age. (c) Text availability on the test set: 55.4% of stays carry a radiology report, and the with-text subgroup has a higher positive rate. (d) Distribution of structured events per stay (mean 217).

Comparison with prior methods
Model	AUROC	AUPRC	F1	Sens.	Spec.	Brier
DeliMF (ours)	0.803 $\pm$ 0.003	0.360 $\pm$ 0.012	0.391 $\pm$ 0.018	0.718 $\pm$ 0.036	0.762 $\pm$ 0.037	0.162 $\pm$ 0.016
DeLLiriuM-style	0.721 $\pm$ 0.000	0.242 $\pm$ 0.001	0.316 $\pm$ 0.006	0.545 $\pm$ 0.037	0.770 $\pm$ 0.032	0.182 $\pm$ 0.015
E-PRE-DELIRIC-style	0.692 $\pm$ 0.000	0.189 $\pm$ 0.000	0.283 $\pm$ 0.000	0.624 $\pm$ 0.000	0.666 $\pm$ 0.000	0.228 $\pm$ 0.000
Single-modality ablation
Structured-only	0.800 $\pm$ 0.002	0.334 $\pm$ 0.007	0.390 $\pm$ 0.003	0.663 $\pm$ 0.057	0.791 $\pm$ 0.031	0.148 $\pm$ 0.013

TABLE I: Comparison on the MIMIC-IV test set (3 seeds, mean

\pm

std; threshold 0.5). The table is organized into blocks: comparison with prior methods, single-modality ablation, fusion design, training schedule, and encoder substitution. Our full model is shaded; best per column in bold blue (lower is better for Brier).

Fig. 3: Subset analysis across text availability and age (3 seeds, mean

\pm

std). (a) AUROC and (b) AUPRC for DeliMF, the structured-only model, and two structured-input baselines. DeliMF’s AUPRC gain concentrates on the with-text and elderly subgroups, while it matches structured-only when no note is present. The DeLLiriuM-style and E-PRE-DELIRIC-style baselines use only structured inputs, so they are unaffected by note availability.

Fig. 4: Gate value distribution over patients with text. The mean of 0.45 and the left tail show per-patient adaptive reliance on the text modality.

Age	Struct.	DeliMF	Evidence in the radiology report
61	0.21	0.56	1. Parietal hemorrhage enlarging: “ $12\times 12$ mm” vs. “ $8\times 5.5$ mm” prior 2. “consistent with diffuse axonal injury”
68	0.29	0.68	1. “EtOH cirrhosis c/b ascites, varices,” HCC s/p liver transplant 2. immunosuppressed “on tacrolimus” 3. “possible infection as precipitant”
91	0.37	0.69	1. “Hypoxic respiratory failure” 2. right-lower-lobe consolidation that “could be acute pneumonia”
75	0.47	0.55	1. S/p “subdural hematoma evacuation,” two burr holes 2. collection decreased: 19 vs. 26 mm
45	0.14	0.63	1. C7 fracture with “extensive edema” 2. study “substantially degraded by patient motion” (agitation)

TABLE II: Representative cases the structured-only model misses but DeliMF catches (true label: delirium). The last column quotes verbatim phrases from the first-24 h radiology report, showing that DeliMF recovers true delirium cases by extracting additional information.

Clinical Prediction

Clinical prediction uses patient records, signals, and images to estimate diagnoses, risks, outcomes, and treatment responses for care.

See all 206 papers

Multimodal Deep Learning for Early Prediction of Patient Deterioration in the ICU: Integrating Time-Series EHR Data with Clinical Notes
Mar 16, 2026Binesh SadanandanClinical Risk Prediction Longitudinal EHR Modeling
Risk Stratification for ICU Delirium using Pervasive Ambient Sensing Information
Jun 17, 2026Jiaqing Zhang, Sabyasachi Bandyopadhyay, Miguel Contreras +8Clinical Prediction Clinical Risk Prediction
Department of Electrical and Computer Engineering, University of Florida, Gainesville, United States · Department of Medicine, Stanford University, Stanford, United States · Department of Biomedical Engineering, University of Florida, Gainesville, United States +1
OC-Distill: Ontology-aware Contrastive Learning with Cross-Modal Distillation for ICU Risk Prediction
Apr 18, 2026Zhongyuan Liang, Junhyung Jo, Hyang-Jung Lee +2Contrastive Learning Cross-Modal Knowledge Distillation
Computational Precision Health UC Berkeley, UCSF · Samsung Advanced Institute of Technology (SAIT) · Computational Precision Health UC Berkeley, UCSF[cs.LG]

Mar 16, 2026J/K move · Enter open · S save

Multimodal Deep Learning for Early Prediction of Patient Deterioration in the ICU: Integrating Time-Series EHR Data with Clinical Notes

Binesh Sadanandan

Clinical Risk Prediction Longitudinal EHR Modeling Multimodal Learning

Open paper AskPDF arXiv

Early identification of patients at risk for clinical deterioration in the intensive care unit (ICU) remains a critical challenge. Delayed recognition of impending adverse events, including mortality, vasopressor initiation, and mechanical ventilation, contributes to preventable morbidity and mortality. We present a multimodal deep learning approach that combines structured time-series data (vital signs and laboratory values) with unstructured clinical notes to predict patient deterioration within 24 hours. Using the MIMIC-IV database, we constructed a cohort of 74,822 ICU stays and generated 5.7 million hourly prediction samples. Our architecture employs a bidirectional LSTM encoder for temporal patterns in physiologic data and ClinicalBERT embeddings for clinical notes, fused through a cross-modal attention mechanism. We also present a systematic review of existing approaches to ICU deterioration prediction, identifying 31 studies published between 2015 and 2024. Most existing models rely solely on structured data and achieve area under the curve (AUC) values between 0.70 and 0.85. Studies incorporating clinical notes remain rare but show promise for capturing information not present in structured fields. Our multimodal model achieves a test AUROC of 0.7857 and AUPRC of 0.1908 on 823,641 held-out samples, with a validation-to-test gap of only 0.6 percentage points. Ablation analysis validates the multimodal approach: clinical notes improve AUROC by 2.5 percentage points and AUPRC by 39.2% relative to a structured-only baseline, while deep learning models consistently outperform classical baselines (XGBoost AUROC: 0.7486, logistic regression: 0.7171). This work contributes both a thorough review of the field and a reproducible multimodal framework for clinical deterioration prediction.

Abstract

Clinical Prediction

Explore similar work

Multimodal Deep Learning for Early Prediction of Patient Deterioration in the ICU: Integrating Time-Series EHR Data with Clinical Notes

Risk Stratification for ICU Delirium using Pervasive Ambient Sensing Information

OC-Distill: Ontology-aware Contrastive Learning with Cross-Modal Distillation for ICU Risk Prediction