Delirium affects a large fraction of intensive care unit (ICU) patients and foreshadows longer stays, lasting cognitive decline, and higher mortality. Flagging the patients who will become delirious within the first day of admission would give clinicians a window to intervene. Yet state-of-the-art predictors rely only on the structured electronic health record (EHR) of vital signs, laboratory values, and medications, while overlooking the free-text clinical notes written in the same window, which carry information the numbers cannot encode. To this end, we present DeliMF, a multimodal framework that fuses the structured time series with clinical text. It encodes the structured signals with a temporal Transformer and the time-stamped radiology reports with a clinical text encoder, and fuses the two through a gated cross-attention module. The gate learns a per-patient weight on the text; when a patient has no note the gate closes and DeliMF reduces to its structured-only form, giving robustness to missing text by construction. On an incident-only MIMIC-IV cohort, DeliMF improves significantly over the baselines, particularly on patients who carry clinical text, while losing nothing on those without. Ablation studies confirm the contribution of multimodal fusion and gated cross-attention, and a case study further shows that fusing clinical text with the structured stream recovers delirium cases that the structured data alone misses. We hope DeliMF encourages early ICU prediction to make fuller use of the overlooked clinical text and contributes to the early identification and timely intervention of delirium.
Figures & tables
Fig. 1: Overview of DeliMF , traced through a real MIMIC-IV case (61-year-old admitted after a fall, incident-delirium label). The structured tower embeds each event from its value, feature id, and timestamp and encodes the sequence with a temporal Transformer, pooled to hs . The text tower encodes the first-24 h radiology report with ClinicalBERT, pooled to ht . In the gated cross-attention module, the structured states query the text, and a per-patient gate weights the two representations into the fused hf ; when no note is available the gate closes and DeliMF reduces to its structured-only form. A two-layer MLP head maps hf to the delirium probability. For this case the report names a subarachnoid hemorrhage with diffuse axonal injury, precipitants absent from the tabular record, and DeliMF raises the probability to 0.56 while the structured-only baseline assigns 0.21 and misses the case.
Fig. 2: Data and cohort overview. (a) Cohort construction from all brain-status-assessed ICU stays down to the incident-only cohort of 23,431; the final (red) bar is the analyzed cohort. (b) Age distribution (bars) and delirium rate (line), which rises monotonically with age. (c) Text availability on the test set: 55.4% of stays carry a radiology report, and the with-text subgroup has a higher positive rate. (d) Distribution of structured events per stay (mean 217).
Model
AUROC
AUPRC
F1
Sens.
Spec.
Brier
Comparison with prior methods
DeliMF (ours)
0.803 ± 0.003
0.360 ± 0.012
0.391 ± 0.018
0.718 ± 0.036
0.762 ± 0.037
0.162 ± 0.016
DeLLiriuM-style
0.721 ± 0.000
0.242 ± 0.001
0.316 ± 0.006
0.545 ± 0.037
0.770 ± 0.032
0.182 ± 0.015
E-PRE-DELIRIC-style
0.692 ± 0.000
0.189 ± 0.000
0.283 ± 0.000
0.624 ± 0.000
0.666 ± 0.000
0.228 ± 0.000
Single-modality ablation
Structured-only
0.800 ± 0.002
0.334 ± 0.007
0.390 ± 0.003
0.663 ± 0.057
0.791 ± 0.031
0.148 ± 0.013
TABLE I: Comparison on the MIMIC-IV test set (3 seeds, mean ± std; threshold 0.5). The table is organized into blocks: comparison with prior methods, single-modality ablation, fusion design, training schedule, and encoder substitution. Our full model is shaded; best per column in bold blue (lower is better for Brier).
Fig. 3: Subset analysis across text availability and age (3 seeds, mean ± std). (a) AUROC and (b) AUPRC for DeliMF, the structured-only model, and two structured-input baselines. DeliMF’s AUPRC gain concentrates on the with-text and elderly subgroups, while it matches structured-only when no note is present. The DeLLiriuM-style and E-PRE-DELIRIC-style baselines use only structured inputs, so they are unaffected by note availability.
Fig. 4: Gate value distribution over patients with text. The mean of 0.45 and the left tail show per-patient adaptive reliance on the text modality.
Age
Struct.
DeliMF
Evidence in the radiology report
61
0.21
0.56
1. Parietal hemorrhage enlarging: “ 12×12 mm” vs. “ 8×5.5 mm” prior 2. “consistent with diffuse axonal injury”
1. “Hypoxic respiratory failure” 2. right-lower-lobe consolidation that “could be acute pneumonia”
75
0.47
0.55
1. S/p “subdural hematoma evacuation,” two burr holes 2. collection decreased: 19 vs. 26 mm
45
0.14
0.63
1. C7 fracture with “extensive edema” 2. study “substantially degraded by patient motion” (agitation)
TABLE II: Representative cases the structured-only model misses but DeliMF catches (true label: delirium). The last column quotes verbatim phrases from the first-24 h radiology report, showing that DeliMF recovers true delirium cases by extracting additional information.
Early identification of patients at risk for clinical deterioration in the intensive care unit (ICU) remains a critical challenge. Delayed recognition of impending adverse events, including mortality, vasopressor initiation, and mechanical ventilation, contributes to preventable morbidity and mortality. We present a multimodal deep learning approach that combines structured time-series data (vital signs and laboratory values) with unstructured clinical notes to predict patient deterioration within 24 hours. Using the MIMIC-IV database, we constructed a cohort of 74,822 ICU stays and generated 5.7 million hourly prediction samples. Our architecture employs a bidirectional LSTM encoder for temporal patterns in physiologic data and ClinicalBERT embeddings for clinical notes, fused through a cross-modal attention mechanism. We also present a systematic review of existing approaches to ICU deterioration prediction, identifying 31 studies published between 2015 and 2024. Most existing models rely solely on structured data and achieve area under the curve (AUC) values between 0.70 and 0.85. Studies incorporating clinical notes remain rare but show promise for capturing information not present in structured fields. Our multimodal model achieves a test AUROC of 0.7857 and AUPRC of 0.1908 on 823,641 held-out samples, with a validation-to-test gap of only 0.6 percentage points. Ablation analysis validates the multimodal approach: clinical notes improve AUROC by 2.5 percentage points and AUPRC by 39.2% relative to a structured-only baseline, while deep learning models consistently outperform classical baselines (XGBoost AUROC: 0.7486, logistic regression: 0.7171). This work contributes both a thorough review of the field and a reproducible multimodal framework for clinical deterioration prediction.
Delirium is a common and serious complication in the Intensive Care Unit (ICU), associated with increased morbidity, prolonged hospital stays, and higher healthcare costs. Despite its prevalence, early prediction and prevention remain challenging. Environmental factors such as ambient sound and light may influence the onset of delirium, yet they are often overlooked in risk assessments. In this study, we examined whether light intensity and sound pressure levels can independently predict delirium across multiple prediction horizons. We evaluated four efficient sequential neural network models on data collected from 9 ICUs across 309 patients to predict delirium for 10 prediction-window sizes. We reported feature importance and direction of influence using Shapley Additive Explanations analysis. The convolutional model achieved the strongest discrimination, with AUC = 0.80 on sound data and on combined data. Sound features were the dominant predictors overall. Integrating sound with light improved short-term (<1 week) prediction, with the combined model assigning the highest risk immediately after the sensing period. These findings suggest that passive ambient sensing, especially sound, can add a clinically meaningful, interpretable signal for delirium risk estimation and offer a practical pathway to enrich multimodal ICU prediction and prevention strategies.
Jiaqing Zhang, Sabyasachi Bandyopadhyay, Miguel Contreras +8
Department of Electrical and Computer Engineering, University of Florida, Gainesville, United States · Department of Medicine, Stanford University, Stanford, United States · Department of Biomedical Engineering, University of Florida, Gainesville, United States +1
Early prediction of severe clinical deterioration and remaining length of stay can enable timely intervention and better resource allocation in high-acuity settings such as the ICU. This has driven the development of machine learning models that leverage continuous streams of vital signs and other physiological signals for real-time risk prediction. Despite their promise, existing methods have important limitations. Contrastive pretraining treats all patients as equally strong negatives, failing to capture clinically meaningful similarity between patients with related diagnoses. Meanwhile, downstream fine-tuning typically ignores complementary modalities such as clinical notes, which provide rich contextual information unavailable in physiological signals alone. To address these challenges, we propose OC-Distill, a two-stage framework that leverages multimodal supervision during training while requiring only vital signs at inference. In the first stage, we introduce an ontology-aware contrastive objective that exploits the ICD hierarchy to quantify patient similarity and learn clinically grounded representations. In the second stage, we fine-tune the pretrained encoder via cross-modal knowledge distillation, transferring complementary information from clinical notes into the model. Across multiple ICU prediction tasks on MIMIC, OC-Distill demonstrates improved label efficiency and achieves state-of-the-art performance among methods that use only vital signs at inference.
Zhongyuan Liang, Junhyung Jo, Hyang-Jung Lee +2
Computational Precision Health UC Berkeley, UCSF · Samsung Advanced Institute of Technology (SAIT) · Computational Precision Health UC Berkeley, UCSF[cs.LG]