Clinical prognosis requires integrating structured measurements, physiological waveforms, longitudinal history, and population context. We introduce PRISM (Population-Retrieved Inference for Structured Multimodal Prognosis), a multimodal language model that combines per-feature numerical tokenization, recency-weighted aggregation of longitudinal ECGs, and a dual-modality population memory bank. Retrieval operates in representation space: the model receives mean-pooled embeddings of similar training-cohort encounters, while no individual retrieved record enters the prompt. On the 1,443-task MDS-ED benchmark, PRISM achieves AUROCs of 86.2% for diagnosis, 93.1% for deterioration, 91.8% for ICU admission, and 92.8% for mortality. These results use a 96-hour pre-arrival window unavailable to reported baselines. With benchmark-window inputs only, PRISM reaches 0.8895 AUROC, compared with 0.8937 for UniPACT and 0.8889 for MDS-ED. A matched-input compact fusion model using the same five evidence streams and frozen encoders reaches 0.7652 AUROC, versus 0.9092 for PRISM. Replacing numerical EHR encoding with text serialization reduces AUROC by 3.66 points; removing pre-arrival history reduces it by 1.97 points; and zeroing retrieval reduces it by 0.99 points. Genuine retrieval outperforms random-neighbor retrieval by 0.97 macro-average points, although a paired task-level sign test does not establish a uniform advantage (p=0.067). On the external MC-MED cohort, supervised fine-tuning reaches 0.8611 AUROC, compared with 0.8227 for locally trained gradient boosting and 0.7841 for source-only PRISM. These results support structure-aware multimodal integration and supervised external adaptation.
Figures & tables
Figure 1: Overview of the PRISM architecture. Five modality streams (structured EHR, current ECG, longitudinal ECG history, retrieved EHR context, and retrieved ECG context) are independently encoded and projected as soft tokens into a shared sequence consumed by a LoRA-adapted MedGemma-4B decoder. The population memory bank (center) retrieves the k=3 nearest training-fold encounters from dual EHR and ECG indices constructed after Stage 1 and passes their mean-pooled embeddings, not their outcome labels, to the decoder. The temporal history aggregator (right) pools prior ECG embeddings under exponential recency weighting. Each task question is answered in one forward pass, via either the generative or the classification pathway, with no task-specific heads.
Method
Tasks · AUROC (%)
Diagnosis
Deterioration
ICU
Mortality
ECG Foundation Models · waveform input
HeartBERT ( Tahery et al., 2025 )
56.35
58.24
57.28
61.11
ECGFM-KED ( Tian et al., 2024 )
58.75
64.76
61.32
65.76
ESI ( Yu et al., 2024 )
60.98
63.34
62.52
66.28
ECGFounder ( Li et al., 2025 )
61.52
69.35
64.13
69.42
Table 1: Benchmark results across all 1,443 MDS-ED tasks. AUROC (%) is macro-averaged within each task group: Diagnosis (1,428 ICD-10 discharge tasks), Deterioration (6 tasks), ICU admission (2 tasks), and Mortality (7 tasks). PRISM has the highest reported values among the listed methods in each task group. Its full-input configuration additionally uses 96-hour pre-arrival history; the matched-window comparison is reported in the text. Bold with underline : best. † Zero-shot, text-only input.
Model / condition
Diagnosis
Deterioration
ICU
Mortality
Overall AUROC
Source-only transfer
PRISM, original bank
0.7086
0.7112
0.8188
0.8977
0.7841 [0.751, 0.814]
Local-bank adaptation (no parameter updates)
PRISM, MC-MED bank
0.7080
0.7133
0.8193
0.9025
0.7858 [0.754, 0.815]
Supervised local training / adaptation
Gradient boosting
0.7012
0.7509
0.8924
0.9465
0.8227 [0.796, 0.851]
Table 2: External evaluation on MC-MED. All entries are AUROC; PRISM uses the generative pathway. Overall AUROC is the unweighted mean of the four task-group means. Brackets show reported 95% patient-bootstrap confidence intervals. The first two rows use no target-site parameter training. The final two rows perturb retrieval at inference using the fine-tuned checkpoint and identical evaluation records. Bold identifies the highest point estimate among the supervised rows, not statistical significance.
Figure 2: PRISM performance vs. MDS-ED baseline across clinical prediction categories. Comparison of AUROC for (a) mortality at multiple horizons, (b) clinical deterioration endpoints, (c) ICU admission, and (d) ICD-chapter-level diagnosis groups. Red and blue markers denote PRISM and MDS-ED, respectively. Green annotations indicate relative AUROC improvement. Gains are largest on laboratory-dependent outcomes (e.g., blood and immune disorders, musculoskeletal conditions) and on deterioration endpoints where metabolic measurements are clinically relevant.
Figure 3: Qualitative analysis of the population memory bank. A test encounter with ICD-10 label Z79 (long-term drug therapy, positive outcome) and three training-fold encounters selected from the dual-modality memory bank for illustration. The EHR and ECG indices are queried independently; the displayed encounters are returned by both indices. The model itself receives the two mean-pooled retrieval vectors defined in Equation 5 , rather than the individual encounters shown here. All displayed encounters share the same discharge label and outcome. Similarity values are reported within the index from which they are obtained and are not directly comparable across modalities. Shared abnormalities, including elevated glucose and heart rate, illustrate local clinical coherence in the learned embedding space.
Ablation / configuration
AUROC
Δ
Encoder ablations
Text-serialized EHR (no tokenization) †
0.8726
−3.66
w/o EHR encoder
0.8874
−2.20
w/o current-visit ECG †
0.8931
−1.61
w/o longitudinal ECG history †
0.8970
−1.22
Memory bank ablations, three-seed reference
Table 3: Architectural and training ablations. AUROC is reported as macro-of-macro over the four task groups under the generative pathway. Rows marked † use the three-seed full-model reference ( 0.9092±0.0032 ); unmarked rows use the single-run reference (0.9094). Individual ablations are single training runs unless stated otherwise. Δ is calculated against the corresponding full-model reference.
Model
LLM
AUROC
AUPRC
ECE
Params
k NN outcome prevalence ( k=50 )
No
0.5262
0.1235
–
0
FT-Transformer, structured EHR only
No
0.6870±0.0058
0.1864
0.4362
3.1M
ECG and longitudinal history only
No
0.7527±0.0031
0.2378
0.4308
4.1M
Fusion, EHR + ECG + history
No
0.7759±0.0051
0.2727
0.4261
4.6M
Fusion + retrieval (all five streams)
No
0.7652±0.0043
0.2500
0.4043
5.9M
PRISM
Yes
0.9092±0.0032
0.5504
0.1134
4B
Table 4: Architectures with and without a language model. The fusion+retrieval control is matched to PRISM on all five input streams; the remaining compact controls use the subsets indicated in their row labels. Frozen upstream encoders are shared where applicable, and all models use the same test fold and evaluation protocol. AUROC is reported as mean ± standard deviation over five seeds for the compact models and over three seeds for PRISM. Expected calibration error uses ten equal-count bins weighted by bin size. Average precision is reported at true cohort prevalence. Parameter counts are totals including frozen components.
Configuration
AUROC
AUPRC
ECE
Δ
Structured EHR as serialized text
0.8726
0.4632
0.1586
−3.66
w/o 96-hour pre-arrival window
0.8895
0.5358
0.1290
−1.97
w/o current-visit ECG
0.8931
0.5031
0.1266
−1.61
w/o longitudinal ECG history
0.8970
0.5418
0.1260
−1.22
w/o retrieval (tokens zeroed)
0.8993
0.5450
0.1239
−0.99
Retrieval with random neighbors
0.8995
0.5470
0.1224
−0.97
Table 5: Contribution of each evidence stream. One evidence stream is removed or replaced per row, with the remaining model components held fixed. Δ is the change in AUROC relative to full PRISM, in points. Random-neighbor and zeroed-token controls retain the retrieval architecture while varying the content supplied through it.
ECG Encoder
Diagnosis
Deterioration
ICU
Mortality
HeartBERT ( Tahery et al., 2025 )
83.65
91.07
90.79
91.61
ECGFM-KED ( Tian et al., 2024 )
82.71
90.57
90.27
90.90
ECGFounder ( Li et al., 2025 )
82.54
91.71
90.60
90.73
D-BETA (PRISM) ( Hung et al., 2025 )
86.24
93.12
91.79
92.82
Δ vs. next-best
+2.59
+1.41
+1.00
+1.21
Table 6: ECG encoder backbone ablation. All other architectural components and hyperparameters are held fixed; only the waveform encoder varies. Best AUROC (%) across both inference pathways is reported per task group. Values are per-task-group percentages from the original single-run evaluation, on the same scale as Table 1 and not on the macro-of-macro scale used in Tables 4 , 5 and 3 . Δ denotes the gap between D-BETA and the next-best encoder. Bold : best per column.
Group
Tasks
Rows
AUROC
AP@eval
AP@true
Brier
ECE
PRISM gen
Diagnoses
1,428
253,055
0.8619
0.8532
0.3400
0.1720
0.1558
Deterioration
6
729
0.9291
0.9357
0.5855
0.1186
0.1183
ICU
2
3,476
0.9180
0.9204
0.7265
0.1383
0.0954
Mortality
7
4,898
0.9279
0.9172
0.5495
0.1126
0.0840
Macro-of-macro
1,443
0.9092
0.5504
0.1134
Table 7: Per-group discrimination, average precision and calibration. AP@eval is measured at the retained evaluation prevalence after ECG exclusions; its chance reference is the retained positive fraction for each task. AP@true rescales to the cohort prevalence of each endpoint. ECE uses ten equal-count bins weighted by bin size.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Stage 1
Stage 2
LLM backbone
MedGemma-4B (frozen)
MedGemma-4B + LoRA
LoRA rank / scaling
–
16 / 32
LoRA dropout
–
0.05
LoRA target modules
–
q, k, v, o, gate, up, down
EHR encoder dmodel
192
frozen
EHR encoder layers / heads
3 / 4
frozen
Appendix
Table 8: Training hyperparameters for PRISM Stage 1 and Stage 2.
Modality stream
Input dim.
Stage
Current-visit ECG
decg
1
Structured EHR
512
1
Longitudinal ECG history
512
1
Retrieved EHR context
512
2
Retrieved ECG context
decg
2
Appendix
Table 9: Soft-token projector specifications. All projectors have a hidden dimension of 1,024 and produce an output shape of 8×2,560 (MedGemma-4B embedding size).
Property
EHR Index
ECG Index
Vector dimension
512
decg
Distance metric
Cosine similarity
Cosine similarity
Indexed encounters
116,186
116,186
Neighbors retrieved ( k )
3
3
Context aggregation
Mean pooling
Mean pooling
Index construction
After Stage 1
After Stage 1
Appendix
Table 10: Population memory bank configuration for PRISM.
Group
Endpoint
Test n
PRISMdisc [95% CI]
PRISMgen [95% CI]
Deterioration
Cardiac arrest
68
0.9706 [0.9187, 0.9965]
0.9870 [0.9627, 1.0000]
Inotrope administration
27
0.9615 [0.8765, 1.0000]
0.9286 [0.8076, 1.0000]
Mechanical ventilation
437
0.9597 [0.9438, 0.9759]
0.9666 [0.9517, 0.9789]
ECMO cannulation
18
0.9383 [0.8051, 1.0000]
0.9383 [0.7916, 1.0000]
Vasopressor initiation
125
0.9286 [0.8717, 0.9701]
0.9247 [0.8742, 0.9672]
Severe hypoxemia
54
0.8285 [0.7069, 0.9293]
0.8285 [0.7129, 0.9286]
Appendix
Table 11: Per-task AUROC for all Deterioration, ICU, and Mortality tasks on the held-out test set, with 95% bootstrap confidence intervals (1,000 resamples). The stronger pathway per task is bolded . Point estimates are computed on the retained sampled evaluation records; the bootstrap estimates uncertainty. The macro-averages in the final three rows are the exact means of the per-task values above them and reconcile with the corresponding entries of Table 1 .
ICD-10 Chapter
Ch.
N
Mean
Range
Pregnancy, childbirth, puerperium
O
3
0.9825
0.9475–1.0000
Certain infectious diseases
A
15
0.9189
0.8493–0.9835
Blood and immune disorders
D
50
0.8952
0.6600–1.0000
Circulatory system
I
181
0.8900
0.7128–1.0000
Poisoning and toxic effects
T
43
0.8888
0.6317–0.9889
Neoplasms
C
48
0.8886
0.2800–1.0000
Appendix
Table 12: Diagnosis AUROC by ICD-10 chapter ( PRISMgen ), sorted by mean AUROC. N : number of ICD-10 codes per chapter.
AUROC Range
Tasks
Proportion
Cumulative
≥0.90
506
35.4%
35.4%
0.80 – 0.89
613
42.9%
78.4%
0.70 – 0.79
260
18.2%
96.6%
0.60 – 0.69
41
2.9%
99.4%
<0.60
8
0.6%
100%
Total
1,428
100%
Appendix
Table 13: Distribution of per-task AUROC across all 1,428 diagnosis tasks ( PRISMdisc ). Cumulative percentage is computed from the highest bin downward.
Prior ECGs
Encounters
Share
0
5,752
89.33%
1
565
8.77%
2
86
1.34%
3
26
0.40%
4
7
0.11%
5
2
0.03%
Appendix
Table 14: Prior ECG acquisitions per test encounter within the 96-hour window.
Quantity
Value
Initialization of λ
1.0000
Converged ∣λ∣
0.9891
Change from initialization
1.1%
Half-life τln2/∣λ∣
67.3 h
τ/∣λ∣
97.1 h
Weight retained at 96 h
0.372
Appendix
Table 15: Converged temporal decay parameter and derived quantities.
Aggregation rule
All encounters
≥2 prior ECGs
Oldest acquisition only
0.9083
0.8903
Most recent acquisition only
0.9087
0.8998
Uniform average
0.9091
0.9026
Fixed decay, half-life 1 h
0.9090
0.9041
Fixed decay, half-life 2 h
0.9089
0.9069
Learned decay (PRISM)
0.9092
0.9136
Appendix
Table 16: Alternative rules for pooling prior ECG embeddings.
Metric
Value
Unique patients
71,098
ECG-linked encounters
129,095
Training samples (task instances)
4.5M
Training set
116,186
Validation set
6,455
Test set
6,454
Appendix
Table 17: MDS-ED cohort and partition statistics.
Group
Tasks
Examples / Endpoints
Diagnosis
1,428
Full ICD-10 nosology (circulatory, musculoskeletal, etc.)
Deterioration
6
Intubation, cardiac arrest, vasopressors, severe hypoxemia
ICU Admission
2
Overall ICU admission, admission within 24 hours
Mortality
7
In-hospital; within 1, 7, 28, 90, 180, 365 days
Appendix
Table 18: Summary of the 1,443 binary clinical tasks predicted by PRISM.
Attribute
Subgroup
Patients
Rows
Event rate
AUROC [95% CI]
ECE
Sex
Female
3,329
132,751
0.478
0.9177 [0.9037, 0.9270]
0.1143
Male
3,027
129,407
0.523
0.8974 [0.8738, 0.9125]
0.1442
Age
< 40
1,028
30,289
0.265
0.9277 [0.8867, 0.9481]
0.0962
40–54
1,073
38,031
0.416
0.9240 [0.9054, 0.9363]
0.1153
55–69
1,938
82,045
0.522
0.8969 [0.8745, 0.9141]
0.1378
70–84
1,598
76,755
0.565
0.8868 [0.8602, 0.9064]
0.1356
Appendix
Table 19: PRISM performance stratified by sex, age band and self-reported ethnicity. Confidence intervals are bootstrap estimates over patients. Of the twenty-one pairwise contrasts within attributes, twenty have overlapping intervals; the exception is the 40–54 band against the 85+ band.
Cat.
Analyte
Lo
Hi
Unit
Cat.
Analyte
Lo
Hi
Unit
Vitals
Temperature
36.0
38.0
∘ C
Vitals
Systolic BP
90
140
mmHg
Heart rate
60
100
bpm
Diastolic BP
60
90
mmHg
Resp. rate
12
20
br/min
O 2 sat.
95
–
%
Hem.
WBC
4.5
11.0
103 /µL
Hem.
Neutrophils
50
70
%
hemoglobin
12.0
17.5
g/dL
Lymphocytes
20
40
%
hematocrit
36
50
%
Eosinophils
1.0
4.0
%
Appendix
Table 20: Clinical reference intervals used for EHR abnormality flagging. Open bounds are indicated by a dash. Applied in conjunction with a population z-score threshold of ∣z∣>2.0 (Section K.2 ).
Figure 4: Population memory bank retrieval across four clinical phenotypes (Cases A–D). For each test encounter, the 50 nearest neighbors of the EHR index are used, so every percentage below is computed against a denominator of 50; clinical status (PRESENT/ABSENT) reflects the ground-truth outcome label of the index patient, and percentages report outcome prevalence among the 50 retrieved neighbors.
Setting
Partition
Encounters/visits
Patients
Temporal
Training
67,562
43,238
Validation
3,524
2,342
Test
1,435
1,002
Memory bank
71,086
45,580
MC-MED
Training
31,239
22,908
Validation
4,579
3,982
Appendix
Table 21: Cohort sizes for external and temporal evaluation. Counts describe the underlying visit/encounter tables, rather than the sampled encounter–task records. Banks comprise training and validation rows.
Task group
Retrieval zeroed
Random retrieval
Diagnosis
0.7920 [0.779, 0.807]
0.7919 [0.778, 0.806]
Deterioration
0.7389 [0.641, 0.831]
0.7310 [0.628, 0.828]
ICU
0.8681 [0.830, 0.903]
0.8657 [0.826, 0.901]
Mortality
0.9215 [0.887, 0.949]
0.9236 [0.889, 0.950]
Overall
0.8301 [0.802, 0.857]
0.8280 [0.799, 0.856]
Appendix
Table 22: MC-MED retrieval controls. AUROC and 95% patient-level percentile-bootstrap confidence intervals for inference-time perturbations of fine-tuned PRISM. Overall is the unweighted mean of the four task-group AUROCs.
Model / condition
Diagnosis
Deterioration
ICU
Mortality
Overall AUROC
Fusion MLP
0.6617
0.8049
0.7858
0.8153
0.7669 [0.730, 0.801]
Fusion MLP + retrieval
0.6488
0.7941
0.7798
0.7935
0.7541 [0.716, 0.788]
PRISM
0.8119
0.8501
0.8726
0.9004
0.8587 [0.821, 0.892]
PRISM, retrieval zeroed
0.8064
0.8431
0.8665
0.8957
0.8529 [0.815, 0.887]
PRISM, random neighbors
0.8099
0.8525
0.8717
0.8988
0.8582 [0.821, 0.891]
Appendix
Table 23: Temporal generalization to later encounters. All entries are AUROC; PRISM uses the generative pathway. Brackets show reported 95% patient-bootstrap confidence intervals. The later-era records are retained without rebalancing after the original test-set ECG exclusions. Zeroed and random retrieval are inference-time controls. Bold indicates the highest point estimate in each column, not statistical significance.
Emergency department (ED) decision-making relies on heterogeneous clinical information, including patient history, vital signs, laboratory results, and electrocardiograms (ECGs). Vision--language models (VLMs) can jointly process these modalities, but strong predictive performance does not necessarily imply meaningful use of the correct patient's ECG. We term this failure mode ECG Mirage: apparent multimodal capability without useful dependence on patient-specific ECG information. We distinguish two forms: ECG neglect, where ECGs provide little predictive benefit, and ECG confusion, where matched ECGs outperform no-image inputs but not mismatched ECGs. To evaluate these behaviours, we compare predictions obtained with matched ECGs, outcome-discordant mismatched ECGs, and no-image inputs while holding the clinical text and prediction targets fixed. Across four VLMs on MDS-ED, matched ECGs provide no consistent advantage for either ICU admission or clinical deterioration prediction. We then train four restricted visual prompts using supervised learning followed by conditional direct preference optimisation, while keeping the VLM backbone frozen. The resulting models achieve balanced accuracies of 70.6% for ICU admission and 67.5% for deterioration and increase the matched-versus-mismatched performance gap to approximately 16.5 and 5.5 percentage points, respectively. Overall, our study identifies ECG Mirage in multimodal clinical prediction and introduces visual prompt tuning as an efficient mitigation strategy.
Autoregressive foundation models trained on tokenized electronic health records (EHRs) can support zero-shot clinical prediction, yet most operate on structured event codes alone, and do not incorporate multiple modalities in a principled way. We present a framework for conditioning such models on auxiliary clinical modalities, including ECG waveforms, chest X-ray images, and clinical notes, using modality-specific latent compression and gated cross-attention with temporal alignment. We investigate two key design choices: (1) how to compress long per-modality sequences (e.g., ECG time series) before they enter the multi-modal cross-attention. This feature may be essential to reduce compute overheads and may be beneficial for generalization; (2) how the choice of pretrained encoder for each modality impacts downstream performance. Through controlled ablations on MIMIC-IV, we show that the best latent-compression configurations outperforms both uncompressed cross-attention and mean pooling. Encoder choice has a clear within-modality effect, with stronger pretrained encoders consistently outperforming weaker alternatives. We further show that merely adding auxiliary modalities does not guarantee improvement on ICU mortality prediction over an EHR-only baseline. This implies that careful design of the fusion architecture and an appropriate evaluation in the clinical context are required.
Yuxuan Liu, Joshua Placidi, Jinpei Han +3
Dept. of Computing, Imperial College London, London, United Kingdom · UKRI Centres in AI for Health, United Kingdom · School of Public Health, Imperial College London, London, United Kingdom +2
Accurate assessment of patients in intensive care units (ICUs) is essential for timely clinical intervention and improved patient outcomes. Multimodal electronic health records (EHRs), including structured physiological time series and longitudinal clinical notes, provide complementary information for critical care prediction. However, in real-world clinical settings, individual modalities may be partially observed or entirely unavailable, resulting in substantial performance degradation for existing multimodal models. To address this challenge, we propose a multimodal prompt-learning framework for robust clinical prediction under diverse missing-modality scenarios. The proposed framework introduces four complementary types of prompts: generative prompts, missing-signal prompts, missing-type prompts, and temporal prompts. Generative prompts construct surrogate latent representations for unavailable modalities, while missing-signal prompts distinguish observed representations from generated ones. Missing-type prompts condition the model on different modality-availability configurations, whereas temporal prompts perform condition-specific aggregation over temporally encoded clinical sequences. Together, these prompts enable the model to capture missingness-aware intramodal dependencies and cross-modal interactions within a unified architecture. Extensive experiments demonstrate that our method outperforms existing approaches across evaluation metrics on two missingness settings. Ablation and robustness analyses further verify the complementary contributions of the four prompt types and the effectiveness of the proposed framework for clinical prediction from incomplete multimodal EHR data.
Yixin Yang, Yueyang Sun, Weichen Liu +2
Faculty of Engineering, Shenzhen MSU-BIT University · International School, Beijing University of Posts and Telecommunications · School of Computer Science and Engineering, Southeast University +1