Retrieval-Augmented Multimodal Language Model with Heterogeneous Evidence Integration for Clinical Prognosis

Hamza Shafiq; Jialu Tang; Ting Dang; Jun Hu; Aaqib Saeed

cs.AIOct 9, 2026

Retrieval-Augmented Multimodal Language Model with Heterogeneous Evidence Integration for Clinical Prognosis

Authors: Hamza Shafiq, Jialu Tang, Ting Dang, Jun Hu, Aaqib Saeed

Organizations: Eindhoven University of Technology · University of Melbourne

Abstract

Clinical prognosis requires integrating structured measurements, physiological waveforms, longitudinal history, and population context. We introduce PRISM (Population-Retrieved Inference for Structured Multimodal Prognosis), a multimodal language model that combines per-feature numerical tokenization, recency-weighted aggregation of longitudinal ECGs, and a dual-modality population memory bank. Retrieval operates in representation space: the model receives mean-pooled embeddings of similar training-cohort encounters, while no individual retrieved record enters the prompt. On the 1,443-task MDS-ED benchmark, PRISM achieves AUROCs of 86.2% for diagnosis, 93.1% for deterioration, 91.8% for ICU admission, and 92.8% for mortality. These results use a 96-hour pre-arrival window unavailable to reported baselines. With benchmark-window inputs only, PRISM reaches 0.8895 AUROC, compared with 0.8937 for UniPACT and 0.8889 for MDS-ED. A matched-input compact fusion model using the same five evidence streams and frozen encoders reaches 0.7652 AUROC, versus 0.9092 for PRISM. Replacing numerical EHR encoding with text serialization reduces AUROC by 3.66 points; removing pre-arrival history reduces it by 1.97 points; and zeroing retrieval reduces it by 0.99 points. Genuine retrieval outperforms random-neighbor retrieval by 0.97 macro-average points, although a paired task-level sign test does not establish a uniform advantage (p=0.067). On the external MC-MED cohort, supervised fine-tuning reaches 0.8611 AUROC, compared with 0.8227 for locally trained gradient boosting and 0.7841 for source-only PRISM. These results support structure-aware multimodal integration and supervised external adaptation.

Figures & tables

Figure 1: Overview of the PRISM architecture. Five modality streams (structured EHR, current ECG, longitudinal ECG history, retrieved EHR context, and retrieved ECG context) are independently encoded and projected as soft tokens into a shared sequence consumed by a LoRA-adapted MedGemma-4B decoder. The population memory bank (center) retrieves the $k{=}3$ nearest training-fold encounters from dual EHR and ECG indices constructed after Stage 1 and passes their mean-pooled embeddings, not their outcome labels, to the decoder. The temporal history aggregator (right) pools prior ECG embeddings under exponential recency weighting. Each task question is answered in one forward pass, via either the generative or the classification pathway, with no task-specific heads.

Method	Tasks · AUROC (%)
Method	Diagnosis	Deterioration	ICU	Mortality
ECG Foundation Models · waveform input
HeartBERT ( Tahery et al., 2025 )	$56.35$	$58.24$	$57.28$	$61.11$
ECGFM-KED ( Tian et al., 2024 )	$58.75$	$64.76$	$61.32$	$65.76$
ESI ( Yu et al., 2024 )	$60.98$	$63.34$	$62.52$	$66.28$
ECGFounder ( Li et al., 2025 )	$61.52$	$69.35$	$64.13$	$69.42$

Table 1: Benchmark results across all 1,443 MDS-ED tasks. AUROC (%) is macro-averaged within each task group: Diagnosis (1,428 ICD-10 discharge tasks), Deterioration (6 tasks), ICU admission (2 tasks), and Mortality (7 tasks). PRISM has the highest reported values among the listed methods in each task group. Its full-input configuration additionally uses 96-hour pre-arrival history; the matched-window comparison is reported in the text. Bold with underline : best. † Zero-shot, text-only input.

Model / condition	Diagnosis	Deterioration	ICU	Mortality	Overall AUROC
Source-only transfer
PRISM, original bank	0.7086	0.7112	0.8188	0.8977	0.7841 [0.751, 0.814]
Local-bank adaptation (no parameter updates)
PRISM, MC-MED bank	0.7080	0.7133	0.8193	0.9025	0.7858 [0.754, 0.815]
Supervised local training / adaptation
Gradient boosting	0.7012	0.7509	0.8924	0.9465	0.8227 [0.796, 0.851]

Table 2: External evaluation on MC-MED. All entries are AUROC; PRISM uses the generative pathway. Overall AUROC is the unweighted mean of the four task-group means. Brackets show reported 95% patient-bootstrap confidence intervals. The first two rows use no target-site parameter training. The final two rows perturb retrieval at inference using the fine-tuned checkpoint and identical evaluation records. Bold identifies the highest point estimate among the supervised rows, not statistical significance.

Figure 2: PRISM performance vs. MDS-ED baseline across clinical prediction categories. Comparison of AUROC for (a) mortality at multiple horizons, (b) clinical deterioration endpoints, (c) ICU admission, and (d) ICD-chapter-level diagnosis groups. Red and blue markers denote PRISM and MDS-ED, respectively. Green annotations indicate relative AUROC improvement. Gains are largest on laboratory-dependent outcomes (e.g., blood and immune disorders, musculoskeletal conditions) and on deterioration endpoints where metabolic measurements are clinically relevant.

Figure 3: Qualitative analysis of the population memory bank. A test encounter with ICD-10 label Z79 (long-term drug therapy, positive outcome) and three training-fold encounters selected from the dual-modality memory bank for illustration. The EHR and ECG indices are queried independently; the displayed encounters are returned by both indices. The model itself receives the two mean-pooled retrieval vectors defined in Equation 5 , rather than the individual encounters shown here. All displayed encounters share the same discharge label and outcome. Similarity values are reported within the index from which they are obtained and are not directly comparable across modalities. Shared abnormalities, including elevated glucose and heart rate, illustrate local clinical coherence in the learned embedding space.

Ablation / configuration	AUROC	$\bm{\Delta}$
Encoder ablations
Text-serialized EHR (no tokenization) †	0.8726	$-3.66$
w/o EHR encoder	0.8874	$-2.20$
w/o current-visit ECG †	0.8931	$-1.61$
w/o longitudinal ECG history †	0.8970	$-1.22$
Memory bank ablations, three-seed reference

Table 3: Architectural and training ablations. AUROC is reported as macro-of-macro over the four task groups under the generative pathway. Rows marked

\dagger

use the three-seed full-model reference (

0.9092\pm 0.0032

); unmarked rows use the single-run reference (0.9094). Individual ablations are single training runs unless stated otherwise.

\bm{\Delta}

is calculated against the corresponding full-model reference.

Model	LLM	AUROC	AUPRC	ECE	Params
$k$ NN outcome prevalence ( $k{=}50$ )	No	0.5262	0.1235	–	0
FT-Transformer, structured EHR only	No	$0.6870\pm 0.0058$	0.1864	0.4362	3.1M
ECG and longitudinal history only	No	$0.7527\pm 0.0031$	0.2378	0.4308	4.1M
Fusion, EHR + ECG + history	No	$0.7759\pm 0.0051$	0.2727	0.4261	4.6M
Fusion + retrieval (all five streams)	No	$0.7652\pm 0.0043$	0.2500	0.4043	5.9M
PRISM	Yes	$\mathbf{0.9092\pm 0.0032}$	0.5504	0.1134	4B

Table 4: Architectures with and without a language model. The fusion+retrieval control is matched to PRISM on all five input streams; the remaining compact controls use the subsets indicated in their row labels. Frozen upstream encoders are shared where applicable, and all models use the same test fold and evaluation protocol. AUROC is reported as mean

\pm

standard deviation over five seeds for the compact models and over three seeds for PRISM. Expected calibration error uses ten equal-count bins weighted by bin size. Average precision is reported at true cohort prevalence. Parameter counts are totals including frozen components.

Configuration	AUROC	AUPRC	ECE	$\bm{\Delta}$
Structured EHR as serialized text	0.8726	0.4632	0.1586	$-3.66$
w/o 96-hour pre-arrival window	0.8895	0.5358	0.1290	$-1.97$
w/o current-visit ECG	0.8931	0.5031	0.1266	$-1.61$
w/o longitudinal ECG history	0.8970	0.5418	0.1260	$-1.22$
w/o retrieval (tokens zeroed)	0.8993	0.5450	0.1239	$-0.99$
Retrieval with random neighbors	0.8995	0.5470	0.1224	$-0.97$

Table 5: Contribution of each evidence stream. One evidence stream is removed or replaced per row, with the remaining model components held fixed.

\Delta

is the change in AUROC relative to full PRISM, in points. Random-neighbor and zeroed-token controls retain the retrieval architecture while varying the content supplied through it.

ECG Encoder	Diagnosis	Deterioration	ICU	Mortality
HeartBERT ( Tahery et al., 2025 )	83.65	91.07	90.79	91.61
ECGFM-KED ( Tian et al., 2024 )	82.71	90.57	90.27	90.90
ECGFounder ( Li et al., 2025 )	82.54	91.71	90.60	90.73
D-BETA (PRISM) ( Hung et al., 2025 )	86.24	93.12	91.79	92.82
$\Delta$ vs. next-best	+2.59	+1.41	+1.00	+1.21

Table 6: ECG encoder backbone ablation. All other architectural components and hyperparameters are held fixed; only the waveform encoder varies. Best AUROC (%) across both inference pathways is reported per task group. Values are per-task-group percentages from the original single-run evaluation, on the same scale as Table 1 and not on the macro-of-macro scale used in Tables 4 , 5 and 3 .

\Delta

denotes the gap between D-BETA and the next-best encoder. Bold : best per column.

Group	Tasks	Rows	AUROC	AP@eval	AP@true	Brier	ECE
PRISM ${}_{\textit{gen}}$
Diagnoses	1,428	253,055	0.8619	0.8532	0.3400	0.1720	0.1558
Deterioration	6	729	0.9291	0.9357	0.5855	0.1186	0.1183
ICU	2	3,476	0.9180	0.9204	0.7265	0.1383	0.0954
Mortality	7	4,898	0.9279	0.9172	0.5495	0.1126	0.0840
Macro-of-macro	1,443		0.9092		0.5504		0.1134

Table 7: Per-group discrimination, average precision and calibration. AP@eval is measured at the retained evaluation prevalence after ECG exclusions; its chance reference is the retained positive fraction for each task. AP@true rescales to the cohort prevalence of each endpoint. ECE uses ten equal-count bins weighted by bin size.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Stage 1	Stage 2
LLM backbone	MedGemma-4B (frozen)	MedGemma-4B + LoRA
LoRA rank / scaling	–	16 / 32
LoRA dropout	–	0.05
LoRA target modules	–	q, k, v, o, gate, up, down
EHR encoder $d_{\mathrm{model}}$	192	frozen
EHR encoder layers / heads	3 / 4	frozen

Appendix

Table 8: Training hyperparameters for PRISM Stage 1 and Stage 2.

Modality stream	Input dim.	Stage
Current-visit ECG	$d_{\mathrm{ecg}}$	1
Structured EHR	512	1
Longitudinal ECG history	512	1
Retrieved EHR context	512	2
Retrieved ECG context	$d_{\mathrm{ecg}}$	2

Appendix

Table 9: Soft-token projector specifications. All projectors have a hidden dimension of 1,024 and produce an output shape of

8\times 2{,}560

(MedGemma-4B embedding size).

Property	EHR Index	ECG Index
Vector dimension	512	$d_{\mathrm{ecg}}$
Distance metric	Cosine similarity	Cosine similarity
Indexed encounters	116,186	116,186
Neighbors retrieved ( $k$ )	3	3
Context aggregation	Mean pooling	Mean pooling
Index construction	After Stage 1	After Stage 1

Appendix

Table 10: Population memory bank configuration for PRISM.

Group	Endpoint	Test $n$	$\textbf{PRISM}_{\textbf{disc}}$ [95% CI]	$\textbf{PRISM}_{\textbf{gen}}$ [95% CI]
Deterioration	Cardiac arrest	68	0.9706 [0.9187, 0.9965]	0.9870 [0.9627, 1.0000]
	Inotrope administration	27	0.9615 [0.8765, 1.0000]	0.9286 [0.8076, 1.0000]
	Mechanical ventilation	437	0.9597 [0.9438, 0.9759]	0.9666 [0.9517, 0.9789]
	ECMO cannulation	18	0.9383 [0.8051, 1.0000]	0.9383 [0.7916, 1.0000]
	Vasopressor initiation	125	0.9286 [0.8717, 0.9701]	0.9247 [0.8742, 0.9672]
	Severe hypoxemia	54	0.8285 [0.7069, 0.9293]	0.8285 [0.7129, 0.9286]

Appendix

Table 11: Per-task AUROC for all Deterioration, ICU, and Mortality tasks on the held-out test set, with 95% bootstrap confidence intervals (1,000 resamples). The stronger pathway per task is bolded . Point estimates are computed on the retained sampled evaluation records; the bootstrap estimates uncertainty. The macro-averages in the final three rows are the exact means of the per-task values above them and reconcile with the corresponding entries of Table 1 .

ICD-10 Chapter	Ch.	$\bm{N}$	Mean	Range
Pregnancy, childbirth, puerperium	O	3	0.9825	0.9475–1.0000
Certain infectious diseases	A	15	0.9189	0.8493–0.9835
Blood and immune disorders	D	50	0.8952	0.6600–1.0000
Circulatory system	I	181	0.8900	0.7128–1.0000
Poisoning and toxic effects	T	43	0.8888	0.6317–0.9889
Neoplasms	C	48	0.8886	0.2800–1.0000

Appendix

Table 12: Diagnosis AUROC by ICD-10 chapter (

\text{PRISM}_{\text{gen}}

), sorted by mean AUROC.

N

: number of ICD-10 codes per chapter.

AUROC Range	Tasks	Proportion	Cumulative
$\geq 0.90$	506	35.4%	35.4%
$0.80$ – $0.89$	613	42.9%	78.4%
$0.70$ – $0.79$	260	18.2%	96.6%
$0.60$ – $0.69$	41	2.9%	99.4%
$<0.60$	8	0.6%	100%
Total	1,428	100%

Appendix

Table 13: Distribution of per-task AUROC across all 1,428 diagnosis tasks (

\text{PRISM}_{\text{disc}}

). Cumulative percentage is computed from the highest bin downward.

Prior ECGs	Encounters	Share
0	5,752	89.33%
1	565	8.77%
2	86	1.34%
3	26	0.40%
4	7	0.11%
5	2	0.03%

Appendix

Table 14: Prior ECG acquisitions per test encounter within the 96-hour window.

Quantity	Value
Initialization of $\lambda$	1.0000
Converged $\|\lambda\|$	0.9891
Change from initialization	1.1%
Half-life $\tau\ln 2/\|\lambda\|$	67.3 h
$\tau/\|\lambda\|$	97.1 h
Weight retained at 96 h	0.372

Appendix

Table 15: Converged temporal decay parameter and derived quantities.

Aggregation rule	All encounters	$\geq 2$ prior ECGs
Oldest acquisition only	0.9083	0.8903
Most recent acquisition only	0.9087	0.8998
Uniform average	0.9091	0.9026
Fixed decay, half-life 1 h	0.9090	0.9041
Fixed decay, half-life 2 h	0.9089	0.9069
Learned decay (PRISM)	0.9092	0.9136

Appendix

Table 16: Alternative rules for pooling prior ECG embeddings.

Metric	Value
Unique patients	71,098
ECG-linked encounters	129,095
Training samples (task instances)	4.5M
Training set	116,186
Validation set	6,455
Test set	6,454

Appendix

Table 17: MDS-ED cohort and partition statistics.

Group	Tasks	Examples / Endpoints
Diagnosis	1,428	Full ICD-10 nosology (circulatory, musculoskeletal, etc.)
Deterioration	6	Intubation, cardiac arrest, vasopressors, severe hypoxemia
ICU Admission	2	Overall ICU admission, admission within 24 hours
Mortality	7	In-hospital; within 1, 7, 28, 90, 180, 365 days

Appendix

Table 18: Summary of the 1,443 binary clinical tasks predicted by PRISM.

Attribute	Subgroup	Patients	Rows	Event rate	AUROC [95% CI]	ECE
Sex	Female	3,329	132,751	0.478	0.9177 [0.9037, 0.9270]	0.1143
	Male	3,027	129,407	0.523	0.8974 [0.8738, 0.9125]	0.1442
Age	$<$ 40	1,028	30,289	0.265	0.9277 [0.8867, 0.9481]	0.0962
	40–54	1,073	38,031	0.416	0.9240 [0.9054, 0.9363]	0.1153
	55–69	1,938	82,045	0.522	0.8969 [0.8745, 0.9141]	0.1378
	70–84	1,598	76,755	0.565	0.8868 [0.8602, 0.9064]	0.1356

Appendix

Table 19: PRISM performance stratified by sex, age band and self-reported ethnicity. Confidence intervals are bootstrap estimates over patients. Of the twenty-one pairwise contrasts within attributes, twenty have overlapping intervals; the exception is the 40–54 band against the 85+ band.

Cat.	Analyte	Lo	Hi	Unit	Cat.	Analyte	Lo	Hi	Unit
Vitals	Temperature	36.0	38.0	∘ C	Vitals	Systolic BP	90	140	mmHg
	Heart rate	60	100	bpm		Diastolic BP	60	90	mmHg
	Resp. rate	12	20	br/min		O 2 sat.	95	–	%
Hem.	WBC	4.5	11.0	$10^{3}$ /µL	Hem.	Neutrophils	50	70	%
	hemoglobin	12.0	17.5	g/dL		Lymphocytes	20	40	%
	hematocrit	36	50	%		Eosinophils	1.0	4.0	%

Appendix

Table 20: Clinical reference intervals used for EHR abnormality flagging. Open bounds are indicated by a dash. Applied in conjunction with a population z-score threshold of

|z|>2.0

(Section K.2 ).

Figure 4: Population memory bank retrieval across four clinical phenotypes (Cases A–D). For each test encounter, the 50 nearest neighbors of the EHR index are used, so every percentage below is computed against a denominator of 50; clinical status (PRESENT/ABSENT) reflects the ground-truth outcome label of the index patient, and percentages report outcome prevalence among the 50 retrieved neighbors.

Setting	Partition	Encounters/visits	Patients
Temporal	Training	67,562	43,238
	Validation	3,524	2,342
	Test	1,435	1,002
	Memory bank	71,086	45,580
MC-MED	Training	31,239	22,908
	Validation	4,579	3,982

Appendix

Table 21: Cohort sizes for external and temporal evaluation. Counts describe the underlying visit/encounter tables, rather than the sampled encounter–task records. Banks comprise training and validation rows.

Task group	Retrieval zeroed	Random retrieval
Diagnosis	0.7920 [0.779, 0.807]	0.7919 [0.778, 0.806]
Deterioration	0.7389 [0.641, 0.831]	0.7310 [0.628, 0.828]
ICU	0.8681 [0.830, 0.903]	0.8657 [0.826, 0.901]
Mortality	0.9215 [0.887, 0.949]	0.9236 [0.889, 0.950]
Overall	0.8301 [0.802, 0.857]	0.8280 [0.799, 0.856]

Appendix

Table 22: MC-MED retrieval controls. AUROC and 95% patient-level percentile-bootstrap confidence intervals for inference-time perturbations of fine-tuned PRISM. Overall is the unweighted mean of the four task-group AUROCs.

Model / condition	Diagnosis	Deterioration	ICU	Mortality	Overall AUROC
Fusion MLP	0.6617	0.8049	0.7858	0.8153	0.7669 [0.730, 0.801]
Fusion MLP + retrieval	0.6488	0.7941	0.7798	0.7935	0.7541 [0.716, 0.788]
PRISM	0.8119	0.8501	0.8726	0.9004	0.8587 [0.821, 0.892]
PRISM, retrieval zeroed	0.8064	0.8431	0.8665	0.8957	0.8529 [0.815, 0.887]
PRISM, random neighbors	0.8099	0.8525	0.8717	0.8988	0.8582 [0.821, 0.891]

Appendix

Table 23: Temporal generalization to later encounters. All entries are AUROC; PRISM uses the generative pathway. Brackets show reported 95% patient-bootstrap confidence intervals. The later-era records are retained without rebalancing after the original test-set ECG exclusions. Zeroed and random retrieval are inference-time controls. Bold indicates the highest point estimate in each column, not statistical significance.

Clinical Outcome Prediction

Clinical outcome prediction uses patient records, imaging, signals, and biomarkers to estimate risks, disease trajectories, treatment response, and survival.

See all 265 papers

ECG Mirage: Revealing and Mitigating the Underutilisation of ECGs in Vision-Language Models for Clinical Prediction
Sep 18, 2026Jinning Liang, Mingcheng Zhu, Tingting ZhuMultimodal Model Evaluation Clinical Outcome Prediction
University of Oxford, Oxford, United Kingdom
Autoregressive EHR Foundation Models with Multimodal Inputs
Jul 24, 2026Yuxuan Liu, Joshua Placidi, Jinpei Han +3Clinical Outcome Prediction Multimodal Token Compression
Dept. of Computing, Imperial College London, London, United Kingdom · UKRI Centres in AI for Health, United Kingdom · School of Public Health, Imperial College London, London, United Kingdom +2
Multimodal Prompt Learning with Irregular EHRs for Robust Monitoring of Critical Care Patients
Aug 22, 2026Yixin Yang, Yueyang Sun, Weichen Liu +2Missing-Modality Learning Multimodal Learning
Faculty of Engineering, Shenzhen MSU-BIT University · International School, Beijing University of Posts and Telecommunications · School of Computer Science and Engineering, Southeast University +1

Sep 18, 2026J/K move · Enter open · S save

ECG Mirage: Revealing and Mitigating the Underutilisation of ECGs in Vision-Language Models for Clinical Prediction

Jinning Liang, Mingcheng Zhu, Tingting Zhu

University of Oxford, Oxford, United Kingdom

Multimodal Model Evaluation Clinical Outcome Prediction

Open paper AskPDF arXiv

Emergency department (ED) decision-making relies on heterogeneous clinical information, including patient history, vital signs, laboratory results, and electrocardiograms (ECGs). Vision--language models (VLMs) can jointly process these modalities, but strong predictive performance does not necessarily imply meaningful use of the correct patient's ECG. We term this failure mode ECG Mirage: apparent multimodal capability without useful dependence on patient-specific ECG information. We distinguish two forms: ECG neglect, where ECGs provide little predictive benefit, and ECG confusion, where matched ECGs outperform no-image inputs but not mismatched ECGs. To evaluate these behaviours, we compare predictions obtained with matched ECGs, outcome-discordant mismatched ECGs, and no-image inputs while holding the clinical text and prediction targets fixed. Across four VLMs on MDS-ED, matched ECGs provide no consistent advantage for either ICU admission or clinical deterioration prediction. We then train four restricted visual prompts using supervised learning followed by conditional direct preference optimisation, while keeping the VLM backbone frozen. The resulting models achieve balanced accuracies of 70.6% for ICU admission and 67.5% for deterioration and increase the matched-versus-mismatched performance gap to approximately 16.5 and 5.5 percentage points, respectively. Overall, our study identifies ECG Mirage in multimodal clinical prediction and introduces visual prompt tuning as an efficient mitigation strategy.

Abstract

Clinical Outcome Prediction

Explore similar work

ECG Mirage: Revealing and Mitigating the Underutilisation of ECGs in Vision-Language Models for Clinical Prediction

Autoregressive EHR Foundation Models with Multimodal Inputs

Multimodal Prompt Learning with Irregular EHRs for Robust Monitoring of Critical Care Patients