Is Clinical Text Enough? A Multimodal Study on Mortality Prediction in Heart Failure Patients

Oumaima El Khettari; Virgile Barthet; Guillaume Hocquet; Joconde Weller; Emmanuel Morin; Pierre Zweigenbaum

cs.CLApr 2, 2026

Is Clinical Text Enough? A Multimodal Study on Mortality Prediction in Heart Failure Patients

Authors: Oumaima El Khettari, Virgile Barthet, Guillaume Hocquet, Joconde Weller, Emmanuel Morin, Pierre Zweigenbaum

Organizations: Nantes Univ., École Centrale Nantes, CNRS, LS2N, UMR 6004, 44000 Nantes, France · Université Paris-Saclay, CNRS, LISN, Orsay, France · Hôpital Saint Joseph, DIMID, et Service de Cardiologie, Paris, France

Abstract

Accurate short-term mortality prediction in heart failure (HF) remains challenging, particularly when relying on structured electronic health record (EHR) data alone. We evaluate transformer-based models on a French HF cohort, comparing text-only, structured-only, multimodal, and LLM-based approaches. Our results show that enriching clinical text with entity-level representations improves prediction over CLS embeddings alone, and that supervised multimodal fusion of text and structured variables achieves the best overall performance. In contrast, large language models perform inconsistently across modalities and decoding strategies, with text-only prompts outperforming structured or multimodal inputs. These findings highlight that entity-aware multimodal transformers offer the most reliable solution for short-term HF outcome prediction, while current LLM prompting remains limited for clinical decision support.

Figures & tables

Figure 1: Distribution of clinical notes per size.

Label	Precision	Recall	F1
Age	1.00	1.00	1.00
Social circle	0.87	1.00	0.93
Time	1.00	0.80	0.89
Mode	0.86	0.86	0.86
Treatment	0.90	0.82	0.86
Dose	0.77	0.92	0.84

Table 1: Evaluation of the supervised NER results against expert annotations

Methods	CamemBERT-bio				DrBERT
Methods	P	R	F1	AUC	P	R	F1	AUC
Baseline: CLS only	28.92	48.77	36.24	75.09	28.70	29.02	28.79	67.85
Average fusion	36.88	40.33	38.42	74.38	31.94	34.68	33.20	71.97
Sum fusion	36.49	33.92	34.79	73.97	37.02	29.91	32.60	70.68
Concatenation	31.19	48.78	37.99	74.90	33.15	27.03	29.74	71.40
Weighted average	32.77	47.56	38.74	75.82	31.67	26.22	28.50	70.86

Table 2: Performance of text-only methods. The best F1 and ROC-AUC results are bolded, and the second best are underlined

Methods	P	R	F1	AUC
Reference	24.80	63.74	35.67	76.18
Baseline	26.24	69.38	38.02	79.04

Table 3: Performance of structured-data only models. The best F1 and AUC results are bolded.

Methods	CamemBERT-bio				DrBERT
Methods	P	R	F1	AUC	P	R	F1	AUC
Baseline (CLS + All)	27.84	63.29	38.65	78.32	26.62	39.49	31.67	69.26
Baseline (CLS + Sel.)	28.70	66.94	40.15	8 1.94	27.21	41.10	32.64	70.11
Gated Fusion (Sel.)	48.99	48.43	48.48	81.46	36.14	48.75	41.11	75.22
Dual-XAttn	35.14	31.48	33.16	75.75	37.47	28.18	32.10	75.59
Gated Dual-XAttn	44.99	42.29	42.72	80.28	46.12	48.78	46.84	7 9.65

Table 4: Performance of multimodal models. The best F1 and AUC scores are bolded, and the second best are underlined

Modality	Qwen2.5-7B-Instruct						Mistral-7B-Instruct			MedGemma-4B-it
	Greedy			Constrained			Greedy			Greedy
	P	R	F1	P	R	F1	P	R	F1	P	R	F1
Text	13.23	58.00	21.54	10.92	90.73	19.50	15.85	67.21	25.65	41.38	19.35	26.37
Structured	11.93	59.09	19.85	10.99	100.00	19.80	0.00	0.00	0.00	10.94	99.19	19.70
Multimodal	11.10	98.59	19.95	11.00	100.00	19.82	12.75	86.64	22.23	12.03	84.68	21.06

Table 5: Performance of LLMs for binary mortality prediction across data modalities and decoding strategies. “Text” uses the raw clinical notes, “Structured” uses tabular patient data, and “Multimodal” combines both sources in a multimodal prompt.

Model	P	R	F1
Qwen 7B it	10.78	90.48	19.27
Mistral 7B it	10.92	87.85	19.42
MedGemma 4B it	10.81	96.77	19.45

Table 6: Performance of LLMs when structured variables are transformed into textual descriptions using a templated representation.

Feature Type	Feature Names
Demographics	ADMIN_AGE, ADMIN_T4
Biological Measurements	BIO_UREE, BIO_BNP, BIO_hco3a, BIO_CRP, BIO_GGT, BIO_pha, BIO_TROPOHS, BIO_BILI, BIO_CCMH, BIO_TSH, BIO_LDL, BIO_POT, BIO_ASAT, BIO_PLAQ, BIO_CK, BIO_PAL, BIO_VGM, BIO_sao2a, BIO_RBC, BIO_kul
Vital Signs / Clinical Indicators	INF_pas, INF_delta, SIGNES_Si dyspnée
Diagnoses / Comorbidities	DIAG_Cerebrovascular Disease, DIAG_Weight Loss, DIAG_Maladie_hepatique, DIAG_Obesity, DIAG_Renal Disease
Medications	MED_DIURETIQUES – DIURETIQUE NON EPARGNEUR POTASSIUM – DIURETIQUE THIAZIDIQUE; MED_RENINE-ANGIOTENSINE – ARA II – AUTRE; MED_RENINE-ANGIOTENSINE – IEC; MED_BETABLOQUANTS; MED_ANTIDIABETIQUES, INSULINES EXCLUES – SULFAMIDES; MED_ANTIBACTERIENS A USAGE SYSTEMIQUE; MED_ANTITHROMBOTIQUE – ANTICOAGULANT – HEPARINE
Procedures / ECG / Admission Information	ECG_Si QRS fin, ECG_Si Pacemaker, MVT_ENTREE_TRANSFERT, TRANS_TRANSFUSION_SANG

Table 7: Structured variables used in the structured-only and multimodal models.

Comparison	$\boldsymbol{\Delta}$ (Mean AUC)	95% CI	p-value
Late Fusion vs Baseline (Structured data model)	+0.0427	[0.0203, 0.0696]	0.0409
Textual Gated fusion vs Baseline (Structured data model)	$-0.0450$	[ $-0.0730$ , $-0.0169$ ]	0.0524
Late Fusion vs Textual Gated fusion	+0.0904	[0.0724, 0.1072]	0.0009

Table 8: Paired significance analysis of ROC-AUC across 5 cross-validation folds.

\Delta

denotes the difference between mean AUC values of the two compared models (Model A – Model B). Confidence intervals are obtained via bootstrap resampling over folds; p-values correspond to paired t-tests.

Mortality Prediction

Mortality prediction estimates the risk, timing, or cause of death from clinical records, images, signals, and population data.

See all 30 papers

Narrative Feature or Structured Feature? A Study of Large Language Models to Identify Cancer Patients at Risk of Heart Failure
Mar 18, 2024Ziyi Chen, Mengyuan Zhang, Mustafa Mohammed Ahmed +4Healthcare Oncology
Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FL, USA · Division of Cardiovascular Medicine, Department of Medicine, College of Medicine, University of Florida, Gainesville, FL, USA · Division of Hematology & Oncology, Department of Medicine, College of Medicine, University of Florida, Gainesville, FL, USA
Autoregressive EHR Foundation Models with Multimodal Inputs
Jul 24, 2026Yuxuan Liu, Joshua Placidi, Jinpei Han +3Clinical Outcome Prediction Multimodal Token Compression
Dept. of Computing, Imperial College London, London, United Kingdom · UKRI Centres in AI for Health, United Kingdom · School of Public Health, Imperial College London, London, United Kingdom +2
ECG Mirage: Revealing and Mitigating the Underutilisation of ECGs in Vision-Language Models for Clinical Prediction
Sep 18, 2026Jinning Liang, Mingcheng Zhu, Tingting ZhuMultimodal Model Evaluation Clinical Outcome Prediction
University of Oxford, Oxford, United Kingdom

Mar 18, 2024J/K move · Enter open · S save

Narrative Feature or Structured Feature? A Study of Large Language Models to Identify Cancer Patients at Risk of Heart Failure

Ziyi Chen, Mengyuan Zhang, Mustafa Mohammed Ahmed, Yi Guo+3

Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FL, USA · Division of Cardiovascular Medicine, Department of Medicine, College of Medicine, University of Florida, Gainesville, FL, USA · Division of Hematology & Oncology, Department of Medicine, College of Medicine, University of Florida, Gainesville, FL, USA

Healthcare Oncology Longitudinal EHR Modeling Medical LLMs Clinical Risk Prediction

Open paper AskPDF arXiv

Cancer treatments are known to introduce cardiotoxicity, negatively impacting outcomes and survivorship. Identifying cancer patients at risk of heart failure (HF) is critical to improving cancer treatment outcomes and safety. This study examined machine learning (ML) models to identify cancer patients at risk of HF using electronic health records (EHRs), including traditional ML, Time-Aware long short-term memory (T-LSTM), and large language models (LLMs) using novel narrative features derived from the structured medical codes. We identified a cancer cohort of 12,806 patients from the University of Florida Health, diagnosed with lung, breast, and colorectal cancers, among which 1,602 individuals developed HF after cancer. The LLM, GatorTron-3.9B, achieved the best F1 scores, outperforming the traditional support vector machines by 39%, the T-LSTM deep learning model by 7%, and a widely used transformer model, BERT, by 5.6%. The analysis shows that the proposed narrative features remarkably increased feature density and improved performance.

Abstract

Mortality Prediction

Explore similar work

Narrative Feature or Structured Feature? A Study of Large Language Models to Identify Cancer Patients at Risk of Heart Failure

Autoregressive EHR Foundation Models with Multimodal Inputs

ECG Mirage: Revealing and Mitigating the Underutilisation of ECGs in Vision-Language Models for Clinical Prediction