Integrating Heterogeneous Echocardiography AI Outputs in Language Space: Learnable Late Fusion with a Supervised LLM

Ryo Takizawa; Tempei Kabayama; Chiemi Yamazaki; Shun Kitamura; Ryo Matsuoka; Yuta Ando; Satoshi Kodera; Norihiko Takeda

cs.CLOct 29, 2026

Integrating Heterogeneous Echocardiography AI Outputs in Language Space: Learnable Late Fusion with a Supervised LLM

Authors: Ryo Takizawa, Tempei Kabayama, Chiemi Yamazaki, Shun Kitamura, Ryo Matsuoka, Yuta Ando, Satoshi Kodera, Norihiko Takeda

Organizations: The University of Tokyo Hospital

Abstract

View- and target-specific echocardiography AI models capture useful but complementary information and may not achieve sufficient predictive performance in isolation. A learned ensemble is a natural way to combine these outputs, but heterogeneous models typically require each output to be converted into a binary decision, probability, confidence score, or fixed-length feature vector. We developed a supervised large language model (LLM)-based late-fusion module that directly aggregates view-specific predicted finding statements and predicted measurements in language space and learns a case-level fusion function. In patient-level three-fold cross-validation of 859 examinations from 646 patients, the proposed method integrated heterogeneous outputs within a single framework, performed comparably to the best-performing reference strategy evaluated, and outperformed a general-purpose LLM without task-specific training. These results suggest that a supervised LLM can combine multiple upstream models without model-specific output conversion or feature engineering that requires medical knowledge.

Figures & tables

Figure 1: Study pipeline. View-specific report models produce finding statements, and measurement models produce continuous values. These heterogeneous outputs are formatted as examination-level text and passed to a supervised MedGemma-based fusion module, which jointly returns four binary findings in a fixed JSON schema.

Fold	Train		Test		LV dil.	LA dil.	Root dil.	LV dysf.
	Exams	Patients	Exams	Patients	Pos.	Pos.	Pos.	Pos.
1	573	431	286	215	26	95	49	23
2	572	430	287	216	26	95	49	23
3	573	431	286	215	25	94	49	24
Overall	–	–	859	646	77	284	147	70

Table 1: Patient-level three-fold cross-validation splits. Positive counts refer to each test fold.

Method	Macro-F1	Micro-F1	Exact match	Hamming acc.	Fold Macro-F1, mean $\pm$ SD
Reports OR	0.703	0.709	0.639	0.891	0.703 $\pm$ 0.042
Reports AND	0.576	0.587	0.653	0.894	0.571 $\pm$ 0.052
Measurements OR	0.565	0.586	0.483	0.832	0.567 $\pm$ 0.047
Measurements AND	0.559	0.598	0.531	0.851	0.557 $\pm$ 0.018
All signals OR	0.615	0.629	0.477	0.826	0.619 $\pm$ 0.059
All signals AND	0.473	0.525	0.637	0.886	0.469 $\pm$ 0.048

Table 3: Overall performance of the fusion methods. Bold values mark the highest value in each column.

Target	Reports OR	LightGBM	SFT all	SFT report only	GPT-5.5
LV dilatation	0.733	0.727	0.696	0.721	0.731
LA dilatation	0.767	0.771	0.778	0.770	0.766
Aortic root dilatation	0.569	0.542	0.545	0.545	0.528
Global LV dysfunction	0.741	0.713	0.742	0.735	0.405
Macro-F1	0.703	0.688	0.690	0.693	0.608

Table 4: Target-specific F1. Bold values mark the highest point estimate in each row.

Panel of Figure 2 — Figure 2: Exploratory paired patient-cluster bootstrap differences in Macro-F1. Points show paired differences, horizontal lines show 95% CIs, and numeric annotations give the point estimate followed by the CI. Analyses use fixed OOF predictions.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Characteristic	Summary
Sex, male, $n$ (%)	496 (57.7%)
Sex, female, $n$ (%)	363 (42.3%)
Age, median [IQR], years	63.0 [49.0–74.0]
Age, range, years	8–93

Appendix

Table 5: Examination-level cohort characteristics.

Target	Report inputs	Measurement inputs	Positive direction
LV dilatation	4CH, 2CH, LAX	$D_{d}$	Larger
LA dilatation	4CH, 2CH, LAX	LAD	Larger
Aortic root dilatation	3CH, LAX	Valsalva diameter	Larger
Global LV dysfunction	4CH, 2CH, 3CH, SAX	EF MOD , EF Teichholz	Smaller

Appendix

Table 6: Report and measurement inputs associated with each evaluation target.

Hyperparameter	Setting
Base model	google/medgemma-4b-it
LoRA rank / alpha / dropout	8 / 16 / 0.05
Learning rate	$1\times 10^{-5}$
Warmup ratio / scheduler	0.05 / linear
Epochs	5
Per-GPU batch size	4

Appendix

Table 7: Supervised fine-tuning configuration.

Echocardiography

Echocardiography uses ultrasound to image the heart and assess its structure, motion, blood flow, and clinical function.

See all 58 papers

Multimodal Generative AI Assists Comprehensive Echocardiographic Interpretation
Oct 23, 2026Yi Qin, Marawan Elbatel, Shanshan Song +28Echocardiography Medical VLMs
Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China · Guangdong Cardiovascular Institute, Guangdong Provincial People’s Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, Guangdong Province, China · Li Ka Shing Institute of Health Sciences, Faculty of Medicine, The Chinese University of Hong Kong, Hong Kong SAR, China +19
MARCUS: An agentic, multimodal vision-language model for cardiac diagnosis and management
Mar 23, 2026Jack W O'Sullivan, Mohammad Asadi, Lennart Elbe +10Medical VLMs Multimodal Understanding
Division of Cardiology, Department of Medicine, Stanford University, CA, USA · Department of Biomedical Data Science, Stanford University, CA, USA · Department of Electrical Engineering, Stanford University, CA, USA +7
Integrating Machine Learning Ensembles and Large Language Models for Heart Disease Prediction Using Voting Fusion
Feb 25, 2026Md. Tahsin Amin, Tanim Ahmmod, Zannatul Ferdus +5Cardiovascular Risk Prediction Clinical Risk Prediction
Faculty of Computer Science & Engineering, Patuakhali Science & Technology University, Bangladesh · Dept. of Electrical & Electronic Engineering, Rajshahi University of Engineering & Technology, Bangladesh · Dept. of Industrial & Production Engineering, Ahsanullah University of Science & Technology, Bangladesh +3

Oct 23, 2026J/K move · Enter open · S save

Multimodal Generative AI Assists Comprehensive Echocardiographic Interpretation

Yi Qin, Marawan Elbatel, Shanshan Song, Taoran Huang+27

Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China · Guangdong Cardiovascular Institute, Guangdong Provincial People’s Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, Guangdong Province, China · Li Ka Shing Institute of Health Sciences, Faculty of Medicine, The Chinese University of Hong Kong, Hong Kong SAR, China+19

Echocardiography Medical VLMs Medical Report Generation

Open paper AskPDF arXiv

Echocardiography is the cornerstone of cardiovascular diagnostics, yet its global utility is constrained by a critical shortage of experienced interpreters. Current artificial intelligence (AI) solutions remain limited, typically operating as opaque "black boxes" or focusing on narrow tasks that fail to capture the holistic, guideline-driven reasoning of a clinician. Here we introduce EchoChat, a generative multimodal vision-language model that performs comprehensive echocardiographic analysis to support clinician interpretation. By emulating the cognitive workflow of a cardiologist, EchoChat integrates vast medical knowledge with a chain-of-thought reasoning mechanism to explicitly ground its conclusions in visual evidence. Trained on echocardiography-instruction pairs from 46,358 examinations, our model establishes a new state-of-the-art across diverse cohorts from six international medical centers. It supports the broadest diagnostic scope of 33 cardiac conditions and findings, including 15 newly added conditions for which it achieves an average F1 of 73.7% across available external evaluations. It surpasses existing best-performing foundation models in cardiac condition recognition (improving average recall by 13.54% on shared tasks) and quantification (average R2 score increase of 0.275 on shared tasks). Furthermore, in a source-blinded echocardiographer reader study, echocardiography reports co-generated with EchoChat were rated higher in diagnostic accuracy and completeness by two senior echocardiographers, and improved diagnostic performance from 72.78% to 82.75% in terms of F1 score over routine-reader only reports. These results demonstrate that integrating clinical reasoning into multimodal AI can democratize specialist expertise and transform the standard of cardiovascular care.

Abstract

Echocardiography

Explore similar work

Multimodal Generative AI Assists Comprehensive Echocardiographic Interpretation

MARCUS: An agentic, multimodal vision-language model for cardiac diagnosis and management

Integrating Machine Learning Ensembles and Large Language Models for Heart Disease Prediction Using Voting Fusion