cs.CVOct 23, 2026

Multimodal Generative AI Assists Comprehensive Echocardiographic Interpretation

Authors: Yi Qin, Marawan Elbatel, Shanshan Song, Taoran Huang, Jiarong Guo, Ningshu Li, Lehan Wang, Chenxu Zhao, 

Organizations: Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China · Guangdong Cardiovascular Institute, Guangdong Provincial People’s Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, Guangdong Province, China · Li Ka Shing Institute of Health Sciences, Faculty of Medicine, The Chinese University of Hong Kong, Hong Kong SAR, China · Department of Cardiology, Amsterdam Cardiovascular Sciences, Amsterdam University Medical Center, University of Amsterdam, Amsterdam, the Netherlands · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Echocardiography is the cornerstone of cardiovascular diagnostics, yet its global utility is constrained by a critical shortage of experienced interpreters. Current artificial intelligence (AI) solutions remain limited, typically operating as opaque "black boxes" or focusing on narrow tasks that fail to capture the holistic, guideline-driven reasoning of a clinician. Here we introduce EchoChat, a generative multimodal vision-language model that performs comprehensive echocardiographic analysis to support clinician interpretation. By emulating the cognitive workflow of a cardiologist, EchoChat integrates vast medical knowledge with a chain-of-thought reasoning mechanism to explicitly ground its conclusions in visual evidence. Trained on echocardiography-instruction pairs from 46,358 examinations, our model establishes a new state-of-the-art across diverse cohorts from six international medical centers. It supports the broadest diagnostic scope of 33 cardiac conditions and findings, including 15 newly added conditions for which it achieves an average F1 of 73.7% across available external evaluations. It surpasses existing best-performing foundation models in cardiac condition recognition (improving average recall by 13.54% on shared tasks) and quantification (average R2 score increase of 0.275 on shared tasks). Furthermore, in a source-blinded echocardiographer reader study, echocardiography reports co-generated with EchoChat were rated higher in diagnostic accuracy and completeness by two senior echocardiographers, and improved diagnostic performance from 72.78% to 82.75% in terms of F1 score over routine-reader only reports. These results demonstrate that integrating clinical reasoning into multimodal AI can democratize specialist expertise and transform the standard of cardiovascular care.

Figures & tables

Figure 1: The overall workflow of EchoChat. a, EchoChat streamlines the clinical workflow by functioning as an interactive assistant that enables conversational queries. The system processes these queries to execute comprehensive echocardiographic tasks, thereby supporting the echocardiographer’s decision-making. b, EchoChat is a multi-modal large language model trained and tested on over 2 million echocardiographic videos and images from six international hospitals. By integrating imaging and textual data, it performs recognition on 33 types of cardiac conditions and findings, 22 types of measurement estimation, report generation, and visual question answering, outperforming existing echocardiographic foundation models with new capabilities. □\Box refers to the report retrieval capacity of EchoPrime, which matches existing reports to echocardiographic content based on similarity rather than generating a new report from scratch. c, By employing echocardiography-centered, evidence-based reasoning alongside a vast knowledge library in the training corpus, EchoChat enhances disease recognition accuracy and provides interpretable supporting evidence.
Figure 2: Cardiac condition recognition task performance of EchoChat and baseline methods. a, Distribution of the 33 cardiac conditions supported by EchoChat, grouped into six clinical categories. b, Mean F1 scores of EchoChat and baseline models in the four external test sets (the Netherlands, Spain, Mainland A China and Mainland B China), using the common subset of available conditions supported by all three models within each center. c, Condition-level accuracy comparison for conditions with an available baseline result. Each point represents one condition, with accuracy averaged equally across available external centers. The best-performing baseline is selected separately for each condition, and EchoChat and that baseline are compared on the same available centers. The x axis shows baseline accuracy and the y axis shows EchoChat accuracy; points above the diagonal favor EchoChat. d-g, Condition-level F1 scores of EchoChat and baseline models on four external test sets. IVC: Inferior Vena Cava, RV: Right Ventricle, RA: Right Atrial, LA: Left Atrial, LV: Left Ventricle, MV: Mitral Valve, TV: Tricuspid Valve, AV: Aortic Valve, PV: Pulmonary Valve, PA: Pulmonary Artery, HCM: Hypertrophic Cardiomyopathy, TAVR: Transcatheter Aortic Valve Replacement, SWMA: Segmental Wall Motion Abnormality, PH: Pulmonary Hypertension, VSD: Ventricular Septal Defect, PE: Pericardial Effusion. NA: Not Available.
Figure 3: Heart measurement estimation performance comparison of EchoChat against baselines using Mean Absolute Error (MAE) and coefficient of determination ( R2R^{2} ). a-e, Performance comparison on five external test sets from the Netherlands ( a ), Spain ( b ), Mainland China A ( c ), Mainland China B ( d ), and Austria ( e ). For each test set, the left panel shows the MAE (lower is better), while the right panel shows the R2R^{2} score (higher is better). Error bars represent the 95% confidence interval for each data point in the metrics. RV: Right Ventricle, RA: Right Atrial, LA: Left Atrial, LV: Left Ventricle, MV: Mitral Valve, AV: Aortic Valve, AAO: Ascending Aorta, PA: Pulmonary Artery, PV: Pulmonary Valve, LVEDD: Left Ventricular End-Diastolic Diameter, LVOT: Left Ventricular Outflow Tract, TV: Tricuspid Valve, TR: Tricuspid Regurgitation, PR: Pulmonary Regurgitation, AR: Aortic Regurgitation, IVS: Interventricular Septum, LVPWd: Left Ventricular Posterior Wall diameter at end-diastole, LVEF: Left Ventricular Ejection Fraction, LAAPD: Left Atrial Anteroposterior Diameter, TAPSE: Tricuspid Annular Plane Systolic Excursion, LAESV: LA end-systolic volume. NA: Not Available.
Figure 4: Echocardiographer-EchoChat collaboration and individual echocardiographer benchmarks. a, Precision, recall and micro-F1 for two routine echocardiographers, two senior echocardiographers, EchoChat, and two routine echocardiographer-EchoChat collaborations on 100 examinations. All conditions use the same 21 clinician-defined primary diagnostic targets. Symbols identify individual outputs; horizontal lines show patient-bootstrap 95% confidence intervals. All three axes span 50-100%. b, Patient-paired micro-F1 gains for each collaboration over the comparator named on the y axis: EchoChat alone, routine echocardiographer A, or routine echocardiographer B. Positive differences favour collaboration. Lines show 95% patient-bootstrap confidence intervals; pp denotes percentage points. c, Source-blinded senior assessment based on the six-dimension rubrics. Means and patient-bootstrap intervals are shown separately by dimension. Each point averages both raters within patient and then across patients; error bars show 95% patient-bootstrap confidence intervals. Scores range from 1 to 5, with higher values indicating better quality. d, Label changes from the initial EchoChat draft to each final collaborative report, evaluated against the primary-diagnosis reference labels. Omitted diagnoses added are FN-to-TP transitions; false positives removed are FP-to-TN; correct diagnoses lost are TP-to-FN; false positives introduced are TN-to-FP. Rightward positive bars indicate corrections and leftward negative bars indicate new disagreements. e, Reports with at least one edit of each clinician-recorded reason. Denominators are 100 saved reports per collaborator; categories overlap, so percentages need not sum to 100. Counts refer to reports, not editing operations. f, All reports partitioned into unchanged, only edits marked non-significant, or at least one edit marked clinically significant by the revising clinician. Numbers inside bars are counts out of 100.
Figure 5: Representative case illustrating EchoChat-routine echocardiographer collaboration. a, Representative echocardiographic input images, and clinical reference report from PACS. b, Standalone report from the routine echocardiographer. c, Report generated by EchoChat from the echocardiographic examination. d, Collaboration report, produced by a routine echocardiographer revising the EchoChat draft. e, Independent assessment by senior raters across finding accuracy, diagnostic accuracy, measurement/severity accuracy, completeness, internal consistency, and clarity/report expression. Each cell reports the scores of the two senior raters.
More in this topic

Echocardiography

Echocardiography uses ultrasound to image the heart and assess its structure, motion, blood flow, and clinical function.

See all 58 papers

Explore similar work

CardsList
Jun 26, 2026

EchoSonar-R: A Multi-View Reasoning-Enabled Model for Disease Classification and Report Generation in Echocardiography

Echocardiography is the most widely used non-invasive cardiac imaging modality, providing essential information for cardiovascular diagnosis. Interpreting an echocardiogram requires synthesizing complementary evidence across multiple heart views to identify abnormalities and produce structured clinical reports. While recent efforts focus on improving classification performance, most models lack explicit diagnostic reasoning and spatially grounded anatomical evidence, limiting clinician trust. We present EchoSonar-R, a multi-view reasoning-enabled vision-language model that jointly performs multi-label disease classification and report generation from echocardiography studies. EchoSonar-R combines a spatiotemporal video encoder with a structure-aware cardiac detector that provides spatially grounded anatomical cues to improve interpretability and clinician trust during cross-view reasoning. EchoSonar-R is trained in two stages: supervised fine-tuning (SFT) on reasoning-annotated targets, followed by Group Relative Policy Optimization (GRPO) with task-specific rewards that jointly align classification and report generation within a unified reinforcement-learning framework. Across a private multi-view dataset and two public benchmarks, EchoSonar-R improves macro balanced accuracy by 17.1% on the private set and 6.1% on MIMICEchoQA over the strongest baseline, achieves a GREEN clinical faithfulness score of 0.800, and produces interpretable reasoning traces grounded in multi-view visual evidence.

Darya Taratynova, Ahmed Aly, Numan Saeed +1

Division of Computing and Mathematical Sciences, Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, UAE

Medical Report GenerationEchocardiographyMedical VLMsExplainable Medical Image Analysis
AskPDF
Apr 7, 2026

EchoAgent: Towards Reliable Echocardiography Interpretation with "Eyes","Hands" and "Minds"

Reliable interpretation of echocardiography (Echo) is crucial for assessing cardiac function, which demands clinicians to synchronously orchestrate multiple capabilities, including visual observation (eyes), manual measurement (hands), and expert knowledge learning and reasoning (minds). While current task-specific deep-learning approaches and multimodal large language models have demonstrated promise in assisting Echo analysis through automated segmentation or reasoning, they remain focused on restricted skills, i.e., eyes-hands or eyes-minds, thereby limiting clinical reliability and utility. To address these issues, we propose EchoAgent, an agentic system tailored for end-to-end Echo interpretation, which achieves a fully coordinated eyes-hands-minds workflow that learns, observes, operates, and reasons like a cardiac sonographer. First, we introduce an expertise-driven cognition engine where our agent can automatically assimilate credible Echo guidelines into a structured knowledge base, thus constructing an Echo-customized mind. Second, we devise a hierarchical collaboration toolkit to endow EchoAgent with eyes-hands, which can automatically parse Echo video streams, identify cardiac views, perform anatomical segmentation, and quantitative measurement. Third, we integrate the perceived multimodal evidence with the exclusive knowledge base into an orchestrated reasoning hub to conduct explainable inferences. We evaluate EchoAgent on CAMUS and MIMIC-EchoQA datasets, which cover 48 distinct echocardiographic views spanning 14 cardiac anatomical regions. Experimental results show that EchoAgent achieves optimal performance across diverse structure analyses, yielding overall accuracy of up to 80.00%. Importantly, EchoAgent empowers a single system with abilities to learn, observe, operate and reason like an echocardiologist, which holds great promise for reliable Echo interpretation.

Qin Wang, Zhiqing He, Yu Liu +8

Fudan University · Fudan Zhongshan Hospital · Fuwai Yunnan Hospital +1

EchocardiographyUltrasound ImagingMultimodal AgentsExplainable Medical Image Analysis
AskPDF
Mar 23, 2026

MARCUS: An agentic, multimodal vision-language model for cardiac diagnosis and management

Cardiovascular disease remains the leading cause of global mortality, with progress hindered by human interpretation of complex cardiac tests. Current AI vision-language models are limited to single-modality inputs and are non-interactive. We present MARCUS (Multimodal Autonomous Reasoning and Chat for Ultrasound and Signals), an agentic vision-language system for end-to-end interpretation of electrocardiograms (ECGs), echocardiograms, and cardiac magnetic resonance imaging (CMR) independently and as multimodal input. MARCUS employs a hierarchical agentic architecture comprising modality-specific vision-language expert models, each integrating domain-trained visual encoders with multi-stage language model optimization, coordinated by a multimodal orchestrator. Trained on 13.5 million images (0.25M ECGs, 1.3M echocardiogram images, 12M CMR images) and our novel expert-curated dataset spanning 1.6 million questions, MARCUS achieves state-of-the-art performance surpassing frontier models (GPT-5 Thinking, Gemini 2.5 Pro Deep Think). Across internal (Stanford) and external (UCSF) test cohorts, MARCUS achieves accuracies of 87-91% for ECG, 67-86% for echocardiography, and 85-88% for CMR, outperforming frontier models by 34-45% (P<0.001). On multimodal cases, MARCUS achieved 70% accuracy, nearly triple that of frontier models (22-28%), with 1.7-3.0x higher free-text quality scores. Our agentic architecture also confers resistance to mirage reasoning, whereby vision-language models derive reasoning from unintended textual signals or hallucinated visual content. MARCUS demonstrates that domain-specific visual encoders with an agentic orchestrator enable multimodal cardiac interpretation. We release our models, code, and benchmark open-source.

Jack W O'Sullivan, Mohammad Asadi, Lennart Elbe +10

Division of Cardiology, Department of Medicine, Stanford University, CA, USA · Department of Biomedical Data Science, Stanford University, CA, USA · Department of Electrical Engineering, Stanford University, CA, USA +7

Medical VLMsMultimodal UnderstandingElectrocardiogram InterpretationEchocardiographyElectrocardiographyCardiac MRI
AskPDF