cs.CVOct 23, 2026

Multimodal Generative AI Assists Comprehensive Echocardiographic Interpretation

Authors: Yi Qin, Marawan Elbatel, Shanshan Song, Taoran Huang, Jiarong Guo, Ningshu Li, Lehan Wang, Chenxu Zhao, 

Organizations: Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China · Guangdong Cardiovascular Institute, Guangdong Provincial People’s Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, Guangdong Province, China · Li Ka Shing Institute of Health Sciences, Faculty of Medicine, The Chinese University of Hong Kong, Hong Kong SAR, China · Department of Cardiology, Amsterdam Cardiovascular Sciences, Amsterdam University Medical Center, University of Amsterdam, Amsterdam, the Netherlands · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Echocardiography is the cornerstone of cardiovascular diagnostics, yet its global utility is constrained by a critical shortage of experienced interpreters. Current artificial intelligence (AI) solutions remain limited, typically operating as opaque "black boxes" or focusing on narrow tasks that fail to capture the holistic, guideline-driven reasoning of a clinician. Here we introduce EchoChat, a generative multimodal vision-language model that performs comprehensive echocardiographic analysis to support clinician interpretation. By emulating the cognitive workflow of a cardiologist, EchoChat integrates vast medical knowledge with a chain-of-thought reasoning mechanism to explicitly ground its conclusions in visual evidence. Trained on echocardiography-instruction pairs from 46,358 examinations, our model establishes a new state-of-the-art across diverse cohorts from six international medical centers. It supports the broadest diagnostic scope of 33 cardiac conditions and findings, including 15 newly added conditions for which it achieves an average F1 of 73.7% across available external evaluations. It surpasses existing best-performing foundation models in cardiac condition recognition (improving average recall by 13.54% on shared tasks) and quantification (average R2 score increase of 0.275 on shared tasks). Furthermore, in a source-blinded echocardiographer reader study, echocardiography reports co-generated with EchoChat were rated higher in diagnostic accuracy and completeness by two senior echocardiographers, and improved diagnostic performance from 72.78% to 82.75% in terms of F1 score over routine-reader only reports. These results demonstrate that integrating clinical reasoning into multimodal AI can democratize specialist expertise and transform the standard of cardiovascular care.

Figures & tables

Figure 1: The overall workflow of EchoChat. a, EchoChat streamlines the clinical workflow by functioning as an interactive assistant that enables conversational queries. The system processes these queries to execute comprehensive echocardiographic tasks, thereby supporting the echocardiographer’s decision-making. b, EchoChat is a multi-modal large language model trained and tested on over 2 million echocardiographic videos and images from six international hospitals. By integrating imaging and textual data, it performs recognition on 33 types of cardiac conditions and findings, 22 types of measurement estimation, report generation, and visual question answering, outperforming existing echocardiographic foundation models with new capabilities. □\Box refers to the report retrieval capacity of EchoPrime, which matches existing reports to echocardiographic content based on similarity rather than generating a new report from scratch. c, By employing echocardiography-centered, evidence-based reasoning alongside a vast knowledge library in the training corpus, EchoChat enhances disease recognition accuracy and provides interpretable supporting evidence.
Figure 2: Cardiac condition recognition task performance of EchoChat and baseline methods. a, Distribution of the 33 cardiac conditions supported by EchoChat, grouped into six clinical categories. b, Mean F1 scores of EchoChat and baseline models in the four external test sets (the Netherlands, Spain, Mainland A China and Mainland B China), using the common subset of available conditions supported by all three models within each center. c, Condition-level accuracy comparison for conditions with an available baseline result. Each point represents one condition, with accuracy averaged equally across available external centers. The best-performing baseline is selected separately for each condition, and EchoChat and that baseline are compared on the same available centers. The x axis shows baseline accuracy and the y axis shows EchoChat accuracy; points above the diagonal favor EchoChat. d-g, Condition-level F1 scores of EchoChat and baseline models on four external test sets. IVC: Inferior Vena Cava, RV: Right Ventricle, RA: Right Atrial, LA: Left Atrial, LV: Left Ventricle, MV: Mitral Valve, TV: Tricuspid Valve, AV: Aortic Valve, PV: Pulmonary Valve, PA: Pulmonary Artery, HCM: Hypertrophic Cardiomyopathy, TAVR: Transcatheter Aortic Valve Replacement, SWMA: Segmental Wall Motion Abnormality, PH: Pulmonary Hypertension, VSD: Ventricular Septal Defect, PE: Pericardial Effusion. NA: Not Available.
Figure 3: Heart measurement estimation performance comparison of EchoChat against baselines using Mean Absolute Error (MAE) and coefficient of determination ( R2R^{2} ). a-e, Performance comparison on five external test sets from the Netherlands ( a ), Spain ( b ), Mainland China A ( c ), Mainland China B ( d ), and Austria ( e ). For each test set, the left panel shows the MAE (lower is better), while the right panel shows the R2R^{2} score (higher is better). Error bars represent the 95% confidence interval for each data point in the metrics. RV: Right Ventricle, RA: Right Atrial, LA: Left Atrial, LV: Left Ventricle, MV: Mitral Valve, AV: Aortic Valve, AAO: Ascending Aorta, PA: Pulmonary Artery, PV: Pulmonary Valve, LVEDD: Left Ventricular End-Diastolic Diameter, LVOT: Left Ventricular Outflow Tract, TV: Tricuspid Valve, TR: Tricuspid Regurgitation, PR: Pulmonary Regurgitation, AR: Aortic Regurgitation, IVS: Interventricular Septum, LVPWd: Left Ventricular Posterior Wall diameter at end-diastole, LVEF: Left Ventricular Ejection Fraction, LAAPD: Left Atrial Anteroposterior Diameter, TAPSE: Tricuspid Annular Plane Systolic Excursion, LAESV: LA end-systolic volume. NA: Not Available.
Figure 4: Echocardiographer-EchoChat collaboration and individual echocardiographer benchmarks. a, Precision, recall and micro-F1 for two routine echocardiographers, two senior echocardiographers, EchoChat, and two routine echocardiographer-EchoChat collaborations on 100 examinations. All conditions use the same 21 clinician-defined primary diagnostic targets. Symbols identify individual outputs; horizontal lines show patient-bootstrap 95% confidence intervals. All three axes span 50-100%. b, Patient-paired micro-F1 gains for each collaboration over the comparator named on the y axis: EchoChat alone, routine echocardiographer A, or routine echocardiographer B. Positive differences favour collaboration. Lines show 95% patient-bootstrap confidence intervals; pp denotes percentage points. c, Source-blinded senior assessment based on the six-dimension rubrics. Means and patient-bootstrap intervals are shown separately by dimension. Each point averages both raters within patient and then across patients; error bars show 95% patient-bootstrap confidence intervals. Scores range from 1 to 5, with higher values indicating better quality. d, Label changes from the initial EchoChat draft to each final collaborative report, evaluated against the primary-diagnosis reference labels. Omitted diagnoses added are FN-to-TP transitions; false positives removed are FP-to-TN; correct diagnoses lost are TP-to-FN; false positives introduced are TN-to-FP. Rightward positive bars indicate corrections and leftward negative bars indicate new disagreements. e, Reports with at least one edit of each clinician-recorded reason. Denominators are 100 saved reports per collaborator; categories overlap, so percentages need not sum to 100. Counts refer to reports, not editing operations. f, All reports partitioned into unchanged, only edits marked non-significant, or at least one edit marked clinically significant by the revising clinician. Numbers inside bars are counts out of 100.
Figure 5: Representative case illustrating EchoChat-routine echocardiographer collaboration. a, Representative echocardiographic input images, and clinical reference report from PACS. b, Standalone report from the routine echocardiographer. c, Report generated by EchoChat from the echocardiographic examination. d, Collaboration report, produced by a routine echocardiographer revising the EchoChat draft. e, Independent assessment by senior raters across finding accuracy, diagnostic accuracy, measurement/severity accuracy, completeness, internal consistency, and clarity/report expression. Each cell reports the scores of the two senior raters.
More in this topic

Echocardiography

Echocardiography uses ultrasound to image the heart and assess its structure, motion, blood flow, and clinical function.

See all 58 papers

Explore similar work

CardsList

  1. EchoSonar-R: A Multi-View Reasoning-Enabled Model for Disease Classification and Report Generation in Echocardiography

    Jun 26, 2026Darya Taratynova, Ahmed Aly, Numan Saeed +1Medical Report GenerationEchocardiography

    Division of Computing and Mathematical Sciences, Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, UAE

  2. EchoAgent: Towards Reliable Echocardiography Interpretation with "Eyes","Hands" and "Minds"

    Apr 7, 2026Qin Wang, Zhiqing He, Yu Liu +8EchocardiographyUltrasound Imaging

    Fudan University · Fudan Zhongshan Hospital · Fuwai Yunnan Hospital +1

  3. MARCUS: An agentic, multimodal vision-language model for cardiac diagnosis and management

    Mar 23, 2026Jack W O'Sullivan, Mohammad Asadi, Lennart Elbe +10Medical VLMsMultimodal Understanding

    Division of Cardiology, Department of Medicine, Stanford University, CA, USA · Department of Biomedical Data Science, Stanford University, CA, USA · Department of Electrical Engineering, Stanford University, CA, USA +7