MARCUS: An agentic, multimodal vision-language model for cardiac diagnosis and management
Authors: Jack W O'Sullivan, Mohammad Asadi, Lennart Elbe, Akshay Chaudhari, Tahoura Nedaee, Francois Haddad, Ivan Lopez, Fang Cao, +5 more
Organizations: Division of Cardiology, Department of Medicine, Stanford University, CA, USA · Department of Biomedical Data Science, Stanford University, CA, USA · Department of Electrical Engineering, Stanford University, CA, USA · Department of Medicine, Radiology, and Pediatrics, UCSF, CA, USA · Bakar Institute, UCSF, CA, USA · UCSF–UC Berkeley Joint Program in Computational Precision Health, CA, USA · Department of Radiology, Stanford University, CA, USA · Department of Biology, Stanford University, CA, USA · Department of Computer Science, Stanford University, CA, USA · Department of Psychiatry and Behavioral Sciences, Stanford University, CA, USA
Cardiovascular disease remains the leading cause of global mortality, with progress hindered by human interpretation of complex cardiac tests. Current AI vision-language models are limited to single-modality inputs and are non-interactive. We present MARCUS (Multimodal Autonomous Reasoning and Chat for Ultrasound and Signals), an agentic vision-language system for end-to-end interpretation of electrocardiograms (ECGs), echocardiograms, and cardiac magnetic resonance imaging (CMR) independently and as multimodal input. MARCUS employs a hierarchical agentic architecture comprising modality-specific vision-language expert models, each integrating domain-trained visual encoders with multi-stage language model optimization, coordinated by a multimodal orchestrator. Trained on 13.5 million images (0.25M ECGs, 1.3M echocardiogram images, 12M CMR images) and our novel expert-curated dataset spanning 1.6 million questions, MARCUS achieves state-of-the-art performance surpassing frontier models (GPT-5 Thinking, Gemini 2.5 Pro Deep Think). Across internal (Stanford) and external (UCSF) test cohorts, MARCUS achieves accuracies of 87-91% for ECG, 67-86% for echocardiography, and 85-88% for CMR, outperforming frontier models by 34-45% (P<0.001). On multimodal cases, MARCUS achieved 70% accuracy, nearly triple that of frontier models (22-28%), with 1.7-3.0x higher free-text quality scores. Our agentic architecture also confers resistance to mirage reasoning, whereby vision-language models derive reasoning from unintended textual signals or hallucinated visual content. MARCUS demonstrates that domain-specific visual encoders with an agentic orchestrator enable multimodal cardiac interpretation. We release our models, code, and benchmark open-source.
Figures & tables
Figure 1: Study design and evaluation framework for MARCUS. a. Overview of the MARCUS architecture and training pipeline. The model integrates three cardiac diagnostic modalities: electrocardiography (ECG), echocardiography (Echo), and cardiac magnetic resonance (CMR). ECG data (0.25M ECGs) are processed via a SigLIP vision encoder and a 2-layer MLP projection into a 3B parameter language model. Echo (1.27M images) and CMR (12.19M images) data utilize multi-view visual encoders with temporal aggregation and cross-view fusion modules, feeding into a language model optimized via Group Relative Policy Optimization (GRPO). A Multimodal Orchestrating Agent synthesizes these inputs into a unified text output. The training process employs a multi-stage optimization strategy, including supervised fine-tuning on 741,000 visual Q&A pairs and GRPO reinforcement learning on 879,000 diagnostic multiple-choice questions. b. Evaluation and validation workflow. The model’s performance was benchmarked using an internal test set (Stanford) and an external validation cohort (UCSF). The evaluation encompasses two distinct tracks: (1) Visual Multi-choice Questions across all three cardiac modalities to assess precision, and (2) Open-ended Clinical Reasoning Text Questions to evaluate the model’s capacity for expert-level free-text synthesis and interaction.
Figure 2: Comparative performance evaluation of MARCUS against frontier models. a. Accuracy (%) of MARCUS, Gemini 2.5 (Deep Think), and GPT-5 (Thinking) on multiple-choice questions (MCQ) across four domains: Echocardiography (Echo), Cardiovascular Magnetic Resonance (CMR), Electrocardiography (ECG), and Multimodal (All). b. Expert-rated performance on free-text clinical reasoning Visual Question Answering (VQA) tasks, represented by Mean Likert scores (scale 1–5). c. Radar plot illustrating the holistic performance profile across all evaluation metrics. d. Representative qualitative examples of VQA and MCQ prompts for Echo, CMR, and ECG, including model-generated answers.
Figure 3: Granular performance breakdown across clinical sub-categories in free-text, open-ended clinical reasoning VQA. a–d. Mean Likert scores (scale 1–5) for MARCUS, Gemini 2.5, and GPT-5 evaluated across specific diagnostic domains within Echocardiography (a), Cardiovascular Magnetic Resonance (CMR) (b), Electrocardiography (ECG) (c), and Multimodal integration (d).
Figure 4: External validation of MARCUS and GPT-5 across Stanford and UCSF datasets. a. Visual question answering (VQA) performance, reported as mean Likert score (scale 1–5). b. Multiple-choice question (MCQ) accuracy (%). MARCUS substantially outperforms GPT-5 across all modalities and both task types, with consistent performance across institutions demonstrating robust external generalisability.
Figure 5: Discrete disease-category diagnostic accuracy for MARCUS, Gemini 2.5 Pro, and GPT-5 when answering multiple-choice questions (MCQs) on cardiac imaging. a. Multimodal. b. CMR. c. Echocardiography. d. ECG. The radial gridlines indicate accuracy levels of 25%, 50%, 75%, and 100%.
Figure 6: MARCUS versus purpose-built single-modality cardiac AI models, on the Stanford internal test set and on five external public test sets spanning four countries. a , Multiple-choice accuracy (%) on the Stanford test set for MARCUS vs each single modality model. b , Free-text mean Likert (1–5; 1 = clinically wrong, 5 = fully correct) on the Stanford test set. c , Multiple-choice accuracy on five external public test sets: MIMIC-IV-ECG and PTB-XL (12-lead ECG; USA and Germany), MIMIC-IV-ECHO (echocardiography; USA), and ACDC and UK Biobank (cardiac MRI; France and the UK). d , Free-text mean Likert on the same five external test sets. EchoNext’s twelve binary structural-heart-disease probabilities were formatted as a clinical pseudo-report (matching the procedure used for ViTa and PanEcho) and graded on the same 1–5 Likert rubric as MARCUS. Brackets indicate paired hypothesis tests with significance stars: McNemar’s exact test for multiple choice ( a , c ), Wilcoxon signed-rank test for mean Likert ( b , d ); ∗∗∗P<0.001 , ∗∗P<0.01 , ∗P<0.05 , ns = not significant. Dashed grey line indicates chance performance on four-option multiple-choice questions.
Modality
VQA Questions
VQA Unique Studies
MCQ Questions
MCQ Unique Studies
Echo
100
82
50
46
CMR
100
78
50
46
ECG
100
82
50
41
Multimodal
100
—
40
—
Total
400
190
Supplementary Table 1 : Stanford test set composition by modality and question type.
Modality
Unique Studies
VQA Questions
MCQ Questions
Avg VQA Q/study
Avg MCQ Q/study
Echo
411–414
8,119
10,345
19.8
25.0
CMR
455–457
6,874
6,226
15.0
13.7
ECG
406–409
4,572
2,691
11.3
6.6
Total
∼ 1,274
19,565
19,262
Supplementary Table 2 : UCSF dataset composition by modality and question type.
Dataset
Modality
Country
n MCQ
n VQA
ACDC
Cardiac MRI
France
520
50
UK Biobank
Cardiac MRI
UK
337
29
MIMIC-IV-ECHO
Echo
USA
4,795
750
MIMIC-IV-ECG
12-lead ECG
USA
3,000
479
PTB-XL
12-lead ECG
Germany
1,676
321
Supplementary Table 3 : Public-dataset external validation cohort. Source modality, country of origin, and item counts per dataset.
Parameter
Stage 1 (Pretrain)
Stage 2 (SFT)
Stage 3 (GRPO)
Base model
Qwen2.5-VL-3B-Instruct
Qwen2.5-VL-3B-Instruct
Qwen2.5-VL-3B-Instruct
Trainable
Vision encoder + MLP
Full model
Full model (policy)
Frozen
Qwen2 LLM
None
Reference model
Learning rate
2.0e-4
1.0e-5
1.0e-6
LR scheduler
Cosine
Cosine
—
Warmup ratio
0.05
0.05
—
Supplementary Table 4 : Training hyperparameters for all three stages of the MARCUS training pipeline.
Cohort
Modality / dataset
Single-modality model
n MCQ
n VQA (binary)
n VQA (mean Likert)
Stanford internal test set
Stanford
ECG (in-scope)
EchoNext
153
62
62
Stanford
Echo
EchoPrime
9,008
4,980
4,980
Stanford
Echo
PanEcho
9,017
3,088
3,088
Stanford
CMR
ViTa
3,357
2,705
2,705
External multi-country public test sets
Supplementary Table 5 : Paired sample sizes for every MARCUS-versus-single-modality-model comparison reported in the main text. Stanford ECG sample sizes are restricted to the in-scope paired subset (in-scope items only; see Section 5.2 ); external sample sizes use the full test set with EchoNext’s out-of-scope items marked incorrect under the clinical-deployment-fairness rule.
Supplementary Figure 1 : An overview of the vision-language model for each modality expert. Each ECG trace, echocardiography video, or MRI video is broken into 16×16 patches and fed into the vision transformer. The vision embeddings are then run through a cross-attention layer and then integrated into the LLM blocks by (1) concatenation with the text tokens and (2) residual connections between various depths of the vision transformers (ViT) and the LLM blocks.
Supplementary Figure 2 : Agentic orchestrator architecture. The orchestrator interacts with the user, decomposes the questions into sub-questions for each of the single modalities (if necessary), interacts with the tools (single-modality experts) to infer information from each modality, combines the observations and findings, manages conflicts, and aggregates the results into the final answer to return to the user.
Supplementary Figure 3 : Multi-stage training pipeline for MARCUS. Stage 1: Pre-training (Frozen LLM). A specialized visual encoder is pre-trained for distinct cardiac modalities (ECG, Echo, CMR) to extract dense latent representations from medical imaging data. During this stage, the parameters of the decoder-only large language model (LLM block) are held frozen to preserve its pre-existing linguistic knowledge while aligning the new visual features. Stage 2: Fine-tuning (Active Models). The pre-trained vision encoder and the LLM block are simultaneously fine-tuned on visual question answering tasks. Stage 3: RL (GRPO Optimization). Model precision for diagnostics-focused queries is further enhanced through reinforcement learning using Group-Relative Policy Optimization (GRPO).
Supplementary Figure 4 : Susceptibility of medical expert models to mirage reasoning and its mitigation via agentic orchestration in MARCUS. A. Mechanism of mirage reasoning. B. Quantification of mirage rates in modality-specific expert models within MARCUS. C. System-level mitigation via the agentic Orchestrator within MARCUS achieving a zero-percent mirage rate.
Modality
Model
N
Accuracy
95% CI / p-value
ECG
MARCUS
50
88.0%
(78.0–96.0%)
ECG
GPT-5
50
48.0%
(34.0–62.0%)
ECG
Gemini 2.5 Pro
47
46.8%
(34.0–61.7%)
ECG
MARCUS vs GPT-5 (McNemar p)
50
—
< 0.001
ECG
MARCUS vs Gemini (McNemar p)
47
—
< 0.001
Echo
MARCUS
50
64.0%
(50.0–78.0%)
Supplementary Table 6 : MCQ accuracy with bootstrapped 95% confidence intervals and McNemar’s exact test p-values.
Supplementary Figure 5 : Confusion matrices for MCQ evaluation across modalities. Rows represent ground-truth answer options (A–E); columns represent predicted answer options. Cell values show raw counts; colour intensity reflects the row-normalised conditional probability (0–1 scale). Diagonal cells indicate correct predictions. Three models are shown: MARCUS (left), GPT-5 (centre), Gemini 2.5 Pro (right).
Supplementary Figure 6 : Per-category mean Likert score heatmaps for VQA tasks. A: ECG categories. B: Echocardiography categories. C: CMR categories. D: Multimodal categories. Cell values are mean Likert scores (scale 1–5). Colour scale: red (1) to green (5). All three models (MARCUS, GPT-5, Gemini 2.5 Pro) are shown as column groups within each panel.
Supplementary Figure 7 : Violin plots of Likert score distributions per modality for MARCUS (blue), GPT-5 (red), and Gemini 2.5 Pro (green). Individual responses are overlaid as jittered strip plots (semi-transparent points). Horizontal bars within each violin indicate the median. Annotated means ( μ ) are shown above each violin. Likert scale: 1 = poor, 5 = excellent.
Supplementary Figure 8 : Frequency histograms of Likert score distributions. Bars show the percentage of responses assigned each integer score (1–5) for MARCUS (blue), GPT-5 (red), and Gemini 2.5 Pro (green) within each modality. Scores 1–2 indicate clinically inadequate responses; 3 indicates acceptable; 4–5 indicate good to expert quality.
Supplementary Figure 9 : Error type distribution for MARCUS failure cases. Left panel: MCQ failures ( n =40 across all modalities), categorised by error type and stratified by modality. Right panel: VQA failures (Likert ≤ 2, n =169), with the same stratification.
While Multimodal Large Language Models (MLLMs) show promising performance in automated electrocardiogram interpretation, it remains unclear whether they genuinely perform actual step-by-step reasoning or just rely on superficial visual cues. To investigate this, we introduce ECG-Reasoning-Benchmark, a novel multi-turn evaluation framework comprising over 6,400 samples to systematically assess step-by-step reasoning across 17 core ECG diagnoses. Our comprehensive evaluation of state-of-the-art models reveals a critical failure in executing multi-step logical deduction. Although models possess the medical knowledge to retrieve clinical criteria for a diagnosis, they exhibit near-zero success rates (6% Completion) in maintaining a complete reasoning chain, primarily failing to ground the corresponding ECG findings to the actual visual evidence in the ECG signal. These results demonstrate that current MLLMs bypass actual visual interpretation, exposing a critical flaw in existing training paradigms and underscoring the necessity for robust, reasoning-centric medical AI. The code and data are available at https://github.com/Jwoo5/ecg-reasoning-benchmark.
Jungwoo Oh, Hyunseung Chung, Junhee Lee +6
KAIST · Ajou University School of Medicine · Yonsei University College of Medicine +1
Echocardiography is the most widely used non-invasive cardiac imaging modality, providing essential information for cardiovascular diagnosis. Interpreting an echocardiogram requires synthesizing complementary evidence across multiple heart views to identify abnormalities and produce structured clinical reports. While recent efforts focus on improving classification performance, most models lack explicit diagnostic reasoning and spatially grounded anatomical evidence, limiting clinician trust. We present EchoSonar-R, a multi-view reasoning-enabled vision-language model that jointly performs multi-label disease classification and report generation from echocardiography studies. EchoSonar-R combines a spatiotemporal video encoder with a structure-aware cardiac detector that provides spatially grounded anatomical cues to improve interpretability and clinician trust during cross-view reasoning. EchoSonar-R is trained in two stages: supervised fine-tuning (SFT) on reasoning-annotated targets, followed by Group Relative Policy Optimization (GRPO) with task-specific rewards that jointly align classification and report generation within a unified reinforcement-learning framework. Across a private multi-view dataset and two public benchmarks, EchoSonar-R improves macro balanced accuracy by 17.1% on the private set and 6.1% on MIMICEchoQA over the strongest baseline, achieves a GREEN clinical faithfulness score of 0.800, and produces interpretable reasoning traces grounded in multi-view visual evidence.
Darya Taratynova, Ahmed Aly, Numan Saeed +1
Division of Computing and Mathematical Sciences, Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, UAE
Cardiovascular diagnosis and treatment rest on integrating complementary modalities, such as electrocardiogram, echocardiography, and chest X-rays, each capturing distinct but complementary aspects of cardiac pathophysiology. Yet most medical foundation models remain modality-specific, combining modalities only for finetuning or post-training. This discards the cross-modal evidence clinicians naturally integrate and ignores the structure within each modality. We introduce Latent-Attention Masked Autoencoder (LAMAE), a multimodal, structure-aware masked autoencoder that jointly learns patient-level representations during self-supervised pretraining. Instead of fusing modalities post hoc, LAMAE exchanges information directly in the latent space through a shared latent-attention module operating over a study-view-entity hierarchy, enabling aggregation of variable observations and handling of missing modalities. Pretrained on over 500'000 MIMIC-IV hospital stays, LAMAE outperforms modality-specific pretraining and strong contrastive and vision-language baselines across multimodal hospital-stay tasks, such as in-hospital mortality, ICD-10 and DRG coding, and length of stay, while remaining competitive on unimodal tasks. Modeling this structure also pays off within a single modality: even without cross-modal information, the latent-attention module improves representations over modality-specific pretraining.
Andrea Agostini, Simon Böhi, Moritz Vandenhirtz +7
Department of Computer Science, ETH Zurich, Switzerland · Department of Biomedical Engineering, University of Basel, Switzerland