cs.AIMar 23, 2026
SaveMARCUS: An agentic, multimodal vision-language model for cardiac diagnosis and management
Organizations: Division of Cardiology, Department of Medicine, Stanford University, CA, USA · Department of Biomedical Data Science, Stanford University, CA, USA · Department of Electrical Engineering, Stanford University, CA, USA · Department of Medicine, Radiology, and Pediatrics, UCSF, CA, USA · Bakar Institute, UCSF, CA, USA · UCSF–UC Berkeley Joint Program in Computational Precision Health, CA, USA · Department of Radiology, Stanford University, CA, USA · Department of Biology, Stanford University, CA, USA · Department of Computer Science, Stanford University, CA, USA · Department of Psychiatry and Behavioral Sciences, Stanford University, CA, USA
Abstract
Cardiovascular disease remains the leading cause of global mortality, with progress hindered by human interpretation of complex cardiac tests. Current AI vision-language models are limited to single-modality inputs and are non-interactive. We present MARCUS (Multimodal Autonomous Reasoning and Chat for Ultrasound and Signals), an agentic vision-language system for end-to-end interpretation of electrocardiograms (ECGs), echocardiograms, and cardiac magnetic resonance imaging (CMR) independently and as multimodal input. MARCUS employs a hierarchical agentic architecture comprising modality-specific vision-language expert models, each integrating domain-trained visual encoders with multi-stage language model optimization, coordinated by a multimodal orchestrator. Trained on 13.5 million images (0.25M ECGs, 1.3M echocardiogram images, 12M CMR images) and our novel expert-curated dataset spanning 1.6 million questions, MARCUS achieves state-of-the-art performance surpassing frontier models (GPT-5 Thinking, Gemini 2.5 Pro Deep Think). Across internal (Stanford) and external (UCSF) test cohorts, MARCUS achieves accuracies of 87-91% for ECG, 67-86% for echocardiography, and 85-88% for CMR, outperforming frontier models by 34-45% (P<0.001). On multimodal cases, MARCUS achieved 70% accuracy, nearly triple that of frontier models (22-28%), with 1.7-3.0x higher free-text quality scores. Our agentic architecture also confers resistance to mirage reasoning, whereby vision-language models derive reasoning from unintended textual signals or hallucinated visual content. MARCUS demonstrates that domain-specific visual encoders with an agentic orchestrator enable multimodal cardiac interpretation. We release our models, code, and benchmark open-source.
Figures & tables
Figure 1: Study design and evaluation framework for MARCUS. a. Overview of the MARCUS architecture and training pipeline. The model integrates three cardiac diagnostic modalities: electrocardiography (ECG), echocardiography (Echo), and cardiac magnetic resonance (CMR). ECG data (0.25M ECGs) are processed via a SigLIP vision encoder and a 2-layer MLP projection into a 3B parameter language model. Echo (1.27M images) and CMR (12.19M images) data utilize multi-view visual encoders with temporal aggregation and cross-view fusion modules, feeding into a language model optimized via Group Relative Policy Optimization (GRPO). A Multimodal Orchestrating Agent synthesizes these inputs into a unified text output. The training process employs a multi-stage optimization strategy, including supervised fine-tuning on 741,000 visual Q&A pairs and GRPO reinforcement learning on 879,000 diagnostic multiple-choice questions. b. Evaluation and validation workflow. The model’s performance was benchmarked using an internal test set (Stanford) and an external validation cohort (UCSF). The evaluation encompasses two distinct tracks: (1) Visual Multi-choice Questions across all three cardiac modalities to assess precision, and (2) Open-ended Clinical Reasoning Text Questions to evaluate the model’s capacity for expert-level free-text synthesis and interaction.
Figure 2: Comparative performance evaluation of MARCUS against frontier models. a. Accuracy (%) of MARCUS, Gemini 2.5 (Deep Think), and GPT-5 (Thinking) on multiple-choice questions (MCQ) across four domains: Echocardiography (Echo), Cardiovascular Magnetic Resonance (CMR), Electrocardiography (ECG), and Multimodal (All). b. Expert-rated performance on free-text clinical reasoning Visual Question Answering (VQA) tasks, represented by Mean Likert scores (scale 1–5). c. Radar plot illustrating the holistic performance profile across all evaluation metrics. d. Representative qualitative examples of VQA and MCQ prompts for Echo, CMR, and ECG, including model-generated answers.
Figure 3: Granular performance breakdown across clinical sub-categories in free-text, open-ended clinical reasoning VQA. a–d. Mean Likert scores (scale 1–5) for MARCUS, Gemini 2.5, and GPT-5 evaluated across specific diagnostic domains within Echocardiography (a), Cardiovascular Magnetic Resonance (CMR) (b), Electrocardiography (ECG) (c), and Multimodal integration (d).
Figure 4: External validation of MARCUS and GPT-5 across Stanford and UCSF datasets. a. Visual question answering (VQA) performance, reported as mean Likert score (scale 1–5). b. Multiple-choice question (MCQ) accuracy (%). MARCUS substantially outperforms GPT-5 across all modalities and both task types, with consistent performance across institutions demonstrating robust external generalisability.
Figure 5: Discrete disease-category diagnostic accuracy for MARCUS, Gemini 2.5 Pro, and GPT-5 when answering multiple-choice questions (MCQs) on cardiac imaging. a. Multimodal. b. CMR. c. Echocardiography. d. ECG. The radial gridlines indicate accuracy levels of 25%, 50%, 75%, and 100%.
Figure 6: MARCUS versus purpose-built single-modality cardiac AI models, on the Stanford internal test set and on five external public test sets spanning four countries. a , Multiple-choice accuracy (%) on the Stanford test set for MARCUS vs each single modality model. b , Free-text mean Likert (1–5; 1 = clinically wrong, 5 = fully correct) on the Stanford test set. c , Multiple-choice accuracy on five external public test sets: MIMIC-IV-ECG and PTB-XL (12-lead ECG; USA and Germany), MIMIC-IV-ECHO (echocardiography; USA), and ACDC and UK Biobank (cardiac MRI; France and the UK). d , Free-text mean Likert on the same five external test sets. EchoNext’s twelve binary structural-heart-disease probabilities were formatted as a clinical pseudo-report (matching the procedure used for ViTa and PanEcho) and graded on the same 1–5 Likert rubric as MARCUS. Brackets indicate paired hypothesis tests with significance stars: McNemar’s exact test for multiple choice ( a , c ), Wilcoxon signed-rank test for mean Likert ( b , d ); , , , ns = not significant. Dashed grey line indicates chance performance on four-option multiple-choice questions.
| Modality | VQA Questions | VQA Unique Studies | MCQ Questions | MCQ Unique Studies |
|---|---|---|---|---|
| Echo | 100 | 82 | 50 | 46 |
| CMR | 100 | 78 | 50 | 46 |
| ECG | 100 | 82 | 50 | 41 |
| Multimodal | 100 | — | 40 | — |
| Total | 400 | 190 |
Supplementary Table 1 : Stanford test set composition by modality and question type.
| Modality | Unique Studies | VQA Questions | MCQ Questions | Avg VQA Q/study | Avg MCQ Q/study |
|---|---|---|---|---|---|
| Echo | 411–414 | 8,119 | 10,345 | 19.8 | 25.0 |
| CMR | 455–457 | 6,874 | 6,226 | 15.0 | 13.7 |
| ECG | 406–409 | 4,572 | 2,691 | 11.3 | 6.6 |
| Total | 1,274 | 19,565 | 19,262 |
Supplementary Table 2 : UCSF dataset composition by modality and question type.
| Dataset | Modality | Country | MCQ | VQA |
|---|---|---|---|---|
| ACDC | Cardiac MRI | France | 520 | 50 |
| UK Biobank | Cardiac MRI | UK | 337 | 29 |
| MIMIC-IV-ECHO | Echo | USA | 4,795 | 750 |
| MIMIC-IV-ECG | 12-lead ECG | USA | 3,000 | 479 |
| PTB-XL | 12-lead ECG | Germany | 1,676 | 321 |
Supplementary Table 3 : Public-dataset external validation cohort. Source modality, country of origin, and item counts per dataset.
| Parameter | Stage 1 (Pretrain) | Stage 2 (SFT) | Stage 3 (GRPO) |
|---|---|---|---|
| Base model | Qwen2.5-VL-3B-Instruct | Qwen2.5-VL-3B-Instruct | Qwen2.5-VL-3B-Instruct |
| Trainable | Vision encoder + MLP | Full model | Full model (policy) |
| Frozen | Qwen2 LLM | None | Reference model |
| Learning rate | 2.0e-4 | 1.0e-5 | 1.0e-6 |
| LR scheduler | Cosine | Cosine | — |
| Warmup ratio | 0.05 | 0.05 | — |
Supplementary Table 4 : Training hyperparameters for all three stages of the MARCUS training pipeline.
| Cohort | Modality / dataset | Single-modality model | MCQ | VQA (binary) | VQA (mean Likert) |
|---|---|---|---|---|---|
| Stanford internal test set | |||||
| Stanford | ECG (in-scope) | EchoNext | 153 | 62 | 62 |
| Stanford | Echo | EchoPrime | 9,008 | 4,980 | 4,980 |
| Stanford | Echo | PanEcho | 9,017 | 3,088 | 3,088 |
| Stanford | CMR | ViTa | 3,357 | 2,705 | 2,705 |
| External multi-country public test sets | |||||
Supplementary Table 5 : Paired sample sizes for every MARCUS-versus-single-modality-model comparison reported in the main text. Stanford ECG sample sizes are restricted to the in-scope paired subset (in-scope items only; see Section 5.2 ); external sample sizes use the full test set with EchoNext’s out-of-scope items marked incorrect under the clinical-deployment-fairness rule.
Supplementary Figure 1 : An overview of the vision-language model for each modality expert. Each ECG trace, echocardiography video, or MRI video is broken into patches and fed into the vision transformer. The vision embeddings are then run through a cross-attention layer and then integrated into the LLM blocks by (1) concatenation with the text tokens and (2) residual connections between various depths of the vision transformers (ViT) and the LLM blocks.
Supplementary Figure 2 : Agentic orchestrator architecture. The orchestrator interacts with the user, decomposes the questions into sub-questions for each of the single modalities (if necessary), interacts with the tools (single-modality experts) to infer information from each modality, combines the observations and findings, manages conflicts, and aggregates the results into the final answer to return to the user.
Supplementary Figure 3 : Multi-stage training pipeline for MARCUS. Stage 1: Pre-training (Frozen LLM). A specialized visual encoder is pre-trained for distinct cardiac modalities (ECG, Echo, CMR) to extract dense latent representations from medical imaging data. During this stage, the parameters of the decoder-only large language model (LLM block) are held frozen to preserve its pre-existing linguistic knowledge while aligning the new visual features. Stage 2: Fine-tuning (Active Models). The pre-trained vision encoder and the LLM block are simultaneously fine-tuned on visual question answering tasks. Stage 3: RL (GRPO Optimization). Model precision for diagnostics-focused queries is further enhanced through reinforcement learning using Group-Relative Policy Optimization (GRPO).
Supplementary Figure 4 : Susceptibility of medical expert models to mirage reasoning and its mitigation via agentic orchestration in MARCUS. A. Mechanism of mirage reasoning. B. Quantification of mirage rates in modality-specific expert models within MARCUS. C. System-level mitigation via the agentic Orchestrator within MARCUS achieving a zero-percent mirage rate.
| Modality | Model | Accuracy | 95% CI / p-value | |
|---|---|---|---|---|
| ECG | MARCUS | 50 | 88.0% | (78.0–96.0%) |
| ECG | GPT-5 | 50 | 48.0% | (34.0–62.0%) |
| ECG | Gemini 2.5 Pro | 47 | 46.8% | (34.0–61.7%) |
| ECG | MARCUS vs GPT-5 (McNemar p) | 50 | — | 0.001 |
| ECG | MARCUS vs Gemini (McNemar p) | 47 | — | 0.001 |
| Echo | MARCUS | 50 | 64.0% | (50.0–78.0%) |
Supplementary Table 6 : MCQ accuracy with bootstrapped 95% confidence intervals and McNemar’s exact test p-values.
| Modality | Model | Mean | 95% CI (mean) | Median | 95% CI (median) | p-value | |
|---|---|---|---|---|---|---|---|
| ECG | MARCUS | 100 | 3.65 | (3.37–3.91) | 4.0 | (4.0–4.0) | — |
| ECG | GPT-5 | 100 | 2.60 | (2.34–2.87) | 3.0 | (2.0–3.0) | 0.001 |
| ECG | Gemini 2.5 Pro | 100 | 2.55 | (2.29–2.81) | 2.0 | (2.0–3.0) | 0.001 |
| Echo | MARCUS | 100 | 2.41 | (2.14–2.69) | 2.0 | (2.0–3.0) | — |
| Echo | GPT-5 | 100 | 1.97 | (1.75–2.20) | 2.0 | (1.0–2.0) | 0.041 |
| Echo | Gemini 2.5 Pro | 100 | 1.47 | (1.31–1.65) | 1.0 | (1.0–1.0) | 0.001 |
Supplementary Table 7 : VQA Likert score statistics.
Supplementary Figure 5 : Confusion matrices for MCQ evaluation across modalities. Rows represent ground-truth answer options (A–E); columns represent predicted answer options. Cell values show raw counts; colour intensity reflects the row-normalised conditional probability (0–1 scale). Diagonal cells indicate correct predictions. Three models are shown: MARCUS (left), GPT-5 (centre), Gemini 2.5 Pro (right).
Supplementary Figure 6 : Per-category mean Likert score heatmaps for VQA tasks. A: ECG categories. B: Echocardiography categories. C: CMR categories. D: Multimodal categories. Cell values are mean Likert scores (scale 1–5). Colour scale: red (1) to green (5). All three models (MARCUS, GPT-5, Gemini 2.5 Pro) are shown as column groups within each panel.
Supplementary Figure 7 : Violin plots of Likert score distributions per modality for MARCUS (blue), GPT-5 (red), and Gemini 2.5 Pro (green). Individual responses are overlaid as jittered strip plots (semi-transparent points). Horizontal bars within each violin indicate the median. Annotated means ( ) are shown above each violin. Likert scale: 1 = poor, 5 = excellent.
Supplementary Figure 8 : Frequency histograms of Likert score distributions. Bars show the percentage of responses assigned each integer score (1–5) for MARCUS (blue), GPT-5 (red), and Gemini 2.5 Pro (green) within each modality. Scores 1–2 indicate clinically inadequate responses; 3 indicates acceptable; 4–5 indicate good to expert quality.
Supplementary Figure 9 : Error type distribution for MARCUS failure cases. Left panel: MCQ failures ( =40 across all modalities), categorised by error type and stratified by modality. Right panel: VQA failures (Likert 2, =169), with the same stratification.