Learning Where to Look: Anatomical Grounding and Guided Attention for Cardiac MRI Vision-Language Models

Bangwei Guo; Xiao Chen; Boris Mailhe; Jia Yao; Yiqing Wang; Ankush Mukherjee; Yikang Liu; Zheyuan Zhang

cs.CVSep 30, 2026

Learning Where to Look: Anatomical Grounding and Guided Attention for Cardiac MRI Vision-Language Models

Authors: Bangwei Guo, Xiao Chen, Boris Mailhe, Jia Yao, Yiqing Wang, Ankush Mukherjee, Yikang Liu, Zheyuan Zhang, +3 more

Organizations: Rutgers University, NJ, USA · United Imaging Intelligence, Boston, MA, USA · University of Texas Southwestern Medical Center, TX, USA · Duke University, NC, USA

Abstract

Cardiac magnetic resonance imaging (CMR) enables assessment of cardiac anatomy, ventricular function, and myocardial tissue characteristics. Clinicians interpret these images by identifying cardiac structures and focusing on the regions relevant to each clinical question, motivating anatomically guided vision-language models (VLMs). Yet CMR-specific supervision for anatomical localisation and clinical question answering remains limited. To address this gap, we investigate fine-grained CMR visual question answering through anatomical grounding and guided attention. We construct 128,915 anatomical-grounding and 42,799 clinical QA pairs across short-axis cine, late gadolinium enhancement, and long-axis cine. These datasets support anatomical recognition, localisation, and clinical assessment without requiring paired reports for individual training images. To help the model learn where to look, we introduce Cardiac Anatomy-Routed Attention (CARA), which selects predicted anatomical priors according to the question and guides decoder attention with learned task-specific strengths. Combining anatomical grounding pretraining with CARA yields our model, CARA-VL. Experiments demonstrate CARA-VL's strengths in clinical assessment and regional localisation across CMR imaging settings, with promising generalization to an external clinical cohort. Together, our data and method provide a practical framework for studying and advancing cardiac visual understanding in VLMs. We will release the QA data derived from public datasets upon publication.

Figures & tables

Figure 1: CMR VQA dataset construction pipeline. (A) Clinical report analysis guides the selection of 14 assessable clinical QA tasks. (B–E) Multi-source CMR images are standardised and paired with expert or quality-controlled automatic masks. (F) Expert masks and acquisition metadata generate nine types of anatomical-grounding QA for identifying and localising cardiac structures. (G) Image-derived measurements and task-specific decision rules generate clinical QA on cardiac structure, function, and myocardial scar.

Figure 2: Two-stage training with CARA. Stage 1 learns anatomical grounding while keeping the language model frozen. Stage 2 jointly trains clinical answer generation and anatomical occupancy prediction. CARA selects the question-relevant occupancy map and injects the resulting anatomical prior into decoder attention with a learned task-specific strength.

(a) SAX cine
Model	Hyper.	LV size	RV size	EF	GRS	GCS	Avg.	Reg. hyper. $\dagger$	Reg. motion $\dagger$
General medical VLMs
LLaVA-Med-7B	50.0 / 9.9	50.0 / 24.3	50.0 / 42.4	47.5 / 39.9	50.0 / 41.0	50.0 / 38.8	49.6 / 32.7	9.2	31.2
MedGemma-4B-it	56.6 / 55.7	60.7 / 46.0	36.7 / 35.5	50.0 / 27.8	50.0 / 23.4	51.1 / 39.4	50.9 / 38.0	9.2	31.9
Huatuo-Vision	50.0 / 47.1	50.0 / 24.3	50.0 / 20.9	50.0 / 27.8	52.1 / 35.5	59.5 / 59.5	51.9 / 35.9	0.9	24.1
Lingshu-7B	50.0 / 47.1	44.4 / 27.6	50.8 / 22.7	50.0 / 27.8	39.2 / 39.5	45.7 / 43.5	46.7 / 34.7	3.2	16.9

Table 1: QA-level clinical VQA results on the ACDC test set (SAX), the 50-patient LGE test set, and the M&Ms-2 test set (LAX). Classification cells show binary BA / macro-

F_{1}

; LV size, EF, and EDV are evaluated as normal versus abnormal. Regional tasks (

\dagger

) report micro-

F_{1}

. All scores are percentages(%).

General medical VLMs
Model	Reference	Hyper.	LV size	RV size	EF	GRS	GCS	Avg.	Reg. hyper.	Reg. motion
LLaVA-Med-7B	Rule-based	50.0 / 13.3	50.6 / 16.2	50.0 / 49.4	49.0 / 47.3	50.0 / 47.6	50.0 / 48.5	49.9 / 37.0	7.2	10.5
LLaVA-Med-7B	Report-based	50.0 / 23.0	50.4 / 41.2	50.0 / 40.4	49.6 / 49.4	50.0 / 42.8	50.0 / 43.0	50.0 / 40.0	32.2	28.9
MedGemma-4B-it	Rule-based	58.3 / 48.6	56.1 / 43.0	52.2 / 48.9	49.8 / 27.6	48.9 / 13.3	46.5 / 36.1	51.9 / 36.2	7.2	12.7
MedGemma-4B-it	Report-based	52.9 / 50.7	55.9 / 55.7	51.3 / 48.8	50.1 / 21.8	48.0 / 23.5	47.4 / 44.0	50.9 / 40.7	32.2	39.0
Huatuo-Vision	Rule-based	49.5 / 48.9	50.0 / 11.1	50.0 / 2.3	50.0 / 24.1	47.3 / 19.3	55.8 / 46.7	50.4 / 25.4	3.3	9.2

Table 2: External VQA results on In-house B. Classification cells show BA / macro-

F_{1}

; regional cells show micro-

F_{1}

(%). Avg. covers six classification tasks. Report-based GRS/GCS use wall-motion-score proxies; reference definitions, regional coverage, and report-field availability are detailed in Appendix C.6 .

Model	Stage-1	CARA	BA	Macro- $F_{1}$
CARA-VL (Ours)	$\checkmark$	$\checkmark$	90.7	90.0
w/o Stage-1	$\times$	$\checkmark$	88.4	87.9
w/o CARA	$\checkmark$	$\times$	86.8	86.5
w/o Stage-1 and CARA	$\times$	$\times$	86.2	86.0

Table 3: Ablation on ACDC SAX clinical QA after Stage-2 training. BA and macro-

F_{1}

are averaged across the six global classification tasks (%).

Figure 3: Attention maps from models trained without and with CARA for hypertrophy and LV-size questions on the same SAX image. Attention maps average all heads in the final eight decoder layers at the last prompt token. Attention maps share a colour scale within each row; anatomical priors are separately normalised to unit peak. ROI attention quantifies the fraction of visual attention allocated to the target anatomy.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Representative CMR views and spatially matched reference annotations. Top: end-diastolic SAX cine (ACDC), short-axis LGE with myocardial scar (MyoPS), and end-diastolic four-chamber LAX cine (M&Ms-2). Bottom: the corresponding segmentations. SAX shows ventricular cross-sections, LAX displays chamber geometry along the ventricular long axis, and LGE delineates hyperenhanced scar. Colours denote the LV cavity (blue), LV myocardium (orange), RV cavity (green), and myocardial scar (red). These are reference annotations, not model predictions.

Figure 5: Structured report audit over 3,215 CMR reports. (a) For each template field, the share of reports in which the field is not mentioned, explicitly normal, or carries an abnormal value, ordered by mention rate (explicitly normal

+

abnormal). A high mention rate with a low abnormality rate marks a finding that radiologists address routinely but rarely find abnormal. (b) Abnormal statements matching no template field.

Domain	Task and answer categories	Frames	Measurement and labelling rule
SAX	Hypertrophy: present / absent	ED	Present if any AHA segment has ED wall thickness $\geq 13\,\mathrm{mm}$ ; source-labelled normal cases are assigned negative.
	Regional hypertrophy: segment list	ED	Segments with ED wall thickness $\geq 13\,\mathrm{mm}$ ; six segments at basal and mid-ventricular levels, four at the apical level.
	LV size: normal / mild / moderate / severe dilation	ED+ES	Patient-level LV EDV: $<200$ , $[200,240)$ , $[240,300)$ , and $\geq 300\,\mathrm{mL}$ , respectively.
	EF: normal / mild / moderate / severe reduction	ED+ES	Whole-stack area-ratio EF: $\geq 50\%$ , $[40,50)\%$ , $[30,40)\%$ , and $<30\%$ , respectively.
	RV size: dilated / normal	ED+ES	Dilated if RV EDV $\geq 200\,\mathrm{mL}$ and RV/LV EDV ratio $\geq 1.3$ ; single-criterion positives require visual confirmation. Normal if both are below threshold; unconfirmed candidates are excluded.
	GRS surrogate: normal / reduced	ED+ES	Mean systolic wall thickening: $\geq 45\%$ normal, $<35\%$ reduced; exclude $[35,45)\%$ .

Appendix

Table 4: Definitions and labelling criteria for the 14 clinical QA tasks. ED and ES denote end-diastole and end-systole, respectively. The table summarises answer categories, measurement rules, and exclusion intervals. Segmentation sources, quality control, measurement derivation, and training-set statistics are detailed in Appendix C.5 .

Domain	Source	Mask source	Stage-1	Stage-2 train	Held out
Public datasets
SAX	ACDC	Expert	100 (8,845)	89 (1,803)	50
	M&Ms	Expert	320 (26,846)	281 (4,815)	–
	M&Ms-2	Expert	200 (16,449)	200 (3,841)	160
	CMR-MULTI cine	Expert	105 (10,093)	95 (1,459)	–
	MyoPS-related bSSFP cine	Auto. + QC	–	64 (994)	–

Appendix

Table 5: Actual source contributions to training and held-out evaluation.

Figure 6: Mid-ventricular AHA partitioning in a canonical anatomical orientation. The six

60^{\circ}

myocardial sectors carry AHA numbers 7–12; these are anatomical identifiers, not the generator’s local mask indices. The dashed ray runs from the LV-cavity centroid towards the septal anchor, separating the anteroseptal and inferoseptal sectors adjacent to the RV. Dots illustrate the two RV insertion locations. This is a geometric schematic, not a patient segmentation or a fixed scanner display orientation. Colours distinguish sectors only.

Figure 7: Examples of the nine Stage-1 grounding tasks from public ACDC, M&Ms-2, and MyoPS data. Each panel shows the frozen $448\times 448$ input, generated question, and target answer; phase uses two input frames. Coordinate targets appear only in text, whereas the highlights in highlight and segment are part of the input.

Task	Question $\rightarrow$ answer	Domains	QA
view	imaging view and sequence $\rightarrow$ “Short-axis, cine (bSSFP).”	SAX, LAX, LGE	7,005
phase	two shuffled frames of one slice $\rightarrow$ which frame is ED and which is ES	SAX, LAX	9,978
detect	enumerate all structures $\rightarrow$ name and box for each	SAX, LAX	11,738
bbox	named structure $\rightarrow$ $[x_{\min},y_{\min},x_{\max},y_{\max}]$	SAX, LAX	33,024
point	coordinate $[x,y]$ $\rightarrow$ structure name, or background	SAX, LAX, LGE	35,796
highlight	region highlighted in a random colour $\rightarrow$ structure name	SAX, LAX, LGE	15,038

Appendix

Table 6: Stage-1 grounding tasks. Answers are generated from the ground-truth mask.

Domain	Task	QA pairs	Answer counts
SAX	Hypertrophy	4,733	Absent / present: 4,243 / 490
	Regional hypertrophy	4,733	Unaffected / affected: 4,243 / 490
	LV size	3,985	2,963 / 412 / 304 / 306
	EF	3,971	2,291 / 732 / 418 / 530
	RV size	1,221	Normal / dilated: 994 / 227
	GRS surrogate	3,713	Normal / reduced: 2,939 / 774

Appendix

Table 7: Clinical training QA counts before oversampling. Four-grade counts follow the order normal, mild, moderate, and severe.

Figure 8: Clinical QA examples for the eight SAX tasks from ACDC, covering cardiac structure, ventricular function, and regional findings. Each row presents a task-specific question alongside two example inputs and their corresponding reference answers.

Figure 9: Clinical QA examples for the three LGE tasks from MyoPS and three LAX tasks from M&Ms-2, covering scar assessment and ventricular function. Each row presents a task-specific question alongside two example inputs and their corresponding reference answers.

A. Implemented criteria and reference scope
Task	Rule-based reference	Report-based reference	Difference
Hypertrophy	Any slice AHA wall thickness $\geq 13$ mm; reviewed basal/apical positives retained.	Septal or posterior wall thickness $\geq 11$ mm.	Cut-off gap: 2 mm; slice-wide vs. two report regions.
LV size	Product LV EDVi: normal $\leq 105$ ; mild $(105,130]$ ; moderate $(130,160]$ ; severe $>160$ .	Report LV EDVi above its recorded upper limit: 62 (female) or 75 (male); binary dilation.	Normal-limit gap: 43/30 mL/m 2 . No matched report four-grade scale.
RV size	Product RV EDVi $>110$ mL/m 2 ; if BSA is absent, RV EDV $>205$ mL.	Explicit normal/enlarged RV description in the report conclusion.	Quantitative rule vs. qualitative finding; not numerically comparable.
EF	Product EF: normal $\geq 50\%$ ; mild $[40,50)$ ; moderate $[30,40)$ ; severe $<30\%$ .	Report EF value mapped using the same 50/40/30% bands.	Cut-off gap: 0 percentage points; different measurement sources.
GRS	Mean mask-derived systolic wall thickening: reduced $<35\%$ ; normal $\geq 45\%$ ; $[35,45)$ excluded.	Any report-mapped middle-segment WMS $\geq 2$ indicates abnormal motion.	Thickening vs. WMS proxy; not equivalent strain references.

Appendix

Table 9: Reference-standard differences and paired-label disagreement in In-house B . These are comparisons between reference labels, not model scores. Panel B uses only documented, comparable reference units; rates are percentages.

Domain	Clinical task	Anatomical prior
SAX	Hypertrophy	MYO
SAX	Regional hypertrophy	MYO
SAX	LV size	LV
SAX	EF	LV
SAX	RV size	RV
SAX	Radial motion (GRS)	MYO

Appendix

Table 10: Anatomical priors for the 14 clinical tasks. LV and RV denote ventricular cavities, and MYO denotes myocardium. Each task has an independent learned injection strength.

Cohort	View	Reference	Task	$N$	ACC	BA	Macro- $F_{1}$
ACDC-50	SAX	Internal	LV size	171	78.4	63.9	62.1
ACDC-50	SAX	Internal	EF	171	79.5	59.3	53.4
M&Ms-2	LAX	Internal	EF	160	56.9	53.5	51.3
M&Ms-2	LAX	Internal	EDV	160	75.0	71.5	62.6
In-house B	SAX	Rule-based	LV size	1,783	70.1	41.4	31.6
In-house B	SAX	Rule-based	EF	1,795	52.4	51.6	44.4

Appendix

Table 11: Four-grade QA classification by CARA-VL on the internal and external test sets. Grades are normal, mild, moderate, and severe. All scores are percentages.

ACDC-50 (SAX)
Task	$N$	ACC	Sens.	Spec.	PPV	BA	Macro- $F_{1}$
Hypertrophy	457	93.4	78.0	95.3	67.2	86.7	84.3
LV size	171	89.5	90.9	88.8	79.4	89.9	88.4
RV size	170	91.8	75.6	97.6	91.9	86.6	88.8
EF	171	90.6	98.5	85.7	81.2	92.1	90.4
GRS	160	93.8	89.8	95.5	89.8	92.6	92.6

Appendix

Table 12: Additional binary classification metrics for CARA-VL on the internal test sets, using the same evaluations as Table 1 . All scores are percentages.

Rule-based
Task	$N$	ACC	Sens.	Spec.	PPV	BA	Macro- $F_{1}$
Hypertrophy	1,945	95.6	74.9	99.3	95.3	87.1	90.7
LV size	1,783	76.1	91.0	74.0	33.2	82.5	66.6
RV size	1,795	94.9	28.6	96.5	16.4	62.5	59.1
EF	1,795	66.5	92.6	54.4	48.5	73.5	66.3
GRS	1,708	95.1	77.7	96.9	71.8	87.3	86.0

Appendix

Table 13: Additional binary classification metrics for CARA-VL on In-house B under the two reference standards. All scores are percentages.

Cohort	Reference	Task	$N$	Prec.	Rec.	Micro- $F_{1}$	Exact	Jaccard
ACDC-50	Internal	Hypertrophy	457	65.8	58.3	61.8	88.2	91.3
ACDC-50	Internal	Wall motion	171	51.1	61.5	55.8	57.9	71.1
In-house B	Rule-based	Hypertrophy	1,831	71.6	44.8	55.1	91.2	93.3
In-house B	Rule-based	Wall motion	1,795	33.9	40.1	36.7	76.7	80.2
In-house B	Report-based	Hypertrophy	1,822	94.9	18.5	30.9	74.8	78.0
In-house B	Report-based	Wall motion	1,795	69.4	22.9	34.5	69.4	74.6

Appendix

Table 14: Additional regional-task metrics for CARA-VL. Precision, recall, and micro-

F_{1}

are pooled over question–region/segment pairs; exact match and mean Jaccard compare the affected-region sets per QA. All scores are percentages.

Explore similar work

CineMR: Tool-Integrated Vision-Language Reasoning for Quantitative Cardiac MRI Assessment
Oct 1, 2026Kunyang Li, Hai Nguyen, Joshua Lowe +6Cardiac Magnetic Resonance Imaging Films
Institute for Artificial Intelligence, University of Central Florida · Department of Clinical Sciences, College of Medicine, University of Central Florida · Nemours Cardiac Center, Nemours Children’s Hospital, Florida +1
CMRVision: A Foundation Model for Cardiac MR Image Analysis
Sep 1, 2026Athira J. Jacob, Puneet Sharma, Daniel RueckertCardiac Magnetic Resonance Imaging Echocardiography
Digital Technology and Innovation, Siemens Healthineers, Princeton, NJ, USA · Chair for Al in Healthcare and Medicine, Technical University of Munich (TUM) and TUM University Hospital, Munich, Germany · Department of Computing, Imperial College London, UK
CardioLens: Revealing the Clinical Reality Gap of MLLMs via Multi-Sequence Cardiac MRI Evaluations
May 28, 2026Zixian Su, Hongkai Zhang, Fan Gao +12Cardiac Magnetic Resonance Imaging Medical Vision-Language Models
Beijing Academy of Artificial Intelligence · Beijing Anzhen Hospital · Beihang University +1

Oct 1, 2026, cs.CVJ/K move · Enter open · S save

CineMR: Tool-Integrated Vision-Language Reasoning for Quantitative Cardiac MRI Assessment

Kunyang Li, Hai Nguyen, Joshua Lowe, Chenguang Zhao+5

Institute for Artificial Intelligence, University of Central Florida · Department of Clinical Sciences, College of Medicine, University of Central Florida · Nemours Cardiac Center, Nemours Children’s Hospital, Florida+1

Cardiac Magnetic Resonance Imaging Films

Open paper AskPDF arXiv

Cardiovascular magnetic resonance (CMR), including cine imaging, is a reference standard for the noninvasive assessment of cardiac morphology and ventricular function. Cine CMR interpretation integrates qualitative visual assessment with quantitative measurements of ventricular volumes, ejection fraction, myocardial mass, wall thickness, and regional wall motion. Current medical vision-language models (VLMs) cannot reliably derive quantitative measurements from multidimensional cine images without analysis tools. We present CineMR, a tool-augmented VLM that invokes cardiac image-analysis tools and integrates their outputs into interleaved reasoning for quantitative CMR assessment. We also construct a multi-cohort visual question answering benchmark covering quantitative metric extraction, multiclass diagnosis, and differential diagnosis, together with tools for segmentation, phase selection, volumetry, morphometry, and regional wall motion analysis. CineMR is trained with supervised fine-tuning (SFT) on tool-interaction traces followed by Group Relative Policy Optimization (GRPO) with conditional tool-use rewards. On the multi-cohort cine CMR benchmark, CineMR achieves 35.9% pass@1 and 58.9% pass@4, compared with 1.5% pass@1 for the Qwen3-VL-8B backbone and 0.0% and 7.0% pass@1 for LLaVA-Med v1.5 and MedGemma-4B, respectively. Correct tool invocation reaches 99.8% after GRPO, up from 78.9% after SFT. Live tool outputs improve ventricular measurement accuracy by 20.4--23.7% over direct model predictions, and removing all tools reduces pass@1 from 35.9% to 27.9%. These results highlight the importance of reliable tool use for quantitative cine CMR reasoning and support CineMR as a promising approach for assistive cardiac image assessment. Code, benchmark resources, and model weights are available at https://github.com/AI-MIND-Lab/CineMR.

Abstract

Explore similar work

CineMR: Tool-Integrated Vision-Language Reasoning for Quantitative Cardiac MRI Assessment

CMRVision: A Foundation Model for Cardiac MR Image Analysis

CardioLens: Revealing the Clinical Reality Gap of MLLMs via Multi-Sequence Cardiac MRI Evaluations