Beyond Accuracy: Evaluating Visual Grounding In Multimodal Medical Reasoning

Anas Zafar; Leema Krishna Murali; Ashish Vashist

cs.CVMar 3, 2026

Beyond Accuracy: Evaluating Visual Grounding In Multimodal Medical Reasoning

Authors: Anas Zafar, Leema Krishna Murali, Ashish Vashist

Organizations: The University of Texas MD Anderson Cancer Center · Cohere Labs Community · Eisai Inc. · Indian Institute of Science, Bangalore

Abstract

Recent work shows that text-only reinforcement learning with verifiable rewards (RLVR) can match or outperform image-text RLVR on multimodal medical VQA benchmarks, suggesting current evaluation protocols may fail to measure causal visual dependence. We introduce a counterfactual evaluation framework using real, blank, and shuffled images across four medical VQA benchmarks: PathVQA, PMC-VQA, SLAKE, and VQA-RAD. Beyond accuracy, we measure Visual Reliance Score (VRS), Image Sensitivity (IS), and introduce Hallucinated Visual Reasoning Rate (HVRR) to detect cases where models generate visual claims despite producing image-invariant answers. Our findings reveal that RLVR improves accuracy while degrading visual grounding: text-only RLVR achieves negative VRS on PathVQA (-0.09), performing better with mismatched images, while image-text RLVR reduces image sensitivity to 39.8% overall despite improving accuracy. On VQA-RAD, both variants achieve 63% accuracy through different mechanisms: text-only RLVR retains 81% performance with blank images, while image-text RLVR shows only 29% image sensitivity. Models generate visual claims in 68-74% of responses, yet 38-43% are ungrounded (HVRR). These findings demonstrate that accuracy-only rewards enable shortcut exploitation, and progress requires grounding-aware evaluation protocols and training objectives that explicitly enforce visual dependence.

Figures & tables

Model	Acc	VRS	BD	IS	VBR	VHR
Baseline	56.5%	0.127	0.130	48.2%	26.8%	14%
RL(text)	56.2%	0.105	0.115	50.0%	27.0%	16.5%
RL(image)	58.8%	0.100	0.125	39.8%	23.2 %	13.3%
Acc = Accuracy on real images; VRS = Visual Reliance Score; BD = Blank Drop;
IS = Image Sensitivity; VBR = Visual Benefit Rate; VHR = Visual Harm Rate.

Table 1: Overall model performance averaged across four medical VQA benchmarks (PathVQA, PMC-VQA, SLAKE, VQA-RAD;

n=100

each). RL(image) achieves highest accuracy but shows degraded visual grounding metrics across VRS, IS, and VBR.

Panel of Figure 1 — Figure 1: The Modality Skeptic Paradox. Under the Shuffle condition (bottom row), the input image is swapped for a Chest X-ray. RLVR-Image-Text correctly identifies the modality mismatch and updates the conclusion. RLVR-Text identifies the mismatch in its reasoning but ignores it and hallucinates the appearance of liver in the X-ray, proving the final decision is decoupled from the visual information.

Model	VRS	95% CI	Significant?
Baseline	0.00	[-0.12, 0.13]	No
RL(text)	-0.09	[-0.21, 0.05]	No
RL(image)	0.04	[-0.07, 0.16]	No

Table 3: Statistical significance tests for VRS on PathVQA (

n=100

). No model achieves significant visual grounding; all 95% bootstrap CIs overlap zero. Negative VRS for RL(text) indicates reverse dependence (performs better with wrong images).

Model	NVCR	HVRR	Cond. Prob.	Acc
Baseline	68%	38%	54.5%	57%
RL(text)	74%	40%	53.4%	56%
RL(image)	70%	43%	60.9%	59%

Table 4: Hallucinated visual reasoning rates averaged across benchmarks. NVCR = Novel Visual Claim Rate (frequency of visual language); HVRR = Hallucinated VR Rate (visual claims with invariant answers); Cond. Prob. =

P(\text{invariant}\mid\text{visual claim})

. RL(image) shows highest hallucination rates despite training on images. Full per-benchmark breakdown in Appendix Table 5 .

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Model	NVCR	HVRR	Cond. Prob.	Acc
PathVQA	Baseline	80%	51%	63.8%	62%
	RL(text)	88%	52%	59.1%	56%
	RL(image)	83%	55%	66.3%	60%
PMC-VQA	Baseline	60%	24%	40.0%	50%
	RL(text)	66%	27%	40.9%	44%
	RL(image)	64%	31%	48.4%	57%

Appendix

Table 5: Complete per-benchmark hallucinated visual reasoning rates. NVCR = Novel Visual Claim Rate (frequency of visual language); HVRR = Hallucinated VR Rate (visual claims with invariant answers); Cond. Prob. =

P(\text{invariant}\mid\text{visual claim})

. RL(image) shows highest hallucination rates despite training on images.

Figure 2: Modality-Specific Reasoning Collapse (VQA-RAD). A text-only RL-trained vision–language model produces identical answers and visually detailed reasoning when evaluated on the correct image and a shuffled image. Despite explicit references to radiological features, the model’s prediction remains invariant, resulting in a positive Hallucinated Visual Reasoning Rate (HVRR).

Explore similar work

Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs
Jul 4, 2026Anas Zafar, Leema Krishna Murali, Siddhant Bharadwaj +2VLM Evaluation Counterfactual Evaluation
The University of Texas MD Anderson Cancer Center · Eisai Inc. · Cohere Labs Community +1
MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering
Apr 10, 2026Suyang Xi, Songtao Hu, Yuxiang Lai +4Medical VQA Medical VLMs
Department of Radiation Oncology and Winship Cancer Institute, Emory University School of Medicine, Atlanta, 30322, GA, USA · Department of Biostatistics and Bioinformatics, Emory University, Atlanta, 30322, GA, USA
MedQA-MM: Shortcuts Behind Medical Visual Reasoning
Sep 3, 2026Benlu Wang, Yifan Zhang, Jiaqing Yu +7Medical VQA Shortcut Learning
University of Massachusetts Amherst · 4Yale University · University of Massachusetts Lowell +6

Jul 4, 2026J/K move · Enter open · S save

Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs

Anas Zafar, Leema Krishna Murali, Siddhant Bharadwaj, Ashish Vashist+1

The University of Texas MD Anderson Cancer Center · Eisai Inc. · Cohere Labs Community+1

VLM Evaluation Counterfactual Evaluation Contrastive Learning Medical VQA Vision-Language Grounding Medical VLMs

Open paper AskPDF arXiv

Large vision language models (VLMs) report strong accuracy on medical question-answering, yet it remains unclear whether they reason from visual evidence or exploit textual shortcuts. We introduce a counterfactual evaluation framework that decouples visual and textual contributions by substituting input images with controlled surrogates blank, pixel-shuffled, image-absent, and CLIP-retrieved hard negatives and derive a suite of grounding metrics including the Visual Reliance Score (VRS) and Visual Hallucination Rate (VHR). We further introduce CORAL (COntrastive Retrieval-Augmented Learning), a 7B-parameter LoRA fine-tune of Qwen2.5-VL-7B trained with a Contrastive Grounding Objective (CGO) that penalises answer invariance under hard-negative image swaps. On a paired controlled evaluation across four closed-form medical VQA benchmarks (PathVQA, PMC-VQA, SLAKE, VQA-RAD; n=400 total), CORAL improves macro accuracy by +6.7 pp (P(Delta>0)=0.988) and reduces VHR by 8.0 pp (P<0.001) over the matched Qwen2.5-VL-7B base; neither MedVLThinker RL variant achieves a significant gain on either metric. Cross-domain diagnostics further reveal that image substitution costs only <=6.5 pp on medical benchmarks versus 48-61 pp on general-domain tasks, situating the grounding gap that CGO targets. We discuss evaluation limitations openly including train/eval benchmark overlap and underpowered secondary metrics and release our framework, training code, and model weights to support reproducible grounding audits of medical VLMs.

Abstract

Explore similar work

Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs

MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering

MedQA-MM: Shortcuts Behind Medical Visual Reasoning