Better Eyes, Better Thoughts: Why Vision Chain-of-Thought Fails in Medicine

Yuan Wu; Zongxian Yang; Jiayu Qian; Songpan Gao; Guanxing Chen; Qiankun Li; Yu-An Huang; Zhi-An Huang

cs.CVMar 2, 2026

Better Eyes, Better Thoughts: Why Vision Chain-of-Thought Fails in Medicine

Authors: Yuan Wu, Zongxian Yang, Jiayu Qian, Songpan Gao, Guanxing Chen, Qiankun Li, Yu-An Huang, Zhi-An Huang

Organizations: Department of Computer Science, City University of Hong Kong (Dongguan) · College of Computing and Data Science (CCDS), Nanyang Technological University · School of Computer Science, Northwestern Polytechnical University

Abstract

Large vision-language models (VLMs) often benefit from chain-of-thought (CoT) prompting in general domains, yet its efficacy in medical vision-language tasks remains underexplored. We report a counter-intuitive trend: on medical visual question answering, CoT frequently underperforms direct answering (DirA) across general-purpose and medical-specific models. We attribute this to a medical perception bottleneck: subtle, domain-specific cues can weaken visual grounding, and CoT may compound early perceptual uncertainty rather than correct it. To probe this hypothesis, we introduce two training-free, inference-time grounding interventions: (i) perception anchoring via region-of-interest cues and (ii) description grounding via high-quality textual guidance. Across multiple benchmarks and model families, these interventions improve accuracy, mitigate CoT degradation, and in several settings reverse the CoT--DirA inversion. Our findings suggest that reliable clinical VLMs require robust visual grounding and cross-modal alignment, beyond extending text-driven reasoning chains. Code is available here.

Figures & tables

Figure 1: The three-stage Medical VLM CoT framework and targeted interventions.

Figure 2: Main results across RQ1–RQ3. (a) CoT improves general benchmarks but degrades medical benchmarks. (b) CoT is more sensitive to progressive visual degradation than DirA. (c) Supplementing models with expert-level image descriptions alone effectively mitigates CoT degradation. (d,e) Counterfactual inputs reveal pseudo-robustness in DirA and stronger visual dependence in CoT. (f) Incorrect RoI and descriptions degrade CoT performance, confirming its reliance on accurate visual grounding.

Model		Path-VQA	SLAKE	VQA-RAD	PMC-VQA	OM.VQA	Avg
Qwen3VL -8B ∗	DirA	45.15	71.91	63.41	54.38	77.31	62.43
Qwen3VL -8B ∗	CoT	44.82	66.16	61.19	50.01	73.31	59.10
InternVL3 -8B ∗	DirA	43.49	73.99	66.74	53.21	78.41	63.17
InternVL3 -8B ∗	CoT	43.57	68.71	54.23	49.84	74.62	58.19
Lingshu -7B †	DirA	61.73	82.00	67.62	54.31	86.28	70.41
Lingshu -7B †	CoT	58.16	75.77	60.31	47.58	67.54	61.87

Table 1: Quantitative comparison of DirA and CoT on medical benchmarks. All values report accuracy (%). Superscripts ∗ , † , and ‡ denote general, medical-specific, and closed-source models, respectively. Bold indicates the better score between DirA and CoT for each model-dataset pair.

Model		Baseline	+ BBox RoI	+ Expert Desc.	+ Combined Prior
Qwen3VL -8B	DirA	71.91	77.95 ( +6.04 )	82.00 ( +10.09 )	84.35 ( +12.44 )
Qwen3VL -8B	CoT	66.16	76.15 ( +9.99 )	81.24 ( +15.08 )	84.45 ( +18.29 )
InternVL3 -8B	DirA	73.99	78.29 ( +4.30 )	82.00 ( +8.01 )	85.11 ( +11.12 )
InternVL3 -8B	CoT	68.71	75.68 ( +6.97 )	80.02 ( +11.31 )	85.29 ( +16.58 )
Lingshu -7B	DirA	82.00	84.07 ( +2.07 )	82.37 ( +0.37 )	85.57 ( +3.57 )
Lingshu -7B	CoT	75.77	82.75 ( +6.98 )	78.51 ( +2.74 )	85.77 ( +10.00 )

Table 2: Targeted interventions on the SLAKE dataset. Bold indicates the best score within each row (DirA or CoT) across intervention settings.

Figure 3: Qualitative case study. Standard CoT exhibits misaligned attention patterns and incorrect conclusions, while grounded interventions provide additional spatial and semantic priors that yield more visually consistent reasoning trajectories.

Explore similar work

Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs
Jul 4, 2026Anas Zafar, Leema Krishna Murali, Siddhant Bharadwaj +2VLM Evaluation Counterfactual Evaluation
The University of Texas MD Anderson Cancer Center · Eisai Inc. · Cohere Labs Community +1
ClinCoT: Clinical-Aware Visual Chain-of-Thought for Medical Vision Language Models
Mar 1, 2026Xiwei Liu, Yulong Li, Xinlin Zhuang +5Vision-Language Models Medical VLMs
Mohamed bin Zayed University of Artificial Intelligence · East China Normal University
Is There Knowledge Left to Extract? Evidence of Fragility in Medically Fine-Tuned Vision-Language Models
Apr 10, 2026Oliver McLaughlin, Daniel Shubin, Carsten Eickhoff +3Medical VLMs VLM Reasoning
Brown University · University of Washington · University of Tübingen +1

Jul 4, 2026J/K move · Enter open · S save

Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs

Anas Zafar, Leema Krishna Murali, Siddhant Bharadwaj, Ashish Vashist+1

The University of Texas MD Anderson Cancer Center · Eisai Inc. · Cohere Labs Community+1

VLM Evaluation Counterfactual Evaluation Contrastive Learning Medical VQA Vision-Language Grounding Medical VLMs

Open paper AskPDF arXiv

Large vision language models (VLMs) report strong accuracy on medical question-answering, yet it remains unclear whether they reason from visual evidence or exploit textual shortcuts. We introduce a counterfactual evaluation framework that decouples visual and textual contributions by substituting input images with controlled surrogates blank, pixel-shuffled, image-absent, and CLIP-retrieved hard negatives and derive a suite of grounding metrics including the Visual Reliance Score (VRS) and Visual Hallucination Rate (VHR). We further introduce CORAL (COntrastive Retrieval-Augmented Learning), a 7B-parameter LoRA fine-tune of Qwen2.5-VL-7B trained with a Contrastive Grounding Objective (CGO) that penalises answer invariance under hard-negative image swaps. On a paired controlled evaluation across four closed-form medical VQA benchmarks (PathVQA, PMC-VQA, SLAKE, VQA-RAD; n=400 total), CORAL improves macro accuracy by +6.7 pp (P(Delta>0)=0.988) and reduces VHR by 8.0 pp (P<0.001) over the matched Qwen2.5-VL-7B base; neither MedVLThinker RL variant achieves a significant gain on either metric. Cross-domain diagnostics further reveal that image substitution costs only <=6.5 pp on medical benchmarks versus 48-61 pp on general-domain tasks, situating the grounding gap that CGO targets. We discuss evaluation limitations openly including train/eval benchmark overlap and underpowered secondary metrics and release our framework, training code, and model weights to support reproducible grounding audits of medical VLMs.

Abstract

Explore similar work

Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs

ClinCoT: Clinical-Aware Visual Chain-of-Thought for Medical Vision Language Models

Is There Knowledge Left to Extract? Evidence of Fragility in Medically Fine-Tuned Vision-Language Models