To Agree or To Be Right? The Grounding-Sycophancy Tradeoff in Medical Vision-Language Models

OFM Riaz Rahman Aranya; Kevin Desai

cs.CVMar 23, 2026

To Agree or To Be Right? The Grounding-Sycophancy Tradeoff in Medical Vision-Language Models

Authors: OFM Riaz Rahman Aranya, Kevin Desai

Organizations: Department of Computer Science, The University of Texas at San Antonio

Abstract

Vision-language models (VLMs) adapted to the medical domain have shown strong performance on visual question answering benchmarks, yet their robustness against two critical failure modes, hallucination and sycophancy, remains poorly understood, particularly in combination. We evaluate six VLMs (three general-purpose, three medical-specialist) on three medical VQA datasets and uncover a grounding-sycophancy tradeoff: models with the lowest hallucination propensity are the most sycophantic, while the most pressure-resistant model hallucinates more than all medical-specialist models. To characterize this tradeoff, we propose three metrics: L-VASE, a logit-space reformulation of VASE that avoids its double-normalization; CCS, a confidence-calibrated sycophancy score that penalizes high-confidence capitulation; and Clinical Safety Index (CSI), a unified safety index that combines grounding, autonomy, and calibration via a geometric mean. Across 1,151 test cases, no model achieves a CSI above 0.35, indicating that none of the evaluated 7-8B parameter VLMs is simultaneously well-grounded and robust to social pressure. Our findings suggest that joint evaluation of both properties is necessary before these models can be considered for clinical use. Code is available at https://github.com/UTSA-VIRLab/AgreeOrRight

Figures & tables

Figure 1: The grounding–sycophancy tradeoff on VQA-RAD ( $n{=}451$ ). Each point represents one model; the x-axis shows L-VASE (hallucination propensity, lower is better) and the y-axis shows CCS (confidence-calibrated sycophancy, lower is better). No model reaches the lower-left desired quadrant. Models that hallucinate less (Qwen3-VL, MedGemma) are the most sycophantic, while the most resistant model (IDEFICS2) hallucinates substantially.

Figure 2: Overview of the evaluation pipeline. (a) L-VASE computes hallucination propensity by passing weakly-augmented ( $\sigma{=}3$ ) and heavily-distorted ( $\sigma{=}15$ ) versions of each medical image through the VLM, extracting raw logit vectors $\boldsymbol{\ell}_{\text{weak}}$ and $\boldsymbol{\ell}_{\text{dist}}$ , and measuring the entropy of their contrastive combination in logit space, avoiding the double-normalization issue of operating on probability vectors. The score is averaged over $N{=}5$ stochastic samples ( $\tau{=}1.0$ ). (b) CCS measures confidence-calibrated sycophancy by first recording the model’s baseline answer and logit-derived confidence $c$ , then probing resistance under three clinically motivated pressure types (expert correction, peer consensus, and authority citation). Each capitulation is weighted by $c$ , capturing the most dangerous failure mode: abandoning high-confidence diagnoses. (c) CSI unifies both axes into a single deployment-readiness score via a geometric mean inspired by FMEA methodology [ 20 ] , ensuring that failure on any individual axis (grounding, autonomy, or calibration) collapses the overall safety score.

FMEA Factor	VLM Equivalent	Metric
Occurrence	Hallucination freq.	$1-\text{{L-VASE}{}}$
Severity	Confident capitulation	$1-\text{{CCS}{}}$
Detection	Self-correction ability	Resistance $R$

Table 3

		VQA-RAD ( $n{=}451$ )				SLAKE ( $n{=}500$ )				PathVQA ( $n{=}200$ )
	Model	L-VASE $\downarrow$	$R\uparrow$	CCS $\downarrow$	CSI $\uparrow$	L-VASE $\downarrow$	$R\uparrow$	CCS $\downarrow$	CSI $\uparrow$	L-VASE $\downarrow$	$R\uparrow$	CCS $\downarrow$	CSI $\uparrow$
General	LLaVA-1.5	0.944	0.006	0.751	0.052	0.925	0.005	0.763	0.056	1.046	0.008	0.725	0.030
	Qwen3-VL	0.309	0.000	0.915	0.084	0.300	0.000	0.913	0.085	0.372	0.000	0.877	0.092
	IDEFICS2	0.711	0.303	0.554	0.339	0.747	0.185	0.628	0.259	0.912	0.125	0.663	0.155
Medical	LLaVA-Med	0.626	0.139	0.675	0.257	0.604	0.220	0.614	0.323	1.040	0.212	0.618	0.093
	MedVLM-R1	0.663	0.050	0.575	0.192	0.705	0.079	0.664	0.198	0.870	0.120	0.604	0.184

Table 1: Main results across three medical VQA benchmarks. L-VASE : hallucination score.

R

: sycophancy resistance rate. CCS : confidence-calibrated sycophancy. CSI : Clinical Safety Index.

\downarrow

: lower is better,

\uparrow

: higher is better. Best in bold , second best underlined , worst in red .

Figure 3: CSI distribution across all models and datasets. All 18 evaluation points fall within the Critical, High Risk, or Moderate Risk zones. No model reaches the Cautionary or Safe regions, with a maximum CSI of 0.339 (IDEFICS2 on VQA-RAD).

	Model	Expert	Consensus	Authority	Confidence
VQA-RAD	LLaVA-1.5	0.4	0.7	0.7	0.755
	Qwen3-VL	0.0	0.0	0.0	0.914
	IDEFICS2	21.5	32.6	36.8	0.833
	LLaVA-Med	22.4	4.2	15.1	0.786
	MedVLM-R1	3.8	8.0	3.1	0.602
	MedGemma	0.2	8.0	0.0	0.943

Table 2: Sycophancy resistance rate (%) by pressure type and mean baseline confidence across all benchmarks. Higher resistance = safer. Best in bold , second best underlined , worst in red .

Figure 4: Side-by-side comparison of VASE and L-VASE formulations. Left: VASE operates on softmax probability vectors ( $\mathbf{p}\in[0,1]$ ). Contrastive subtraction produces negative entries that are invalid in probability space; empirically, 98.6% of token-level vectors exhibit this issue (LLaVA-1.5, 30 VQA-RAD images, 5 samples, $\alpha{=}1.0$ , $n{=}15{,}187$ vectors). An outer softmax masks these negatives but yields corrupted entropy estimates. Right: L-VASE operates on raw logit vectors ( $\boldsymbol{\ell}\in\mathbb{R}$ ). The same subtraction produces negative entries that are mathematically valid in logit space. A single softmax converts the result into a proper distribution with no mass corruption. Bar charts are schematic.

Sycophancy in VLMs

See all 11 papers

Position, Not Provenance: Separating Reasoning Mediation from Sycophancy in Medical Vision-Language Models
Jul 29, 2026Supratik Bhowal, Subhrajyoti Basu, Aritra Gir Mahanta +1Medical VQA Medical VLMs
Department of Computer Science (AIML) IEM Kolkata, School of UEMK Kolkata, India · Department of Computer Science and Business System Heritage Institute of Technology Kolkata, India · Department of Computer Science and Engineering Indian Institute of Information Technology, Kalyani Kalyani, India
Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs
Jul 4, 2026Anas Zafar, Leema Krishna Murali, Siddhant Bharadwaj +2VLM Evaluation Counterfactual Evaluation
The University of Texas MD Anderson Cancer Center · Eisai Inc. · Cohere Labs Community +1
Overconfidence and Calibration in Medical VQA: Empirical Findings and Hallucination-Aware Mitigation
Apr 2, 2026Ji Young Byun, Young-Jin Park, Jean-Philippe Corbeil +1Vision-Language Models VLM Hallucination
Johns Hopkins University, School of Medicine · Massachusetts Institute of Technology · Microsoft Healthcare & Life Sciences

Jul 29, 2026J/K move · Enter open · S save

Position, Not Provenance: Separating Reasoning Mediation from Sycophancy in Medical Vision-Language Models

Supratik Bhowal, Subhrajyoti Basu, Aritra Gir Mahanta, Anik Pal Chowdhury

Department of Computer Science (AIML) IEM Kolkata, School of UEMK Kolkata, India · Department of Computer Science and Business System Heritage Institute of Technology Kolkata, India · Department of Computer Science and Engineering Indian Institute of Information Technology, Kalyani Kalyani, India

Medical VQA Medical VLMs VLM Reasoning Sycophancy in VLMs Faithfulness in VLMs

Open paper AskPDF arXiv

Medical vision-language models (VLMs) generate chain-of-thought (CoT) reasoning before answering clinical questions, but whether this reasoning causally influences predictions remains unclear. We present CoT-Mediate, a behavioral framework that perturbs a single clinically meaningful attribute within a model's own generated reasoning and measures whether the resulting prediction follows the edited reasoning. Our framework combines a dual-arm protocol comparing re-prompted evidence with prefix-forced continuation, together with a provenance-controlled intervention that varies only the attributed source of identical reasoning to disentangle reasoning mediation from sycophancy. We evaluate LLaVA-Med and MedGemma on 1,000 VQA-RAD samples each. Prefix-forced continuation consistently yields higher mediation faithfulness than re-prompting, while the provenance analysis reveals distinct model-specific deference behaviors. Across both models, removing visual evidence increases reliance on injected reasoning, whereas laterality is the least faithfully tracked clinical attribute. These results show that the mechanism used to inject reasoning substantially affects measured faithfulness and that contextual position, rather than stated provenance, is the primary determinant of whether medical VLMs use their generated reasoning.

Abstract

Sycophancy in VLMs

Explore similar work

Position, Not Provenance: Separating Reasoning Mediation from Sycophancy in Medical Vision-Language Models

Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs

Overconfidence and Calibration in Medical VQA: Empirical Findings and Hallucination-Aware Mitigation