Vision-language models (VLMs) in medical retrieval-augmented generation must be able to say when a retrieved report contradicts the X-ray, so understanding whether they can detect such conflicts is important for safe deployment. The sycophancy mask hypothesis holds that these models internally detect conflicts but refuse to report them, concealing a detected contradiction. We introduce a three-part diagnostic that separates concealment from uncertainty: representation probing, activation steering, and two-way forced choice. Across four instruction-tuned VLMs on chest X-rays, we confirm behavioral suppression of contradiction and find that a linear probe separates matched from mismatched inputs, beyond what a simple CLIP model achieves. Steering the learned conflict direction never elicits contradiction, and forcing a binary choice does not recover one; instead, cross-modal attention to the image drops on mismatched pairs. We conclude that these models fail to contradict because the mismatch registers as hedged, gated uncertainty rather than concealed knowledge, calling for re-grounding and uncertainty surfacing. Code is available at https://github.com/mohamed-eldagla/Uncertainty-Not-Concealment/.
Figures & tables
Figure 1: The diagnostic framework. A single observation (internal signal present, behavioral CONTRADICTS absent) is ambiguous between concealment (H1) and uncertainty (H2). Three tests (steering, forced choice, and cross-modal attention) disambiguate them; in every case the observation matches H2.
Model
CONDITION
SUPPORT
CONTRADICT
INCONCLUSIVE
INVALID
LLaVA-7B
C2 / C3 / C4
15.8 / 8.5 / 5.9
0.0 / 0.0 / 0.0
0.0 / 0.0 / 0.0
84.2 / 91.5 / 94.1
InternVL2-8B
C2 / C3 / C4
92.1 / 90.5 / 73.7
0.9 / 0.2 / 0.0
7.0 / 9.3 / 26.3
0.0 / 0.0 / 0.0
Qwen2-VL
C2 / C3 / C4
86.0 / 73.8 / 49.3
0.0 / 0.0 / 0.0
14.0 / 26.2 / 50.7
0.0 / 0.0 / 0.0
LLaVA-13B
C2 / C3 / C4
59.6 / 56.4 / 79.5
0.0 / 0.0 / 0.0
0.0 / 0.0 / 0.0
40.4 / 43.6 / 20.5
Table 1: Behavioral verdicts (%) by condition, robust first-verdict parsing. INVALID = non-compliant outputs. CONTRADICTS is ≈0% everywhere.
Model
AUC
LLaVA-7B
0.789±0.032
InternVL2-8B
0.798±0.010
Qwen2-VL
0.813±0.007
LLaVA-13B
0.797±0.028
Table 2: CLS probe AUC (mean ± std over 5 folds).
Figure 2: Activation steering (InternVL2-8B): amplifying the mismatch direction leaves the robust verdict distribution unchanged (CONTRADICTS 0% , SUPPORTS ≈88% , INCONCLUSIVE ≈11% ); it only reduces naive “contradict” hedging (gray dashed). No contradiction is surfaced.
Vision-language models (VLMs) adapted to the medical domain have shown strong performance on visual question answering benchmarks, yet their robustness against two critical failure modes, hallucination and sycophancy, remains poorly understood, particularly in combination. We evaluate six VLMs (three general-purpose, three medical-specialist) on three medical VQA datasets and uncover a grounding-sycophancy tradeoff: models with the lowest hallucination propensity are the most sycophantic, while the most pressure-resistant model hallucinates more than all medical-specialist models. To characterize this tradeoff, we propose three metrics: L-VASE, a logit-space reformulation of VASE that avoids its double-normalization; CCS, a confidence-calibrated sycophancy score that penalizes high-confidence capitulation; and Clinical Safety Index (CSI), a unified safety index that combines grounding, autonomy, and calibration via a geometric mean. Across 1,151 test cases, no model achieves a CSI above 0.35, indicating that none of the evaluated 7-8B parameter VLMs is simultaneously well-grounded and robust to social pressure. Our findings suggest that joint evaluation of both properties is necessary before these models can be considered for clinical use. Code is available at https://github.com/UTSA-VIRLab/AgreeOrRight
OFM Riaz Rahman Aranya, Kevin Desai
Department of Computer Science, The University of Texas at San Antonio
A medical vision-language model may analyze a given CT scan and correctly detect both kidneys in view. Continuing this conversation, if a user provides a prescription with an opposite direction, medical VLMs often sycophantically choose to realign their diagnosis. In this work, we study regressive sycophancy under fabricated evidence, where a model abandons a correct answer when pushback comes with a fake document. We test four open VLMs on 400 VQA-RAD yes/no questions in 19,200 multi-turn conversations, with and without a fabricated prescription. Without a disclosure that documents are synthetic, the models abandon 80.1% of 6,488 correct answers. Notably, under one script each, the same prescription raises the flip rate of Qwen2.5-VL-7B from 45.9% to 89.6% and lowers that of Qwen3-VL-8B from 77.7% to 50.9%. A model rejected the prescription while still preserving the correct answer in 915 of 3,232 conversations with the prescription (28.3%), and 40% of these flipped in later turns. System-prompt sentences that disclose the documents as synthetic lower one model's flip rate from 77.8% to 47.3%. Thus, we conclude that awareness about a forgery does not prevent a model from adopting its answer.
Many medical conditions require clinical assessment of symptoms, history, and functioning rather than judgments from appearance. We study whether vision-language models (VLMs) abstain from facial attribution of autism spectrum disorder (ASD), attention-deficit/hyperactivity disorder (ADHD), major depressive disorder (MDD), and generalized anxiety disorder (GAD). To support this study, we construct PARITY (Paired Assessment with Reused Identity), a demographically balanced dataset for testing whether changing facial expressions alters diagnostic judgments, while holding identity and other visual factors as constant as possible. Our comprehensive evaluation of 27 open- and closed-weight models reveals inconsistent abstention across expressions, conditions, and task formats. Models turn facial expressions into unsupported medical judgments: this behavior is widespread among open-weight VLMs and extends to safety-trained frontier models that abstain on individual portraits yet make diagnostic judgments when comparing portraits of the same identity. Safeguard checks reveal further gaps: clinical instructions do not consistently prevent these judgments, and a guardrail model rarely flags paired requests as unsafe. Semantic analysis further shows that most baseline open-model refusals fail to explain both the insufficiency of facial evidence and the need for clinical assessment. Unsupported judgments and seemingly authoritative explanations can cause distress, reinforce stigma, and offer false reassurance, underscoring the need for stronger safety measures in model training and deployment.
Karan Dua, Amit Agarwal, Hitesh Laxmichand Patel +8