cs.CVNov 6, 2026

Uncertainty, Not Concealment: A Mechanistic Reappraisal of Sycophancy in Vision-Language Models

Authors: Mohamed I. Eldagla, Zeyad H. ElShafei, Ahmed M. Anter

Organizations: Egypt-Japan University of Science and Technology

Abstract

Vision-language models (VLMs) in medical retrieval-augmented generation must be able to say when a retrieved report contradicts the X-ray, so understanding whether they can detect such conflicts is important for safe deployment. The sycophancy mask hypothesis holds that these models internally detect conflicts but refuse to report them, concealing a detected contradiction. We introduce a three-part diagnostic that separates concealment from uncertainty: representation probing, activation steering, and two-way forced choice. Across four instruction-tuned VLMs on chest X-rays, we confirm behavioral suppression of contradiction and find that a linear probe separates matched from mismatched inputs, beyond what a simple CLIP model achieves. Steering the learned conflict direction never elicits contradiction, and forcing a binary choice does not recover one; instead, cross-modal attention to the image drops on mismatched pairs. We conclude that these models fail to contradict because the mismatch registers as hedged, gated uncertainty rather than concealed knowledge, calling for re-grounding and uncertainty surfacing. Code is available at https://github.com/mohamed-eldagla/Uncertainty-Not-Concealment/.

Figures & tables

Explore similar work

CardsList
  1. To Agree or To Be Right? The Grounding-Sycophancy Tradeoff in Medical Vision-Language Models

    Mar 23, 2026OFM Riaz Rahman Aranya, Kevin DesaiSycophancy in VLMsMedical VLMs

  2. Seeing Through the Forgery Is Not Enough: Regressive Sycophancy under Fabricated Evidence in Medical Vision-Language Models

    Oct 13, 2026Sadif Ahmed, Soumya Swagata Biswas, Rifat Shahriyar +1Sycophancy in VLMsMedical VLMs