cs.CVOct 25, 2026

Conviction Without Evidence: Verdict Stability Is Independent of Visual Grounding in Medical Vision--Language Models

Authors: Federico Felizzi, Francesco Andrea Causio, Bianca Destro Castaniti, Olivia Riccomi, Michele Ferramola, Vittorio De Vita, Alessandro Tosi, Antonio Cristiano, 

Organizations: SIIAM, Rome, Italy · Dept. of Computer Science, University of Pennsylvania, USA · UCSC, Rome, Italy · NSBProject, Mantova, Italy · 

Abstract

Vision--language models applied to medical question answering can reach correct diagnoses without using the diagnostic image, inferring instead from the clinical vignette. We ask whether this is visible in how firmly a model holds its answer: does a model defend a diagnosis derived from a blank placeholder as tenaciously as one derived from a real medical image? Crossing an image-substitution protocol with two levels of adversarial pressure on 60 Italian State Examination items, we find no evidence that ungrounded verdicts are more fragile. In a paired item-level analysis, fabricated consensus (L4) overturns 94.4% of verdicts with the real image and 98.1% without (Δ=+0.037Δ= +0.037; one-sided 95% upper bound +0.105+0.105); contentless doubt (L1) overturns 62.5% and 51.8% (Δ=−0.107Δ= -0.107, 95% CI [−0.209,−0.001][-0.209, -0.001]), the point estimate running opposite to the direction a grounding detector requires. The two levels nonetheless produce opposite dynamics: under L4 the model flips once, always to the exact option argued for (110/110 first-turn flips), and then freezes, averaging 1.9 verdict changes over ten turns; under L1 it never settles, averaging 7.4 changes. Instability under contentless doubt tracks intrinsic item ambiguity rather than evidence availability, and so does not serve as a grounding signal. Flips are almost entirely corrupting (99/100 across both levels). The model is not unaware of the missing evidence---it flags the blank image at baseline on 35 of 60 items and abstains on 4--6---yet those abstentions do not survive contact with pressure. None of the epistemic signals a user could plausibly consult distinguishes a grounded diagnosis from a confabulated one.

Figures & tables

Explore similar work

CardsList
  1. MedVIGIL: Evaluating Trustworthy Medical VLMs Under Broken Visual Evidence

    May 8, 2026Hanqi Jiang, Junhao Chen, Mingyu Kang +12VLM EvaluationVLM Robustness

  2. Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs

    Jul 4, 2026Anas Zafar, Leema Krishna Murali, Siddhant Bharadwaj +2VLM EvaluationCounterfactual Evaluation