cs.CVOct 20, 2026

Beyond the Evidence Boundary: Behavioral Mirage Under Progressive Degradation in Medical VLMs

Authors: Edric Castel C. Hao, Rez Samantha Z. Floresca

Organizations: Ateneo de Manila Senior High School

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

VLMs for medical image interpretation are typically evaluated on clean inputs, despite the progressive evidence loss that can occur in high stakes clinical settings. Existing robustness studies show that performance can degrade while confidence remains high, but do not establish when a previously reliable model-case pair ceases to behave reliably. We introduce a baseline-anchored framework that traces baseline-correct cases through progressive image degradation to identify an empirical evidence boundary, and define mirage as continued commitment at this boundary, regardless of correctness. Across three VLMs on three medical datasets, only MedGemma provides a sufficiently large baseline-correct pool for trajectory analysis. On VQA-RAD, the only dataset in which the question text varies, language-only answerability is associated with baseline correctness (AUROC 0.677) and with commitment at the boundary (AUROC 0.653 to 0.670). MedGemma's post-boundary behavior is heterogeneous, including direct mirage, correct recovery, mirage dissolution, consistent abstention, and oscillation, with mirage rates of 16.0 to 100.0% across dataset-modality pairs. Despite this behavioral heterogeneity, residual-stream probes decode commit-versus-abstain behavior at the boundary with AUROC of 0.895 to 0.994 under nested cross-validation. These results show that progressive evidence loss can produce persistent commitment that output entropy and abstention behavior do not consistently reflect, motivating boundary-aware evaluation, option-order controls, and improved abstention mechanisms for reliable medical foundation models operating under partial observation.

Figures & tables

Figure 4: Full per-example trajectory rasters across the severity schedule for every dataset-modality pair, MedGemma. Each row is one baseline-correct example, grouped by trajectory type (Persistent correct, Direct mirage, Correct recovery, Mirage dissolves, Consistent abstention, Oscillatory, as applicable to each pair). Color denotes state (correct / wrong / IDK-abstain) at each severity level. Aggregate trajectory counts underlying these panels are reported in Appendix Table 14 . See Appendix A for the trajectory taxonomy.
Figure 5: IDK (abstention) rate by ground-truth class and severity for MedGemma across each dataset-modality pair. Color encodes the percentage of examples in that class abstaining at that severity level, and labels are rounded to integers. HAM10000 panels show a uniformly low IDK rate across all classes and severities, consistent with the main text’s finding that HAM10000 almost never abstains under image degradation (see “Class-level trajectories” in Appendix K). APTOS and VQA-RAD panels show class-dependent onset of abstention, complementing the aggregate class-level breakdowns discussed in the main text.
Figure 6: Summary of trajectory-type proportions across all dataset-modality pairs for MedGemma. Bars aggregate the per-example rasters shown in Appendix M.2 into trajectory-type counts, corresponding to the columns of Appendix Table 14 .
More in this topic

Medical VLMs

Medical vision-language models connect clinical images with text for diagnosis, reporting, question answering, and image grounding.

See all 479 papers

Explore similar work

CardsList
May 8, 2026

MedVIGIL: Evaluating Trustworthy Medical VLMs Under Broken Visual Evidence

Medical vision--language models (VLMs) are usually evaluated on intact image--question pairs, but trustworthy clinical use requires a stronger property: a model must recognise when the evidential basis for an answer has failed. We study this through silent failures under perturbed evidence, where a vision-required medical question is paired with a false premise, wording perturbation, knowledge-only rewrite, or ROI-corrupted image, yet the model returns a fluent non-refusal answer. We introduce medvigil, a 300-case evaluation suite drawn from four public medical VQA sources, supervised end to end by four board-certified radiologists: every gold answer, refusal option, candidate-answer set, paraphrase, false-premise trap, ROI box, and clinical risk tier is clinician-authored. Two attending radiologists annotate every case in parallel, a senior radiologist consolidates the released manifest, and a separate fourth radiologist independent of construction answers every probe to provide the human reference baseline. The release contains 2556 MCQ probes, 240 counterfactual triplets, physician-adjudicated risk-tier and answerability flags, ROI boxes, and a paired open-ended variant. We report seven correctness-conditioned audit metrics that summarise into the medvigil Composite Score (MCS), and audit 16 vision-capable models plus two text-only baselines. The independent radiologist scores MCS 83.3 at silent-failure rate 5.8%, leaving a 14.1-point composite headroom above the strongest audited model (Claude Opus 4.7 at 69.2). The benchmark and evaluation harness are publicly released.

Hanqi Jiang, Junhao Chen, Mingyu Kang +12

University of Georgia · Chungbuk National University · Chungnam National University Hospital +5

VLM EvaluationVLM RobustnessHealthcareMedical VQAMedical VLMsMedical Image Benchmarks
AskPDF
Mar 24, 2026

MedObvious: Exposing the Medical Moravec's Paradox in VLMs via Clinical Triage

Vision Language Models (VLMs) are increasingly used for tasks like medical report generation and visual question answering. However, fluent diagnostic text does not guarantee safe visual understanding. In clinical practice, interpretation begins with pre-diagnostic sanity checks: verifying that the input is valid to read (correct modality and anatomy, plausible viewpoint and orientation, and no obvious integrity violations). Existing benchmarks largely assume this step is solved, and therefore miss a critical failure mode: a model can produce plausible narratives even when the input is inconsistent or invalid. We introduce MedObvious, a 1,880-task benchmark that isolates input validation as a set-level consistency capability over small multi-panel image sets: the model must identify whether any panel violates expected coherence. MedObvious spans five progressive tiers, from basic orientation/modality mismatches to clinically motivated anatomy/viewpoint verification and triage-style cues, and includes five evaluation formats to test robustness across interfaces. Evaluating 17 different VLMs, we find that sanity checking remains unreliable: several models hallucinate anomalies on normal (negative-control) inputs, performance degrades when scaling to larger image sets, and measured accuracy varies substantially between multiple-choice and open-ended settings. These results show that pre-diagnostic verification remains unsolved for medical VLMs and should be treated as a distinct, safety-critical capability before deployment.

Ufaq Khan, Umair Nawaz, L D M S S Teja +5

Mohamed bin Zayed University of Artificial Intelligence, UAE · National Institute of Technology, Silchar · Birmingham City University, UK

VLM EvaluationMedical VLMsVLM Robustness
AskPDF
May 23, 2026

Med-R2: An Adversarial Benchmark for Evidence-Grounded Reasoning in Medical VLMs

Vision-language models have demonstrated impressive capabilities in general medical visual question answering, yet due to limited interpretability, it remains unclear whether their predictions reflect evidence-grounded clinical reasoning or reliance on spurious priors. We introduce Med-R2 Bench, a hierarchical benchmark aligned with the clinical workflow to evaluate adversarial robustness with visual grounding. We design stepwise QA tasks to assess whether reasoning chains are strictly grounded in visual evidence across the four clinical stages, and employ adversarial perturbations to test robustness against misleading cues. Med-R2 comprises 42,432 images, 31 task categories, and 110,406 QA pairs. Evaluation across 14 VLMs reveals a sequential performance degradation along the four-stage clinical workflow. Adversarial experiments show that models rely heavily on correct prompts to guess answers. Even when provided with explicit visual cues, the models struggle to accurately align textual descriptions. Finally, we demonstrate stepwise fine-tuning using our hierarchical data significantly improves reasoning robustness, highlighting its potential to drive future improvements in evidence-based medical AI.

Wen Ma, Fucheng Niu, Zhiting Fan +3

Zhejiang University, Zhejiang, China · Guangdong Institute of Intelligence Science and Technology

Adversarial Attacks on VLMsHealthcareMedical VLMsVLM ReasoningVisually Grounded ReasoningMedical Image Grounding+1
AskPDF