Authority-Preserving Evaluation of Medical Vision-Language Assistants

Flint Xiaofeng Fan; Cheston Tan; Yew-Soon Ong; Roger Wattenhofer

cs.CVSep 14, 2026

Authority-Preserving Evaluation of Medical Vision-Language Assistants

Authors: Flint Xiaofeng Fan, Cheston Tan, Yew-Soon Ong, Roger Wattenhofer

Organizations: ETH Zurich, Zurich, Switzerland · Agency for Science, Technology and Research (A*STAR), Singapore, Singapore

Abstract

Medical vision-language models can propose how urgently a skin lesion should be reviewed, but the local service retains authority to accept or replace that proposal under referral policy, capacity, and locally held patient context. Proposal quality and selected-action quality are therefore distinct evaluation targets, and benchmark evidence transfers between them only when local review preserves the expected action score. We introduce AuthEval, a logging and evaluation framework that records both actions, scores the selected action under declared local criteria, and, where feasible, scores the declined proposal under the same rule. It reports the resulting authority gap only when the record supports it. Because the gap is the product of the proposal-change rate and the mean score change on changed cases, that rate alone determines neither its magnitude nor its sign. On ISIC 2019, with MedGemma and simulated local review, two constraint regimes with similar change rates produced an optimistic image-equal gap under capacity ( $+0.744$ simulator units) but no detectable gap under safety. The declared evaluation unit also mattered: under mixed constraints the gap reversed from $+0.374$ to $-0.206$ when weighting shifted from image to lesion-aware cluster. AuthEval thus clarifies whether a study's records support claims about the model, the workflow, or both.

Figures & tables

Panel of Figure 1 — Figure 1: Why proposal and selected-action scores can differ, organized by when information is available. At decision time the proposer sees only $x$ , while the label-blind gate $G$ also sees local context $z$ and slate $S$ , but never the reference $y$ . Post-hoc scoring applies one locally parameterized rule to both actions; the declined proposal’s score is the one extra field needed when review changes the action.

Quantity	Setting	$p_{\rm img}$	$\delta_{\rm img}$	Image estimate	Cluster estimate
$\Delta_{\rm authority}$	MedGemma / mixed	$0.470$	$0.80$	$+0.374$	$-0.206$
				$[+0.091,+0.663]$	$[-0.373,-0.037]$
$\Delta_{\rm authority}$	MedGemma / capacity	$0.341$	$2.18$	$+0.744$	$+0.056$
				$[+0.491,+1.010]$	$[-0.107,+0.218]$
$\Delta_{\rm authority}$	MedGemma / safety	$0.281$	$-0.18$	$-0.051$	$-0.224$
				$[-0.262,+0.158]$	$[-0.358,-0.093]$

Table 1: Controlled authority gaps on the ISIC test split (

n=3{,}967

images; 2,090 clusters; simulator units); positive values denote optimistic proposal scoring. The final row is the paired probe-minus-LLaVA-Med gap difference under fixed capacity.

p_{\rm img}

is the image-level proposal-change rate and

\delta_{\rm img}

the mean score change on changed images. Weighting and bootstrap CIs (brackets) are defined in Section 4 .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Split	$n$	Very low	Low	Moderate	High
Train	17,629	9,392	1,839	2,890	3,508
Validation	3,735	1,923	393	648	771
Test	3,967	2,052	392	652	871

Appendix

Table 2: Split sizes and risk-stratum composition. Strata are derived from the recorded diagnosis and are used only as the post-hoc reference state.

Grouping	Components	Images
Exact pHash	95	191
Exact pHash crossing splits	0	0
Threshold-4 pHash	594	1,777
Threshold-4 pHash crossing splits	122	—

Appendix

Table 3: Perceptual-duplicate groupings over all 25,331 images. Exact groups constrain the split; threshold-4 components are diagnostic only.

Risk stratum	Clusters	Images	Images per cluster
Very low	1,333	2,052	1.54
Low	221	392	1.77
Moderate	244	652	2.67
High	292	871	2.98

Appendix

Table 4: Test-split cluster sizes by risk stratum. Image-equal weighting is proportional to the image column, cluster-equal weighting to the cluster column.

Profile	Service character	Capacity	Floor policy	$w_{u}$	$w_{o}$
P1	High-capacity dermatology	3	Standard	1.20	0.75
P2	General mixed source	2	Standard	1.20	1.20
P3	Specialist, safety-oriented	3	Strict	1.80	0.90
P4	Low-capacity, access-constrained	1	Lenient	1.30	2.00

Appendix

Table 5: Workflow-profile parameters. Capacity is the highest admissible action level;

w_{u}

and

w_{o}

are the base under- and over-escalation weights before the private multipliers of Table 6 .

Factor	Level	$w_{u}$ mult.	$w_{o}$ mult.
Sensitivity preference	Sensitivity-seeking	1.25	0.95
	Neutral	1.00	1.00
	Specificity-seeking	0.95	1.25
Follow-up burden	Low	1.00	1.00
	High	1.00	1.15
Invasiveness aversion	Neutral	1.00	1.00

Appendix

Table 6: Private factor levels and their multipliers on the under- and over-escalation weights.

Record level	Additional local-review fields	Quantities recovered
Proposal-only	$\hat{a}$ and proposal metric	none
Role-separated	$a_{\mathrm{sel}}$ , $U_{z}(x,y,a_{\mathrm{sel}})$	$p$ , $J$
Full counterfactual	$U_{z}(x,y,\hat{a})$ on changed cases	$J_{\rm auth}$ , $\Delta_{\mathrm{authority}}$ , $\delta$ if $p>0$

Appendix

Table 7: Cumulative record levels and the AuthEval quantities each one recovers. Fields are additional to the level above.

Review process	$J_{\rm auth}$	$J$	$p$	$\delta$	$\Delta_{\mathrm{authority}}$ , image	$\Delta_{\mathrm{authority}}$ , cluster
Capacity ceiling	$-2.938$	$-3.683$	$0.341$	$2.182$	$+0.744$ $[+0.491,+1.010]$	$+0.056$ $[-0.107,+0.218]$
Safety floor	$-2.938$	$-2.888$	$0.281$	$-0.180$	$-0.051$ $[-0.262,+0.158]$	$-0.224$ $[-0.358,-0.093]$
Mixed	$-2.938$	$-3.313$	$0.470$	$0.797$	$+0.374$ $[+0.091,+0.663]$	$-0.206$ $[-0.373,-0.037]$

Appendix

Table 8: Authority gap by simulated review process for the fixed MedGemma-derived proposal. Scores are in simulator units; brackets give 95% whole-cluster bootstrap intervals.

Proposal	Test balanced acc.	$p$	$\delta$	$\Delta_{\mathrm{authority}}$ , image	$\Delta_{\mathrm{authority}}$ , cluster
LLaVA-Med head	$0.249$	$0.163$	$0.084$	$+0.014$ $[-0.056,+0.086]$	$-0.096$ $[-0.156,-0.037]$
MedGemma readout	$0.420$	$0.341$	$2.182$	$+0.744$ $[+0.491,+1.010]$	$+0.056$ $[-0.107,+0.218]$
CLIP probe	$0.546$	$0.276$	$2.988$	$+0.824$ $[+0.621,+1.033]$	$+0.337$ $[+0.256,+0.422]$
Probe $-$ LLaVA-Med	—	—	—	$+0.810$ $[+0.629,+0.999]$	$+0.433$ $[+0.345,+0.524]$

Appendix

Table 9: Authority gap by proposal mechanism under one fixed capacity review process. The final row is the paired probe-minus-LLaVA-Med contrast, computed on identical clusters within each bootstrap replicate.

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 989 papers

MedVIGIL: Evaluating Trustworthy Medical VLMs Under Broken Visual Evidence
May 8, 2026Hanqi Jiang, Junhao Chen, Mingyu Kang +12VLM Evaluation VLM Robustness
University of Georgia · Chungbuk National University · Chungnam National University Hospital +5
Adapt or Calibrate? How Local Labels Buy Certified Coverage in Medical VQA
Jun 14, 2026Reza Khanmohammadi, Kundan Thind, Mohammad M. GhassemiVLM Evaluation Confidence Estimation for VLMs
Michigan State University East Lansing, MI, USA · Henry Ford Health Detroit, MI, USA
Consistent but Dangerous: Per-Sample Safety Classification Reveals False Reliability in Medical Vision-Language Models
Mar 22, 2026Binesh Sadanandan, Vahid BehzadanVLM Evaluation Medical VLMs
SAIL Lab, University of New Haven, West Haven, CT, USA

May 8, 2026J/K move · Enter open · S save

MedVIGIL: Evaluating Trustworthy Medical VLMs Under Broken Visual Evidence

Hanqi Jiang, Junhao Chen, Mingyu Kang, Hyeokjae Kwon+11

University of Georgia · Chungbuk National University · Chungnam National University Hospital+5

VLM Evaluation VLM Robustness Healthcare Medical VQA Medical VLMs Medical Image Benchmarks

Open paper AskPDF arXiv

Medical vision--language models (VLMs) are usually evaluated on intact image--question pairs, but trustworthy clinical use requires a stronger property: a model must recognise when the evidential basis for an answer has failed. We study this through silent failures under perturbed evidence, where a vision-required medical question is paired with a false premise, wording perturbation, knowledge-only rewrite, or ROI-corrupted image, yet the model returns a fluent non-refusal answer. We introduce medvigil, a 300-case evaluation suite drawn from four public medical VQA sources, supervised end to end by four board-certified radiologists: every gold answer, refusal option, candidate-answer set, paraphrase, false-premise trap, ROI box, and clinical risk tier is clinician-authored. Two attending radiologists annotate every case in parallel, a senior radiologist consolidates the released manifest, and a separate fourth radiologist independent of construction answers every probe to provide the human reference baseline. The release contains 2556 MCQ probes, 240 counterfactual triplets, physician-adjudicated risk-tier and answerability flags, ROI boxes, and a paired open-ended variant. We report seven correctness-conditioned audit metrics that summarise into the medvigil Composite Score (MCS), and audit 16 vision-capable models plus two text-only baselines. The independent radiologist scores MCS 83.3 at silent-failure rate 5.8%, leaving a 14.1-point composite headroom above the strongest audited model (Claude Opus 4.7 at 69.2). The benchmark and evaluation harness are publicly released.

Abstract

VLM Evaluation

Explore similar work

MedVIGIL: Evaluating Trustworthy Medical VLMs Under Broken Visual Evidence

Adapt or Calibrate? How Local Labels Buy Certified Coverage in Medical VQA

Consistent but Dangerous: Per-Sample Safety Classification Reveals False Reliability in Medical Vision-Language Models