Auditing VLM Benchmarks from Existing Evaluation Runs

Denis Degtiarev; Anna Gromova; Sergey Katkov; Artem Mikhailin; Safina Mukhamedshina; Albert Nasybullin; Karina Shaikhutdinova; Artem Tuzov

cs.CVOct 9, 2026

Auditing VLM Benchmarks from Existing Evaluation Runs

Authors: Denis Degtiarev, Anna Gromova, Sergey Katkov, Artem Mikhailin, Safina Mukhamedshina, Albert Nasybullin, Karina Shaikhutdinova, Artem Tuzov

Organizations: Innopolis University

Abstract

Every evaluation run of a vision-language model (VLM) already produces per-question artifacts (predictions, judged correctness, metadata) that leaderboards reduce to a single score and then discard. We reuse those artifacts to audit the benchmark itself, and add an audit layer to VLMEvalKit: six detectors that run as post-processing over saved outputs, plus one additional text-only rerun with visual inputs removed, emitting a per-question report. The layer reports per-question findings together with aggregate audit statistics; for the headline quantities, it attaches permutation-null excess where applicable, interval estimates for proportions, and sensitivity to the model pool. Auditing MMStar, a benchmark curated specifically so that the image is indispensable, with three models, we find zero aggregate accuracy gain from the image on 20.2% of its questions. An item-matched permutation null accounts for 17.0%, leaving a +3.2 pp excess (corrected permutation p = 0.001). This excess is concentrated in the mathematics and science categories and is absent from coarse perception, a localization consistent with knowledge substituting for visual evidence; confirming that mechanism would require item-level review. A second detector flags 134 questions on which every model that answered in parseable form picked the same non-gold option, and these candidates peak in the perception categories, whereas the zero-gain excess is largest in mathematics and science & technology. Replacing the LLM judge with exact string matching changes no conclusion, and every scoring discrepancy between the two traces to one model whose free-form answers a parser cannot read. On a binary yes/no benchmark the detectors measurably fall out of calibration, a scope boundary that we measure and report. The audit layer, the prediction artifacts and the generated reports accompany this submission.

Figures & tables

Figure 1: The audit pipeline in five stages. (1) Inputs: the full model runs with predictions and judged hits, one additional text-only blind run with images removed, and the dataset itself with questions, options and gold labels. (2) All frames are aligned by dataset question ID. (3) Six detectors process the aligned frame. (4) The audit report lists per-question findings with severity levels and summary statistics, including permutation-null excess values and a reproducibility manifest. (5) The report yields actionable insights for improving benchmarks and evaluation practice. The blind run is the only step that requires inference, and it runs without images.

Detector	Signal	Pathology flagged	Judge-dep.
visual_dependency	full vs. blind accuracy	no-gain-from-image items	yes
consensus_error	majority vote vs. label	label-conflict candidates	no
fleiss_kappa_agreement	inter-model $\kappa$	unstable / ambiguous items	no
correctness_agreement	judged-correctness profile	difficulty / discrimination	yes
answer_options_distribution	label-position histogram	position bias	no
distractor_similarity	pairwise option similarity	near-duplicate options	no

Table 1: Detector taxonomy. “Judge-dep.” marks detectors whose output depends on the answer-extraction protocol.

Figure 2: The zero-gain signal on MMStar. (a) Model-averaged full against blind accuracy per question (judged arm, jittered; three models give the

4\times 4

grid). Questions on the diagonal gain nothing from the image. (b) The observed 20.2% share against 999 item-shuffled permutations: the null centres at 17.0%, leaving a

+3.2

pp excess that no permutation reaches (

p=0.001

). (c) The no-gain set is heterogeneous: on 164 of 303 questions every model is wrong with and without the image, so the conservative reading is zero aggregate accuracy gain from the image for this model pool , and genuinely text-answerable items are only one possible component.

Model pool	Observed (%)	Null mean (%)	Excess (pp)	$p$
All three models	20.2	17.0	$+3.2$	0.001
without InternVL3-8B	27.9	24.1	$+3.7$	0.001
without Ristretto-3B	25.5	21.9	$+3.6$	0.001
without gpt-5-nano	34.4	26.7	$+7.7$	0.001

Table 2: Zero-gain share against the permutation null on MMStar (999 permutations, one-sided,

(g{+}1)/(N{+}1)

correction). The excess is markedly more pool-stable than the raw share: raw spread 14.2 pp, excess spread 4.6 pp.

Figure 3: Per-category audit of MMStar (250 items per category), sorted by zero-gain excess. (a) The excess concentrates in mathematics and science & technology and is absent from coarse perception, a pattern consistent with greater scope for non-visual knowledge in the former categories. (b) Unanimous label-conflict candidates peak in the perception categories. The two detectors localize different candidate failure modes.

Figure 4: Judge dependence of the audit on MMStar. (a) Category shares under the gpt-4o-mini judge and under exact letter matching, on identical predictions: every share moves by at most 2.1 pp and no conclusion changes. (b) The disagreement has a single source: for the two open models the judge and the letter parser score every item identically, while gpt-5-nano, whose answers are unparseable on 10% of items, accounts for all of it.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 5: HallusionBench (951 yes/no items, same three models). (a) Ranking items by the continuous visual gain does not recover the benchmark’s own design labels (AUC 0.49 against 0.50 for chance). (b) The one directional signal that survives is consistent with the benchmark’s design: zero-gain excess is approximately 2.2 times larger on visual-supplement items than on visual-dependent items.

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 999 papers

Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?
May 21, 2026Zixuan Lan, Luzhe Sun, Matthew R. Walter +1VLM Evaluation VLM Interpretability
University of Chicago · Toyota Technological Institute at Chicago · Stony Brook University
Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Models
Apr 23, 2026Mohammed Safi Ur Rahman Khan, Sanjay Suryanarayanan, Tushar Anand +1VLM Evaluation VLM Robustness
♠Nilekani Centre at AI4Bharat ♢Indian Institute of Technology Madras · ♠Nilekani Centre at AI4Bharat · ♣BITS Pilani, Hyderabad
When the Judge Acts: Auditing VLM-Guided Image Selection on Culturally Situated Prompts
Oct 1, 2026Huichan SeoVLM Evaluation Position Bias in VLMs
Independent Researcher

May 21, 2026J/K move · Enter open · S save

Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?

Zixuan Lan, Luzhe Sun, Matthew R. Walter, Jiawei Zhou

University of Chicago · Toyota Technological Institute at Chicago · Stony Brook University

VLM Evaluation VLM Interpretability Vision-Language Grounding Visually Grounded Reasoning Fine-Grained Visual Perception Visual Evidence Sensitivity in VLMs

Open paper AskPDF arXiv

Benchmark accuracy is often implicitly assumed to reflect grounded visual understanding in vision-language models (VLMs), yet it remains unclear to what extent such scores truly reflect reliance on visual evidence. Motivated by a surprising observation that removing a substantial fraction of image tokens only degrades model performance very slightly on a widely used hallucination benchmark, we systematically investigate this mismatch in a set of open-source VLMs. Our analysis spans multiple levels of granularity, spanning global visual degradation, localized occlusion, question reformulation, answer-space expansion, and decision-level analyses beyond standard accuracy. We further complement these behavioral results with a layer-wise analysis of vision-token geometry. Throughout the experiments, we find that although VLMs do incorporate visual input, their predictions are less sensitive to the loss of fine-grained visual evidence that standard accuracy should have suggested. Even when the final prediction remains unchanged, the model's internal support for the correct answer may already be weakened. We further complement a representation-level analysis, which shows increasing similarity among visual tokens in deeper layers, providing a possible explanation for our findings. Together, these results suggest that current benchmarks are not sufficient to reliably evaluate fine-grained visual grounding in VLMs.

Abstract

VLM Evaluation

Explore similar work

Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?

Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Models

When the Judge Acts: Auditing VLM-Guided Image Selection on Culturally Situated Prompts