What Paired Evaluations Reveal under Visual Perturbations

Yongda Wei; Chen Zhang; Yifei Wang; Xinyu Wang; Bosen Shao; Hanxi Li; Liping Di

cs.CVSep 28, 2026

What Paired Evaluations Reveal under Visual Perturbations

Authors: Yongda Wei, Chen Zhang, Yifei Wang, Xinyu Wang, Bosen Shao, Hanxi Li, Liping Di

Abstract

Robustness evaluation must examine diverse visual perturbations, while benchmarks cover only some real-world conditions and physical testing is costly. Paired evaluations link clean and perturbed predictions for the same image, capturing changes in correctness, confidence, and acceptance beyond aggregate accuracy. We investigate how this image correspondence supports two needs in robustness evaluation: interpreting paired evaluation results and prioritizing samples for physical testing. To interpret paired evaluation results, we fix both sets of prediction records and vary their correspondence within each class. We prove that classwise correct-correct counts give the same sharp bounds on lost acceptance and mean true-class probability decrease among retained-correct inputs as any feasible five-state refinement. Distinguishing persistent from changed wrong answers can further constrain accepted-error transitions, while shared correspondence can establish policy orderings left unresolved by separate cost intervals. To prioritize samples for physical testing, we retain each image's synthetic responses and rank clean-correct images by their mean true-class probability under corruption. Across 44 classifiers, testing the highest-risk 20% finds 67% and 45% of failures under mild screen and print recaptures, versus 58% and 36% for clean confidence and 60% and 37% for an equal-size natural-transformation average. With both probability averaging and an A3Rank scoring adaptation, the tested corruption set yields higher mean failure recall than the natural-transform set; differences between scores depend on the source and budget. Together, these findings show that the value of correspondence depends on the evaluation objective: classwise counts suffice for specified reliability bounds, while image-specific synthetic responses improve the allocation of physical tests within the evaluated pool.

Figures & tables

State	Response	$C_{i}$	$C^{\prime}_{i}$	Labels
$h$	Harmful flip	$1$	$0$	$\hat{y}_{i}\neq\hat{y}^{\prime}_{i}$
$r$	Correction	$0$	$1$	$\hat{y}_{i}\neq\hat{y}^{\prime}_{i}$
$s$	Persistent error	$0$	$0$	$\hat{y}_{i}=\hat{y}^{\prime}_{i}$
$w$	Changed error	$0$	$0$	$\hat{y}_{i}\neq\hat{y}^{\prime}_{i}$
$k$	Retained correctness	$1$	$1$	$\hat{y}_{i}=\hat{y}^{\prime}_{i}$

Table 1: Five disjoint response states and aggregate metrics under the original single-label definition. The state symbols also denote empirical proportions.

Target	Definition	Interpretation
$dM_{k}$	$\mathbb{E}[M-M^{\prime}\mid T=k]$	Mean true-class probability decrease among inputs correct on both sides.
$ds_{j}$	$\mathbb{E}[S-S^{\prime}\mid T=j],\ j\in\{s,w\}$	Mean decrease in maximum probability within the specified wrong-answer state.
$E_{\tau}$	$\mathbb{P}(C^{\prime}=0,Z^{\prime}=1)$	Fraction of inputs with an accepted error after perturbation.
$L=L_{k}$	$\mathbb{P}(T=k,Z=1,Z^{\prime}=0)$	Fraction remaining correct but changing from accepted to rejected.
$H$	$\mathbb{P}(C=1,Z=1,C^{\prime}=0,Z^{\prime}=1)$	Fraction changing from an accepted correct prediction to an accepted error.

Table 2: Reliability targets. The first two are empirical conditional means, undefined for empty states and positive for decreases. The last three are empirical event proportions with all

N

evaluated inputs as denominator.

Target	Information used	Supported conclusion
$E_{\tau}$	Perturbed records	Exact accepted-error mass.
$L,\ dM_{k}$	Classwise overlaps $\mathcal{I}_{k}$	Sharp bounds equal those under every feasible five-state refinement.
$H$	Five-state counts $\mathcal{I}_{5}$	Can tighten overlap-conditioned bounds.
$J_{j}-J_{\ell}$	Shared correspondence across policies	Can certify an ordering throughout the feasible set despite overlapping cost intervals.

Table 3: Information and supported conclusions for fixed single-label records and within-class reassignment. Both sides’ records remain available;

dM_{k}

requires positive retained-correct support.

Figure 1: Mean identified widths in the 450-cell panel measure correspondence ambiguity.

W_{0}

W_{1}

and

W_{2}

use marginal records, added classwise overlap and added five-state counts. Colors partition

W_{0}

into remaining width and two reductions; bold labels give

W_{0}

. Five-state refinement narrows

H

, but not

L

dM_{k}

. Marginal

dM_{k}

ranges over positive overlaps; later levels fix observed overlaps.

Question	Comparison	Result
Shared correspondence: more orderings?	Shared correspondence vs. separate intervals ( $\mathcal{I}_{5}$ )	312/450 (69.3%)
Correctness overlap: more orderings?	$\mathcal{I}_{0}\rightarrow\mathcal{I}_{k}$	241/450 (53.6%)
Five-state counts: more orderings?	$\mathcal{I}_{k}\rightarrow\mathcal{I}_{5}$	42/450 (9.3%)
Minimax regret: lower realized cost?	Minimax regret vs. feasible mean ( $\mathcal{I}_{5}$ )	4 lower / 3 higher 443 equal

Table 4: Policy comparisons and selection on the frozen 450-cell panel,

\lambda=0.5

. Ordering rows count cells with additional strict comparisons; these counts can overlap. Counted orderings require both correct and incorrect predictions among threshold-switching records. Feasible-mean selection minimizes mean cost over uniformly weighted feasible correspondences.

Figure 2: Screening scores are evaluated by how well they distinguish images that fail under real recapture from those that remain correct. All scores use identical clean-correct image sets. (a) AUROC on a common color scale. Class- $\delta$ is the development-set class mean of $M-U$ ; $g(S,U)$ is fitted on development images. (b) Paired AUROC differences $U-S$ with 95% model-group resampling intervals; the dashed line marks zero gain. Numerical results, class-resampled intervals and cell counts are in Appendix E .

Figure 3: Different rankings are evaluated by the fraction of real failures they uncover at a given recapture budget. Results cover three budgets on mild conditions. Curves average the same model–condition cells;

A^{3}

uses the same corruptions as

U

. Intervals and other tiers are in Appendix E .

Figure 4: At equal inference counts, the tested corruptions yield higher mean failure recall and AUROC than natural transforms under both scoring rules. Comparisons use identical mild-condition image sets and 18 transformed inputs each. Mean denotes probability averaging;

A^{3}

, adapted A3Rank. Lines connect point estimates; selected difference intervals appear in Appendices E.2 and E.3 .

Appendix figures & tables49 assets

Supplementary material from the paper’s appendix.

Appendix

Record	Clean	Perturbed
1	$(0.6,0.2,0.2)$	$(0.6,0.2,0.2)$
2	$(0.1,0.8,0.1)$	$(0.1,0.8,0.1)$
3	$(0.1,0.8,0.1)$	$(0.3,0.3,0.4)$

Appendix

Table 5: Prediction records for the accepted-error counterexample.

		$H$		$L$		$dM_{k}$
Source	Cells	$\mathcal{I}_{0}$	$\mathcal{I}_{k}$	$\mathcal{I}_{0}$	$\mathcal{I}_{k}$	$\mathcal{I}_{k}$
Syn	792	8.44	3.44	6.73	4.63	6.01
Syn-B	792	1.96	0.66	6.33	5.42	2.11
ES	2,376	1.56	0.79	5.52	4.44	3.52
Diverse	7,128	0.86	0.69	2.86	1.99	5.76

Appendix

Table 6: Full-grid mean identification widths (percentage points).

\mathcal{I}_{0}

retains complete classwise marginal records;

\mathcal{I}_{k}

additionally fixes classwise correct–correct counts. For

L,dM_{k}

, the five-state bounds equal

\mathcal{I}_{k}

by Theorem 1 . Conditional-confidence means exclude zero-

K

cells. Cells are weighted equally within each source.

Source	Cells	$\mathcal{I}_{0}$	$\mathcal{I}_{k}$	$\mathcal{I}_{5}$	Extra	Narrower
Syn	90	8.45	3.47	2.40	1.07	90/90
Syn-B	90	1.92	0.69	0.54	0.15	25/90
ES	135	1.12	0.77	0.68	0.09	38/135
Diverse	135	0.97	0.78	0.68	0.10	42/135

Appendix

Table 7: Expanded panel: mean

H

widths (pp), additional narrowing from wrong-state refinement, and number of cells with strictly narrower bounds. Panel means use the same cells at every level.

	$H$		$L$		$dM_{k}$
Source	Original	Matched	Original	Matched	Original	Matched
Syn	2.40	0.87	4.61	2.52	5.97	2.76
Syn-B	0.54	0.41	5.47	4.74	2.13	1.76
ES	0.68	0.54	3.03	2.38	5.11	3.90
Diverse	0.68	0.52	2.66	2.04	5.84	4.27

Appendix

Table 8: Mean five-state width before and after matching to 200 common true classes and two evaluation records per class (pp). Matched values first average defined cells within each seed and then the five seeds equally. Candidate-probability spaces retain their original sizes of 1,000 classes for Syn and 200 for the other sources.

Source	$K>0$ cells	Pair $-$ ind.	Mean $\|W\|$	Mean $\|V\|$	$\|V\|>\|W\|$
Syn	792	+1.00	0.08	0.96	98.6%
Syn-B	792	+0.38	0.09	0.39	85.0%
ES	2,374	+0.52	0.21	0.67	86.3%
Diverse	7,010	+0.66	0.50	1.57	76.8%

Appendix

Table 9: Signed confidence decomposition on positive-

K

cells. Means are cell-weighted, confidence differences are in percentage points, and the last column reports the percentage of cells where the absolute within-class selection term exceeds the absolute composition term.

Source	Tier	Cells	$C$	$S$	$M$
Syn	all tiers	792	0.32 [0.28, 0.35]	0.32 [0.27, 0.37]	0.20 [0.16, 0.23]
Syn	mild	792	0.32 [0.28, 0.35]	0.32 [0.27, 0.37]	0.20 [0.16, 0.23]
Syn-B	all tiers	792	0.33 [0.29, 0.37]	0.31 [0.25, 0.36]	0.20 [0.17, 0.23]
Syn-B	mild	792	0.33 [0.29, 0.37]	0.31 [0.25, 0.36]	0.20 [0.17, 0.23]
ES	all tiers	2376	0.55 [0.53, 0.58]	0.55 [0.50, 0.59]	0.44 [0.41, 0.46]
ES	mild	1936	0.47 [0.44, 0.50]	0.47 [0.41, 0.51]	0.34 [0.31, 0.36]

Appendix

Table 10: Paired-to-unpaired variance ratio with group-resampled 95% intervals, by source and tier, for correctness

C

, maximum probability

S

and true-class probability

M

. Cells are model–condition pairs, weighted equally.

Source	Tier	$K$ width, classwise	$K$ width, global	Observed $h{+}r$	Pinned (%)
Syn	all tiers	13.78	23.31	13.87	48.4
Syn	mild	13.78	23.31	13.87	48.4
Syn-B	all tiers	6.44	11.32	9.33	83.9
Syn-B	mild	6.44	11.32	9.33	83.9
ES	all tiers	5.16	10.39	26.04	87.1
ES	mild	5.90	11.15	16.02	85.2

Appendix

Table 11: Identification of the total correct–correct overlap

K

from accuracies alone (percentage points of

N

). Classwise sums per-class Fréchet intervals; global uses only

A

B

and

N

. Gross flips

h+r=A+B-2K

inherit twice the width of

K

. Pinned is the share of classes whose interval is a single value.

Source	Cells	Shared $\pi$	$+k$	$+5$
Syn	90	82	77	28
Syn-B	90	60	75	6
ES	135	92	47	2
Diverse	135	78	42	6
Total	450	312	241	42

Appendix

Table 12: Cells with at least one additional strict policy comparison at

\lambda=0.5

, restricted to threshold-switching records containing both correct and incorrect predictions. Shared

\pi

compares common-correspondence differences against separate sharp policy intervals within

\mathcal{I}_{5}

;

+k

and

+5

instead add information (

\mathcal{I}_{0}\to\mathcal{I}_{k}

and

\mathcal{I}_{k}\to\mathcal{I}_{5}

). Counts summarize cells of the frozen panel.

Figure 5: Sensitivity to the frozen cost-weight grid. Fractions give the share of cells in each source’s fixed panel with an additional certified ordering. Shared-correspondence comparisons use separate sharp cost intervals as the control; information increments compare successive information levels. The primary weight is

\lambda=.5

Source	Cells	Feasible mean	Minimax	Better	Worse	Tied
Syn	90	0.004778	0.004222	1	1	88
Syn-B	90	0.005556	0.005556	0	0	90
ES	135	0.012593	0.004444	3	0	132
Diverse	135	0.003704	0.005185	0	2	133

Appendix

Table 13: Mean realized regret in percentage points at

\lambda=.5

\mathcal{I}_{5}

, averaging cells equally within source. Both rules use the same feasible permutations and measure regret relative to the lowest-cost policy under the observed correspondence. Better/Worse/Tied count cells where minimax has lower/higher/equal realized cost than the feasible-permutation mean rule.

$\lambda$	$\Delta$ (pp)	Better	Worse	Tied
0.0	0.000000	0	0	450
0.1	0.000044	1	0	449
0.2	-0.000542	0	3	447
0.3	-0.002400	2	5	443
0.4	-0.003111	4	7	439
0.5	0.002111	4	3	443

Appendix

Table 14: Realized policy-selection results over the full frozen weight grid at

\mathcal{I}_{5}

\Delta

is feasible-permutation mean regret minus minimax regret, in percentage points, averaged equally over all 450 cells; negative values favor the mean rule. Better/Worse/Tied count lower/higher/equal realized costs for minimax.

Target	$\mathcal{I}_{0}$	$\mathcal{I}_{k}$	$\mathcal{I}_{5}$	Observed
$H$	[0.4, 0.8]	[0.4, 0.8]	[0.4, 0.6]	0.6
$L$	[17.8, 26.2]	[19.0, 26.0]	[19.0, 26.0]	23.2
$dM_{k}$	[19.52, 30.02]	[20.58, 29.43]	[20.58, 29.43]	26.24

Appendix

Table 15: Sharp intervals and observed values in the real-record illustration. Units are percentage points for

H,L

and probability points for

dM_{k}

. Actual correspondence determines the observed column.

Source	Tier	$S$	$U$	$\delta$	ret	Class- $\delta$	$g(S,U)$	$U-S$ [95%]
ES	mild	0.778	0.854	0.768	0.823	0.580	0.855	+0.076 [+0.070, +0.080]
ES	severe	0.593	0.647	0.630	0.614	0.546	0.647	+0.054 [+0.044, +0.063]
ES	extreme	0.538	0.578	0.572	0.560	0.535	0.576	+0.040 [+0.027, +0.056]
Diverse	mild	0.678	0.801	0.781	0.755	0.587	0.809	+0.123 [+0.110, +0.131]
Diverse	severe	0.601	0.700	0.698	0.656	0.565	0.710	+0.099 [+0.084, +0.109]
Diverse	extreme	0.504	0.512	0.512	0.529	0.478	0.517	+0.008 [-0.006, +0.024]

Appendix

Table 16: AUROC for ranking real failures among clean-correct images, on identical sets for every score. Class-

\delta

is the development-set class mean of

M-U

;

g(S,U)

is fitted on development images. The gap uses 95% model-group resampling intervals. Class-resampled intervals and cell counts are in Appendix E .

Source	Tier	Cells	$U-S$	$\delta-U$	$g(S,U)-S$
ES	mild	1935	+0.076 [+0.060,+0.093]	-0.086 [-0.107,-0.065]	+0.077 [+0.060,+0.097]
ES	severe	219	+0.054 [+0.035,+0.070]	-0.016 [-0.028,-0.005]	+0.054 [+0.034,+0.072]
ES	extreme	214	+0.040 [+0.020,+0.057]	-0.005 [-0.020,+0.011]	+0.038 [+0.017,+0.056]
Diverse	mild	2244	+0.123 [+0.107,+0.142]	-0.020 [-0.031,-0.007]	+0.131 [+0.110,+0.153]
Diverse	severe	1099	+0.099 [+0.084,+0.116]	-0.002 [-0.013,+0.009]	+0.108 [+0.091,+0.128]
Diverse	extreme	3174	+0.008 [-0.013,+0.039]	+0.001 [-0.015,+0.017]	+0.013 [-0.013,+0.052]

Appendix

Table 17: Per-cell AUROC differences with 95% intervals from resampling the 200 true classes (point estimates are the plain cell means; cells is the mean number of cells retained per draw).

		$q=0.1$					$q=0.2$
Source	Tier	$S$	$U$	$\delta$	ret	random	$S$	$U$	$\delta$	ret	random
ES	mild	0.406	0.489	0.395	0.487	0.096	0.578	0.672	0.568	0.648	0.199
ES	severe	0.126	0.133	0.134	0.135	0.100	0.241	0.254	0.252	0.248	0.201
ES	extreme	0.104	0.106	0.106	0.106	0.100	0.208	0.210	0.209	0.208	0.200
Diverse	mild	0.208	0.260	0.271	0.271	0.099	0.357	0.449	0.452	0.441	0.200
Diverse	severe	0.130	0.144	0.150	0.147	0.100	0.248	0.274	0.280	0.270	0.199

Appendix

Table 18: Screening: share of real failures among the top-

q

fraction of clean-correct images ranked by each score (cells weighted equally within source and tier). Random selection is one seeded draw per cell; its expectation is

q

Source	Tier	$S$	$U$	$U-S$ [95%]	$U-\delta$ [95%]	$U-\mathrm{ret}$ [95%]
ES	mild	0.578	0.672	+0.094 [+0.078,+0.106]	+0.104 [+0.079,+0.140]	+0.024 [+0.013,+0.042]
ES	severe	0.241	0.254	+0.013 [+0.011,+0.017]	+0.002 [-0.001,+0.007]	+0.007 [+0.001,+0.016]
ES	extreme	0.208	0.210	+0.002 [+0.002,+0.003]	+0.001 [-0.000,+0.002]	+0.001 [+0.000,+0.004]
Diverse	mild	0.357	0.449	+0.092 [+0.075,+0.109]	-0.003 [-0.009,+0.007]	+0.008 [-0.006,+0.036]
Diverse	severe	0.248	0.274	+0.026 [+0.021,+0.032]	-0.006 [-0.008,-0.005]	+0.004 [-0.002,+0.016]
Diverse	extreme	0.203	0.204	+0.001 [+0.001,+0.002]	-0.000 [-0.001,-0.000]	+0.000 [-0.000,+0.001]

Appendix

Table 19: Failure coverage at a 20% recapture budget and paired differences with 95% model-group resampling intervals (ties prorated; cells equally weighted). Random selection covers approximately 20% in expectation, subject to integer budget rounding. Other budgets and class-resampled intervals are in Appendix E .

Source	Tier	Budget	$U-S$ [classes]	$U-\delta$ [classes]	$U-\mathrm{ret}$ [classes]
ES	mild	0.1	[+0.064,+0.111]	[+0.077,+0.116]	[-0.009,+0.013]
ES	mild	0.2	[+0.075,+0.114]	[+0.076,+0.125]	[+0.008,+0.038]
ES	mild	0.3	[+0.079,+0.120]	[+0.057,+0.115]	[+0.015,+0.048]
ES	severe	0.1	[+0.004,+0.011]	[-0.004,+0.004]	[-0.005,+0.001]
ES	severe	0.2	[+0.006,+0.020]	[-0.003,+0.006]	[+0.001,+0.011]
ES	severe	0.3	[+0.008,+0.024]	[-0.001,+0.009]	[+0.008,+0.021]

Appendix

Table 20: Class-resampled 95% intervals for the paired coverage differences at each budget (200 draws of the 200 true classes, one draw shared by every cell of a source).

Source	Tier	$p$	attainable ceiling			$U$ / ceiling
			$q{=}0.1$	$q{=}0.2$	$q{=}0.3$	$q{=}0.1$	$q{=}0.2$
ES	mild	0.167	0.748	0.883	0.941	0.676	0.760
ES	severe	0.675	0.160	0.319	0.478	0.864	0.830
ES	extreme	0.908	0.112	0.223	0.335	0.956	0.947
Diverse	mild	0.360	0.373	0.643	0.815	0.777	0.739
Diverse	severe	0.652	0.170	0.339	0.500	0.891	0.853

Appendix

Table 21: Failure share

p

among clean-correct images, attainable coverage ceiling

\min(1,k/n_{\mathrm{fail}})

for budgets

q

, and the coverage of

U

as a fraction of the ceiling. Each column averages cellwise quantities equally within source and tier; the last is the mean of cellwise ratios, not the ratio of the displayed means.

Source	Tier	$S$	$U$	$U_{\mathrm{rank}}$	$U^{0.5}$	$U^{2}$	$U_{\mathrm{rank}}-S$ [groups]
ES	mild	0.778	0.854	0.832	0.854	0.851	+0.054 [+0.049,+0.058]
ES	severe	0.593	0.647	0.628	0.649	0.644	+0.035 [+0.027,+0.044]
ES	extreme	0.538	0.578	0.565	0.580	0.575	+0.028 [+0.016,+0.045]
Diverse	mild	0.678	0.801	0.757	0.805	0.794	+0.079 [+0.069,+0.086]
Diverse	severe	0.601	0.700	0.662	0.704	0.695	+0.061 [+0.052,+0.069]
Diverse	extreme	0.504	0.512	0.506	0.511	0.512	+0.003 [-0.008,+0.016]

Appendix

Table 22: Protocol sensitivity: AUROC of

U

, of its rank-based version and of power-transformed versions, with the

U_{\mathrm{rank}}-S

gap and a 95% interval from resampling model groups.

Source	Tier	$S$	$U$	$\delta$	ret	$g(S{,}U)$	AUROC $U{-}S$	Cov. $U{-}S$
ES	mild	0.778	0.856	0.779	0.831	0.856	+0.078 [+0.072,+0.083]	+0.111 [+0.097,+0.129]
ES	severe	0.572	0.641	0.646	0.608	0.644	+0.069 [+0.059,+0.075]	+0.012 [+0.010,+0.015]
ES	extreme	0.504	0.560	0.577	0.553	0.563	+0.056 [+0.034,+0.072]	+0.003 [+0.002,+0.003]
Diverse	mild	0.673	0.801	0.790	0.754	0.815	+0.128 [+0.119,+0.135]	+0.093 [+0.075,+0.120]
Diverse	severe	0.588	0.701	0.709	0.663	0.718	+0.113 [+0.096,+0.128]	+0.026 [+0.020,+0.036]
Diverse	extreme	0.475	0.507	0.536	0.518	0.520	+0.032 [+0.019,+0.049]	+0.001 [+0.001,+0.002]

Appendix

Table 23: Check on the development images with fitting roles exchanged: AUROC of each score, the

U-S

gap, and the budget-0.2 coverage difference, with 95% intervals from resampling model groups (class-resampled intervals also exclude zero on mild and severe conditions).

Source	Tier (eval-defined)	Conditions	$S$	$U$	AUROC $U{-}S$ [95%]	Cov. $U{-}S$ [95%]
ES	mild	44	0.778	0.856	+0.078 [+0.072,+0.083]	+0.111 [+0.097,+0.129]
ES	severe	5	0.572	0.641	+0.069 [+0.059,+0.075]	+0.012 [+0.010,+0.015]
ES	extreme	5	0.504	0.560	+0.056 [+0.034,+0.072]	+0.003 [+0.002,+0.003]
Diverse	mild	56	0.667	0.794	+0.128 [+0.118,+0.135]	+0.088 [+0.071,+0.114]
Diverse	severe	20	0.583	0.695	+0.112 [+0.093,+0.127]	+0.023 [+0.018,+0.032]
Diverse	extreme	86	0.475	0.507	+0.032 [+0.019,+0.049]	+0.001 [+0.001,+0.002]

Appendix

Table 24: Swapped-split confirmation regrouped by tiers defined from evaluation-set accuracy (development images evaluated; fitting on evaluation images). AUROC of

S

and

U

, and the

U-S

gaps in AUROC and in coverage at

q=0.2

, with 95% intervals from resampling model groups.

Source	Tier	Cells	$S$	$U$	Derangement	LOO mean	Gap [95%]	Wins (%)
ES	mild	1936	0.775	0.853	0.553	0.548	+0.299 [+0.275,+0.319]	99.9
ES	severe	218	0.594	0.647	0.523	0.519	+0.124 [+0.119,+0.130]	99.1
ES	extreme	216	0.543	0.581	0.504	0.504	+0.077 [+0.058,+0.103]	80.5
Diverse	mild	2244	0.675	0.800	0.581	0.582	+0.219 [+0.205,+0.230]	100.0
Diverse	severe	1099	0.599	0.699	0.537	0.536	+0.162 [+0.148,+0.171]	99.2
Diverse	extreme	3485	0.498	0.508	0.461	0.453	+0.047 [+0.038,+0.059]	56.0

Appendix

Table 25: Identity ablation: AUROC of clean confidence

S

, of

U_{i}

, of its within-class derangement control and of its leave-one-out class mean for ranking failing against retained-correct clean-correct images (failure as the positive class), all on the same non-singleton subset; gap is

U_{i}

minus the derangement control with a 95% interval from resampling the ten model groups; wins is the share of cells in which

U_{i}

exceeds its control.

Source	Tier	$S$	$U$	Class (cc. dev)	Class (all dev)	$U-$ Class [95%]	Cov. $U-$ Class [95%]
ES	mild	0.778	0.854	0.597	0.590	+0.257 [+0.238,+0.277]	+0.374 [+0.332,+0.422]
ES	severe	0.593	0.646	0.557	0.543	+0.089 [+0.080,+0.100]	+0.037 [+0.027,+0.052]
ES	extreme	0.540	0.578	0.545	0.541	+0.033 [+0.012,+0.050]	+0.006 [+0.005,+0.008]
Diverse	mild	0.678	0.801	0.604	0.596	+0.197 [+0.185,+0.206]	+0.177 [+0.148,+0.213]
Diverse	severe	0.601	0.701	0.567	0.554	+0.134 [+0.122,+0.143]	+0.051 [+0.041,+0.063]
Diverse	extreme	0.502	0.510	0.449	0.419	+0.061 [+0.036,+0.093]	+0.003 [+0.003,+0.004]

Appendix

Table 26: Class-level baselines matched to

U

: AUROC of

S

U

and the development-set class mean of

U

over clean-correct development images (cc. dev) or over all development images (all dev), on the images whose class has such a value; the last two columns give the

U

minus clean-correct development class-mean gap in AUROC and in coverage at

q=0.2

with 95% intervals from resampling model groups.

Source	Stratum	$S$	$U$	Derangement	Gap [95%]	Retention gap
ES	low	0.683	0.771	0.517	+0.255 [+0.235,+0.274]	+0.225
ES	mid	0.548	0.749	0.455	+0.294 [+0.255,+0.332]	+0.100
ES	high	0.530	0.755	0.460	+0.295 [+0.252,+0.325]	+0.075
Diverse	low	0.581	0.676	0.541	+0.136 [+0.110,+0.156]	+0.150
Diverse	mid	0.527	0.633	0.490	+0.144 [+0.117,+0.170]	+0.127
Diverse	high	0.517	0.646	0.507	+0.138 [+0.122,+0.158]	+0.116

Appendix

Table 27: Image-level separation within clean-confidence terciles (all conditions of each source, cells weighted equally; failure as the positive class). Gap is

U_{i}

minus its derangement control on the non-singleton subset with a 95% interval from resampling the ten model groups; the last column is the retention rate of images with above-median

U_{i}

minus that of images below the median within the stratum.

Source	$m$	Subsets	$n$	Coverage [min,max]	$U_{m}-S$ [min]	AUROC [min,max]
ES	1	random	18	0.597 [0.569,0.619]	+0.019 [-0.009]	0.799 [0.785,0.816]
ES	2	random	20	0.625 [0.609,0.638]	+0.047 [+0.030]	0.818 [0.801,0.833]
ES	3	random	20	0.640 [0.626,0.652]	+0.062 [+0.048]	0.830 [0.818,0.844]
ES	6	random	20	0.654 [0.641,0.666]	+0.076 [+0.063]	0.840 [0.826,0.852]
ES	9	random	20	0.663 [0.655,0.669]	+0.085 [+0.077]	0.849 [0.837,0.854]
ES	9	severity 2 only	1	0.658	+0.080	0.837

Appendix

Table 28: Coverage at

q=0.2

, coverage gain over

S

and AUROC of

U_{m}

computed from

m

of the 18 Syn-B conditions on mild conditions (cells weighted equally); for random subsets the mean over the prefixed subsets is followed by the minimum and maximum over subsets (minimum only for the gain).

Source	Tier	$q$	$U^{+0}-\mathrm{TTA}^{+0}$	native $U-\mathrm{TTA}$	$U_{18}-\mathrm{TTA}^{\mathrm{sel}}$	$\mathrm{TTA}^{\mathrm{sel}}-\mathrm{TTA}_{18}$
ES	mild	0.1	+0.057 [+0.046,+0.074]	+0.060 [+0.046,+0.082]	+0.054 [+0.043,+0.069]	+0.002 [-0.002,+0.007]
ES	mild	0.2	+0.076 [+0.062,+0.086]	+0.073 [+0.061,+0.082]	+0.072 [+0.060,+0.081]	+0.004 [+0.002,+0.007]
ES	mild	0.3	+0.078 [+0.062,+0.090]	+0.076 [+0.062,+0.088]	+0.076 [+0.061,+0.089]	+0.002 [+0.001,+0.003]
ES	severe	0.1	+0.007 [+0.006,+0.009]	+0.006 [+0.005,+0.007]	+0.007 [+0.005,+0.009]	+0.000 [-0.001,+0.001]
ES	severe	0.2	+0.012 [+0.009,+0.016]	+0.010 [+0.006,+0.013]	+0.012 [+0.009,+0.015]	+0.001 [+0.000,+0.002]
ES	severe	0.3	+0.017 [+0.012,+0.021]	+0.014 [+0.008,+0.019]	+0.017 [+0.011,+0.021]	+0.001 [-0.001,+0.002]

Appendix

Table 29: Paired coverage differences for the variants of the natural-augmentation comparison at budgets

q\in\{0.1,0.2,0.3\}

(95% intervals from resampling model groups): both averages with the untransformed image added; both under the native scoring rule;

U_{18}

against the development-selected natural set; the development-selected against the fixed natural set (the development-selected Syn-B set against the historical

U

is given in the text).

Implementation	Models	Pred. agreement (min)	Scale (range)	Within tolerances
DINOv2	1	1.0000	–	1/1
DINOv3	1	1.0000	–	1/1
OpenCLIP	35	1.0000	14.3–117.3	35/35
timm	6	1.0000	–	6/6
torchvision	1	1.0000	–	1/1

Appendix

Table 30: Agreement between the re-inference and the historical clean and corruption records, by model implementation (minimum prediction agreement, maximum mean absolute probability difference and clean accuracy difference over models and conditions; range of learned logit scales), and the number of models meeting the prespecified agreement tolerances.

Source	Tier	$S$	$U_{18}$	$\mathrm{TTA}_{18}$	$U_{18}-\mathrm{TTA}_{18}$ [groups]	[classes]	$\Delta$ AUROC [groups]	$U_{18}-\mathrm{TTA}^{\mathrm{sel}}$
ES	mild	0.561	0.671	0.583	+0.088 [+0.076,+0.103]	[+0.058,+0.125]	+0.065 [+0.060,+0.069]	+0.085 [+0.073,+0.100]
ES	severe	0.228	0.241	0.229	+0.012 [+0.009,+0.015]	[+0.006,+0.018]	+0.065 [+0.055,+0.071]	+0.011 [+0.009,+0.015]
ES	extreme	0.204	0.207	0.204	+0.002 [+0.002,+0.003]	[+0.001,+0.004]	+0.057 [+0.037,+0.072]	+0.002 [+0.002,+0.003]
Diverse	mild	0.334	0.428	0.345	+0.082 [+0.066,+0.109]	[+0.069,+0.100]	+0.117 [+0.108,+0.123]	+0.080 [+0.064,+0.105]
Diverse	severe	0.239	0.265	0.242	+0.024 [+0.018,+0.033]	[+0.019,+0.030]	+0.104 [+0.087,+0.118]	+0.024 [+0.018,+0.033]
Diverse	extreme	0.202	0.203	0.202	+0.001 [+0.001,+0.002]	[+0.001,+0.002]	+0.035 [+0.021,+0.049]	+0.001 [+0.001,+0.002]

Appendix

Table 31: Natural-augmentation comparison on the development half of the image pool with the selection of

\mathrm{TTA}^{\mathrm{sel}}

redone on evaluation images (coverage at

q=0.2

; intervals from resampling model groups and classes).

Source	Pool	$m$	Subsets	Coverage: mean [min, max]	Gain over $S$ : mean [min]	AUROC: mean [min, max]
ES	$U$	1	18	0.597 [0.569,0.619]	+0.019 [-0.009]	0.799 [0.785,0.816]
ES	$U$	2	20	0.625 [0.609,0.638]	+0.047 [+0.030]	0.818 [0.802,0.833]
ES	$U$	3	20	0.640 [0.626,0.652]	+0.062 [+0.048]	0.830 [0.818,0.844]
ES	$U$	6	20	0.654 [0.641,0.666]	+0.076 [+0.063]	0.840 [0.826,0.852]
ES	$U$	9	20	0.663 [0.655,0.669]	+0.085 [+0.077]	0.849 [0.837,0.854]
ES	$U$	18	1	0.672	+0.094	0.854

Appendix

Table 32: Coverage at

q=0.2

, gain over

S

and AUROC of the corruption average and of the natural-augmentation average computed from

m

versions, mild conditions, with the prefixed subsets of Table 28 applied by index to both lists (mean over subsets followed by the minimum and maximum; minimum only for the gain).

Failure recall at 20% budget
	Natural transforms		Corruptions
Scoring rule	ES	Diverse	ES	Diverse
Probability mean	0.596	0.366	0.672	0.449
$A^{3}$ score	0.588	0.360	0.654	0.447
AUROC
Probability mean	0.791	0.688	0.854	0.801

Appendix

Table 33: Two probe sets crossed with two scoring rules on identical mild-condition image sets. Each uses 18 transformed inputs;

A^{3}

is the A3Rank scoring adaptation. Recall uses a 20% recapture budget. Difference intervals are in Appendix E.2 .

Source	A3 probes	Metric	$U-A^{3}$	Group interval	Class interval
ES	Natural	Recall@.2	+8.38	[+7.15, +9.40]	[+6.72, +10.06]
ES	Natural	AUROC	+6.95	[+6.30, +7.48]	[+5.51, +8.47]
ES	Corruption	Recall@.2	+1.85	[+0.73, +2.74]	[+0.69, +2.86]
ES	Corruption	AUROC	+1.77	[+1.37, +2.07]	[+1.30, +2.31]
Diverse	Natural	Recall@.2	+8.91	[+7.41, +10.45]	[+7.06, +10.97]
Diverse	Natural	AUROC	+12.09	[+10.67, +13.04]	[+10.56, +14.00]

Appendix

Table 34: Mild-condition differences between

U

and the A3Rank scoring adaptations. Differences and separate 95% resampling intervals are in percentage points. Group resampling uses 2,000 draws of ten model groups; class resampling uses 200 draws of 200 classes and recomputes metrics on weighted images.

Source	Budget	$U$	$A^{3}$	Difference	Group interval
ES	10%	0.489	0.479	+0.93	[-0.84, +2.13]
ES	20%	0.672	0.654	+1.85	[+0.73, +2.74]
ES	30%	0.771	0.758	+1.29	[+0.71, +2.16]
Diverse	10%	0.260	0.268	-0.80	[-1.37, -0.44]
Diverse	20%	0.449	0.447	+0.23	[-0.40, +1.18]
Diverse	30%	0.591	0.577	+1.41	[+0.59, +2.64]

Appendix

Table 35: Budget sensitivity of the corruption-based ranking adaptation.

U

and

A^{3}

columns report failure recall; differences

U-A^{3}

and their 95% model-group intervals are in percentage points. The primary budget is 20%;

A^{3}

has higher recall on Diverse at 10%.

Source	Probe set	Evaluations	Recall@.2	AUROC	$\Delta$ vs. TTA
ES	Clean	1	0.578	0.778	-0.018
ES	Natural TTA	18	0.596	0.791	+0.000
ES	Severity 2	9	0.658	0.837	+0.062
ES	Severity 4	9	0.669	0.854	+0.073
ES	All corruptions	18	0.672	0.854	+0.076
Diverse	Clean	1	0.357	0.678	-0.009

Appendix

Table 36: Mild screening by probe set. Evaluations count transformed inputs beyond the common clean prediction; the clean-only row counts that prediction. The severity subsets use different numbers of transformed inputs. Differences are in recall units.

Figure 6: Failure recall at a 20% testing budget for each corruption probe on mild recaptures. Circles and squares distinguish severities 2 and 4. Vertical lines mark the 18-corruption average, 18-transform natural average and clean confidence. Each point uses one transformed input, while the averages use 18. The probe sets differ in both transformation family and response strength.

Source	Tier	$S$	$U_{18}$	$\mathrm{TTA}_{18}$	[groups]	[classes]	$U_{18}$	$\mathrm{TTA}_{18}$	$\Delta$ [groups]
		Coverage at $q{=}0.2$			$U_{18}-\mathrm{TTA}_{18}$		AUROC
ES	mild	0.578	0.672	0.596	+0.076 [+0.063,+0.086]	[+0.059,+0.088]	0.854	0.791	+0.063 [+0.058,+0.067]
ES	severe	0.241	0.254	0.242	+0.013 [+0.010,+0.016]	[+0.007,+0.018]	0.647	0.596	+0.051 [+0.041,+0.059]
ES	extreme	0.208	0.210	0.208	+0.002 [+0.002,+0.003]	[+0.001,+0.004]	0.578	0.540	+0.038 [+0.026,+0.052]
Diverse	mild	0.357	0.449	0.366	+0.083 [+0.066,+0.098]	[+0.066,+0.103]	0.801	0.688	+0.114 [+0.100,+0.123]
Diverse	severe	0.248	0.274	0.250	+0.023 [+0.019,+0.028]	[+0.018,+0.031]	0.700	0.607	+0.093 [+0.078,+0.104]

Appendix

Table 37: Corruption versus natural-transformation averages across all tiers, using the same model evaluations. Coverage is failure recall at

q=0.2

; paired differences carry separate 95% group and class intervals.

Source	Score	Recall@.2	AUROC	$U-$ score	Group CI	Class CI
ES	$U$	0.672	0.854	–	–	–
ES	$S$	0.578	0.778	+0.094	[+0.079,+0.106]	[+0.076,+0.115]
ES	TTA	0.596	0.791	+0.076	[+0.063,+0.086]	[+0.059,+0.088]
ES	Margin	0.577	0.779	+0.096	[+0.079,+0.110]	[+0.073,+0.115]
ES	Entropy	0.574	0.773	+0.098	[+0.079,+0.114]	[+0.081,+0.121]
ES	Agreement	0.423	0.648	+0.249	[+0.208,+0.284]	[+0.212,+0.283]

Appendix

Table 38: Additional simple baselines on mild recaptures.

\Delta

is the original mean recall difference

U-

baseline at

q=0.2

; the final two columns give separate 95% group/class resampling intervals for that difference. AUROC is the secondary metric; all three baselines were specified before calculation.

Source	Group	Cells	$U-S$	$U-\mathrm{TTA}$
Diverse	SSL	204	+0.085	+0.074
Diverse	DataComp	204	+0.107	+0.093
Diverse	DFN	102	+0.115	+0.091
Diverse	LAION	612	+0.093	+0.091
Diverse	MetaCLIP	153	+0.123	+0.115
Diverse	OpenAI	255	+0.072	+0.064

Appendix

Table 39: Mild-condition recall differences at

q=0.2

by the ten fixed model groups, with cells weighted equally within each group. The statistical groups organize the evaluated model pool and have unequal sizes.

Figure 7: Examples of three perturbation families for the same base image. Rows 1–2 show synthetic corruptions used in Syn and Syn-B. Row 3 shows representative natural transformations used for comparison. Row 4 shows original physical recaptures from ImageNet-ES and ES Diverse.

		$H$		$L$		Sel. risk		Coverage
Source	Protocol	raw	Platt	raw	Platt	raw	Platt	raw	Platt
ES	Q80	1.39	1.39	9.90	9.90	10.48	10.48	0.531	0.531
ES	A0.9	0.65	0.26	9.48	10.81	7.92	5.44	0.484	0.476
ES	A0.7	2.31	1.82	7.48	6.45	14.03	13.26	0.602	0.608
ES	A0.5	5.39	5.27	4.20	3.26	19.78	20.60	0.703	0.712
Diverse	Q80	2.05	2.05	8.16	8.16	18.95	18.95	0.186	0.186

Appendix

Table 40: Acceptance events under the quantile protocol (Q80) and absolute-probability thresholds (A

p

), for raw and Platt-scaled scores fitted on clean development records (real sources, all conditions, cells weighted equally).

H

L

and selective risk are expressed as percentages; coverage is the fraction of perturbed inputs accepted. Selective risk is the error rate among accepted perturbed inputs.

Model identifier	Group	Params. / M
DINOv2-vitg14-lc__meta	SSL	1144
DINOv3-ViT7B16-lc__meta	SSL	6716
EVA02-L-14__merged2b_s4b_b131k	other-clip	428
MAE-ViT-B-ft__meta	SSL	87
MAE-ViT-L-ft__meta	SSL	304
PE-Core-L-14-336__meta	pe-core	671

Appendix

Table 41: Complete model pool with checkpoint identifiers and parameter counts. Groups define the statistical holdouts and may contain checkpoints trained on different datasets.

Set / step	Features
$B_{0}$	Clean accuracy on synthetic base images and real development base images
$B_{1}$	$B_{0}$ , synthetic $a_{1}$ , mean $S$ on each side, mean $M$ among each side’s correct images, perturbed clipped log loss, and target-matched synthetic risk
$B_{\mathrm{ind}}$	$B_{1}$ plus independent counterparts of $F,A,Q,k,dM_{k},ds_{s},ds_{w},L_{k}^{\rm syn}$
C1	Add $F,A,Q$ to $B_{1}$
C2	Add $k$ to C1
C3a	Add $dM_{k}$ to C2 (primary hypothesis)

Appendix

Table 42: Feature definitions.

L_{k}^{\rm syn}

is the mass of synthetic retained-correct inputs that lose acceptance. All features are aggregated at the model level.

Target source	Target	Baseline	C1 gain	C1 interval	$dM_{k}$ gain
ES	$a_{1}$	2.88	-0.12	[-0.52, +0.15]	+0.11
ES	$R_{80}$	3.26	+0.22	[+0.12, +0.32]	+0.24
ES	$E_{\tau}$	0.61	+0.00	[-0.09, +0.07]	-0.02
ES	$L$	2.42	-0.06	[-0.15, -0.00]	-0.00
Diverse	$a_{1}$	3.35	+1.10	[+0.40, +2.12]	+0.01
Diverse	$R_{80}$	2.60	+0.38	[-0.22, +1.29]	+0.02

Appendix

Table 43: Syn summaries predicting mild real targets. Baseline is

B_{\mathrm{ind}}

; changes are MAE reductions in pp, so positive is better. C1 adds

(F,A,Q)

; the last column instead adds

dM_{k}

alone. Brackets are conditional group-reweighting intervals of stored out-of-fold errors, not full fitting-uncertainty intervals. All sources and deletion refits are in Tables 44 and 45 below.

Features	Source	Target	$B_{\mathrm{ind}}+\mathrm{C1}$	Original $dM_{k}$	$B_{\mathrm{ind}}+dM_{k}$
Syn	ES	$a_{1}$	-0.12 [-0.52, +0.15]	+0.52 [-0.43, +2.56]	+0.11 [-0.00, +0.27]
Syn	ES	$R_{80}$	+0.22 [+0.12, +0.32]	+0.05 [-0.07, +0.22]	+0.24 [-0.00, +0.70]
Syn	ES	$E_{\tau}$	+0.00 [-0.09, +0.07]	-0.04 [-0.11, +0.00]	-0.02 [-0.05, +0.00]
Syn	ES	$L$	-0.06 [-0.15, -0.00]	-0.55 [-1.88, +0.05]	-0.00 [-0.06, +0.04]
Syn	Diverse	$a_{1}$	+1.10 [+0.40, +2.12]	+0.02 [+0.00, +0.05]	+0.01 [-0.19, +0.13]
Syn	Diverse	$R_{80}$	+0.38 [-0.22, +1.29]	-0.03 [-0.11, +0.01]	+0.02 [-0.04, +0.07]

Appendix

Table 44: All primary source/target settings: model-weighted MAE reductions (pp) with conditional OOF group-reweighting intervals. Original

dM_{k}

conditions on the preceding paired summaries and its independent counterpart;

B_{\mathrm{ind}}+dM_{k}

is a separate feature-order sensitivity. These are correlated comparisons on the same model pool.

Features	Source	Target	Baseline	C1	Gain (%)	Deletion gain (pp)	Positive
Syn	ES	$a_{1}$	2.88	3.00	-4.1	[-0.20, +0.19]	6/10
Syn	ES	$R_{80}$	3.26	3.04	+6.7	[-0.24, +0.34]	7/10
Syn	ES	$E_{\tau}$	0.61	0.61	+0.0	[-0.01, +0.39]	6/10
Syn	ES	$L$	2.42	2.48	-2.5	[-0.16, +0.06]	2/10
Syn	Diverse	$a_{1}$	3.35	2.25	+33.0	[+0.50, +1.55]	10/10
Syn	Diverse	$R_{80}$	2.60	2.22	+14.5	[+0.30, +1.67]	10/10

Appendix

Table 45: C1 against

B_{\mathrm{ind}}

: complete-source MAE and the range of MAE reductions after removing each group from both training and evaluation and refitting all inner/outer folds. The deletion range is a sensitivity range, not a confidence interval. Relative gain uses the matched baseline in that row.

Figure 8: Syn-to-Diverse C1 sensitivity. Diamonds use all 44 models; dots remove one group from both training and evaluation and rerun nested fitting. Current-summary control is

B_{1}+(F,A,Q)_{\mathrm{ind}}

; all-summary control is

B_{\mathrm{ind}}

. The dashed line is the original

0.1\,\mathrm{IQR}

criterion. Dots show the individual group-deletion refits.

Target	Source	$B_{0}$ MAE	$B_{1}$ MAE	Gain [95% interval]
$a_{1}$	ES	4.87	2.69	+2.18 [+0.68, +4.40]
$R_{80}$	ES	5.32	2.68	+2.65 [+1.19, +5.01]
$a_{1}$	DIV	6.62	3.03	+3.58 [+1.54, +6.87]
$R_{80}$	DIV	8.07	3.96	+4.11 [+1.59, +8.32]

Appendix

Table 46: Unpaired synthetic information versus clean-performance features (Syn; nested prediction across held-out model groups). MAEs and improvements are in percentage points; intervals resample stored out-of-fold errors by model group.

Domain	Matched images	Original acc.	ReaL acc.	$\Delta$ (pp)	Original $s$	ReaL $s$
ES	473	66.88	63.33	-3.55	3.99	8.01
Diverse	473	26.11	24.96	-1.16	1.16	2.77

Appendix

Table 47: Label sensitivity on the same 473 validation images with nonempty ReaL labels; percentages use the matched cohort rather than all 500 images.

Source	Labels	$h$	$r$	$k$	$s$	$w$	$H$	$L$	$h$ moved (%)	$w$ moved (%)
ES	original	24.17	1.55	65.33	3.99	4.96	1.40	9.93	–	–
ES	ReaL	22.02	1.30	62.03	8.01	6.64	1.27	9.38	18.4	16.0
Diverse	original	64.00	0.62	25.49	1.16	8.73	2.08	8.31	–	–
Diverse	ReaL	59.65	0.56	24.40	2.77	12.63	1.95	7.87	10.2	18.6

Appendix

Table 48: Five-state shares and acceptance events under original versus ReaL record-attached correctness (original/ReaL, percentages of the 473 matched images, averaged over models and all conditions). The last two columns give the share of original

h

and

w

records that change state under ReaL.

Domain	High (%)	Low (%)	High events	Low events
ES	61.45	27.10	16416	79704
Diverse	18.69	6.40	49248	239112

Appendix

Table 49: Persistent same-wrong rate conditional on original-label clean-wrong events among all 500 images, stratified by other-family clean-side consensus (high

\geq 0.75

; low

\leq 0.25

Explore similar work

Nov 24, 2025cs.CV

Diagnosing Corruption-Induced Reliability Failures in Vision-Language Models

Visual corruptions can change vision--language model (VLM) behavior in ways that top-1 accuracy does not capture. A model may keep the same answer while losing distributional support, or improve accuracy through unstable wrong-to-correct changes. We introduce Bench-C, a controlled multiple-choice testbed for studying these effects. It selects semantically diverse samples whose predictions respond to corruption, and evaluates them under 19 corruption types and five severity levels. To measure how corruption changes the option distribution, we introduce the Robustness Alignment Score (RAS), which combines confidence-correctness alignment with uncertainty direction. We further separate originally correct samples from originally wrong samples, and track whether changes are temporary or persistent across severity. Experiments across 13 VLMs reveal a counterintuitive pattern: mild corruptions can improve top-1 accuracy while degrading prediction structure. These failures include silent degradation, erroneous overconfidence, and severity-dependent persistence. Bench-C therefore supports robustness evaluation that goes beyond final answers and attributes where reliability changes occur. Code and data are available at https://github.com/xiangjieSui/Bench-C.

Xiangjie Sui, Songyang Li, Hanwei Zhu +3

City University of Macau, Macau SAR, China · Jiangxi University of Finance and Economics, Nanchang, China · Nanyang Technological University, Singapore

Overconfidence

Ask PDF

Apr 23, 2026cs.CV

Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Models

Large Vision-Language Models (VLMs) are increasingly used to evaluate outputs of other models, for image-to-text (I2T) tasks such as visual question answering, and text-to-image (T2I) generation tasks. Despite this growing reliance, the reliability of these Evaluator VLMs remains under explored. In this work, we systematically evaluate the reliability of Evaluator VLMs across both I2T and T2I tasks. We introduce targeted perturbations that degrade output quality along key error dimensions, including object hallucinations, spatial reasoning, factual grounding, and visual fidelity. These perturbations test whether Evaluator VLMs can reliably account for these quality degrading errors in their evaluations. Using a comprehensive benchmark of over 4000 perturbed instances spanning 40 perturbation dimensions, we evaluate 4 prominent VLMs using single-answer scoring, pairwise comparison, and reference-guided paradigms. Our findings reveal that current VLM evaluators exhibit substantial blind spots: they often fail to detect perturbed outputs - in some cases exceeding 50%, struggle particularly with fine-grained compositional and spatial errors, and are often insensitive to hallucinated content that contradicts the input image. Pairwise comparison proves more reliable, though failure rates persist. These results highlight the unreliable nature of current Evaluator VLMs and urge caution in their deployment for benchmarking and development decisions. Code and data have been made publicly available.

Mohammed Safi Ur Rahman Khan, Sanjay Suryanarayanan, Tushar Anand +1

♠Nilekani Centre at AI4Bharat ♢Indian Institute of Technology Madras · ♠Nilekani Centre at AI4Bharat · ♣BITS Pilani, Hyderabad

Large Language Model Evaluation Blind Spots Text-To-Image Destination-Only Scoring Visual Fidelity

Ask PDF

Jun 24, 2026cs.CV

How Robust is OCR-Reasoning? Evaluating OCR-Reasoning Robustness of Vision-Language Models under Visual Perturbations

Vision-language models (VLMs) have achieved strong performance on OCR-based benchmarks and increasingly focused on text-rich understanding, but their robustness under controlled visual degradation remains insufficiently understood. This gap is critical for OCR reasoning, where visual corruption can induce OCR errors and structural distortions, thereby introducing uncertainty into the reasoning task. To systematically study this problem, we introduce OCR-Robust, a benchmark designed for evaluating OCR reasoning robustness under visual perturbations. It contains 812 samples across two complementary subsets: OCR1.0, covering documents, scene text, receipts, handwriting, and mathematical content, and OCR2.0, focusing on charts, geometry diagrams, and tables. To enable efficient yet informative evaluation, we conduct a pilot study over 18 candidate perturbations and select 5 representative types at 3 severity levels each based on their impact and cross-model discriminability. We evaluate robustness using clean accuracy, Relative Corruption Retention (RCR), Worst-Case Retention (WCR), and a composite Corruption Robustness Index (CRI), and benchmark 18 models spanning proprietary systems, open-source VLMs, and OCR+LLM pipelines. Our results show that higher clean accuracy does not necessarily imply stronger robustness, and that models can suffer pronounced degradation in the worst case on OCR tasks that are sensitive to structure, and charts and tables are substantially more fragile than document-like inputs under perturbation.

Yuxing Cheng, Yuan Wu, Yi Chang

School of Artificial Intelligence, Jilin University · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · International Center of Future Science, Jilin University

Lingdt-Vl-Ocr Optical Character Recognition Reasoning Skills

Ask PDF