Measured or Real? When Model-Derived Measurements Can Be Corrected or Compared

Yifeng Li; Ziang Ni

cs.LGOct 27, 2026

Measured or Real? When Model-Derived Measurements Can Be Corrected or Compared

Authors: Yifeng Li, Ziang Ni

Organizations: University of Trento · Delft University of Technology

Abstract

Numbers computed from trained models, such as an example's difficulty or a model's benchmark lead, change with seed and prompt. We ask when such measurements can be corrected or compared. Noise shrinks correlations, and the textbook correction divides by the square root of two estimated reliabilities. On two CIFAR-100 difficulty scores, it returned correlations above one in 70 of 100 classes. We prove that its mean squared error is infinite even when the correlation and the reliabilities use separate seeds. Our rule corrects only when both reliabilities pass a significance test, then divides by their upper confidence bounds. Its output is always bounded, and its coverage guarantees assume Gaussian noise. It helps once reliability is well estimated, more than halving the error with 500 simulated units. Prompt settings can reverse rankings. On ARC, Qwen2-1.5B trails Llama-3.2-3B by 3.8 points with no examples in the prompt and leads by 16.6 with two. Across four benchmarks, averaging the two settings names one winner for 15 of 16 such reversals. We compare models within each setting, controlling family-wise error asymptotically in the number of items for a fixed prompt grid. Aligning embedding coordinates across seeds raises held-out per-dimension reliability from 0.003 to 0.813.

Figures & tables

Figure 1: In each of the three analyses the baseline fails, and the check in this paper gives an outcome that the data support. Left: all three rest on one measurement model and one design rule. Measuring unit

i

under replicate

s

gives its target value

\theta^{\star}_{i}

plus replicate noise

e_{i,s}

. A facet is a protocol, reported per level, when the reported result names its level or when we are unsure. Otherwise, if its levels are exchangeable, it is a replicate and is averaged over. (a) The plug-in correction of the correlation between EL2N and dynamic uncertainty, computed within each CIFAR-100 class, exceeds 1 in magnitude in 70 of the 100 classes (vermilion cells). Algorithm 1 returns a bounded value in all 100 and declines in 3 (open cells). Cells count classes and are not matched between the two grids. This demonstration uses the same two seeds for the correlation and the reliabilities, with the gate at

\alpha{=}0.25

. With the correlation computed across seeds on all 50,000 training examples, Algorithm 1 corrects 0.64 to 0.88. (b) On ARC-Easy, the pooled comparison puts Qwen2-1.5B ahead of Llama-3.2-3B , while the simultaneous 95% intervals of Algorithm 2 exclude zero with opposite signs in the two protocols. Intervals are drawn to scale. The counts of reversals and verdicts cover 270 pairs on four benchmarks. (c) Without alignment, dimension

j

of one seed need not match dimension

j

of another. Orthogonal Procrustes alignment of ResNet-18 features across seeds, fitted on half of the CIFAR-100 test images, raises per-dimension reliability on the other half from 0.003 to 0.813.

Problem	Example in this paper	Unit	Replicate (protocol)	What we estimate	Check	Response	Sec.
Noise shrinks a correlation	EL2N vs. dynamic uncertainty per CIFAR-100 class: $\|\hat{r}_{\mathrm{plug}}\|>1$ in 70/100 classes	training example	seed	correlation of seed-mean scores	$F$ -test gate on each reliability; variance–mean check	Alg. 1 : correct only if both gates pass, then divide by upper bounds	3
Errors from one run correlate	the two scores from the same seed: error correlation +0.28	training example	seed	as above	error correlation within a seed	compute $r_{\mathrm{obs}}$ from different seeds	3.5
A replicate shifts every unit	one wording makes all items easier; synthetic case (App. B ): within-run error 0.44 vs. 0.0007 with independent readouts	item	wording $\times$ option order	paired gap on one shared grid	a model against itself on two halves of the grid: 3.6% false winners (pooled 18.7%)	Alg. 2 on one grid shared by all models	4.4 , App. B
The setting changes the answer	ARC, Qwen2-1.5B minus Llama-3.2-3B : $-$ 3.8 points zero-shot, +16.6 two-shot	item	configuration (protocol: zero- vs. two-shot)	one gap per protocol, $\Delta_{p,\pi}$	sign of each protocol’s simultaneous interval	Alg. 2 : three verdicts; pooling names a stable winner for 15/16 reversals	4
Coordinates are arbitrary across runs	ResNet-18 penultimate features on CIFAR-100	image (per dimension)	seed	per-dimension reliability after alignment	per-dimension reliability before vs. after alignment: 0.003 vs. 0.813	held-out Procrustes alignment; distances; CKA	5

Table 1: Each problem needs its own check and response.

Result	Needs	Gives	Measured here	Not covered
Prop. 1 (known)	(i) Gaussian one-way model; (ii) $\Pr(\hat{\rho}\leq 0)>0$ ; (iii) $(\hat{\rho}_{X},\hat{\rho}_{Y})\in(0,\eta)^{2}$ with positive prob. given $r_{\mathrm{obs}}$	closed form of $\Pr(\hat{\rho}_{K}\leq 0)>0$ ; floored second moment $\Theta(1/\epsilon)$ ; $\Pr(\|\hat{r}_{\mathrm{plug}}\|>1)>0$	plug-in exceeds 1 in 70/100 CIFAR-100 classes (same seeds)	–
Prop. 2 (new)	(H1)–(H3); no Gaussian assumption; $\hat{\rho}_{X},\hat{\rho}_{Y}$ may be dependent	infinite second moment with $r_{\mathrm{obs}}$ on separate seeds; floored rate $\Theta(\epsilon^{-2})$ in the Gaussian disjoint-seed design	synthetic check (App. D.7 )	whether real scores meet (H2)
Alg. 1 : bound and cap	none (algebra)	$\|\hat{r}_{\mathrm{gate}}\|\leq 1$ ; accepted factor $\leq(1-1/(q_{\mathrm{gate}}q_{\mathrm{ucb}}))^{-1}$ if $m{=}K$ , 2.12 in the base setting (App. C.4 )	bounded in 100/100 classes	–
Alg. 1 : levels; Prop. 3	balanced Gaussian one-way model (Gaussian target values and noise); max rule: for each measure, a bound that covers on the branch where it is used	gate level $\alpha$ ; each bound covers with prob. $\geq 1-\alpha$ ; $\|\hat{r}_{\mathrm{gate}}\|\leq$ correction with true reliabilities with prob. $\geq 1-2\alpha$	MSE (Table 3 , Fig. D.1 ); coverage given acceptance 0.90–1.00; exact- $F$ coverage 0.88–0.90 up to $\|S\|\approx 0.5$ , 0.79–0.84 near 0.74	an interval for $r^{\star}$ ; optimality; same-run correlated errors; MSE or coverage of the max rule
Corrected value (0.88)	mean-zero errors; independent errors in $X$ and $Y$ ; exchangeable seeds	a corrected value with no Gaussian assumption; the Gaussian model enters only through the gate and $q_{\mathrm{ucb}}\approx 1.01$	agrees with the joint estimate (0.93) and the plug-in over 12 seed partitions (0.89–0.98)	accuracy on real data (no ground truth)
Alg. 2	i.i.d. items; paired-item CLT; consistent $\hat{R}$ ; one shared grid fixed in advance; every model scored alike	simultaneous coverage $1-\alpha$ of all $2P$ gaps, asymptotic in items; family-wise error control of all sign claims	Tables 4 and 5 ; new wordings 1/93 (App. G.6 )	unseen prompts, harnesses, scoring or tasks; finite samples; rank sets across wordings (0–2 of 6 splits); single configurations (5/93 pairs flip)

Table 2: What each result needs, what it gives, what we measured beyond its assumptions, and what it does not cover. Among the guarantees for Algorithm 1 , only the gate’s level and the coverage levels need Gaussian noise; Proposition 2 needs none.

Figure 2: Algorithm 1 declines wherever the plug-in factor is undefined or grows without bound, and dividing by an upper bound caps the factor that remains. The curves show the factor that one reliability estimate

\hat{\rho}_{\ell}=\mathrm{ICC}(1,K)

contributes to the correction, computed exactly at

N{=}50

K{=}3

\alpha{=}0.1

. Left of the gate, where

\mathrm{MSB}_{\ell}/\mathrm{MSW}_{\ell}\leq q_{\mathrm{gate}}

, Algorithm 1 returns

\mathrm{clip}(r_{\mathrm{obs}})

, so its factor is 1 (thick dashed line). Right of it, if the other gate also passes, Algorithm 1 divides by an upper bound that is at least

u_{\ell}

, so each factor is below

1/\sqrt{L}

. For

m{=}K

the two factors together are therefore at most

1/L

, which is 2.12 here (Proposition 3 (a)). Bottom: the sampling density of

\hat{\rho}_{\ell}

under the balanced Gaussian model when the reliability of the 3-replicate mean is 0.2, as in the base row of Table 3 . Most of it lies left of the gate, and in that row a gate fails in 87% of runs.

Figure 3: Where the plug-in correction returns impossible values, Algorithm 1 mostly declines to correct (a, b), and averaging two prompt settings hides a ranking reversal (c). Panels (a) and (b) simulate a true correlation of 0.4 with

N{=}50

units,

K{=}3

replicates per unit and Gaussian noise, using 4,000 simulation runs per point. The correlation and the reliabilities use the same replicates, and Algorithm 1 uses

\alpha{=}0.1

. (a) Share of runs in which the plug-in exceeds 1 in magnitude, and in which Algorithm 1 declines. (b) Mean squared error on a broken log scale. The plug-in floors the product of the two reliabilities at

10^{-9}

. The clipped plug-in cuts its value back to

[-1,1]

, and the fixed floor raises each reliability to at least 0.1 before dividing and clipping. At reliability 0.8 the plug-in (open ring) matches both. (c) ARC-Easy, Qwen2-1.5B against Llama-3.2-3B , with the simultaneous 95% interval of Algorithm 2 in each setting and the pooled comparison’s interval for their average. A gap is the first-named model’s accuracy minus the second-named model’s, in percentage points.

Scenario	Uncorr.	Joint	Boot. floor	Alg. 1	Best floor †	Declined (%)
Base	.12	.31	.22	.12	.14	87
Asym. $.3/.8$	.06	.11	.10	.06	.06	42
$t_{3}$ noise	.12	.30	.21	.12	.13	84
$r^{\star}{=}0$	.02	.28	.19	.03	.08	87
$r^{\star}{=}.8$	.43	.45	.29	.42	.27	86
$N{=}20$	.16	.33	.29	.16	.26	94

Table 3: At low reliability Algorithm 1 stays within 0.01 of the uncorrected estimate, and with 500 units it more than halves the uncorrected error. Entries are the mean squared error (MSE) against the true correlation over 2,000 simulation runs per row, rounded to two decimals. Base row:

N{=}50

units,

K{=}3

replicates, reliability of the 3-replicate mean 0.2 for both measurements, true correlation 0.4, Gaussian noise. Each other row changes one setting. Asym.: reliabilities 0.3 and 0.8.

t_{3}

: Student-

t

noise with 3 degrees of freedom. In every row

r_{\mathrm{obs}}

and the reliabilities use the same replicates. Uncorr.:

\mathrm{clip}(r_{\mathrm{obs}})

. Joint: moment estimator from covariances between different replicates, clipped. Boot. floor: floor chosen by a parametric bootstrap. Alg. 1 :

\alpha{=}0.1

. † Best post-hoc floor: the single floor from a grid of eight (Appendix D.2 ) with the lowest MSE against the truth. It is chosen after seeing the truth, so it is not deployable, and it can exceed the uncorrected MSE. Declined: percentage of runs in which a gate fails. Bold: lowest deployable MSE (ties bold). The plug-in with a

10^{-9}

floor on the product of the reliabilities is omitted (MSE

10^{6}

–

10^{7}

). Monte Carlo s.e. of Alg. 1 cells

\leq 0.005

Figure 4: Algorithm 2 averages over prompt configurations, keeps the protocols apart and reads one verdict per pair from the signs of its simultaneous intervals. (a) For a pair

p=(A,B)

, the correctness difference on each item is averaged over the 16 configurations of a protocol, 4 wordings

\times

4 option orders, giving the per-item gap

d^{p,\pi}_{i}

. Its mean and standard error over items give one interval per pair and protocol. All

2P

intervals on a benchmark share one max-

t

critical value

q_{\alpha}

, which controls the family-wise error asymptotically in the number of items, for the fixed prompt grid. The pooled comparison also averages over the two protocols, so it cannot return protocol-dependent . (b) Four ARC pairs with their simultaneous 95% intervals, drawn to scale, in percentage points. A pair is certified when both intervals exclude zero with the same sign, protocol-dependent when both exclude zero with opposite signs, and refused when zero lies inside one interval or both.

Measured error rates (%)
	ARC	HellaSwag	MMLU	OpenBookQA
models / pairs	15/105	11/55	11/55	11/55
$q_{\alpha}$	3.513	3.336	3.344	3.380
certified	93	42	43	43
protocol-dependent	5	4	3	4
refused	7	9	9	8

Table 4: Most pairs are certified, and the measured error rates of Algorithm 2 are low. Level

\alpha{=}0.05

and 2,365 items per benchmark.

q_{\alpha}

: max-

t

critical value. Held-out wording (order): share of certified directions whose sign flips on the two wordings (orders) left out of calibration. Perm.-null FWER: family-wise error rate under a permutation null, the share of 200 label-permuted benchmarks in which some pair is falsely resolved (Monte Carlo s.e. 1.5–1.8% ). Self-comp.: split-grid self-comparison, a model compared with itself on two disjoint halves of its grid, where any stable winner is an error. The last row applies the same test to the pooled comparison.

Real data, 270 pairs
	Alg. 2	Pooled
pairs with a stable winner	221	254
reversals given a stable winner	0/16	15/16
split-grid self-comparison, false winners	3.6%	18.7%
Simulated benchmarks, 1,000 per scenario
stable rankings: correct verdict / resolved	89.5%	$\approx$ 100%

Table 5: Pooling buys power and hides reversals. Alg. 2: per-protocol comparison. Pooled: the pooled comparison, which averages the two protocols per item before the same max-

t

step. Simulated benchmarks have six models and 2,365 binary items. For stable rankings and near-ties, the Alg. 2 column gives the share of pairs with the correct verdict, and the pooled column the share resolved. –: the pooled comparison cannot return protocol-dependent . Simulated-null FWER: family-wise error rate under the global null, in which all models are equal in both protocols (Monte Carlo s.e. about 0.7%).

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
Measurements and reliability
$i=1,\dots,N$	unit
$s=1,\dots,K$	replicate (seed or configuration); never the protocol
$\hat{\theta}_{i,s}$	measurement $\theta^{\star}_{i}+e_{i,s}$ , target value plus replicate noise
$\tau^{2}$ , $\sigma^{2}$	variance of $\theta^{\star}$ across units; variance of $e$
$\rho_{1}$	single-replicate reliability, $\tau^{2}/(\tau^{2}+\sigma^{2})$

Appendix

Table A.1: Each symbol keeps one meaning throughout the paper. Replicates are indexed by

s

alone, and

e

is the replicate noise.

Analysis	Units	Replicates	Numerator	Reliability estimate	Gate $\alpha$
(1) Per-class demo (Sec. 3.5 , App. E.2 )	about 500 examples per class	2 seeds	correlation of two-seed means	one-way $\mathrm{ICC}(1,2)$ on the same two seeds	.25
(2) Cross-seed correction (Sec. 3.5 , App. E.3 )	50,000 examples	seeds 0–3	EL2N on seed 2 vs. dynamic uncertainty on seed 3	one-way $\mathrm{ICC}(1,1)$ from seeds 0 and 1 (in the $K{=}2$ pivot)	.1
(3) Same-seed comparison (App. E.3 )	50,000 examples	seeds 0–3	two-seed means of seeds 2 and 3	one-way $\mathrm{ICC}(1,2)$ from seeds 0 and 1	.1
(4) Score reliability (Sec. 3.5 , App. E.1 )	50,000 examples	8 seeds	–	crossed two-way $\mathrm{ICC}(C,1)$ ; intervals from one-way $\mathrm{ICC}(1,1)$ over random 4-of-8 seed subsets	–
(5) Simulations (Sec. 3.4 , App. D )	$N$ as stated in each row	$K{=}3$	correlation of $K$ -replicate means	one-way $\mathrm{ICC}(1,K)$ on the same replicates	.1
(6) LLM item difficulty, $g(1)$ and $g(32)$ (Sec. 4.3 , App. F.2 )	2,365 items	32 configurations, both protocols pooled	–	crossed item $\times$ configuration G-theory	–

Appendix

Table A.2: Each correction divides by a reliability for the same number of replicates as its numerator. Numerator: the observed correlation

r_{\mathrm{obs}}

that a correction divides. –: not applicable, because the analysis reports a reliability and corrects nothing.

\mathrm{ICC}(C,1)

: single-seed reliability from the crossed example

\times

seed design, which removes the shift that each seed applies to all examples.

Figure B.1: With shared readout noise the within-run estimator reports spurious anti-persistence, and independent readouts recover the truth (synthetic). (a) Within-run estimator. (b) Cross-replicate estimator with disattenuation. Each line is the mean estimate over the 40 simulation runs of a regime, plotted against the true persistence

\phi

, with one line style per readout noise standard deviation (SD). Bands cover the central 95% of runs. In (a) the bands are narrower than the lines. In (b) the three lines coincide, and the bands widen with the readout noise. Dashed line:

y{=}x

Setting	$N$	$K$	$\alpha$	$q_{\mathrm{gate}}$	$q_{\mathrm{ucb}}$	$q_{\mathrm{gate}}q_{\mathrm{ucb}}$	cap
Table 3	20	3	.1	1.615	1.730	2.793	1.56
Table 3 , base	50	3	.1	1.357	1.392	1.889	2.12
Table 3	100	3	.1	1.243	1.258	1.563	2.77
Table 3	500	3	.1	1.103	1.106	1.220	5.55
per-class	500	2	.25	1.062	1.062	1.128	8.79
cross-seed size	50,000	2	.1	1.012	1.012	1.023	44.1

Appendix

Table C.1: The cap on an accepted correction loosens as the number of units grows.

q_{\mathrm{gate}}

and

q_{\mathrm{ucb}}

are the upper-

\alpha

quantiles of

F_{\nu_{2},\nu_{1}}

and

F_{\nu_{1},\nu_{2}}

, with

\nu_{1}=N(K-1)

and

\nu_{2}=N-1

. Cap:

(1-1/(q_{\mathrm{gate}}q_{\mathrm{ucb}}))^{-1}

, the largest factor by which an accepted correction can multiply

r_{\mathrm{obs}}

when

m=K

(Proposition 3 (a)). Settings: rows of Table 3 ; per-class, close to the per-class analysis of Section 3.5 ; cross-seed size,

N

and

K

of the cross-seed data of Section 3.5 . The cross-seed correction itself has

m=1

, so its cap is larger and follows from the bounds after this table.

Scenario	Clipped	Uncorr.	Joint (bad %)	jPSD	Pos.-part	Bayes	Boot. floor	Alg. 1	Best floor †	Declined (%)
Base	.52	.12	.31 (52)	.56	.27	.47	.22	.12	.14	87
Asym. $.3/.8$	.13	.06	.11 (14)	.14	.10	.12	.10	.06	.06	42
$t_{3}$ noise	.46	.12	.30 (48)	.49	.27	.42	.21	.12	.13	84
$r^{\star}{=}0$	.57	.02	.28 (50)	.61	.26	.54	.19	.03	.08	87
$r^{\star}{=}.8$	.37	.43	.45 (60)	.45	.34	.34	.29	.42	.27	86
$N{=}20$	.69	.16	.33 (68)	.72	.31	.65	.29	.16	.26	94

Appendix

Table D.1: Every deployable correction except Algorithm 1 is at least 0.10 worse than the uncorrected estimate in some row. MSE against the true correlation, with the runs and settings of Table 3 . Clipped: clipped plug-in. Uncorr.: uncorrected estimate. Joint (bad %): moment estimator, with the share of runs in which its unclipped value is undefined or outside

[-1,1]

. jPSD: moment covariance projected by eigenvalue truncation. Pos.-part: positive-part rule. Bayes: fiducial Bayesian shrinkage. Boot. floor: bootstrap floor. † Best post-hoc floor, which cannot be deployed. Declined (%): percentage of runs in which a gate of Algorithm 1 fails. Bold: lowest deployable MSE in the row (ties bold). The largest Monte Carlo s.e. over deployable methods and rows is 0.021, and that of Algorithm 1 cells is at most 0.005.

Cell	$\alpha{=}.05$	$.1$	$.25$	$.4$
$N{=}50$ , $\rho{=}.1$	.150	.153	.160	.212
$N{=}50$ , $\rho{=}.2$	.124	.122	.136	.188
$N{=}50$ , $\rho{=}.5$	.051	.053	.063	.076
$N{=}20$ , $\rho{=}.2$	.156	.159	.169	.234
$N{=}500$ , $\rho{=}.2$	.054	.044	.047	.057

Appendix

Table D.2: The three smaller levels give similar MSE, and

\alpha{=}0.4

is worse in every cell. MSE of Algorithm 1 with

K{=}3

r^{\star}{=}0.4

and 3,000 runs per cell.

\rho

: reliability of the 3-replicate mean, the same for both measurements. Bold: lowest MSE in the row.

$N$	$\rho$	Uncorr.	Alg. 1	No gate	Gate + plug-in	Floor .1
20	.1	.184	.185	.471	.187	.703
20	.2	.157	.158	.347	.162	.575
50	.1	.150	.152	.359	.156	.519
50	.2	.123	.124	.209	.131	.356
50	.5	.060	.055	.055	.075	.090
500	.2	.104	.046	.040	.063	.062

Appendix

Table D.3: The gate matters at low reliability and the upper bound at high reliability. MSE against the truth with

r^{\star}{=}0.4

K{=}3

\alpha{=}0.1

and 4,000 runs per cell.

\rho

: reliability of the 3-replicate mean for both measurements. No gate: always divide by the upper bounds. Gate + plug-in: apply the gate, then divide by the point estimates. Floor .1: clipped plug-in with a fixed floor of 0.1. Bold: lowest deployable MSE in the row, the uncorrected estimate included (ties bold).

Figure D.1: Algorithm 1 stays stable under heavy-tailed and contaminated noise. Each panel shows the MSE (log scale) against the reliability of the 3-replicate mean, for

N{=}50

K{=}3

r^{\star}{=}0.4

\alpha{=}0.1

and 4,000 runs per point. Panels (a)–(c) use Gaussian, Student-

t_{3}

and contaminated noise with equal variance, and compare Algorithm 1 , its bootstrap version, the clipped plug-in and the plug-in with a fixed floor of 0.1. Panel (d) takes Algorithm 1 apart under Gaussian noise, with the clipped plug-in, the upper bound without the gate, and the gate with the point ICC. Proposition 3 covers only Gaussian noise, so the Student-

t_{3}

and contaminated panels are empirical checks.

Method	$\rho{=}.1$	$\rho{=}.4$
Clipped plug-in	.733	.176
Upper bound, no gate	.351	.083
Gate + point ICC	.155	.096
Algorithm 1	.152	.075

Appendix

Table D.4: Under Gaussian noise the gate gives most of the gain at low reliability, and the upper bound adds a further gain at

\rho{=}0.4

. MSE from the sweep of Figure D.1 .

\rho

: reliability of the 3-replicate mean. The same variants in Table D.3 come from a different run. Bold: lowest MSE in the column.

Score	CIFAR-100	CIFAR-10
EL2N	.58 [.55, .60]	.55 [.45, .60]
Dynamic uncertainty	.84 [.83, .85]	.79 [.74, .82]

Appendix

Table E.1: Dynamic uncertainty is more reliable than EL2N on both datasets. Single-seed reliability over eight seeds and the 50,000 training examples of each dataset. Point values: crossed two-way

\mathrm{ICC}(C,1)

, which removes shifts a seed applies to every example. Brackets: 2.5th to 97.5th percentiles of the one-way

\mathrm{ICC}(1,1)

over 300 random draws of 4 of the 8 seeds.

Estimator	Estimate	$X$ / $Y$ seeds	Reliabilities: seeds, form
Observed $r_{\mathrm{obs}}$	.64	2 / 3	–
Alg. 1	.88	2 / 3	0, 1; $\mathrm{ICC}(1,1)$
Joint	.93	all four	–
Plug-in over 12 partitions	mean .93, s.d. .03, range .89–.98	all ordered pairs	the other two; $\mathrm{ICC}(1,1)$
Plug-in over 12 partitions, CIFAR-10	mean .94, s.d. .04	all ordered pairs	the other two; $\mathrm{ICC}(1,1)$
Alg. 1 , same seeds	.94	two-seed means on 2, 3	0, 1; $\mathrm{ICC}(1,2)$

Appendix

Table E.2: The cross-seed estimates agree, but real data have no ground truth. Correlation of EL2N and dynamic uncertainty over the 50,000 CIFAR-100 training examples, except the row marked CIFAR-10. Partitions: the 12 ordered choices of the

X

seed and the

Y

seed among four, with reliabilities from the other two. They share seeds, so their spread is not an error bar. Alg. 1 puts

\mathrm{ICC}(1,1)

into the exact-

F

pivot for

K{=}2

, a conservative bound.

Figure E.1: Score variance rises with the score mean strongly only for CIFAR-10 EL2N. Each panel bins the 50,000 training examples of one dataset by their mean over eight seeds ( $x$ ) and their standard deviation across the seeds ( $y$ ); color shows the count on a log scale. The line joins the median standard deviation within 20 equal-count bins of the mean. Its markers crowd where examples are dense, so $S$ follows the branch that holds most examples. $S$ : Spearman correlation across examples between the variance across seeds and the mean. (a) CIFAR-100 EL2N; (b) CIFAR-100 dynamic uncertainty; (c) CIFAR-10 EL2N; (d) CIFAR-10 dynamic uncertainty.

Model	Family	Size	Benchmarks
Qwen2.5-0.5B-Instruct	Qwen	0.5B	all four
Qwen2-0.5B-Instruct	Qwen	0.5B	all four
Qwen2.5-1.5B-Instruct	Qwen	1.5B	all four
Qwen2-1.5B-Instruct	Qwen	1.5B	all four
Qwen2.5-3B-Instruct	Qwen	3B	ARC
Qwen2.5-7B-Instruct	Qwen	7B	ARC

Appendix

Table F.1: The 15 models span four families and 135M to 7B parameters. Qwen2 and Qwen2.5 count as one family. Benchmarks: the benchmarks whose reported comparison family includes the model. “ARC” marks the four models included only in the ARC comparison family, and “all four” adds HellaSwag, MMLU and OpenBookQA.

							$g(1)$ / $g(32)$
Model	acc	spr	wrd	ord	shot	res	(pooled)
Qwen2.5-0.5B	.610	26	.19	.00	.73	.08	.36/.95
Qwen2-0.5B	.613	8	.71	.01	.04	.24	.48/.97
Qwen2.5-1.5B	.886	4	.24	.03	.43	.30	.58/.98
Qwen2-1.5B	.845	2	.27	.21	.16	.36	.65/.98
Qwen2.5-3B	.927	1.5	.34	.13	.05	.48	.59/.98

Appendix

Table F.2: For every model, the reliability of item difficulty under one configuration,

g(1)

, is at most 0.65, and the 32-configuration mean reaches

g(32)\geq 0.85

for every model above chance. ARC-Easy, 2,365 items, 32 configurations. acc: mean accuracy over the 32 configurations. spr: range of accuracy across them, in points. wrd, ord, shot, res: shares of the variance of configuration-level accuracy due to wording, option order, shot count and the residual (they sum to 1).

g(1)

g(32)

: reliability of item difficulty for one configuration and for the 32-configuration mean, pooled over both protocols. SmolLM2-135M and SmolLM2-360M sit near the 0.25 chance floor. The Llama-3.2 spreads come from the protocol, because the fixed two-shot block costs them 21–29 points.

Figure F.1: One prompt configuration measures item difficulty poorly for every model, and the 32-configuration mean (both protocols pooled) is reliable whenever the model has item signal. (a) Shares of the variance of configuration-level ARC-Easy accuracy due to wording, option order, shot count and the residual, for the 15 models. (b)

g(1)

(open circle) and

g(32)

(filled circle) on ARC-Easy, joined by a line. Small grey circles give the same two coefficients on HellaSwag, MMLU and OpenBookQA for the 11 models compared there.

		Wording
Model	Condition	1	2	3	4
Llama-3.2-3B	zero-shot	.883	.881	.883	.882
	two-shot, inline	.782	.752	.775	.366
	two-shot, chat	.895	.895	.893	.895
Llama-3.2-1B	zero-shot	.489	.679	.693	.671
	two-shot, inline	.268	.322	.405	.343

Appendix

Table F.3: Formatting the two examples as chat turns removes the two-shot drop. ARC-Easy accuracy on the 2,365 questions at the identity option order, for each of the four instruction wordings. Inline: the two solved examples written into the prompt before the question, the protocol used throughout the paper. Chat: the same examples given as earlier chat turns.

Figure F.2: Formatting the two examples as chat turns removes the two-shot drop of Llama-3.2-3B . ARC-Easy accuracy at the identity option order under zero-shot, two-shot with the examples written inline (the protocol used throughout), and two-shot with the same examples given as earlier chat turns. Markers show the mean over the four wordings, and bars show their range. The labeled point (0.366) is Llama-3.2-3B with inline examples under wording 4.

Figure G.1: Most pairs agree in direction across prompt settings; 16 are significant with opposite signs. One panel per benchmark. Each point is a model pair, with its zero-shot gap on the horizontal axis and its two-shot gap on the vertical axis, and each pair is ordered so that its zero-shot gap is at least zero. A gap is the first-named model’s accuracy minus the second-named model’s, in percentage points. Filled circles: certified ; triangles: protocol-dependent ; open circles: refused . Thin lines: simultaneous 95% intervals of Algorithm 2 in each protocol. The shaded region marks opposite signs in the two settings, and the diagonal marks equal gaps.

Figure G.2: All but one of the 93 certified ARC directions keep their sign on four wordings written after all verdicts were fixed, a measured rate outside the guarantee. Each point is one ARC pair in one protocol. The horizontal axis gives its gap on the original prompt grid, and the vertical axis its mean gap over the four new wordings at the identity option order. A gap is the first-named model’s accuracy minus the second-named model’s, in percentage points. Squares: certified pairs; triangles: the five protocol-dependent pairs. Open markers: zero-shot; filled markers: two-shot. The highlighted pair, Qwen2.5-0.5B vs. Llama-3.2-1B , is the only certified direction that fails. Points in the shaded quadrants changed direction, and the diagonal marks equal gaps.

Figure G.3: False winners in split-grid self-comparisons grow with a model’s configuration spread, and pooling multiplies them. Each point is one model on one benchmark. The horizontal axis is the model’s configuration spread, the range of its accuracy across the 32 configurations in points (log scale). The vertical axis is the share of 200 random splits of its grid into two halves in which a method names the model a stable winner over itself, which is always an error. Thin vertical lines join the two methods’ rates for the same model. Colors distinguish Algorithm 2 from the pooled comparison, and marker shapes distinguish the benchmarks.

Figure G.4: Pooling the zero-shot and two-shot protocols names a stable winner for 15 of the 16 reversed pairs. Each row is one pair, and rows are grouped by benchmark and sorted by the zero-shot gap. A gap is the first-named model’s accuracy minus the second-named model’s, in percentage points. Open and filled squares: simultaneous 95% intervals of Algorithm 2 at zero-shot and two-shot. Diamonds: the pooled comparison’s interval for the average of the two protocols.

	$\alpha{=}.01$	$\alpha{=}.05$	$\alpha{=}.1$
ARC	92/4/9	93/5/7	93/5/7
HellaSwag	42/4/9	42/4/9	42/5/8
MMLU	43/3/9	43/3/9	43/3/9
OpenBookQA	43/4/8	43/4/8	45/4/6

Appendix

Table G.1: Verdict counts change by at most two pairs between

\alpha=0.01

and

\alpha=0.1

. Each entry gives the number of certified / protocol-dependent / refused pairs. ARC has 105 pairs among 15 models, and the other benchmarks have 55 pairs among 11 models. Table 4 reports

\alpha=0.05

Cross-seed measure	CIFAR-100	CIFAR-10
per-dimension, raw	.003	.005
per-dimension, after held-out Procrustes	.813	.902
pairwise distances	.823	–
agreement: linear CKA	.852	.924

Appendix

Table H.1: Seeds disagree on the coordinates of the features and agree on the representation. Penultimate features of three ResNet-18 seeds, evaluated on the half of the test images not used to fit the alignment. Rows above the rule are reliabilities. Raw: before alignment. CKA is a similarity index, shown as agreement. –: not computed.

Result	Command	Data in the repository
Tab. 3 , Tab. D.1 : simulation MSE	erel disatt2	–
Tab. D.3 : gate ablation	erel ablate	–
Fig. 3 (a, b), Tab. D.4 , Fig. D.1 : robustness sweep	scripts/disatt_robustness.py --repetitions 4000 --bootstrap-repetitions 500 --seed 20270720 ; figures: fig1_teaser.py , figD1_robustness.py	results/disatt_robustness_full_4000x500_seed20270720/
Prop. 2 : simulation check (App. D.7 )	scripts/prop2_check.py	–
Fieller interval and REML closed form (App. D.7 )	scripts/interval_compare.py	–
Persistence study (App. B , Fig. B.1 )	erel synth data/synth.csv ; figure: figB1_persistence.py	data/synth.csv

Appendix

Table I.1: Most results run from the released files, and two need regenerated CIFAR-100 inputs. Each row gives a result, the command that produces it, and the data it reads or writes. Every Python command is prefixed with uv run , and a bare script path stands for uv run python followed by that path. erel stands for cargo run --release -p erel-cli -- . Flags and seeds not shown here are in REPRODUCIBILITY.md . Figure scripts are in figscripts/ , and the schematic sources in figscripts/schematics/ . –: the command generates its own data. Stored output: the file is released, but no released script writes it.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,427 papers

What Single-Prompt Accuracy Misses: A Multi-Variant Reliability Audit of Language Models
May 3, 2026Ranit Karmakar, Jayita ChatterjeeLLM Evaluation Language Model Calibration
Harvard University, Boston, MA 02115 · Boston, MA 02134
Brittlebench: Quantifying LLM robustness via prompt sensitivity
Feb 27, 2026Angelika Romanou, Mark Ibrahim, Candace Ross +8Prompt Sensitivity LLM Evaluation
FAIR at Meta · EPFL · Northeastern University
When Calibration Rankings Reverse: Accuracy-Controlled Evaluation for Fair Comparison of LLMs
Jun 29, 2026Zhichao Yang, Caiqi Zhang, Ruihan Yang +3LLM Evaluation Confidence Estimation in Language Models
Fudan University · University of Cambridge

May 3, 2026J/K move · Enter open · S save

What Single-Prompt Accuracy Misses: A Multi-Variant Reliability Audit of Language Models

Ranit Karmakar, Jayita Chatterjee

Harvard University, Boston, MA 02115 · Boston, MA 02134

LLM Evaluation Language Model Calibration LLM Reliability Benchmark Validity Language Model Robustness Prompt Robustness

Open paper AskPDF arXiv

Single-prompt accuracy is the dominant way to benchmark language models, but it can miss reliability failures that matter. We evaluate a 15-model open-weight corpus, with the main reliability analyses focused on 10 instruct models across five classification and reasoning benchmarks under five prompt variants each, measuring accuracy, token-probability calibration, verbal-confidence calibration, verbal parse rate, and prompt-perturbation spread for every (model x dataset x variant) cell. We find three broad results. First, evaluation design can materially change the conclusion. Switching Expected Calibration Error (ECE) token from a raw to a label-set-normalised definition changes per-cell calibration by a mean absolute 0.149. More strikingly, pairing a chain-of-thought prompt with a first-character evaluator on ARC-Challenge reduces apparent accuracy by 72-88% across all five primary models; two independent repair procedures recover 93.8% and 102.7% of the lost performance, indicating an evaluator-side rather than model-side failure. Second, confidence signals are fragile. On MMLU-Pro, every primary model verbally reports confidence substantially above both its accuracy and its token-probability confidence on the same rows, and verbal parse rate can collapse for a single model on a single prompt variant. Third, prompt robustness does not track parameter count reliably. Across 10 instruct models, the correlation between model size and prompt-perturbation spread ranges from -0.244 to 0.474 across benchmarks. Taken together, these results show that reliability conclusions for small language models depend not only on the model being evaluated, but also on the evaluation pipeline used to measure it. We argue that calibration definitions, evaluator logic, verbal parseability, and prompt robustness should be reported explicitly when making reliability claims.

Abstract

LLM Evaluation

Explore similar work

What Single-Prompt Accuracy Misses: A Multi-Variant Reliability Audit of Language Models

Brittlebench: Quantifying LLM robustness via prompt sensitivity

When Calibration Rankings Reverse: Accuracy-Controlled Evaluation for Fair Comparison of LLMs