Equivalence testing and run-count design for stochastic machine-learning experiments

Dmytro Knopov; Nadiya Shvai

cs.LGOct 16, 2026

Equivalence testing and run-count design for stochastic machine-learning experiments

Authors: Dmytro Knopov, Nadiya Shvai

Organizations: Faculty of Computer Sciences, Department of Mathematics, National University of Kyiv-Mohyla Academy, Kyiv, Ukraine. · Cyclope.ai, Paris, France.

Abstract

Claims that two training methods perform equally well often rest on a non-significant difference between two or three runs per method. We develop a protocol, with its power theory, for certifying practical equivalence in per-run metrics such as the AUROC (area under the receiver operating characteristic curve). It rests on a two-level (seed x item) variance model separating between-run dispersion from item-level noise. Equivalence within a pre-specified margin is tested by two one-sided tests (TOST), equivalently by a confidence interval inside the margin; a two-level bootstrap covers composite statistics. A closed-form formula converts a margin and a target power into runs per method; its shifted-t refinement is calibrated against the realised two-level process. Simulations on a 270-cell grid and under non-Gaussian seed laws confirm the formula within its design-relevant range, from five runs per method. For four published runs of a joint energy-based model, the failure to certify equivalence was largely decided by design: with the variance calibrated to those runs, two per method certify with probability near three per cent, whereas certifying 0.01 AUROC at 80 per cent power needs twelve even under exact equality. A second case study on a thousand-seed CIFAR-10 zoo and the MultiBERTs checks the formula against real seed laws (within 0.05 in that range) and resolves comparisons that two or three runs leave undecided. In a two-stage design (pilot, then formula), our experiments suggest that an upper confidence bound on the pilot variance substantially reduces the undersizing seen with its point estimate.

Figures & tables

Figure 1: Runs per method required for

80\%

TOST power at

\alpha=0.05

and true

\Delta=0

, as a function of the equivalence margin

\delta

(logarithmic vertical axis). The three thin curves with open markers show the design formula ( 10 ) (shifted-

t

version) for three seed-level dispersions with the image-level term of the simulation study at

\theta=0.75

(

m_{0}=m_{1}=10^{4}

): solid line with circles for

\sigma_{\mathrm{seed}}=0.002

, dashed line with squares for

\sigma_{\mathrm{seed}}=0.005

, dotted line with triangles for

\sigma_{\mathrm{seed}}=0.01

; the thick solid curve is the case study of Section 6 (

s=0.0080

, image-level term absorbed into

s^{2}

). The star marks

\delta=0.01

, where

n^{*}=12

. The hatched band marks the region below five runs per method, in which we do not rely on the formula; values falling there are raised to five (Remark 3 ). The shaded band is the approximate

95\%

interval for the case-study curve induced by the scaled-

\chi^{2}

(Satterthwaite) uncertainty of the plug-in

s^{2}

(

\hat{\nu}\approx 1.2

, Section 6 ); at

\delta=0.01

it spans

4

3542

runs per method

Figure 2: Empirical probability that TOST declares equivalence (markers,

\pm 3

Monte-Carlo standard errors,

2000

replications per point) against the shifted-

t

approximation ( 11 ) (lines), at

\theta=0.75

and margin

\delta=0.01

. Panels (a) and (b) show the power at true differences

\Delta=0

and

\Delta=0.005

; panels (c)–(e) show the realised size at the boundary

\Delta=\delta

for the three seed-level dispersions

\sigma_{\mathrm{seed}}=0.002

0.005

and

0.01

, one panel each, on an enlarged vertical scale (

0

0.1

). Line style and marker encode

\sigma_{\mathrm{seed}}

as in Fig. 1 . The horizontal axes are logarithmic, and in panels (a) and (b) the markers of the three series are offset slightly so that their error bars stay apart. The approximation is accurate where the power is practically useful, conservative in the near-zero-power regime, most visibly at

n=2

, and misses in both directions at

n=3

(Remark 3 ). The dash-dotted lines mark

\alpha=0.05

Figure 3: Coverage of the true difference

\Delta=0

by nominal

95\%

intervals on a shared test set (

\theta=0.6

m_{0}=m_{1}=2000

\rho=0.8

B=400

1000

replications per point,

\pm 3

Monte-Carlo standard errors) as a function of the number of runs: (a) the two-level bootstrap of Section 3.3 against (b) the naive per-image bootstrap that ignores the seed level; panel (a) enlarges the band shaded in panel (b), the dashed horizontal line marks the nominal level

0.95

, the horizontal axis is logarithmic, and the markers are offset slightly so that their error bars stay apart. Line style and marker encode

\sigma_{\mathrm{seed}}

as in Fig. 1 . The naive interval is far below nominal whenever the seed level matters and does not improve with more runs; the two-level interval departs from nominal only in the regimes of Remark 2

Run	SVHN	CIFAR-100	DTD	LSUN-R	iSUN	macro-5
SGLD, seed 42	$0.5676$	$0.6396$	$0.7250$	$0.7805$	$0.7863$	$0.6998$
SGLD, seed 123	$0.6056$	$0.6244$	$0.7349$	$0.7688$	$0.7899$	$0.7047$
PC, seed 42	$0.6045$	$0.6418$	$0.7374$	$0.7511$	$0.7668$	$0.7003$
PC, seed 123	$0.6237$	$0.6391$	$0.7549$	$0.7737$	$0.7859$	$0.7155$

Table 1: Per-run AUROCs of the case study: two training methods, two seeds each, evaluated at the checkpoints prescribed by the source study, five OOD test sets and their macro-average. Recomputed from the raw per-item scores of Knopov ((2026))

	$n^{*}$ ( $80\%$ )		$n^{*}$ ( $90\%$ )
$\delta$	formula	exact	formula	exact
$0.005$	$45$	$45$	$56$	$56$
$0.0075$	$21$	$21$	$26$	$26$
$0.01$	$12$	$12$	$15$	$15$
$0.015$	$6$	$6$	$8$	$7$
$0.02$	$4$	$4$	$5$	$5$

Table 2: Runs per method

n^{*}

\alpha=0.05

and true

\Delta=0

with the case-study plug-in

s=0.0080

, from the shifted-

t

design formula of Corollary 5 (formula) and from the exact pooled-model power of Shieh ((2016)) inverted for

n

(exact). Entries below

n=5

lie outside the simulation-validated range of the formula (Remark 3 ) and should be read as “at least five”

Collection	$R$	$m$	mean	$s$	skew	kurt	$\bar{v}/s^{2}$	$\hat{\rho}$	always	$n^{*}$
CIFAR-10 zoo	1000	10 000	0.487	0.0136	$-1.80$	$5.40$	0.14	0.57	0.04	33
MultiBERTs, RTE	25	277	0.628	0.0609	$-0.83$	$0.24$	0.22	0.39	0.00	636
MultiBERTs, SST-2	25	872	0.920	0.0049	$-0.43$	$0.14$	3.59	0.70	0.82	5
MultiBERTs, MNLI	25	9 815	0.843	0.0021	$0.38$	$-0.33$	2.94	0.66	0.64	3

Table 3: Two-level structure of four single-configuration collections: runs

R

, test items

m

, mean accuracy, seed-level standard deviation

s

, skewness and excess kurtosis of the per-run accuracies, ratio of the mean binomial item-level variance

\bar{v}

s^{2}

, between-run correlation of per-item correctness

\hat{\rho}

, share of items classified correctly by every run (column always ), and runs per method

n^{*}

for

80\%

power at

\delta=0.01

from the design formula with the plug-in

s

(shared test set); values of

n^{*}

below

5

are to be read as “at least five” (Remark 3 )

Figure 4: The design formula against a real seed law: (a) probability that the seed-level TOST declares equivalence at true

\Delta=0

and (b) realised size at the boundaries

\Delta=\pm\delta

when

n

runs per method are drawn without replacement from the

1000

-seed CIFAR-10 zoo, at

\delta=0.01

(

n^{*}=33

, solid line with diamonds) and

\delta=0.0173

(

n^{*}=12

, dashed line with pentagons), and from the

25

SST-2 runs of MultiBERTs at

\delta=0.01

(

n^{*}=5

, dotted line with inverted triangles). Markers are realised rates with

\pm 3

Monte-Carlo standard errors (in panel (b) the mean of the two boundary rates), and lines in panel (a) show the shifted-

t

formula ( 11 ) with the plug-in

s

of each collection; the thin dash-dotted lines mark the target power

0.8

in panel (a) and the level

\alpha=0.05

in panel (b), the horizontal axis is logarithmic, and the markers are offset slightly so that their error bars stay apart

Collection	Comparison	$n$	$\hat{\Delta}$	$90\%$ interval	$0.01$	$0.02$	$n^{*}$	$n=2$	$n=3$	$n=5$
					TOST verdict			certification rate
zoo	Adam vs SGD	10	$-0.0002$	$[-0.0080,+0.0077]$	yes	yes	14	0.04	0.03	0.07
zoo	GELU vs tanh	10	$+0.0012$	$[-0.0056,+0.0081]$	yes	yes	10	0.02	0.07	0.09
RTE	lr	25	$+0.0983$	$[+0.0737,+0.1229]$	no	no	397	0.00	0.00	0.00
SST-2	lr	25	$+0.0084$	$[+0.0067,+0.0101]$	no	yes	6	0.04	0.04	0.08
MNLI	lr	25	$+0.0040$	$[+0.0028,+0.0052]$	yes	yes	3	0.28	0.67	0.99

Table 4: Eight two-method comparisons under the protocol, all matched by seed and analysed with the paired seed-level TOST on the within-seed differences: matched pairs

n

, estimated difference

\hat{\Delta}

(first method minus second), paired

90\%

interval, TOST verdict at

\delta=0.01

and

\delta=0.02

, runs per method

n^{*}

for

80\%

power at

\delta=0.01

in a fresh experiment with independent arms, from the pooled

s

of the two arms and computed under the counterfactual

\Delta=0

(the requirement grows without bound as the anticipated

|\Delta|

approaches

\delta

, Section 4 ), and the share of the subsets of

2

3

5

matched pairs in which the paired TOST at

\delta=0.01

declares equivalence (exact enumeration over all

\binom{10}{n}

\binom{25}{n}

subsets; a subset whose differences coincide counts as a non-certification; not computed for the five-pair comparisons); the MultiBERTs rows use learning rates

2\cdot 10^{-5}

against

5\cdot 10^{-5}

at two million pretraining steps (rows lr ) and one against two million pretraining steps at

2\cdot 10^{-5}

(rows steps ); values of

n^{*}

below

5

are to be read as “at least five” (Remark 3 )

Statistical Hypothesis Testing

Statistical hypothesis testing develops methods for assessing evidence against a stated null while controlling false-positive and false-negative rates.

See all 101 papers

May 27, 2026

Learning to Assess the Reliability of Number-of-Runs Estimation in Stochastic Optimization

In large-scale benchmarking of stochastic optimization algorithms, the key challenge is no longer whether repeated runs are needed for reliability, but how to determine when sufficient evidence has been collected without incurring unnecessary computational cost. We study a learning-based extension of a recent empirical online heuristic that adaptively estimates the required number of runs using outlier handling and skewness-based symmetry checks. Using annotated outcomes from 132{,}000 Nevergrad runs on COCO (24 problems in 20 dimensions, 10 instances each, 11 optimizers), we train classifiers on 23 statistical, energy-free, and shape and stability features to predict whether a run-number estimate is reliable, prioritizing detection of incorrect estimates via minority-class recall. We evaluate reliability prediction using a within-configuration learning setup, where models are trained and tested on data sharing the same optimizer. The results show that run-number reliability can be learned in a within-configuration scenario, enabling detection of unreliable estimates with high minority-class recall, although performance remains limited by the restricted data diversity within fixed configurations.

Sara Gjorgjieva, Eva Tuba, Tome Eftimov

Jožef Stefan Institute, Ljubljana, Slovenia

Stochastic Optimization Black-Box Optimization

Ask PDF

Aug 15, 2026

Certifying Compressed Language Models: An Audit and a Statistical Toolkit

A fraction of a point of benchmark accuracy is the usual evidence that a compressed model is equivalent to its original. That quantity is least informative when two models are most alike: a net delta is what survives cancellation between opposing per-item changes, and cancellation is most complete in the regime equivalence claims occupy. Across an atlas of 1,707 paired model-by-task cells mined from public per-item evaluation dumps (1.3B-405B), churn runs roughly five times the net accuracy delta, and cells scoring identically to their baseline still disagree on individual items. In a preregistered audit of 17 equivalence claims from three registered frames (method papers, model cards, vendor documentation), 16 are eligible. None states a prospective numerical equivalence margin, and none releases task-matched per-item outputs, though 3 release outputs for other tasks only; 5 report too little to assess numerically, so a reader cannot check them at any sample size. We audit evidential sufficiency, not truth: no claim is called false. We supply the missing instrument: paired equivalence testing at a declared margin, with certification tables giving the items an evaluation needs, computed from disagreement observed under compression, not from independent-binomial variance. A controlled experiment pairs GPTQ and AWQ on byte-identical calibration samples across five seeds. Under the frozen eight-cell decision rule H3 is supported: changing the calibration draw was sufficient to reverse the observed method ordering in 5 of 8 confirmatory cells. The reporting standard we propose is five lines: declare a margin, run the paired test, report churn beside net delta, cite the sample size you met, release per-item outputs. It applies to any comparison between two models alike enough to be worth comparing. All per-item outputs, protocols and code are released.

Amogh Singh

Georgia Institute of Technology

LLM Evaluation LLM Auditing LLM Compression Statistical Hypothesis Testing

Ask PDF

Jul 5, 2026

evalci: A Python Library for Statistically Rigorous Comparison of Language Model Evaluations

The dominant practice in language model evaluation is to report a single accuracy number per model and declare the higher one better, without testing whether the gap could plausibly be sampling noise. On benchmarks of a few thousand items, and under temperature sampling where a model can differ from itself run to run by more than the reported gap between models, this practice routinely overstates confidence in headline claims. The statistical machinery to fix this -- confidence intervals, paired significance tests, power analysis, clustered standard errors, multiple-comparison correction -- is well established, but no standard, pip-installable tool packages it in the shape an evaluation actually takes: a per-item results table. We present evalci, a pure-Python library (numpy/scipy/pandas only) that turns a per-item results table into a publication-ready claim -- e.g., "Model A beats Model B,

Δ=3.1

pts, 95% CI [1.2, 5.0], paired permutation

p=0.002

n=1{,}319

" -- in one function call, with adapters for lm-evaluation-harness and HELM output. Every routine is validated against an independent reference (statsmodels, or brute-force exact enumeration) rather than only against itself. As a case study, we re-analyze a public comparison of nine language models' MMLU accuracy and find that 3 of the 8 adjacent leaderboard-rank gaps are not statistically significant after correcting for the 36 pairwise comparisons the ranking implies. evalci is available at https://pypi.org/project/evalci/ (source: https://github.com/Shreyaskc/evalci, DOI: https://doi.org/10.5281/zenodo.21201815)

Shreyas K Chandrahas

LLM Evaluation Pairwise Comparison Statistical Power Analysis Statistical Hypothesis Testing

Ask PDF

Abstract

Statistical Hypothesis Testing

Explore similar work

Learning to Assess the Reliability of Number-of-Runs Estimation in Stochastic Optimization

Certifying Compressed Language Models: An Audit and a Statistical Toolkit

evalci: A Python Library for Statistically Rigorous Comparison of Language Model Evaluations