Equivalence testing and run-count design for stochastic machine-learning experiments
Authors: Dmytro Knopov, Nadiya Shvai
Organizations: Faculty of Computer Sciences, Department of Mathematics, National University of Kyiv-Mohyla Academy, Kyiv, Ukraine. · Cyclope.ai, Paris, France.
Claims that two training methods perform equally well often rest on a non-significant difference between two or three runs per method. We develop a protocol, with its power theory, for certifying practical equivalence in per-run metrics such as the AUROC (area under the receiver operating characteristic curve). It rests on a two-level (seed x item) variance model separating between-run dispersion from item-level noise. Equivalence within a pre-specified margin is tested by two one-sided tests (TOST), equivalently by a confidence interval inside the margin; a two-level bootstrap covers composite statistics. A closed-form formula converts a margin and a target power into runs per method; its shifted-t refinement is calibrated against the realised two-level process. Simulations on a 270-cell grid and under non-Gaussian seed laws confirm the formula within its design-relevant range, from five runs per method. For four published runs of a joint energy-based model, the failure to certify equivalence was largely decided by design: with the variance calibrated to those runs, two per method certify with probability near three per cent, whereas certifying 0.01 AUROC at 80 per cent power needs twelve even under exact equality. A second case study on a thousand-seed CIFAR-10 zoo and the MultiBERTs checks the formula against real seed laws (within 0.05 in that range) and resolves comparisons that two or three runs leave undecided. In a two-stage design (pilot, then formula), our experiments suggest that an upper confidence bound on the pilot variance substantially reduces the undersizing seen with its point estimate.
Figures & tables
Figure 1: Runs per method required for 80% TOST power at α=0.05 and true Δ=0 , as a function of the equivalence margin δ (logarithmic vertical axis). The three thin curves with open markers show the design formula ( 10 ) (shifted- t version) for three seed-level dispersions with the image-level term of the simulation study at θ=0.75 ( m0=m1=104 ): solid line with circles for σseed=0.002 , dashed line with squares for σseed=0.005 , dotted line with triangles for σseed=0.01 ; the thick solid curve is the case study of Section 6 ( s=0.0080 , image-level term absorbed into s2 ). The star marks δ=0.01 , where n∗=12 . The hatched band marks the region below five runs per method, in which we do not rely on the formula; values falling there are raised to five (Remark 3 ). The shaded band is the approximate 95% interval for the case-study curve induced by the scaled- χ2 (Satterthwaite) uncertainty of the plug-in s2 ( ν^≈1.2 , Section 6 ); at δ=0.01 it spans 4 to 3542 runs per method
Figure 2: Empirical probability that TOST declares equivalence (markers, ±3 Monte-Carlo standard errors, 2000 replications per point) against the shifted- t approximation ( 11 ) (lines), at θ=0.75 and margin δ=0.01 . Panels (a) and (b) show the power at true differences Δ=0 and Δ=0.005 ; panels (c)–(e) show the realised size at the boundary Δ=δ for the three seed-level dispersions σseed=0.002 , 0.005 and 0.01 , one panel each, on an enlarged vertical scale ( 0 to 0.1 ). Line style and marker encode σseed as in Fig. 1 . The horizontal axes are logarithmic, and in panels (a) and (b) the markers of the three series are offset slightly so that their error bars stay apart. The approximation is accurate where the power is practically useful, conservative in the near-zero-power regime, most visibly at n=2 , and misses in both directions at n=3 (Remark 3 ). The dash-dotted lines mark α=0.05
Figure 3: Coverage of the true difference Δ=0 by nominal 95% intervals on a shared test set ( θ=0.6 , m0=m1=2000 , ρ=0.8 , B=400 , 1000 replications per point, ±3 Monte-Carlo standard errors) as a function of the number of runs: (a) the two-level bootstrap of Section 3.3 against (b) the naive per-image bootstrap that ignores the seed level; panel (a) enlarges the band shaded in panel (b), the dashed horizontal line marks the nominal level 0.95 , the horizontal axis is logarithmic, and the markers are offset slightly so that their error bars stay apart. Line style and marker encode σseed as in Fig. 1 . The naive interval is far below nominal whenever the seed level matters and does not improve with more runs; the two-level interval departs from nominal only in the regimes of Remark 2
Run
SVHN
CIFAR-100
DTD
LSUN-R
iSUN
macro-5
SGLD, seed 42
0.5676
0.6396
0.7250
0.7805
0.7863
0.6998
SGLD, seed 123
0.6056
0.6244
0.7349
0.7688
0.7899
0.7047
PC, seed 42
0.6045
0.6418
0.7374
0.7511
0.7668
0.7003
PC, seed 123
0.6237
0.6391
0.7549
0.7737
0.7859
0.7155
Table 1: Per-run AUROCs of the case study: two training methods, two seeds each, evaluated at the checkpoints prescribed by the source study, five OOD test sets and their macro-average. Recomputed from the raw per-item scores of Knopov ((2026))
n∗ ( 80% )
n∗ ( 90% )
δ
formula
exact
formula
exact
0.005
45
45
56
56
0.0075
21
21
26
26
0.01
12
12
15
15
0.015
6
6
8
7
0.02
4
4
5
5
Table 2: Runs per method n∗ at α=0.05 and true Δ=0 with the case-study plug-in s=0.0080 , from the shifted- t design formula of Corollary 5 (formula) and from the exact pooled-model power of Shieh ((2016)) inverted for n (exact). Entries below n=5 lie outside the simulation-validated range of the formula (Remark 3 ) and should be read as “at least five”
Collection
R
m
mean
s
skew
kurt
vˉ/s2
ρ^
always
n∗
CIFAR-10 zoo
1000
10 000
0.487
0.0136
−1.80
5.40
0.14
0.57
0.04
33
MultiBERTs, RTE
25
277
0.628
0.0609
−0.83
0.24
0.22
0.39
0.00
636
MultiBERTs, SST-2
25
872
0.920
0.0049
−0.43
0.14
3.59
0.70
0.82
5
MultiBERTs, MNLI
25
9 815
0.843
0.0021
0.38
−0.33
2.94
0.66
0.64
3
Table 3: Two-level structure of four single-configuration collections: runs R , test items m , mean accuracy, seed-level standard deviation s , skewness and excess kurtosis of the per-run accuracies, ratio of the mean binomial item-level variance vˉ to s2 , between-run correlation of per-item correctness ρ^ , share of items classified correctly by every run (column always ), and runs per method n∗ for 80% power at δ=0.01 from the design formula with the plug-in s (shared test set); values of n∗ below 5 are to be read as “at least five” (Remark 3 )
Figure 4: The design formula against a real seed law: (a) probability that the seed-level TOST declares equivalence at true Δ=0 and (b) realised size at the boundaries Δ=±δ when n runs per method are drawn without replacement from the 1000 -seed CIFAR-10 zoo, at δ=0.01 ( n∗=33 , solid line with diamonds) and δ=0.0173 ( n∗=12 , dashed line with pentagons), and from the 25 SST-2 runs of MultiBERTs at δ=0.01 ( n∗=5 , dotted line with inverted triangles). Markers are realised rates with ±3 Monte-Carlo standard errors (in panel (b) the mean of the two boundary rates), and lines in panel (a) show the shifted- t formula ( 11 ) with the plug-in s of each collection; the thin dash-dotted lines mark the target power 0.8 in panel (a) and the level α=0.05 in panel (b), the horizontal axis is logarithmic, and the markers are offset slightly so that their error bars stay apart
TOST verdict
certification rate
Collection
Comparison
n
Δ^
90% interval
0.01
0.02
n∗
n=2
n=3
n=5
zoo
Adam vs SGD
10
−0.0002
[−0.0080,+0.0077]
yes
yes
14
0.04
0.03
0.07
zoo
GELU vs tanh
10
+0.0012
[−0.0056,+0.0081]
yes
yes
10
0.02
0.07
0.09
RTE
lr
25
+0.0983
[+0.0737,+0.1229]
no
no
397
0.00
0.00
0.00
SST-2
lr
25
+0.0084
[+0.0067,+0.0101]
no
yes
6
0.04
0.04
0.08
MNLI
lr
25
+0.0040
[+0.0028,+0.0052]
yes
yes
3
0.28
0.67
0.99
Table 4: Eight two-method comparisons under the protocol, all matched by seed and analysed with the paired seed-level TOST on the within-seed differences: matched pairs n , estimated difference Δ^ (first method minus second), paired 90% interval, TOST verdict at δ=0.01 and δ=0.02 , runs per method n∗ for 80% power at δ=0.01 in a fresh experiment with independent arms, from the pooled s of the two arms and computed under the counterfactual Δ=0 (the requirement grows without bound as the anticipated ∣Δ∣ approaches δ , Section 4 ), and the share of the subsets of 2 , 3 or 5 matched pairs in which the paired TOST at δ=0.01 declares equivalence (exact enumeration over all (n10) or (n25) subsets; a subset whose differences coincide counts as a non-certification; not computed for the five-pair comparisons); the MultiBERTs rows use learning rates 2⋅10−5 against 5⋅10−5 at two million pretraining steps (rows lr ) and one against two million pretraining steps at 2⋅10−5 (rows steps ); values of n∗ below 5 are to be read as “at least five” (Remark 3 )
Statistical hypothesis testing develops methods for assessing evidence against a stated null while controlling false-positive and false-negative rates.