Phantom Gains: Auditing Self-Improvement Against a Measured Null

Cheng Xu; Nan Yan; Liming Chen; M-Tahar Kechadi

cs.AIAug 20, 2026

Phantom Gains: Auditing Self-Improvement Against a Measured Null

Authors: Cheng Xu, Nan Yan, Liming Chen, M-Tahar Kechadi

Organizations: University College Dublin · Georgia Institute of Technology · Dalian University of Technology

Abstract

Whether a language model has improved itself is increasingly judged not by mean accuracy but by which individual problems it gains and loses. Tracking these transitions means differencing two noisy estimates, leaving them vulnerable to measurement artifacts. Auditing three rounds of rank- $32$ LoRA self-training on Qwen3-8B against a frozen control pushed through the identical pipeline, we identify seven measurement failures, each of which inverts a reported finding when its control is absent. Several are standard practice. A ledger built on a single greedy decode manufactures capability changes on an untrained model, largely an artifact of inference batching; the expansion statistic separating acquisition from sharpening assigns that same model a rate of $0.280$ . The natural threshold repair does not survive replication: estimated across the frozen comparisons such a design already contains, its null stays non-zero. We replace it with a per-problem exact test against a pooled baseline under false-discovery-rate control, which detects nothing on any held-out replicate and is unchanged under the multiple-testing rule, error rate and pool size. Applied to a ladder of arms matched in stream, volume and evaluation, the audit finds that external distillation improves problems the base model rarely reaches while three forms of self-training do not; a regression rejects this asymmetry as a by-product of distillation's larger overall gain ( $p < 10^{-8}$ ). On the far smaller set of problems the base model never reaches, the evidence is inconclusive, while self-training corrupts problems solved at baseline at rates well above the measured floor. Transition-level auditing therefore requires a separately measured null for every statistic it reports: nulls that cost no new experiments, built from baseline replicates a multi-arm study already owns, though not from as few as most possess.

Figures & tables

Panel of Figure 1 — Figure 1: The audit pipeline, and the seven stages at which a measurement failure enters it. Top: an evolution arm draws candidates from an unlabeled stream, filters them by its own criterion (ground-truth answer for STaR, majority vote for the self-training arm, an external teacher for the distillation control) and takes a LoRA step, for three rounds. Middle: a frozen copy of $\theta_{0}$ is pushed through the identical evaluation path, so every statistic is read against a null measured on the same problems with a model that did not train. Bottom: evaluation draws $k{=}128$ samples per problem per checkpoint, converts them to a solve rate rather than a single decode, discretizes with hysteresis, and reads off a seven-state ledger. F7 sits on the control path itself: the null each statistic is read against is an estimate, and measuring it once is the same error one level up. Every mode is measured here, inverts a reported conclusion when uncontrolled, and is caught by a control at the stage marked (Section 4 ).

	failure	would have concluded	control that catches it	cost
F1	single-decode ledger	“a frozen model has $\textsc{clr}=1.5$ ”	frozen floor $+$ estimator	$27
F2	one-success $\mathrm{ER}$ threshold	“maj.-vote expands $0.143$ of AIME”	frozen floor for $\mathrm{ER}$	$0
F3	fixed token cap	“distillation is the most destructive”	log truncation per ckpt	$0
F4	underpowered ledger	“the two methods do not differ”	prior power analysis	$88
F5	single training seed	“maj.-vote corrupts less than STaR”	$\geq 3$ seeds	$282
F6	underpowered probe	“self-training erodes refusal by $10$ pts”	size the probe first	$2

Table 1: Seven measurement failures, the conclusion each would have supported, and the control that caught it. Every row is measured on a run from this study rather than posited, and F7 is the failure that arises inside the correction for F2. Cost is the marginal spend on the control; three of the seven cost nothing at all, being re-analyses of records the study already holds, and two more would cost nothing in any study that reports more than one arm.

Figure 2: The expansion statistic needs a null, and one measurement of a null is not one. Left:

\mathrm{ER}_{m}

against the evidence threshold. At

m{=}1

, the criterion in use, a single frozen comparison sits at

0.280

, above majority-vote self-training. The shaded band is the range over all

110

frozen comparisons the records contain, and it does not approach zero until

m{=}5

: at

m{=}2

, the natural repair, the pooled null is

0.058

and the value it was meant to certify as null is

0.048

. Right: the threshold-free test, by pooled base rate. Detections against a shared

1{,}408

-draw baseline under FDR control; the frozen null is zero in every stratum on all eleven held-out replicates.

filter	update	seed	accuracy	$\Delta$ (pts)	retained	base $0$	base $1$ – $5$	base $>5$
majority vote	SFT	0	$0.180\to 0.213$	$\uparrow$ $3.3$	250	0	2	17
		1	$0.178\to 0.163$	$\downarrow$ $1.5$	251	1	0	3
		2	$0.179\to 0.186$	$\uparrow$ $0.7$	251	0	0	4
ground truth	SFT	0	$0.177\to 0.195$	$\uparrow$ $1.8$	220	1	1	9
external teacher †	SFT	0	$0.180\to 0.310$	$\uparrow$ $13.0$	247	3	7	26
		1	$0.177\to 0.324$	$\uparrow$ $14.7$	248	4	7	26

Table 2: The matched AIME ladder: one stream, one retained volume, one evaluation, and only the source of the correctness filter changing. Detections are problems whose solve rate rises significantly against a pooled

1{,}408

-draw baseline of the untrained model, pooled over 11 independent evaluations (per-problem exact test, FDR

0.05

), split by how often that baseline reaches the problem at all. The frozen null is zero in every column. Expansion is only interpretable in the two low-base columns: there the self-training arms reach

0

–

2

problems and the external teacher

8

–

11

, on every seed. Majority-vote accuracy also moves

+3.3

-1.5

and

+0.7

points across its three seeds while the teacher’s moves

+13

+16

: the instability of F5, on a second benchmark and under the matched protocol.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

arm	benchmark	gen. $T$	max tokens	samples/prob	retained per round	signal correct
STaR	MATH-500	0.9	2,048	4	229 / 227 / 231	$100\%$ (ground truth)
STaR	band	0.9	2,048	4	232 / 229 / 230	$100\%$ (ground truth)
maj.-vote SFT	MATH-500	1.0	2,048	8 votes	241 / 247 / 246	$92.1/89.5/91.5\%$
maj.-vote SFT	band	1.0	2,048	8 votes	247 / 246 / 240	$92.7/91.1/94.6\%$
Distillation	band	0.7	2,048	4	232 / 239 / 234	$100\%$ (filtered)
matched AIME ladder: one stream ( math-train-hard ), one volume, three seeds

Appendix

Table 3: Arm configuration and training-signal statistics. Evaluation is identical across arms within a benchmark:

k{=}128

T{=}0.8

, top-

p

0.95

, plus one greedy sample, with independent sampling calls. All arms use Qwen3-8B with rank-32 LoRA,

\eta=10^{-4}

, batch

8

3

rounds of

256

stream problems, and at most one training example per problem; SFT arms train

3

epochs per round and the policy-gradient arm one, since its rollouts are off-policy after a single pass. The AIME block above the rule is the matched ladder; the arm below it is the unmatched baseline that trains on the evaluation set itself.

arm	$t{=}0$	$t{=}1$	$t{=}2$	$t{=}3$	verdict
STaR	$16.5\%$	$15.9\%$	$12.4\%$	$14.7\%$	stable; counts stand
maj.-vote SFT	$16.6\%$	$14.6\%$	$16.9\%$	$16.0\%$	stable; counts stand
Distillation	$16.5\%$	$\mathbf{65.3\%}$	$\mathbf{46.1\%}$	$\mathbf{49.6\%}$	drift; corruption counts excluded

Appendix

Table 4: Truncation rate by checkpoint on the difficulty band, the diagnostic that excludes an arm’s corruption counts (F3). A method that changes generation style will be scored as destroying capability under a fixed token budget, and nothing else in the ledger distinguishes the two.

		evaluation, AIME		training stream
seed	round	accuracy	truncated	trained	spread	share	vote
$\mathbf{0}$	$t=0$	$0.1764$	4.2%	—	—	—	—
	$t=1$	$0.2082$	18.3%	$248$	$135$	$0.78$	$77.0\%$
	$t=2$	$0.0643$	24.3%	$251$	$143$	$0.77$	$73.7\%$
	$t=3$	$0.1921$	$\mathbf{76.4}$ %	$243$	$217$	$0.58$	$49.8\%$
$\mathbf{1}$	$t=0$	$0.1760$	4.4%	—	—	—	—

Appendix

Table 5: Three seeds of the policy-gradient arm, on identical data. The evolution stream is drawn under a fixed seed, so the three runs see the same

256

problems in the same order each round; --seed varies adapter initialization, training order and sampling. Round

1

is the same experiment three times over: the vote is

77.0

–

77.9\%

correct and

135

–

136

groups of

{\sim}250

carry any gradient. The runs then diverge completely. Every seed ends in severe truncation, so the collapse belongs to the configuration; the round at which it arrives does not. Trained is the number of problems that formed a majority, of

256

; spread the groups carrying gradient; vote the fraction of those majorities that were in fact correct, a diagnostic that never enters training.

experiment class	runs	samples (M)	tokens (M)	cost (USD)
Primary audits, difficulty band ( $n{=}1{,}163$ )	3	1.80	2,441	1,465
Primary audits, MATH-500 and AIME	4	0.27	414	248
Matched AIME ladder, 3 filters $\times$ seeds	6	0.19	692	415
Policy-gradient arms	4	0.16	443	266
Seed replication (2 methods $\times$ 2 seeds)	4	0.36	470	282
Noise floors (4 benchmarks)	7	0.42	589	340

Appendix

Table 6: Compute by experiment class. Cost is the metered price of sampling and training on the hosted service; token counts are generated tokens, which dominate cost.

Figure 3: Construction of the difficulty band. Left: the held-out pool and the retained band; dashed lines mark the selection interval

[0.25,0.75]

. Right: selection estimate at

k{=}8

against the independent re-measurement at

k{=}128

that serves as checkpoint

0

. Regression to the mean widens the realized band, costing statistical power; because the two measurements are independent it cannot create transitions.

Figure 4: An evaluation set fixes which question it can answer, before any method is applied. Measured distribution of the untrained backbone’s per-problem solve rate at

k{=}128

, from the frozen-control records; shading marks the hysteresis band. Expansion is observable only in the spike at

\hat{\pi}_{0}=0

and corruption only in the mass above

0.5

, so AIME can host an expansion measurement and not a corruption one, MATH-500 the reverse, and the constructed band neither expansion nor a natural forgetting rate.

		transitions		matched floor
method	benchmark	learned	corrupted	L	C	accuracy
STaR	MATH-500	2	1	0 (0)	1 (1)	$0.835\to 0.850$
maj.-vote SFT	MATH-500	3	2	0 (0)	1 (1)	$0.832\to 0.830$
maj.-vote SFT	AIME	4	0	0 (0)	0 (0)	$0.177\to 0.202$
Distillation †	AIME	9	0	0 (0)	0 (0)	$0.180\to 0.310$
STaR	band	149	106	10 (4)	8 (5)	$0.550\to 0.606$

Appendix

Table 7: What each method adds and removes, against a floor measured on the same problems and matched to the same number of checkpoints. Counts are problems. The floor columns give the median over four-checkpoint orderings of the independent evaluations of the untrained model that benchmark carries, with the conventional two-checkpoint floor in parentheses.

		mean accuracy				learned vs $t_{0}$				corrupted vs $t_{0}$
arm	set	$t_{0}$	$t_{1}$	$t_{2}$	$t_{3}$	$t_{0}$	$t_{1}$	$t_{2}$	$t_{3}$	$t_{0}$	$t_{1}$	$t_{2}$	$t_{3}$
STaR	band	0.550	0.590	0.591	0.606	0	83	123	149	0	48	98	106
maj.-vote SFT	band	0.550	0.586	0.605	0.590	0	57	119	145	0	33	59	88
Distillation †	band	0.550	0.610	0.577	0.546	0	151	168	161	0	101	143	177
maj.-vote RL	band (175)	0.530	0.603	0.578	0.583	0	22	21	19	0	13	12	9
STaR	MATH-500	0.835	0.841	0.851	0.850	0	1	4	2	0	1	1	1

Appendix

Table 8: Per-checkpoint accuracy and transition counts. Every headline number in the paper compares

t{=}0

with

t{=}3

; this is the path between them. Each count compares that checkpoint with checkpoint

0

under the ledger of Eq. 2 ; it is a running state, not a cumulative total, so it can fall when a problem re-crosses the band, and the final column of each block reproduces Table 7 . Distillation on the band peaks at round

1

and declines under its own truncation drift (F3); majority-vote SFT on the band peaks at round

2

in accuracy while its corruption count keeps rising. The AIME policy-gradient row is reported for completeness only: that arm collapses into unbounded repetition and is

76.4\%

truncated at

t{=}3

, so its transition counts are excluded from the capability analysis (Appendix A.3 ).

Figure 5: The path the endpoint comparison hides. Mean accuracy and the learned and corrupted counts against checkpoint

0

at every checkpoint on the difficulty band, with the frozen-model floor shaded on the two count panels. Distillation peaks at round

1

and declines under its own truncation drift; majority-vote self-training’s accuracy peaks at round

2

while its corruption count continues to climb.

Figure 6: Migration matrices: where problems actually move. Joint distribution of the per-problem solve rate before and after evolution, binned into fifths, for the three difficulty-band arms; the dashed line is no change and cell counts sum to $1{,}163$ . “Up” and “down” count problems above and below the diagonal. Movement is substantial in both directions for every arm, which is the point that a mean accuracy cannot express.

		stable		changed
arm	benchmark	correct	incorrect	learned	corrupted	recovered	transient	oscillating
STaR	MATH-500	167	25	2	1	2	3	0
maj.-vote SFT	MATH-500	167	28	3	2	0	0	0
maj.-vote SFT	AIME	8	48	4	0	0	0	0
Distillation †	AIME	8	40	9	0	0	3	0
STaR	band	567	299	149	103	25	17	3

Appendix

Table 9: Ledger categories by arm, sampled ledger at

k{=}128

z{=}2

. Learned and corrupted here are the trajectory-based labels; the endpoint counts used for clr in the main text differ only where a problem oscillates.

		solve-rate estimator, $k{=}128$			single greedy sample
benchmark	backbone	learned	corrupted	matched (L/C)	L/C	changed [95%]	problems
MATH-500	Qwen3-8B	0–1	0–1	0 / 1	6 / 9	$7.5\%\,[4.6,12.0]$	200
AIME 2025/26	Qwen3-8B	0–0	0–1	0 / 0	2 / 2	$6.7\%\,[2.6,15.9]$	60
difficulty band	Qwen3-8B	0–7	1–9	10 / 8	100 / 120	$\mathbf{18.9\%}\,[16.8,21.3]$	1,163
band (pilot)	Qwen3-8B	0	0	—	—	—	112
AIME 2025/26	gpt-oss-20b	0	0	—	5 / 4	$15.0\%\,[8.1,26.1]$	60

Appendix

Table 10: Noise floors: apparent transitions produced by an untrained model (F1, F7). Estimator columns give the range over every ordered pair of independent frozen evaluations (

110

pairs on AIME,

12

on MATH-500 and

20

on the band) and, in the last estimator column, the median over four-checkpoint orderings that are design-matched to an arm. A conventional single-pair floor is the low end of these ranges. The single-greedy protocol is what a per-example before/after comparison implicitly uses; its changed rate carries a Wilson interval since it is a proportion over problems.

problem	$t_{0}$	$t_{1}$	$\hat{\pi}_{0}$	$\hat{\pi}_{1}$	problem	$t_{0}$	$t_{1}$	$\hat{\pi}_{0}$	$\hat{\pi}_{1}$
counting_prob/199	$\times$	$\checkmark$	0.633	0.672	num_theory/183	$\times$	$\checkmark$	0.875	0.852
geometry/226	$\checkmark$	$\times$	0.719	0.750	num_theory/357	$\checkmark$	$\times$	0.594	0.453
geometry/702	$\times$	$\checkmark$	0.078	0.117	num_theory/516	$\checkmark$	$\times$	0.398	0.328
int_algebra/1063	$\times$	$\checkmark$	0.133	0.023	num_theory/598	$\checkmark$	$\times$	0.898	0.898
int_algebra/1350	$\checkmark$	$\times$	0.367	0.367	prealgebra/1128	$\checkmark$	$\times$	0.688	0.758
int_algebra/1797	$\times$	$\checkmark$	0.859	0.844	precalculus/1313	$\checkmark$	$\times$	0.602	0.695

Appendix

Table 11: Every problem whose greedy verdict changed between two evaluations of the frozen model (F1), MATH-500,

T{=}0

with a fixed seed.

\checkmark

and

\times

are the greedy verdicts;

\hat{\pi}

is the solve rate over

128

samples at the same checkpoint. The point of the table is the last two columns of each block: the underlying solve rate barely moves, since it is a frozen model, while the single-decode verdict flips. Nine of these are scored as corruptions and six as learnings, giving the

\textsc{clr}=1.5

of Section 4.1 .

hysteresis width $z$ (at $k{=}128$ )				sampling budget $k$ (at $z{=}2$ )
$z$	STaR	maj.-vote	frozen	$k$	STaR	maj.-vote	frozen
$0.0$	172/111 (0.65)	167/117 (0.70)	44/39	$16$	115/64 (0.56)	125/69 (0.55)	10/7
$1.0$	165/111 (0.67)	158/103 (0.65)	12/22	$32$	103/83 (0.81)	120/88 (0.73)	6/8
$1.5$	160/109 (0.68)	151/97 (0.64)	7/12	$64$	134/95 (0.71)	143/79 (0.55)	5/7
$2.0$	149/106 (0.71)	145/88 (0.61)	4/5	$128$	149/106 (0.71)	145/88 (0.61)	4/5
$2.5$	136/98 (0.72)	131/85 (0.65)	1/3

Appendix

Table 12: Sensitivity of the ledger to

z

and

k

: learned/corrupted counts with clr in parentheses, difficulty band.

method	seed	$n$ (matched)	learned	corrupted	clr matched	clr as run
STaR	0	175	27	19	$0.704$	$0.711$	range $0.45$ – $0.74$ ; net positive
	1	175	23	17	$0.739$	$0.739$
	2	175	22	10	$0.455$	$0.455$
maj.-vote	0	175	20	11	$0.550$	$0.607$	range $0.55$ – $1.53$ ; $2/3$ net destructive
	1	175	19	29	$\mathbf{1.526}$	$1.526$
	2	175	18	24	$\mathbf{1.333}$	$1.333$

Appendix

Table 13: Seed replication (F5), with every seed scored on the same problems. Seed

0

ran on the full band and seeds

1

–

2

on a

175

-problem subsample, so an as-run comparison confounds seed with evaluation set. Restricting seed

0

to the same

175

problems costs no additional sampling and is given in the matched columns: the reversal is a seed effect, not a set effect.

Figure 7: Two of the seven failures, on the runs where they occurred. Left: apparent transitions produced by a model that never trained, under two definitions of correctness (F1). A single greedy sample manufactures state changes on

6.7

–

18.9\%

of problems; the solve-rate estimator with a hysteresis band reports

0.0

–

0.8\%

. Right: truncation rate per checkpoint (F3). The distillation student adopted its teacher’s verbose style and began hitting the token cap, losing final answers and registering as capability loss; the self-evolution arms are flat, which is why their counts survive.

Figure 8: Transitions clear the noise floor by

{\sim}11\times

, but the comparison between methods is not stable across seeds. Left: learned and corrupted counts on the

1{,}163

-problem difficulty band with bootstrap intervals over problems; the shaded region is the frozen-model floor, design-matched to the arms’ four checkpoints rather than the conventional two-checkpoint control, which understates it by roughly half. Right: clr across three training seeds, every seed scored on the same

175

problems (F5). STaR holds

0.45

–

0.74

; majority-vote self-training spans

0.55

1.53

, crossing the threshold at which a method destroys more than it creates.

arm	BH, per arm	Bonferroni	$p\leq 0.01$	$p\leq 0.05$	joint BH
distillation, seed 0	10	7	10	10	9
distillation, seed 1	11	6	10	11	—
distillation, seed 2	8	7	7	8	—
maj.-vote SFT, seed 0	2	0	2	2	0
maj.-vote SFT, seed 1	1	1	1	1	—
maj.-vote SFT, seed 2	0	0	0	0	—

Appendix

Table 14: The comparison does not depend on the decision rule. Low-base detections (pooled base rate

\leq 5/1{,}408

n=22

) under four rules applied identically to every arm, and under one joint procedure over all arms and problems at once. A false-discovery-rate procedure is adaptive, so an arm with many true effects admits a larger cutoff; the cutoffs actually attained here are

0.0095

for distillation’s

36

detections and

0.0094

for majority-vote self-training’s

19

, so the two arms are held to the same standard. The frozen row is the leave-one-out null over all eleven replicates.

term	$\beta$	95% CI	$p$
low-base $\times$ post (reference arm, STaR)	$+1.17$	$[+0.52,+1.82]$	$10^{-3}$
distillation $\times$ low-base $\times$ post	$+1.91$	$[+1.25,+2.56]$	$10^{-8}$
majority-vote SFT $\times$ low-base $\times$ post	$+0.21$	$[-0.60,+1.03]$	$0.61$
policy gradient $\times$ low-base $\times$ post	$-0.49$	$[-1.51,+0.53]$	$0.34$

Appendix

Table 15: The dissociation is not a by-product of the teacher’s larger gain. Logistic model on the per-problem counts, with terms for arm, base-rate stratum, before/after, and their interactions; the coefficient shown is the additional log-odds an arm moves a low-base problem beyond its own overall lift. A model in which every method improves problems by a common multiplicative factor predicts zero here. It is zero for both self-training arms and is not for the teacher.

	baselines pooled				FDR level
arm	$3$	$5$	$7$	$11$	$0.01$	$0.05$	$0.10$
distillation, seed 0	8	10	10	10	9	10	10
maj.-vote SFT, seed 0	0	1	1	2	0	2	2
STaR	1	2	2	2	2	2	2
policy gradient	1	1	1	1	0	1	1

Appendix

Table 16: Sensitivity of the threshold-free test to the choices that remain. Low-base detections (

n=22

) as the number of pooled baselines and the error rate vary. Pooling fewer baselines costs sensitivity in the arm with signal and never manufactures it in the arms without.

$m$	measured	analytic, plug-in	analytic, beta-binomial
$1$	$0.176$	$0.158$	$0.163$
$2$	$\mathbf{0.058}$	$\mathbf{0.044}$	$0.049$
$3$	$0.023$	$0.013$	$0.015$
$5$	$0.003$	$0.001$	$0.002$

Appendix

Table 17: An analytic null for the expansion rate, and why measuring it still matters.

\mathbb{E}[\mathrm{ER}_{m}]=\mathbb{E}_{q}[(1-q)^{k}\Pr(X\geq m\mid k,q)]/\mathbb{E}_{q}[(1-q)^{k}]

, with the distribution of

q

taken from the pooled

1{,}408

-draw baseline, as a plug-in and under an empirical-Bayes beta prior. The calculation is systematically optimistic and increasingly so as

m

rises, because the base-unreached set is selected on a noisy zero: a problem observed at

0/1{,}408

has a plug-in

q

of exactly zero and is assigned no chance of a later success. At the threshold that appears to repair the statistic, an author trusting the calculation would certify

0.044

and reach the same false conclusion.

replicates $G$	subsets	ordered pairs	min	median	max	spread
$2$	$55$	$2$	$0.000$	$0.048$	$0.120$	$0.120$
$3$	$165$	$6$	$0.015$	$0.056$	$0.103$	$0.088$
$\mathbf{4}$	$330$	$12$	$0.022$	$0.058$	$0.098$	$0.077$
$5$	$462$	$20$	$0.028$	$0.057$	$0.091$	$0.063$
$7$	$330$	$42$	$0.035$	$0.058$	$0.078$	$0.043$
$9$	$55$	$72$	$0.043$	$0.059$	$0.066$	$0.024$

Appendix

Table 18: How many frozen replicates a null needs before it settles. The pooled

\mathrm{ER}_{2}

null re-estimated from every subset of

G

of our eleven baseline evaluations, exhaustively rather than sampled, since

\binom{11}{G}

never exceeds

462

. Four replicates, which is what a three-arm study with a no-op control possesses for free, still admit estimates from

0.022

0.098

, so such a study could certify a null of

0.022

and reproduce F7. The spread halves by seven and again by nine; it takes nine before the estimate stays within

\pm 0.02

of the

0.058

that eleven give.

arm	base $0$ (10)	base $1$ – $5$ (12)	base $>5$ (38)	total
majority-vote SFT	0	2	17	19
distillation †	3	7	26	36
frozen null , 11 replicates	$\mathbf{0}$	$\mathbf{0}$	$\mathbf{0}$	$\mathbf{0}$

Appendix

Table 19: The threshold-free test: per-problem exact tests against a pooled baseline, with FDR control. Each arm’s endpoint count (

x/128

) is tested against the untrained model’s pooled count (

x/1{,}408

, eleven independent evaluations) with a one-sided Fisher exact test at FDR

0.05

across all

60

AIME problems. Strata are the pooled base rate. The null holds out one base evaluation and treats it as the “after”; all eleven replicates return zero detections everywhere. Expansion lives in the two left columns, sharpening in the right one.

	frozen null, AIME			null	AIME arms		MATH-500
$m$	one pair	range over 110	pooled [95%]	MATH-500	maj.-vote	Distillation †	arms
$1$	7/25 $=0.280$	$0.000$ – $0.364$	$0.176\,[0.145,0.206]$	$0.179$	3/21 $=0.143$	12/22 $=0.545$	1/3, 0/3
$2$	0/25 $=0.000$	$0.000$ – $0.136$	$\mathbf{0.058\,[0.038,0.078]}$	$0.128$	1/21 $=0.048$	11/22 $=0.500$	0/3, 0/3
$3$	0/25 $=0.000$	$0.000$ – $0.087$	$0.023\,[0.009,0.038]$	$0.000$	0/21 $=0.000$	10/22 $=0.455$	0/3, 0/3
$5$	0/25 $=0.000$	$0.000$ – $0.040$	$0.003\,[0.000,0.009]$	$0.000$	0/21 $=0.000$	8/22 $=0.364$	0/3, 0/3

Appendix

Table 20: The expansion null is not a number, it is a distribution, and at

m{=}2

it is not zero.

\mathrm{ER}_{m}

counts base-unreached problems that reach

m

correct samples after evolution. The null columns use every ordered pair of independent evaluations of the untrained model (

110

pairs on AIME from eleven such evaluations,

12

on MATH-500), so every count in them is an artifact; “one pair” is the single frozen comparison a conventional study would run, and is what a study with one control reports. At

m{=}2

the pooled null is

0.058

and majority-vote self-training’s measured

0.048

sits below it. Even

m{=}3

has a null of

0.023

; no threshold below

m{=}5

reaches zero.

problem	$t_{0}$	$t_{1}$	$t_{0}$	$t_{3}$	$t_{0}$	$t_{3}$	problem	$t_{0}$	$t_{1}$	$t_{0}$	$t_{3}$	$t_{0}$	$t_{3}$
	frozen		maj.-vote		distill.			frozen		maj.-vote		distill.
2025 / 10	0	0	0	0	0	0	2026 / 10	2	0	0	0	1	1
2025 / 12	0	0	0	0	0	0	2026 / 11	0	1	0	0	0	9
2025 / 13	0	1	2	0	0	1	2026 / 13	0	0	0	0	1	20
2025 / 14	0	0	0	0	0	0	2026 / 14	0	1	0	1	0	9
2025 / 17	0	0	0	0	0	6	2026 / 15	0	0	0	0	0	0

Appendix

Table 21: Every AIME problem the base model does not reach, with its correct-sample count under each arm. Counts are out of

k{=}128

. A row contributes to

\mathrm{ER}_{m}

for an arm when its baseline column is

0

and its endpoint column is at least

m

. The frozen control is the same untrained model evaluated twice, so its seven

0\to 1

rows are the artifact of Section 4.2 ; note that every one of them is exactly

1

, and that no arm’s genuine expansion is. Bold marks a row that counts as expanded at

m\geq 2

Figure 9: Per-problem solve rate before and after evolution. Each point is one evaluation problem; the diagonal is no change. The shaded strip at

\hat{\pi}_{0}=0

contains the only problems whose improvement would constitute expansion rather than consolidation, and the counts in it apply the

m\geq 2

gate against each arm’s own baseline. The AIME panels are the matched-ladder arms of Table 2 , not the unmatched pair of Table 20 , which is why the majority-vote count differs between the two. Self-training leaves the strip almost untouched; the distillation control does not.

baselines pooled	draws	min. successes	$0.01$	$0.02$	$0.03$	$0.05$	$0.08$	$0.12$
$1$	$128$	$7/128$	$0.00$	$0.01$	$0.09$	$0.46$	$0.89$	$1.00$
$3$	$384$	$4/128$	$0.04$	$0.25$	$0.54$	$0.89$	$0.99$	$1.00$
$5$	$640$	$3/128$	$0.14$	$0.47$	$0.74$	$0.96$	$1.00$	$1.00$
$\mathbf{11}$	$1{,}408$	$2/128$	$0.37$	$0.73$	$0.90$	$0.99$	$1.00$	$1.00$

Appendix

Table 22: Power of the per-problem test on a base-unreached problem. Probability of detection against the true post-training pass rate, at

k{=}128

, for a problem with no successes in the pooled baseline, under the cutoff the procedure attains. Pooling is what buys the sensitivity: one baseline requires seven correct samples before a problem can be detected, eleven require two. The null result of Section 5 is therefore a bound, no self-training arm having lifted a base-

0

problem above a

2.4\%

pass rate, and not an absence of measurement.

arm	event	total	$>0.1$	$>0.2$	$>0.234$	$>0.35$	median	max
STaR	corrupted	106	90	69	58	37	$0.258$	$0.781$
STaR	learned	149	133	107	—	61	$0.297$	$0.812$
maj.-vote	corrupted	88	86	69	60	37	$0.320$	$0.750$
maj.-vote	learned	145	133	104	—	62	$0.297$	$0.773$
frozen, worst of $20$ pairs	any change	1,163	108	1	0	0	$0.039$	$\mathbf{0.234}$

Appendix

Table 23: Effect sizes for transition events on the difficulty band. Counts of events whose solve-rate change exceeds each magnitude, against the envelope of the frozen control. The control row is the worst value each column takes over the twenty ordered pairs of independent evaluations of the untrained model, rather than a single pair: across all twenty, one problem moved past

0.2

and none past

0.234

, so

0.234

is the measured zero-false-positive gate and the bolded column is what survives it.

Figure 10: Corruption is large movement, not threshold flicker. Left: distribution of

|\Delta\hat{\pi}(p)|

for the frozen control over all

1{,}163

problems and for each arm’s corruption events, each normalized to its own total; over the twenty independent frozen comparisons the records contain, no problem moves by more than

0.234

. Right: corruption counts surviving a minimum-drop gate. Over half of each arm’s events lie beyond the frozen envelope entirely.

probe	arm	replicate 1 ∗	replicate 2	replicate 3
MMLU-Redux ( $n{=}200$ )	STaR	$+0.063$	$+0.070$ ( $p=0.029$ )	$+0.020$ ( $p=0.60$ )
MMLU-Redux ( $n{=}200$ )	maj.-vote	$+0.106$	$+0.020$ ( $p=0.58$ )	$+0.015$ ( $p=0.72$ )
refusal ( $n{=}50$ prompts)	STaR	$-0.050$	$-0.010\,[-0.085,0.065]$	$-0.005\,[-0.068,0.058]$
refusal ( $n{=}50$ prompts)	maj.-vote	$-0.015$	$\mathbf{-0.115}\,[-0.196,-0.034]$	$-0.005\,[-0.080,0.070]$
refusal, underpowered ( $n{=}10$ )	STaR	$-0.100$ , non-monotonic	—	the F6 probe

Appendix

Table 24: Out-of-domain probes, run three times each against the same stored checkpoints. Any difference between replicates is measurement, not training. Refusal changes are prompt-level means with cluster-robust intervals; MMLU is tested with paired McNemar on the same questions. The one significant refusal result does not replicate on either side of it, and the largest MMLU gain is the largest of six measurements of a smaller effect.

Language Model Auditing

Language model auditing tests whether models behave reliably, fairly, and safely using behavioral probes and internal or deployment evidence.

See all 159 papers

May 20, 2026

Self-Verified Distillation: Your Language Model Is Secretly Its Own Synthetic Data Pipeline

Can post-trained large language models (LLMs) further improve themselves using only unlabeled prompts, without external teachers or feedback from tools? We study this setting starting only from unlabeled seed questions with no ground-truth solutions, across three reasoning domains: math, science, and coding. We propose Self-Verified Distillation, a simple post-training refinement algorithm in which the model generates candidate solutions to these seed questions, filters them using prompt-based self-verification, and trains on the resulting self-curated dataset. Inspired by the UQ benchmark's use of multiple validators to screen candidate answers to hard unsolved questions, we adapt this validation-based filtering idea to self-training: the model filters its own generated solutions through a three-stage cascade of cycle-consistency, factuality, and correctness checks, accepting a solution only if it passes all stages with unanimous judge votes. We find that sampling more candidate generations and using a larger verification budget during training data construction produces higher-quality self-curated data and, in turn, better reasoning models. We then train Qwen3 models at multiple scales with Self-Verified Distillation and obtain gains across all three domains. For Qwen3-4B, our method improves aggregate held-out pass@1 by +16.7 points in math (AIME26 and HMMT), +11.1 points in science (GPQA Diamond and HLE), and +8.3 points in coding (LCBv5 and LCBv6), with gains also extending to 0.6B and 8B models. Compared to our test-time-only baseline (UQ-TTC), which improves performance by spending extra compute at inference time, Self-Verified Distillation achieves better performance in most settings while requiring only a single inference call at test time.

Tony Lee, Percy Liang

Stanford University

Language Model Self-Verification Self-Training for Language Models Language Model Self-Assessment Language Model Distillation Synthetic Data Generation for Language Models Language Model Self-Improvement+1

Ask PDF

Mar 23, 2026

Reliable Self-Improvement Training by Verifying Reasoning, Not Just Answers

Self-improvement training, where models learn from self-generated solutions, promises sustained capability gains but suffers from a pervasive failure mode: across multiple rounds, compounding reasoning errors cause accuracy to stall or degrade. We trace this drift to standard filtering criteria that retain solutions based solely on final answer correctness, which lets lucky guesses (correct answers with flawed reasoning) contaminate the training data. We propose Verified Self-Improvement (VSI), a framework that conditions data retention on step-level structural integrity rather than just the final output. VSI validates solutions by recomputing arithmetic steps via a computer-algebra library (sympy), checking intermediate consistency, and enforcing domain constraints. Evaluating VSI on GSM8K with Qwen3-4B-Thinking across 5 rounds of self-improvement against four baselines (no verification, outcome verification, majority voting, and VSI with DPO) shows that VSI rejects approximately 34% of correct-answer solutions, successfully isolating lucky guesses. This cleaner training signal drives sustained accuracy gains across all rounds (80.5% to 91.0%), whereas outcome verification plateaus and unverified training collapses. Finally, converting VSI checks into DPO preference pairs trains the model to distinguish sound reasoning from lucky answers, boosting reward accuracy from 46% to 63%. VSI offers a simple, reproducible recipe for robust self-improvement whenever automated reasoning checks are available.

Xinyu Zhang

Anyscale

Language Model Self-Improvement Self-Training for Language Models Process Supervision

Ask PDF

Aug 26, 2026

Training Alignment Auditors via Reinforcement Learning

Alignment auditing of frontier models increasingly relies on LLM auditors to surface undesirable behaviors at scale, but current automated auditors can struggle with coherent investigation and audit realism. In this work, we improve LLM auditors with reinforcement learning. In our best training environment, the policy investigates target models that potentially possess hidden behaviors planted via their system prompt. An LLM judge, which knows whether the target has a hidden behavior, holistically compares the policy's investigation to a reference investigation to determine the reward. With systematic ablations, we find that pairwise rewards yield more robust training compared to pointwise rewards, and that adding targets without planted behaviors helps maintain a low false positive rate. Training improves investigation quality against targets with planted behaviors, the rate of concerning behaviors surfaced in unmodified production models, and audit realism, while false-positive rates stay below 1%. Furthermore, auditing capabilities generalize across scaffolds: performance on AuditBench's adversarially fine-tuned targets substantially improves [Sheshadri et al., 2026].

Paul Rosu, Rowan Wang

Anthropic Fellow · Anthropic

LLM Auditing Language Model Auditing RL for Language Models

Ask PDF

Abstract

Language Model Auditing

Explore similar work

Self-Verified Distillation: Your Language Model Is Secretly Its Own Synthetic Data Pipeline

Reliable Self-Improvement Training by Verifying Reasoning, Not Just Answers

Training Alignment Auditors via Reinforcement Learning