Projected Distillation: On Learning to Solve Hard Problems from Close, Correct Targets

Shripad V. Deshmukh; Christian Bock; Kangrui Ruan; Abhijeet Awasthi; Vijay Lingam; Rein Houthooft; Manish Prajapat

cs.LGOct 20, 2026

Projected Distillation: On Learning to Solve Hard Problems from Close, Correct Targets

Authors: Shripad V. Deshmukh, Christian Bock, Kangrui Ruan, Abhijeet Awasthi, Vijay Lingam, Rein Houthooft, Manish Prajapat

Organizations: University of Massachusetts · AWS AI Labs

Abstract

On-policy learning is central to post-training: training on a model's own generations limits distribution shift and helps preserve existing capabilities. On hard problems, however, on-policy RL provides no reward-based learning signal when all attempts fail. On-policy distillation provides next-token supervision from a teacher, but on hard problems, if a rollout goes wrong early, subsequent supervision is conditioned on prefixes that already contain the mistake. Correct behavior can instead come from a gold or teacher solution, but such off-policy targets can differ substantially from the model's generations and, as we show, may erode existing capabilities. We therefore treat on-policyness as a continuum and propose projected distillation, which minimally edits the student's failed attempts into correct targets. The method uses contrastive updates to favor corrections over failed attempts and refreshes these corrections as the student evolves, enabling near-on-policy learning even when all student attempts fail. Theoretically, we prove that the student's learning regret grows with the edit size: it scales linearly with the horizon under minimal correction, as in DAgger, and quadratically under full-answer replacement, as in behavior cloning. Empirically, across two students (Qwen3-8B and OLMo-2-7B) and five domains spanning math, code, science, and tool use, projected distillation gives the highest area under the specialization-retention frontier (best on 8 of 9 cells against GRPO and OPD, and 7 of 9 against SPOT and SODA). Our ablations highlight the importance of online corrections, the contrastive loss, and on-policyness: moving targets farther off-policy steadily worsens the specialization-retention trade-off, with retention falling from 0.916 to 0.744 when minimal corrections are replaced by student-agnostic gold solutions.

Figures & tables

Figure 1: Learning from the nearest correct answer. Correct answers form a set

\mathcal{C}(x)

, but the student assigns them very different likelihoods. On a hard problem, its

k

on-policy attempts ( ) are all wrong and earn no reward. A teacher or gold answer ( ) supplies correctness, but may lie far from what the student writes. Projected distillation instead minimally edits a failed attempt into a newly computed projection ( ) and refreshes that target as the student changes. The green path illustrates repeated online rounds: successive wrong rollouts may move closer until the student produces a correct answer itself (

\checkmark

). The resulting small edits keep supervision near-policy; writing

m

for edit size and

H

for the generation horizon, our analysis bounds their drift by a term proportional to

mH

Figure 2: One round of projected distillation. The student answers each problem

k

times, and its responses split into the wrong ones

y^{-}

and the correct ones

y^{\mathrm{c}}

. A projector rewrites a wrong response into a correct one

y^{+}

with the smallest edit that works. The two are aligned token by token, and the preference term acts only on the spans that differ ( removed , added ), while

y^{\mathrm{c}}

is supervised directly. The updated student begins the next round, so its corrections are recomputed as it changes.

Figure 3: Aggregate specialization–retention Pareto frontier , one curve per method pooled over both students, all benchmarks and seeds, split by projector class. Left: same-family (

32

B) projectors. Right: black-box projector. GRPO uses no projector.

y

-axis is retention measured as held-out accuracy as a fraction of the untrained base.

x

-axis is specialization measured as average hard-set accuracy. Each curve is the monotone Pareto frontier of the pooled trajectory anchored at

\star=(0,1.0)

; up-and-to-the-right is better . Every baseline ends below its untrained base : GRPO and OPD decline from the outset, while SPOT and SODA exceed base early and then fall below it. Projected distillation holds retention at base across the range and attains the highest specialization.

		RLVR	same-family $32$ B teacher		black-box projector
		GRPO	OPD	Ours	SPOT	SODA	Ours
Qwen3-8B	GSM8K	0.061 $\pm$ .011	0.145 $\pm$ .006	0.205 $\pm$ .012	0.209 $\pm$ .030	0.231 $\pm$ .016	0.262 $\pm$ .020
	MATH	0.000 $\pm$ .000	0.003 $\pm$ .001	0.008 $\pm$ .002	0.145 $\pm$ .010	0.159 $\pm$ .009	0.035 $\pm$ .010
	LCB	0.000 $\pm$ .000	0.068 $\pm$ .018	0.138 $\pm$ .003	0.095 $\pm$ .005	0.091 $\pm$ .001	0.123 $\pm$ .006
	SciKnow	0.045 $\pm$ .008	0.027 $\pm$ .002	0.179 $\pm$ .005	0.124 $\pm$ .008	0.155 $\pm$ .013	0.124 $\pm$ .005
	Tool	0.248 $\pm$ .152	0.128 $\pm$ .004	0.676 $\pm$ .013	0.594 $\pm$ .010	0.635 $\pm$ .014	0.667 $\pm$ .015

Table 1: RQ1: area under the specialization–retention frontier (AUPF). Higher is better ( Appendix D ); mean

\pm

s.e. over

3

seeds. GRPO uses no projector and is shown for reference. The OLMo-2-7B LiveCodeBench cell is omitted, since no method moves specialization appreciably above zero there. Bold

=

best per row within each projector block. Ours attains the highest AUPF on

8

9

cells with the same-family teacher and

7

9

with the black-box projector.

	MATH held-out, hard ( $127$ )		Other benchmarks, hard ( $2{,}885$ )
Method	pass@1	pass@16	pass@1	pass@16
Base	0.013	0.150	0.011	0.114
GRPO	0.020	0.205	0.011	0.108
SPOT	0.020	0.197	0.012	0.114
SODA	0.031	0.291	0.016	0.127
Ours	0.056	0.354	0.020	0.154

Table 2: RQ1: learning one benchmark’s hard set, tested on other benchmarks’. Qwen3-8B trained on the MATH hard set with each black-box method and GRPO. Sampled pass@1 over

16

generations per problem at

T{=}0.8

; pass@16 is the share of problems solved at least once in those

16

. Every method is run for one training seed, evaluated every

50

updates, and entered at its own best value for each column, so the checkpoint is selected on the same problems the number is reported on. The rule is applied identically to the baselines and to ours.

Method	$\bar{x}$ $\uparrow$	$\bar{y}_{\mathrm{norm}}$ $\uparrow$	BWT $\uparrow$	$F$ $\downarrow$	SafeSpec 2% $\uparrow$	SafeSpec 5% $\uparrow$
GRPO	0.114 $\pm$ 0.034	0.826 $\pm$ 0.048	$-0.203$ $\pm$ 0.058	0.221 $\pm$ 0.057	0.035 $\pm$ 0.012	0.056 $\pm$ 0.022
OPD	0.118 $\pm$ 0.022	0.951 $\pm$ 0.031	$-0.062$ $\pm$ 0.043	0.136 $\pm$ 0.041	0.101 $\pm$ 0.019	0.135 $\pm$ 0.026
SPOT	0.184 $\pm$ 0.031	0.706 $\pm$ 0.074	$-0.380$ $\pm$ 0.082	0.509 $\pm$ 0.064	0.147 $\pm$ 0.020	0.149 $\pm$ 0.020
SODA	0.224 $\pm$ 0.031	0.776 $\pm$ 0.053	$-0.266$ $\pm$ 0.059	0.355 $\pm$ 0.044	0.140 $\pm$ 0.021	0.190 $\pm$ 0.022
Ours (32B)	0.264 $\pm$ 0.042	1.000 $\pm$ 0.020	0.009 $\pm$ 0.028	0.084 $\pm$ 0.015	0.202 $\pm$ 0.040	0.295 $\pm$ 0.048
Ours (bb)	0.276 $\pm$ 0.042	0.995 $\pm$ 0.024	$-0.004$ $\pm$ 0.034	0.107 $\pm$ 0.027	0.160 $\pm$ 0.037	0.235 $\pm$ 0.040

Table 3: RQ1: continual-learning metrics , pooled over students, benchmarks and seeds (mean

\pm

s.e.), with arrows marking the better direction.

\bar{x}

and

\bar{y}_{\mathrm{norm}}

are specialization and retention averaged over training, BWT is backward transfer against the base,

F

the drop from a run’s retention peak, and

\mathrm{SafeSpec}_{\varepsilon}

the most specialization reached at any checkpoint whose retention is within

\varepsilon

of base ( Appendix D ). Bold

=

best, underline

=

second best per column. Our variants specialize most while holding retention at base, where the baselines either barely specialize or forget.

Figure 4: RQ2: what a larger edit costs. Qwen3-8B student per arm on the GSM8K hard set, varying only how much a verified-correct correction rewrites the student’s own attempt (mean

\pm

s.e. over

3

seeds). Arms sit at their measured edit size, since the projector rather than the prompt sets it, and the dashed line joins the three matched arms. Left to right: hard-set pass@1 relative to the minimal-edit arm, absolute held-out pass@1, and the median likelihood shift

|\Lambda|

. Both behavioural axes fall as the edit grows while

|\Lambda|

rises. Gold appears on the first two panels only: it keeps none of the attempt, so its edit count understates how far it sits, and its

|\Lambda|

3969

is off the scale.

Figure 5: RQ3: online refresh against a static pool , pooled over all model

\times

benchmark cells and

3

seeds. Both arms take minimal edits from the same black-box projector; the static pool mines them once against the base checkpoint and reuses them for the run, the regime of mined-once correction methods. Refreshing holds retention at base throughout, while the pool climbs briefly above base and then falls to

0.60

Figure 6: RQ4: loss recipes. Specialization of each recipe as a fraction of diff-local DPO

+

self-SFT, the recipe adopted in Sec. 3 . Qwen3-8B, GSM8K hard set, mean and stderr on

3

seeds. The adopted recipe is orange , the other contrastive arms amber , and the arms without a contrast violet . Held-out retention stays between

0.88

and

0.94

for every arm, so the ordering reflects specialization and not differences in forgetting. SFT on the projection carries a wider spread across seeds.

Figure 7: RQ5: refining the projector at test time. A frozen Qwen3-32B projector iteratively refines its own correction and the Qwen3-8B student trains on the result (GSM8K hard set,

K\in\{0,1,2\}

3

seeds), with the running correction rate on the left and edit size and off-policyness against the refinement budget on the right. More refinement corrects more wrong attempts with a smaller edit, and the projector is never finetuned.

Projector	Hard set	Held-out
none (untrained student)	0.000	$\approx$ 0.955
the student itself	0.219 $\pm$ 0.012	0.924 $\pm$ 0.021
gold solution	0.489	0.744
black-box projector	0.560	0.916

Table 4: RQ6: a self-projector recovers about

40\%

of the specialization. Qwen3-8B on the GSM8K hard set,

250

updates, mean

\pm

s.e. over

3

seeds. The self-projector is a frozen copy of the student’s own base weights, given the verified answer. Gold replaces the attempt with the dataset’s reference solution. Further arms and the full setup are in Appendix H .

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Problem, student, verifier ( Sec. 3 )
$x$	problem / prompt (drawn from a problem distribution)
$y\in\mathcal{V}^{\leq H}$	answer: a token sequence of length $\leq H$
$\pi_{\theta_{S}}$	student policy (autoregressive, parameters $\theta_{S}$ )
$\pi_{\mathrm{ref}}=\pi_{\theta_{S,0}}$	frozen initial student, used as the DPO reference
$R(x,y)\in\{0,1\}$	verifier (marks an answer correct or not)
$\mathcal{C}(x)$	set of correct answers $\{y:R(x,y)=1\}$

Appendix

Table 5: Notation reference.

Panel of Figure 8 — Figure 8: A minimal-edit projection on GSM8K ( Cindy’s wishing pond , gold answer $200$ ). The student’s wrong attempt (a) and the projector’s correction (b) are token-identical except for one inserted step and the final answer line, which changes from $225$ to $200$ . Only the changed spans $E^{-}\!\to\!E^{+}$ enter the contrast in ( 1 ).

Benchmark	Screened	Hard set	Hard %	Base sample acc.	Eval slice
GSM8K	7,473	122	1.63	0.949	200
MATH	12,000	1,991	16.59	0.703	500
SciKnowEval	3,721	1,003	26.96	0.323	400
ToolAlpaca	3,584	467	13.03	0.845	300
LiveCodeBench	—	100	—	—	50

Appendix

Table 6: Hard-set mining for Qwen3-8B. A problem enters the hard set when all

16

samples drawn from the untrained student at

T{=}0.8

are wrong. “Base sample accuracy” is the fraction of all

16\times

(screened) samples that were correct, i.e. the student’s accuracy over the pool before any training. The evaluation slice is disjoint from the hard set and comes from the benchmark’s test split.

Benchmark	Hard set	Eval slice
GSM8K	230	200
MATH	4,046	200
SciKnowEval	493	200
ToolAlpaca	471	200
LiveCodeBench	810	131

Appendix

Table 7: Hard-set sizes for OLMo-2-7B. The mining summary files were not retained for these runs, so the screened pool size and base sample accuracy are unavailable; the split sizes are taken directly from the mined files.

Figure 9: Per-benchmark specialization–retention Pareto frontier (detailed breakdown of Fig. 3 ). Top: same-family (

32

B) projectors; bottom: black-box projector. Each block has two rows (Qwen3-8B, OLMo-2-7B) and one column per benchmark. Axes: hard-set pass@1 (

x

, specialization) and held-out pass@1 (

y

, retention). Bold line

=

monotone Pareto frontier of the seed-averaged trajectory (larger marker

=

final checkpoint) anchored at the untrained base

\star

(specialization

0

); axes auto-scale per panel. GRPO on Qwen-LiveCodeBench is entered as no learning.

Figure 10: RQ1 pooled learning curves. Top: same-family (

32

B) projectors; bottom: black-box projector. Left panels: specialization (mean hard-set pass@1) over training. Right panels: retention as a fraction of the untrained base (base

=1.0

dashed). Curves pool over students, benchmarks, and seeds (same pooling as Fig. 3 ); shaded bands are s.e. over the pooled runs. The early specialization and subsequent retention loss of SPOT and SODA (retention falls by step

50

) and the retention preservation of both Ours variants are visible directly. GRPO on Qwen-LiveCodeBench entered as no learning.

Student	Benchmark	Method	Hard $\uparrow$	Held $\uparrow$
Qwen3-8B	GSM8K	GRPO	0.044 $\pm$ 0.012	0.950 $\pm$ 0.006
		OPD	0.123 $\pm$ 0.013	0.953 $\pm$ 0.007
		Ours (32B)	0.197 $\pm$ 0.019	0.943 $\pm$ 0.019
		SPOT	0.090 $\pm$ 0.005	0.216 $\pm$ 0.039
		SODA	0.246 $\pm$ 0.021	0.604 $\pm$ 0.140
		Ours (bb)	0.273 $\pm$ 0.022	0.943 $\pm$ 0.009

Appendix

Table 8: RQ1 per-cell final accuracies at the

250

-step checkpoint, mean

\pm

s.e. across seeds. Hard

=

hard-set pass@1 (specialization); Held

=

held-out pass@1 (in-domain retention), measured on a disjoint slice of the same benchmark’s training split, with sizes in Tabs. 6 and 7 . These slices are not the public test suites used in Tab. 11 , so the two are not comparable cell by cell. Higher is better on both, and a method is preferable where it raises Hard without lowering Held. Bold

=

best on each axis within a (student, benchmark) block. Our two variants take the best Hard in

6

9

blocks and the best Held in

5

9

, so the AUPF ordering in Tab. 1 rests on the two axes together rather than on either alone. † This is the one cell reported from a single training seed: OLMo-2-7B/MATH GRPO seeds

0

and

1

stopped with CUDA out-of-memory before the

250

-step checkpoint, so only seed

2

is reported and the

\pm

there is the standard error of one value, not of three. Every other cell is three seeds.

		RLVR	same-family $32$ B teacher		black-box projector
Student	Benchmark	GRPO	OPD	Ours	SPOT	SODA	Ours
Qwen3-8B	GSM8K	$-0.017$ $\pm$ 0.006	$\mathbf{-0.013}$ $\pm$ 0.007	$-0.023$ $\pm$ 0.019	$-0.751$ $\pm$ 0.039	$-0.362$ $\pm$ 0.140	$-0.023$ $\pm$ 0.009
	MATH	$+0.003$ $\pm$ 0.003	$+0.010$ $\pm$ 0.017	$+0.077$ $\pm$ 0.015	$-0.366$ $\pm$ 0.028	$+0.041$ $\pm$ 0.002	$\mathbf{+0.137}$ $\pm$ 0.006
	LiveCodeBench	$+0.000$ $\pm$ 0.000	$-0.004$ $\pm$ 0.018	$+0.016$ $\pm$ 0.013	$\mathbf{+0.042}$ $\pm$ 0.027	$-0.058$ $\pm$ 0.041	$-0.018$ $\pm$ 0.007
	SciKnowEval	$+0.007$ $\pm$ 0.000	$+0.013$ $\pm$ 0.026	$\mathbf{+0.053}$ $\pm$ 0.022	$+0.042$ $\pm$ 0.004	$+0.020$ $\pm$ 0.037	$-0.057$ $\pm$ 0.131
	ToolAlpaca	$\mathbf{-0.013}$ $\pm$ 0.009	$-0.487$ $\pm$ 0.274	$-0.053$ $\pm$ 0.015	$-0.272$ $\pm$ 0.073	$-0.259$ $\pm$ 0.067	$-0.103$ $\pm$ 0.023

Appendix

Table 9: RQ1 per-cell backward transfer ,

\mathrm{BWT}=y_{T}-H_{0}

( Lopez-Paz and Ranzato, 2017 ) , mean

\pm

s.e. across seeds in raw retention points. Higher is better: a positive value means the run ended above the untrained base, a negative one that it ended below, and zero that it returned to base. Bold

=

highest per row. Our variants take the highest value in

5

9

cells and SPOT the lowest in

5

. Every method ends below base on GSM8K and ToolAlpaca for both students, so those two benchmarks carry the retention cost regardless of how the correction is obtained. GRPO on Qwen-LiveCodeBench is entered as no learning.

		RLVR	same-family $32$ B teacher		black-box projector
Student	Benchmark	GRPO	OPD	Ours	SPOT	SODA	Ours
Qwen3-8B	GSM8K	0.020 $\pm$ 0.006	0.016 $\pm$ 0.007	0.030 $\pm$ 0.015	0.751 $\pm$ 0.039	0.362 $\pm$ 0.140	0.024 $\pm$ 0.009
	MATH	0.001 $\pm$ 0.001	0.030 $\pm$ 0.012	0.027 $\pm$ 0.015	0.533 $\pm$ 0.030	0.133 $\pm$ 0.004	0.010 $\pm$ 0.006
	LiveCodeBench	0.000 $\pm$ 0.000	0.073 $\pm$ 0.029	0.047 $\pm$ 0.018	0.040 $\pm$ 0.020	0.070 $\pm$ 0.038	0.053 $\pm$ 0.007
	SciKnowEval	0.040 $\pm$ 0.010	0.047 $\pm$ 0.026	0.023 $\pm$ 0.012	0.084 $\pm$ 0.009	0.109 $\pm$ 0.046	0.153 $\pm$ 0.129
	ToolAlpaca	0.013 $\pm$ 0.009	0.490 $\pm$ 0.272	0.053 $\pm$ 0.015	0.272 $\pm$ 0.073	0.259 $\pm$ 0.067	0.103 $\pm$ 0.023

Appendix

Table 10: RQ1 per-cell forgetting ,

F=\max_{\tau\leq T}y_{\tau}-y_{T}

with

H_{0}

included in the max ( Chaudhry et al., 2018 ) , mean

\pm

s.e. across seeds in raw retention points. Smaller is better:

F

is the retention a run gave up between its best checkpoint and its last, so a small value means retention was held throughout rather than lost and partly regained. Bold

=

smallest per row. SPOT is largest in

5

9

cells. The smallest is split evenly between OPD, GRPO and our same-family variant, so

F

alone does not separate methods and should be read alongside specialization, since a run that moves the student little also has little to give up. GRPO on Qwen-LiveCodeBench is entered as no learning.

Method	GSM8K	MATH-500	AMC23	MMLU	HumanEval
Base (Qwen3-8B)	0.934 $\pm$ 0.003	0.709 $\pm$ 0.005	0.433 $\pm$ 0.030	0.702 $\pm$ 0.002	0.821 $\pm$ 0.002
Ours	0.915 $\pm$ 0.001	0.749 $\pm$ 0.003	0.492 $\pm$ 0.036	0.765 $\pm$ 0.001	0.787 $\pm$ 0.000
Ours -32B	0.917 $\pm$ 0.001	0.615 $\pm$ 0.005	0.400 $\pm$ 0.050	0.724 $\pm$ 0.001	0.827 $\pm$ 0.005
Soda	0.882 $\pm$ 0.004	0.590 $\pm$ 0.019	0.342 $\pm$ 0.022	0.766 $\pm$ 0.002	0.797 $\pm$ 0.005
Spot	0.916 $\pm$ 0.002	0.513 $\pm$ 0.011	0.325 $\pm$ 0.029	0.750 $\pm$ 0.002	0.813 $\pm$ 0.014
GRPO	0.932 $\pm$ 0.004	0.700 $\pm$ 0.008	0.425 $\pm$ 0.029	0.702 $\pm$ 0.002	0.821 $\pm$ 0.002

Appendix

Table 11: Training on math corrections suggests retention is held outside the trained domain. Final-checkpoint evaluation of the training-seed-

0

math-trained Qwen3-8B methods, sampled pass@1 at

T{=}0.8

averaged over three evaluation seeds,

\pm

the standard error of that mean, on full public test splits. These are distinct from the in-domain held-out slices that Tab. 8 uses for retention. GSM8K and MATH-500 are in the trained domain; AMC23, MMLU and HumanEval were never trained on. The black-box variant is the only arm whose score rises on the held-out split of the domain it trained on (

0.709\to 0.749

on MATH-500, where the same recipe with a

32

B projector reaches

0.615

, SPOT

0.513

and SODA

0.590

), and it reaches

0.765

on MMLU while remaining within a few points of base on GSM8K and HumanEval. Bold

=

best and underline

=

second best per column among the six trained methods; the untrained base is a reference row and is not ranked. GRPO ranks high on several columns because it moves the student least, as Tab. 12 makes plain, so these absolute scores are best read alongside that table. These are single checkpoints on one domain, so the pattern is suggestive rather than conclusive, and AMC23 has only

40

problems, where small differences should not be read as meaningful.

Method	GSM8K	MATH-500	AMC23	MMLU	HumanEval
Ours	$-0.019$	$\mathbf{+0.041}$	$\mathbf{+0.058}$	$+0.063$	$-0.035$
Ours -32B	$-0.017$	$-0.093$	$-0.033$	$+0.022$	$\mathbf{+0.006}$
Soda	$-0.052$	$-0.119$	$-0.092$	$\mathbf{+0.064}$	$-0.024$
Spot	$-0.018$	$-0.195$	$-0.108$	$+0.047$	$-0.008$
GRPO	$\mathbf{-0.002}$	$-0.009$	$-0.008$	$-0.001$	$+0.000$
OPD	$-0.006$	$-0.017$	$-0.025$	$-0.025$	$-0.006$

Appendix

Table 12: Only the black-box variant improves in the trained domain. Change relative to the untrained Qwen3-8B, from the same evaluation as Tab. 11 . Ours is alone in moving up on MATH-500 (

+0.041

) and on AMC23 (

+0.058

), and gains

+0.063

on MMLU. The same recipe with a

32

B projector loses

0.093

on MATH-500, and the mined-once and static-snapshot corrections lose

0.119

and

0.195

. GRPO is flat throughout and OPD moves down. Bold

=

largest change and underline

=

second largest per column; GRPO is near zero everywhere, so where it places second it has simply moved the least.

		Sampled pass@1			Greedy
Benchmark	Hard	Base $\to$ Ours	Change	pass@16	change
OlympiadBench	150	$0.011\to 0.017$	$+0.006$ $\pm$ 0.004	$0.133\to 0.147$	$+0.000$ $\pm$ 0.013
Omni-MATH	2,178	$0.010\to 0.019$	$\mathbf{+0.009}$ $\pm$ 0.001	$0.096\to 0.132$	$\mathbf{+0.015}$ $\pm$ 0.003
AIME 1983–2023	199	$0.016\to 0.013$	$-0.003$ $\pm$ 0.003	$0.146\to 0.126$	$-0.010$ $\pm$ 0.010
DAPO-Math	358	$0.018\to 0.033$	$\mathbf{+0.016}$ $\pm$ 0.004	$0.201\to 0.307$	$\mathbf{+0.034}$ $\pm$ 0.010
All four	2,885	$0.011\to 0.020$	$\mathbf{+0.009}$ $\pm$ 0.001	$0.114\to 0.154$	$\mathbf{+0.015}$ $\pm$ 0.003

Appendix

Table 13: Cross-benchmark gain of Ours at its best checkpoint, by benchmark , on each benchmark’s hard problems. Sampled pass@1 over

16

generations and one greedy decode; changes against the base

\pm

paired standard error.

	MATH held-out, hard		Omni-MATH held-out, hard		$3$ others
Trained on	pass@1	pass@16	pass@1	pass@16	hard
Base	0.013	0.150	0.009	0.092	0.016
MATH	0.056 ∗	0.354	0.018 ∗	0.128	0.024 ∗
Omni-MATH	0.045 ∗	0.362	0.022 ∗	0.146	0.030 ∗

Appendix

Table 14: Training on MATH against training on Omni-MATH, on identical problems (our method,

1

training seed each, each at its own best checkpoint). “

3

others”

=

OlympiadBench, AIME 1983–2023 and DAPO-Math, which neither run trained on, pooled over their hard problems. Sampled pass@1 unless marked. Each run’s ∗ is against its own evaluation of the base model; the two base evaluations agree to within

0.001

in pass@1 and

0.008

in pass@16.

Arm	$\|E^{+}\|\!+\!\|E^{-}\|$	$\|E^{+}\|$ / $\|E^{-}\|$	hard	held	$\Lambda$
minimal (ours)	$276$	$160$ / $116$	$\mathbf{0.560}$ $\pm$ 0.015	$\mathbf{0.916}$ $\pm$ 0.016	$\mathbf{-86}$ $\pm$ 4
$3.2\times$	$879$	$308$ / $572$	$0.301$ $\pm$ 0.065	$\underline{0.847}$ $\pm$ 0.067	$\underline{-372}$ $\pm$ 67
$3.9\times$	$1061$	$359$ / $702$	$0.254$ $\pm$ 0.021	$0.807$ $\pm$ 0.069	$-433$ $\pm$ 35
gold (anchor)	$926$	$85$ / $841$	$\underline{0.489}$ $\pm$ 0.090	$0.744$ $\pm$ 0.016	$+3969$ $\pm$ 2471 †

Appendix

Table 15: RQ2 edit-size ablation (Qwen3-8B, GSM8K hard set,

3

seeds, mean

\pm

s.e.). hard and held are final sampled pass@1;

\Lambda

is the median signed likelihood shift over training under the current student (( 5 )); a negative value means the correction is less likely than the attempt.

|E^{+}|,|E^{-}|

are mean edit-span sizes in tokens, and

|E^{+}|+|E^{-}|

is the measured edit size by which the arms are identified. In the three outcome columns bold

=

best and underline

=

second best, with

\Lambda

read as nearer zero is better; the two edit-size columns describe the arms and are not ranked. † gold is the one arm whose shift is positive: it replaces a long attempt with a short reference solution, which the student finds more likely, not less. Its value moves chiefly through length and is not on the scale of the rewrites; its standard error is

62\%

of the value against

5

18\%

for the edit arms.

Figure 11: Full-utterance against diff-localized likelihood shift. Left: every logged step of the three edit arms, on symmetric log axes with the identity line dotted. Right: median

|\Lambda|

per arm under both measurements, with the localized value as a percentage of the full one.

Figure 12: Per-step edit size against the likelihood shift it demands, for the three edit arms (one point per logged step,

n=2249

). Colored lines are per-arm Theil–Sen fits with their slopes; the dashed black line joins the pooled decile medians. Axes are clipped at the

98

th/

99.5

th percentile, so a few staleness-affected steps fall outside the view.

Arm	$\|E^{+}\|$	$\|E^{-}\|$	token diff	length change	$m$
minimal (ours)	$160$	$116$	$276$	$44$	$319$
$3.2\times$	$308$	$572$	$879$	$264$	$1144$
$3.9\times$	$359$	$702$	$1061$	$343$	$1405$
gold	$85$	$841$	$926$	$755$	$1681$

Appendix

Table 16: Token diff against the edit size of Definition 4 . The token diff

|E^{+}|+|E^{-}|

is what Sec. 4.2 reports. The edit size

m

counts over the

H

decisions Definition 2 gives both rollouts, so it also carries the padding that equalizes their lengths:

m=|E^{+}|+|E^{-}|+\big||E^{+}|-|E^{-}|\big|

. Means over training steps and

3

seeds, rounded. The two measures order the arms the same way except for gold , which is intermediate under the token diff and the largest edit of any arm under

m

Figure 13: Insertion-to-deletion balance per arm, per-step median with interquartile range on a logarithmic axis. The dotted line marks equal insertion and deletion.

Figure 14: RQ2 specialization–retention plane (

3

seeds; faint markers are individual seeds). The minimal edit dominates both larger rewrites on both axes; gold is dominated on retention.

Figure 15: Held-out accuracy over training (mean

\pm

s.e. over

3

seeds). The edit arms decline smoothly and in edit-size order; only gold destabilizes.

Arm	hard $\uparrow$	rel. $\uparrow$	held $\uparrow$
Diff-local DPO $+$ self-SFT	$\mathbf{0.601}$ $\pm$ 0.005	$\mathbf{1.000}$	$0.938$ $\pm$ 0.002
Full-sequence DPO $+$ self-SFT	$0.552$ $\pm$ 0.022	$0.918$	$0.911$ $\pm$ 0.014
SFT on projection $+$ self-SFT	$0.484$ $\pm$ 0.078	$0.805$	$0.877$ $\pm$ 0.054
Full-sequence DPO	$0.470$ $\pm$ 0.026	$0.782$	$0.902$ $\pm$ 0.002
Diff-local DPO	$0.440$ $\pm$ 0.015	$0.732$	$0.920$ $\pm$ 0.014
RFT	$0.096$ $\pm$ 0.011	$0.159$	$\mathbf{0.944}$ $\pm$ 0.002

Appendix

Table 17: RQ4 per-arm results on the GSM8K hard set with Qwen3-8B, mean

\pm

s.e. across

3

seeds. hard is hard-set pass@1 and rel. the same quantity as a fraction of the adopted recipe, which is also the best arm; held is held-out pass@1. Bold

=

best per column. SFT on the projection carries the widest spread on both axes.

	Corrections			Student
$K$	corr. (%) $\uparrow$	edit $\|E^{+}\|{+}\|E^{-}\|$ $\downarrow$	median $\|\Lambda\|$ $\downarrow$	hard $\uparrow$	held $\uparrow$
$0$	$9.2$ $\pm$ 0.2	$256$ $\pm$ 70	$38.1$ $\pm$ 1.8	$0.366$ $\pm$ 0.023	$0.911$ $\pm$ 0.014
$1$	$15.4$ $\pm$ 0.7	$182$ $\pm$ 16	$34.2$ $\pm$ 3.0	$\mathbf{0.440}$ $\pm$ 0.010	$0.909$ $\pm$ 0.010
$2$	$\mathbf{16.7}$ $\pm$ 1.1	$\mathbf{130}$ $\pm$ 24	$\mathbf{26.2}$ $\pm$ 4.8	$\mathbf{0.440}$ $\pm$ 0.021	$\mathbf{0.929}$ $\pm$ 0.006

Appendix

Table 18: RQ5 per-budget results on the GSM8K hard set with Qwen3-8B, mean

\pm

s.e. across

3

seeds. The first three columns describe the corrections the frozen projector supplies, averaged over training; the last two the student that learns from them, at the final checkpoint. corr. is the running correction rate

\sum_{t}n^{\mathrm{corr}}_{t}/\sum_{t}n^{\mathrm{wrong}}_{t}

plotted in Fig. 7 (left), and edit and median

|\Lambda|

are the two bars in Fig. 7 (right). Bold

=

best per column. Every column describing the corrections improves monotonically with the budget; downstream, the specialization gain arrives in the first round.

Figure 16: Specialization–retention frontiers over training for

K\in\{0,1,2\}

(

3

seeds pooled, frontier over the pooled points), gold-known, with the shared base marked

\star

. More refinement extends the frontier to higher specialization, and the

K{=}2

frontier dominates the other two: it is the only one that rises above the base in retention. Note the narrow retention axis, which spans

0.925

–

0.960

Figure 17: Per-step correction rate over training (pooled over

3

seeds,

11

-step moving average), gold-known,

K\in\{0,1,2\}

. The rate peaks early and declines as only the hardest problems remain uncorrected; more refinement holds it higher throughout.

Teacher	Correction enters as	Hard set	Held-out
none (base student)	–	0.000	$\approx$ 0.955
the student itself	edited text	0.219 $\pm$ 0.012	0.924 $\pm$ 0.021
the student itself	token distribution	0.046 $\pm$ 0.010	0.937 $\pm$ 0.003
gold solution	full replacement	0.489	0.744
black-box projector	minimal edit	0.560	0.916

Appendix

Table 19: Self-projection recovers about

40\%

of the specialization. Qwen3-8B on the GSM8K hard set,

250

updates, mean

\pm

s.e. over

3

seeds. The two middle rows use the student’s own frozen base weights as the teacher. The reference rows come from Appendix E , on the same hard set and budget.

	MATH(3,4,5)	AMC	MATH-500	GSM8K	Avg.	Chemistry	MMLU
Base (reported)	0.315	0.133	0.245	0.579	0.318	0.283	0.651
SFT	0.243	0.100	0.168	0.455	0.241	0.222	0.648
OPSD ( Zhao et al., 2026 )	0.267	0.084	0.332	0.524	0.302	0.242	0.652
GRPO	0.457	0.249	0.313	0.808	0.457	0.278	0.652
UFT ( Liu et al., 2025 )	0.470	0.293	0.297	0.746	0.452	0.283	0.653
Sampling SFT	0.495	0.277	0.582	0.782	0.534	0.266	0.651

Appendix

Table 20: Black-box projection matches self-projection in the math setting of Karan et al. (2026) (Qwen2.5-3B). Upper block as reported there, not rerun; lower block ours, evaluated with its released scripts and protocol (one sample per problem). Avg. is the mean of the four math columns. ∗

=

paired change of at least two standard errors against our own evaluation of the untrained student (math and chemistry columns).

Arm	Student FLOPs	Against Sampling SFT
Sampling SFT	$8.4\times 10^{18}$	$1\times$
Sampling SFT + RL	$2.1\times 10^{19}$	–
Black-box projection + SFT	$2.8\times 10^{17}$	$30\times$ less
Black-box projection + SFT + RL	$1.3\times 10^{19}$	–
Ours, static pool	$5.3\times 10^{17}$	$16\times$ less
Ours, online refresh	$6.9\times 10^{16}$	$121\times$ less

Appendix

Table 21: Training compute of each arm. Student FLOPs use

2N

per forward token,

6N

per training token and

12N

per RL token (

N{=}3.09

B; attention’s context term omitted, about

5\%

at these lengths). The MCMC sampler is counted with prefix caching, which is the accounting most favourable to it; counted from its released implementation without caching it costs

1.3\times 10^{19}

and the ratios become

46\times

24\times

and

184\times

Language Model Distillation

Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.

See all 508 papers

Oct 13, 2026

Correcting or Continuing? What On-Policy Distillation Learns from Student Errors

On-policy distillation (OPD) trains a student on its own rollouts under a frozen teacher, mitigating the distribution mismatch of off-policy supervision. While prior work has primarily trained on problems a student can largely solve, we ask whether OPD can improve a student when training is restricted to problems it currently fails. We evaluate five representative OPD mechanisms with Qwen-3 and Phi-4 models under a unified competition-mathematics protocol. We find that OPD does not improve accuracy on this hard subset, with most mechanisms instead degrading performance despite continued decreases in training KL divergence. All eight configurations show a decline, and the two-sided sign test gives p=0.0078. In contrast, the same pipeline improves accuracy by 4.6 percentage points when training is not concentrated on failures, showing that the failure is conditional on the training regime rather than systematic. These findings suggest that OPD is effective when supervision is dominated by correct student prefixes, but can reinforce existing errors when training is concentrated on problems the student currently fails.

Shuo Yang, Zheyu Zhang, Mahito Sugiyama +1

Technical University of Munich · National Institute of Informatics · Munich Center for Machine Learning

Language Model Distillation On-Policy Distillation

Ask PDF

Mar 26, 2026

Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes

On-policy distillation (OPD) is increasingly used in LLM post-training because it can leverage a teacher model to provide dense supervision on student rollouts. The standard implementation, however, usually reduces distribution matching to a sampled-token log-ratio, which can make the learning signal fragile on long rollouts whose prefixes drift away from the teacher's typical support. We revisit this formulation from both theoretical and implementation perspectives. Theoretically, token-level OPD is biased relative to sequence-level reverse-KL minimization, but admits a substantially tighter worst-case variance bound; a controlled synthetic study further shows that stronger future-reward coupling increases gradient variance and destabilizes training. Empirically, we identify three failure modes of sampled-token OPD: imbalanced token-level supervision, unreliable teacher guidance on student-generated prefixes, and tokenizer or special-token mismatch. These findings motivate teacher top-K local support matching, a truncated reverse-KL objective that compares teacher and student distributions over a teacher-supported token set at each prefix, together with top-p rollout sampling and special-token masking. Across single-task reasoning and multi-task benchmarks spanning agentic and reasoning settings, this objective improves optimization stability and yields a +19.8% performance gain over standard sampled-token OPD baselines, providing a practical recipe for more stable on-policy distillation.

Yuqian Fu, Haohuan Huang, Kaiwen Jiang +4

State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA · School of Artificial Intelligence, UCAS · Fudan University +1

Language Model Distillation On-Policy Distillation

Ask PDF

Oct 14, 2026

Recovering Reliable Supervision via Critique-Guided On-Policy Distillation

On-policy distillation (OPD) trains a student model on its own rollouts with token-level supervision from a teacher, assuming that the teacher can reach a correct answer from any prefix the student writes. We call this property the recoverability of the teacher and show it fails on incorrect rollouts: the teacher follows student errors, although it recognizes them. We propose CriticOPD, using error recognition to restore recoverability. The teacher critiques an incorrect rollout, allowing the student to rewrite its solution following the error. The resulting output, including the critique as privileged information, is scored by the teacher while the student uses the revision without the critique for training. On two Qwen3 students and six mathematical reasoning benchmarks, CriticOPD improves OPD, with the largest gains on competition problems.

Kaixiang Zhao, Tianrun Yu, Shangzhe Li +4

Brigham Young University · University of North Carolina at Chapel Hill

On-Policy Distillation Knowledge Distillation LLM Self-Correction

Ask PDF

Abstract

Language Model Distillation

Explore similar work

Correcting or Continuing? What On-Policy Distillation Learns from Student Errors

Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes

Recovering Reliable Supervision via Critique-Guided On-Policy Distillation