Demystifying On-Policy Distillation: Where Does the Teacher's Advantage Live?

Yiping Li; Fan Zhao; Zhibo Hou; Zhiyu An; Wan Du

cs.LGOct 13, 2026

Demystifying On-Policy Distillation: Where Does the Teacher's Advantage Live?

Authors: Yiping Li, Fan Zhao, Zhibo Hou, Zhiyu An, Wan Du

Organizations: University of California, Merced, USA

Abstract

On-policy distillation (OPD) is increasingly studied as a cheaper alternative to reinforcement learning, yet it remains a form of knowledge distillation (KD) that relies on teacher supervision. We revisit OPD from the KD perspective by asking where the teacher's advantage lies along student-generated trajectories. Controlled token flips at teacher-student disagreements directly measure the accuracy gained from teacher corrections. Single-token flips yield a modest average gain on selected failed trajectories, whereas repeated flips raise accuracy from 28.4% to 74.5%, close to the teacher's 76.9%. Gains continue to accumulate with more flips, but mass- and KL-based selection outperform random flips by 15-18 percentage points at counts of similar magnitude. Selective intervention remains useful after OPD, but greater argmax agreement does not consistently improve accuracy across model pairs. We advocate training schedules that learn more valuable teacher corrections while maintaining distributional stability. Controlled flips and fork-resolution analysis evaluate OPD beyond accuracy by measuring which teacher preferences the student acquires and what further correction can recover.

Figures & tables

		Flip $-$ Resample				Net Gain $-$ Rest
		Trajectory		Problem		Trajectory		Problem
Fork Group	Share (%)	Gain	$p_{T}$	Gain	$p_{P}$	$\Delta$	$p_{T}$	$\Delta$	$p_{P}$
All Forks	100.0	+1.41 ∗∗∗	$<0.001$	+1.22 ∗	0.015	–	–	–	–
$\mu_{t}^{S}<0.05$	53.9	+2.06 ∗∗∗	$<0.001$	+1.83 ∗∗	0.007	+1.40 ∗∗∗	$<0.001$	+1.35 ∗∗	0.004
Blind	9.0	+5.27 ∗∗∗	$<0.001$	+4.27 ∗	0.012	+4.08 ∗∗∗	$<0.001$	+3.31 ∗	0.015
Reflection/Contrast	6.6	+6.59 ∗	0.045	+5.11	0.116	+1.84 ∗∗	0.008	+3.84	0.189

Table 1: Single-flip gains over resampling and net-gain differences versus remaining positions, in pp. Each estimate is paired with its test at the same aggregation level. Share uses 5,901 eligible forks. Group tests are unadjusted. Full results and eligibility rules appear in Tables 7 and B.1 .

Figure 1: Accuracy versus mean flips per trajectory. Point labels indicate selection thresholds. Full results are in Table 13 .

Flipped	Flips	Introduced Reflection Words	Length	Accuracy (%)
Student alone	0	2.5	1,345	28.4
random 6%	11	3.2	1,506	32.9
reflection words	14	8.4	2,156	34.9
$\mu^{T}_{t}<10^{-5}$ ∗∗	4	3.1	1,594	38.4

Table 2: Reflection-word use, trajectory length, and accuracy under sparse intervention. Reflection counts exclude teacher-inserted tokens. Significance compares accuracy with random flips.

Figure 2: Fork changes on pre-training prefixes (left, % of original forks) and accuracy on each checkpoint’s own rollouts (right).

Figure 3: Fork resolution by pre-training teacher mass and reverse KL. Colors show within-group rates, right-hand values show final rates, and parentheses show shares of original forks.

Figure 4: Mass selection and random flips at three OPD checkpoints. Labels give mass thresholds or random probabilities. Gray lines mark standalone accuracy. Each point uses 800 trajectories.

Figure 5: Entropy on fixed pre-training prefixes (purple, left axes) and accuracy on checkpoint rollouts (gray, right axes). Teacher entropy is dotted. Post-answer repetition is excluded. All six settings appear in Section H.4 .

Fork Group	F/T	F/F	Acc. (%)	$p$
Random	96.6	20.4	57.13	–
Old	100.0	19.7	56.75	0.837
Unchanged	94.5	18.5	57.63	0.761
Random	28.9	5.9	51.00	–
New	31.4	6.8	44.88 ∗∗∗	$<0.001$
Random	52.2	10.8	50.88	–

Table 3: Fork-group interventions at step 16. F/T is flips per trajectory and F/F is the percentage of forks flipped.

p

compares accuracy with Random in each block ( Section B.2 ).

Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Reverse KL on 5.24M base-model positions. (a) Distributions where both argmaxes and the sampled token agree, and at forks where the sampled token differs from the teacher’s argmax. Bars use all positions as the denominator. Other sampling outcomes are omitted. (b) Cumulative loss over all positions ranked by reverse KL.

Figure 7: Token choices at 640k argmax forks where the sampled token also differs from the teacher’s choice. (a,b) Mean probabilities under student and teacher token rankings. (c,d) Rank of the other model’s argmax, with “out” denoting a token outside the top-16.

$L$	$n$	Gain (pp)	95% CI	$p_{T}$	$p_{P}$
1	5901	+1.41	[0.66, 2.15]	$<0.001$	0.015
16	5926	+3.25	[2.15, 4.36]	$<0.001$	0.004
128	5926	+16.51	[12.82, 20.20]	$<0.001$	$<0.001$
$\infty$	5926	+79.68	[74.25, 85.10]	$<0.001$	$<0.001$

Appendix

Table 4: Overall paired gains over the complete resample baseline. Effects and Student-

t

95% intervals are trajectory-weighted, in pp.

p_{T}

tests trajectory means and

p_{P}

tests problem means.

n

is the number of eligible positions ( Section B.1 ).

Group	$n_{L=1}$	$L=1$	$L=16$	$L=128$	$L=\infty$
$0.00\leq\mu_{t}^{T}<0.05$	2774	+2.05	+3.84	+17.17	+78.67 ∗
$0.05\leq\mu_{t}^{T}<0.10$	733	+0.60	+2.22	+14.42 ∗	+81.24
$0.10\leq\mu_{t}^{T}<0.20$	1007	+0.03 ∗	+2.79	+17.63	+80.58
$0.20\leq\mu_{t}^{T}<0.30$	706	+1.55	+2.79	+16.66	+80.83
$0.30\leq\mu_{t}^{T}<0.50$	680	+0.93	+3.02	+14.16 ∗	+79.19
$0.00\leq\mu_{t}^{S}<0.05$	3179	+2.06 ∗∗∗	+3.57	+16.85	+78.97

Appendix

Table 5: Trajectory-weighted gains over resampling by fork group and segment length, in pp.

n_{L=1}

counts eligible changed-token positions. Asterisks compare each group with remaining positions, not with zero, using the trajectory test.

Group	$p_{T}^{16}$	$p_{P}^{16}$	$p_{T}^{128}$	$p_{P}^{128}$	$p_{T}^{\infty}$	$p_{P}^{\infty}$
$0.00\leq\mu_{t}^{T}<0.05$	0.198	0.262	0.058	0.006	0.029	0.202
$0.05\leq\mu_{t}^{T}<0.10$	0.218	0.266	0.049	0.048	0.079	0.356
$0.10\leq\mu_{t}^{T}<0.20$	0.368	0.693	0.186	0.336	0.237	0.029
$0.20\leq\mu_{t}^{T}<0.30$	0.497	0.489	0.900	0.908	0.330	0.422
$0.30\leq\mu_{t}^{T}<0.50$	0.723	0.334	0.030	0.048	0.655	0.828
$0.00\leq\mu_{t}^{S}<0.05$	0.301	0.456	0.270	0.141	0.055	0.108

Appendix

Table 6: Unadjusted group-versus-rest

p

-values for longer teacher segments. Superscripts give segment length. Subscripts give trajectory (

T

) or problem (

P

) aggregation.

		Flip $-$ Resample				Net Gain $-$ Rest
		Trajectory		Problem		Trajectory		Problem
Fork Group	Share (%)	Gain	$p_{T}$	Gain	$p_{P}$	$\Delta$	$p_{T}$	$\Delta$	$p_{P}$	$n_{T}/n_{P}$
All Forks	100.0	+1.41 ∗∗∗	$<0.001$	+1.22 ∗	0.015	–	–	–	–	–
$0.00\leq\mu_{t}^{T}<0.05$	47.0	+2.05 ∗∗∗	$<0.001$	+1.94 ∗	0.032	+1.17	0.057	+1.25	0.145	40/10
$0.05\leq\mu_{t}^{T}<0.10$	12.4	+0.60	0.133	+0.69	0.059	-0.88	0.102	-0.60	0.244	40/10
$0.10\leq\mu_{t}^{T}<0.20$	17.1	+0.03	0.961	+0.04	0.941	-1.56 ∗	0.021	-1.37	0.106	40/10

Appendix

Table 7: Complete single-flip comparisons, with effects in pp and shares of 5,901 eligible forks. Within-group gains use units with eligible positions. Group-versus-rest tests require at least three positions per side.

n_{T}/n_{P}

counts eligible trajectories/problems for these contrasts. Tests are unadjusted ( Section B.1 ).

Fork group	$n$	Flip (%)	Resample (%)	Gain [95% CI] (pp)
All eligible forks	5,901	11.31	9.91	+1.41 [0.66, 2.15]
$\mu_{t}^{S}<0.05$	3,179	11.22	9.16	+2.06 [1.13, 2.99]
$\mu_{t}^{S}\geq 0.05$	2,722	11.50	10.84	+0.66 [0.02, 1.30]
Blind	524	10.19	5.35	+4.84 [2.29, 7.39]
Not blind	5,146	9.56	8.80	+0.77 [0.19, 1.34]
Reflection/contrast	383	8.57	5.79	+2.78 [1.15, 4.41]

Appendix

Table 8: Single-token replacement and resampling success rates. Gain is flip minus resample success at the same positions. Each group pair uses trajectories with at least three eligible positions per side. Rates, gains, and 95% intervals use equal trajectory weights. Group contrasts are reported in Tables 1 and 7 .

Group	Feature	AUC [95% CI]
Token mass	$\mu_{t}^{T}$	0.512 [0.453, 0.537]
	$\mu_{t}^{S}$	0.539 [0.503, 0.557]
Confidence	student	0.480 [0.435, 0.510]
	teacher	0.488 [0.434, 0.517]
Entropy	student	0.486 [0.440, 0.510]
	teacher	0.498 [0.444, 0.519]

Appendix

Table 9: Cross-validated AUC for predicting whether a single flip beats resampling, on 2,598 non-tied forks. Intervals use trajectory-bootstrap refits.

Comparison	Gain [95% CI] (pp)	$p_{T}$	$p_{P}$
True flip $-$ keep	+1.55 [0.79, 2.32]	0.00020	0.01697
Resample $-$ keep	+0.14 [-0.22, 0.50]	0.43082	0.31733

Appendix

Table 10: Keep-token baseline checks. Gains and Student-

t

95% intervals are trajectory-weighted.

p_{T}

and

p_{P}

use trajectory and problem units, respectively, as in Section B.1 .

Figure 8: Success by fork position and teacher-segment length, averaged over positions in each fifth. The vertical axis omits 38–65%.

L=1

retains changed-token samples. Longer segments use the original continuation runs ( App. A ).

Figure 9: Gain over the complete resample baseline by position decile and teacher-segment length. Paired position differences are averaged within each trajectory and then across trajectories. At

L=1

only paths changing the original token are included.

Position	Resample	$L=1$	$L=16$	$L=128$	$L=\infty$
All	0.103	0.113	0.134	0.270	0.887
0.0–0.2	0.163	0.173	0.200	0.332	0.944
0.2–0.4	0.153	0.158	0.171	0.305	0.949
0.4–0.6	0.096	0.114	0.135	0.302	0.920
0.6–0.8	0.073	0.080	0.101	0.238	0.908
0.8–1.0	0.047	0.060	0.081	0.196	0.748

Appendix

Table 11: Position-averaged success rates by trajectory fifth. Resample uses all positions, and

L=1

uses positions with an actual token change. Paired gains and group tests are reported separately in Tables 5 and 7 .

Figure 10: Fork spacing on teacher-generated trajectories. (a) Within-trajectory distances between successive forks. (b) Lengths of maximal consecutive fork runs. Percentages use gaps and runs as their respective denominators.

Figure 11: Token-level diagnostics after a single flip. Each point averages four offsets, with up to 5,901 changed-token forks and 1,495 unchanged-token controls. (a) Student entropy after changed and unchanged tokens. (b) Both models’ entropy on the teacher’s continuation. (c) Their probabilities of the teacher’s tokens.

Trigger	Acc. (%)	F/T	F/F (%)
$\mu^{T}_{t}<0.1$ ∗∗∗	71.0	199	46
Random ( $p=0.46$ )	57.3	157	45
$\mu^{T}_{t}<0.05$ ∗∗∗	65.2	143	35
Random ( $p=0.35$ )	52.0	103	34
$\mu^{T}_{t}<10^{-5}$ ∗∗	38.4	4	2
Random ( $p=0.06$ )	32.9	11	6

Appendix

Table 12: Selected dual-track controls. F/T is mean flips per trajectory and F/F is the percentage of forks flipped. Test details are in Section B.2 .

Trigger	Acc. (%)	F/T
$\mu_{t}^{T}<10^{-5}$	38.38	4.11
$\mu_{t}^{T}<10^{-4}$	42.88	10.27
$\mu_{t}^{T}<10^{-3}$	51.50	25.75
$\mu_{t}^{T}<0.05$	65.25	142.50
$\mu_{t}^{T}<0.1$	71.00	199.01
$\mathrm{KL}>10$	35.12	3.06

Appendix

Table 13: All points in Figure 1 . Each uses 200 problems with 4 trajectories each. F/T counts actual replacements per trajectory. Random controls are shaded.

Figure 12: Accuracy after teacher exit at fraction

f

(left axis) and gain over the student alone per preceding flip, in pp (right). Dotted lines mark standalone student and teacher accuracy.

Figure 13: Accuracy versus the percentage of forks flipped for three teacher–student pairs, with 200 problems and four trajectories per point. Dotted lines mark standalone baselines. MAX estimates the 100% count using each pair’s highest-coverage random run.

Figure 14: Strict prompt accuracy on official IFEval versus the percentage of forks flipped ( Section A.4.3 ). Dotted lines mark standalone baselines. MAX estimates the 100% count using the highest-coverage random run.

Figure 15: Dual-track accuracy on DAPO-Math for Llama 3.2 3B to 1B and Gemma 3 12B to 4B ( Sections A.4.4 and A.4.5 ). Each point uses 200 problems and four trajectories per problem. Dotted lines show Llama’s same-problem baselines with an 8,192-token context limit versus 12,288 for interventions. MAX estimates the 100% count using each pair’s highest-coverage random run.

	$n$	Student	$\tau{=}0.05$	$\tau{=}0.1$	Teacher
S cannot, T can	308	0.065	0.575	0.666	0.821
Both can	388	0.505	0.871	0.905	0.933
Neither can	88	0.034	0.045	0.102	0.000
S can, T cannot	16	0.312	0.188	0.188	0.000

Appendix

Table 14: Accuracy by whether the student (S) and teacher (T) solve a problem on their own (pass@4

>0

n

counts trajectories.

Trigger	Argmax (%)	Sample (%)	Gain (pp)	$p$
$\mu_{t}^{T}<0.1$	71.00	71.88	+0.88	0.5893
$\mu_{t}^{T}<0.05$	65.25	69.00	+3.75	0.0087
$\mu_{t}^{T}<10^{-5}$	38.38	35.62	-2.75	0.1348

Appendix

Table 15: Teacher sampling versus argmax under the same mass trigger, on 200 problems with 4 trajectories each. Sampled replacements use temperature 0.7 and top-

p

0.95. Gain is sample minus argmax, and

p

is from a two-sided test paired over problems ( Section B.2 ).

Step	Accuracy (%)	Solved (%)	Unchanged (%)	Moved (%)	New (%)
0	27.10	0.00	100.00	0.00	0.00
2	28.02	1.46	97.34	1.20	1.10
4	28.38	2.38	95.85	1.76	1.22
8	32.05	5.89	90.02	4.10	2.32
16	44.12	14.86	75.18	9.96	6.32
32	46.73	23.15	63.42	13.43	9.70

Appendix

Table 16: Accuracy on each checkpoint’s own rollouts and fork changes on fixed pre-training prefixes. Fork percentages use the original 729,574 forks as the denominator. New forks are additional positions outside the original set.

Figure 16: Fork changes on fixed pre-training prefixes (left axes) and accuracy on checkpoint-generated rollouts (right axes), at temperature 0.7. Fork percentages use each pair’s original fork count.

Figure 17: Cumulative shares below each pre-training mass or reverse-KL threshold. The gray curve uses all 729,574 original forks as its denominator, and the purple curve uses the 258,841 forks resolved at step 256.

Panel of Figure 18 — Figure 18: Resolution by pre-training mass and reverse KL for the additional Qwen pairs. Colors show within-group rates, right-hand values show final rates, and parentheses show original fork shares.

Figure 19: Resolution by pre-training mass and reverse KL for Llama, using the same display and color scale as Figure 18 .

Figure 20: Resolution by pre-training mass and reverse KL on IFEval, using the same display and color scale as Figure 18 .

Figure 21: KL-based intervention at Final on DAPO-Math (800 trajectories) and IFEval (2,164 trajectories). Dashed lines show the corresponding student-alone accuracy. Error bars are 95% confidence intervals over problem means.

Fork Group	Step 16				Step 64				Final
	F/T	F/F (%)	Acc. (%)	$p$	F/T	F/F (%)	Acc. (%)	$p$	F/T	F/F (%)	Acc. (%)	$p$
Random	96.6	20.4	57.13	–	99.0	17.7	55.63	–	100.4	19.3	58.25	–
Old	100.0	19.7	56.75	0.837	98.1	17.2	57.13	0.353	99.9	18.3	56.88	0.396
Unchanged	94.5	18.5	57.63	0.761	97.6	17.0	58.38	0.086	100.4	18.6	58.50	0.878
Random (Low)	28.9	5.9	51.00	–	–	–	–	–	–	–	–	–
New	31.4	6.8	44.88 ∗∗∗	$<0.001$	78.0	13.8	49.50 ∗∗∗	$<0.001$	82.5	15.9	54.63 ∗	0.024

Appendix

Table 17: Qwen fork-group interventions. F/T is flips per trajectory and F/F is the percentage of forks flipped. At step 16, New and Moved use Random (Low) and Random (Mid) as controls. Other comparisons use Random ( Section B.2 ).

Fork Group	Step 16				Step 64				Final
	F/T	F/F (%)	Acc. (%)	$p$	F/T	F/F (%)	Acc. (%)	$p$	F/T	F/F (%)	Acc. (%)	$p$
Student Alone	0.0	0.0	26.75	–	0.0	0.0	27.63	–	0.0	0.0	32.00	–
New	15.2	14.6	29.00	0.297	16.6	16.7	29.00	0.531	17.0	18.8	32.50	0.849
Moved	7.2	6.7	29.00	0.333	8.4	8.3	28.50	0.711	8.8	9.1	30.50	0.509
Unchanged	74.0	74.3	44.50 ∗∗∗	$<0.001$	66.5	70.4	39.50 ∗∗∗	$<0.001$	60.5	67.8	43.00 ∗∗∗	$<0.001$
Old	78.7	80.1	41.00 ∗∗∗	$<0.001$	72.7	77.6	39.00 ∗∗∗	$<0.001$	66.8	77.0	43.00 ∗∗∗	$<0.001$

Appendix

Table 18: Gemma 12B-to-4B fork-group interventions on 200 problems. F/T and F/F follow Table 17 .

p

compares each policy’s one outcome per problem with the mean of four Student Alone outcomes using a two-sided paired

t

-test.

Fork Group	Step 16				Step 64				Final
	F/T	F/F (%)	Acc. (%)	$p$	F/T	F/F (%)	Acc. (%)	$p$	F/T	F/F (%)	Acc. (%)	$p$
Student Alone	0.0	0.0	3.63	–	0.0	0.0	2.63	–	0.0	0.0	1.63	–
New	10.4	12.1	3.50	0.921	36.9	24.8	4.00	0.289	23.6	16.1	3.50	0.116
Moved	5.3	6.2	3.50	0.927	23.2	9.3	4.00	0.228	26.7	9.4	3.50	0.116
Unchanged	77.1	55.6	3.00	0.620	107.0	39.5	4.00	0.289	94.1	43.7	6.00 ∗∗	0.005
Old	84.1	58.7	6.00	0.149	98.3	41.7	5.00	0.117	87.5	66.0	4.50	0.051

Appendix

Table 19: Llama 3B-to-1B fork-group interventions, using the same columns and problem-paired comparison with Student Alone as Table 18 .

Rule	Step 16		Step 64		Final
	F/T	Acc. (%)	F/T	Acc. (%)	F/T	Acc. (%)
Student Alone	0.00	67.38	0.00	68.99	0.00	65.48
New	0.48	67.51	0.47	68.67	0.50	66.13
Moved	0.44	67.47	0.49	67.93	0.50	65.57
Unchanged	0.55	67.98	0.58	69.13	0.56	65.53
Old	0.56	67.51	0.56	68.21	0.51	65.53

Appendix

Table 20: IFEval fork-group interventions, with 2,164 trajectories per condition. F/T is flips per trajectory. Accuracy uses strict prompt scoring.

Figure 22: Student entropy on fixed pre-training prefixes and accuracy on checkpoint-generated rollouts. Post-answer repetition is excluded from entropy scoring. Settings are in Section A.6.5 .

Pair / Dataset	Positions	$N$ (M)	$H_{\rm Base}$	$H_{\rm Final}$	$\Delta H$	$P_{\rm Base}$	$P_{\rm Final}$	$\Delta P$ (pp)
Qwen 4B–1.7B	Original Non-Forks	4.45	0.135	0.124	-0.011	95.49	95.32	-0.17
	Still Agree	4.36	0.119	0.105	-0.014	96.13	96.80	+0.68
Qwen 4B–0.6B	Original Non-Forks	2.52	0.183	0.126	-0.058	94.17	95.35	+1.18
	Still Agree	2.46	0.162	0.101	-0.061	94.96	97.21	+2.25
Qwen 8B–4B	Original Non-Forks	9.39	0.031	0.108	+0.077	99.17	97.90	-1.27
	Still Agree	9.36	0.028	0.100	+0.072	99.28	98.19	-1.08

Appendix

Table 21: Paired distribution changes from Base to Final at originally agreed positions.

H

is entropy in nats,

P

is the probability of the teacher’s argmax in percent, and

N

is the number of positions in millions. Still Agree uses positions where all three argmax choices agree.

On-Policy Distillation

On-policy distillation trains a model on its own outputs using teacher guidance, aligning supervision with the states it encounters at deployment.

See all 488 papers

Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement
Aug 31, 2026Yi Ding, Ruqi ZhangRL for Language Models Language Model Distillation
Department of Computer Science, Purdue University, USA
When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better
Oct 8, 2026Siyan Zhao, Yonggan Fu, Jindong Jiang +8Language Model Distillation On-Policy Distillation
NVIDIA
Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectives
Sep 29, 2026Qiwei Di, Xuheng Li, Kaixuan Ji +3Multi-Teacher Knowledge Distillation On-Policy Distillation
Department of Computer Science, University of California, Los Angeles, CA 90095, USA

Aug 31, 2026J/K move · Enter open · S save

Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement

Yi Ding, Ruqi Zhang

Department of Computer Science, Purdue University, USA

RL for Language Models Language Model Distillation On-Policy Distillation Language Model Self-Improvement Policy Learning

Open paper AskPDF arXiv

On-policy distillation (OPD) offers dense token-level supervision as an alternative to the sparse outcome-level advantages of reinforcement learning with verifiable rewards (RLVR). However, the teacher scores student-generated trajectories that are inherently off-policy for it, so the reliability of its supervision, and hence the source of the student's improvement, remains unclear. We quantitatively analyze teacher supervision during OPD training and find substantial noise whose prevalence increases with teacher scale. Surprisingly, the student policy is insensitive to such noise, converging to comparable performance regardless of whether noisy supervision is retained or removed. Does OPD distill at all? By analyzing what drives its gains, we find that learning concentrates on low log-probability tokens, and using a single fixed negative advantage matches the performance of teacher-provided ones. This suggests that OPD works largely by suppressing low log-probability tokens, which requires no teacher. These findings motivate On-Policy Self-Adaptation (OPSA), a supervision-free method using entropy-adaptive negative advantages. It assigns stronger learning signals to high-entropy positions, suppressing tail tokens, and evenly redistributing probability mass among head tokens. Compared with the base Qwen3-1.7B, OPSA improves Avg@32 by 35.41 points on AIME24, corresponding to a 263% relative gain, and more than doubles Pass@32 across all three benchmarks. It also outperforms OPD by 16.77 points in Avg@32 on AIME24. Extensive experiments and analyses across model families and tasks further demonstrate its effectiveness and generalizability.

Abstract

On-Policy Distillation

Explore similar work

Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement

When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better

Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectives