On-policy distillation (OPD) is increasingly studied as a cheaper alternative to reinforcement learning, yet it remains a form of knowledge distillation (KD) that relies on teacher supervision. We revisit OPD from the KD perspective by asking where the teacher's advantage lies along student-generated trajectories. Controlled token flips at teacher-student disagreements directly measure the accuracy gained from teacher corrections. Single-token flips yield a modest average gain on selected failed trajectories, whereas repeated flips raise accuracy from 28.4% to 74.5%, close to the teacher's 76.9%. Gains continue to accumulate with more flips, but mass- and KL-based selection outperform random flips by 15-18 percentage points at counts of similar magnitude. Selective intervention remains useful after OPD, but greater argmax agreement does not consistently improve accuracy across model pairs. We advocate training schedules that learn more valuable teacher corrections while maintaining distributional stability. Controlled flips and fork-resolution analysis evaluate OPD beyond accuracy by measuring which teacher preferences the student acquires and what further correction can recover.
Figures & tables
Flip − Resample
Net Gain − Rest
Trajectory
Problem
Trajectory
Problem
Fork Group
Share (%)
Gain
pT
Gain
pP
Δ
pT
Δ
pP
All Forks
100.0
+1.41 ∗∗∗
<0.001
+1.22 ∗
0.015
–
–
–
–
μtS<0.05
53.9
+2.06 ∗∗∗
<0.001
+1.83 ∗∗
0.007
+1.40 ∗∗∗
<0.001
+1.35 ∗∗
0.004
Blind
9.0
+5.27 ∗∗∗
<0.001
+4.27 ∗
0.012
+4.08 ∗∗∗
<0.001
+3.31 ∗
0.015
Reflection/Contrast
6.6
+6.59 ∗
0.045
+5.11
0.116
+1.84 ∗∗
0.008
+3.84
0.189
Table 1: Single-flip gains over resampling and net-gain differences versus remaining positions, in pp. Each estimate is paired with its test at the same aggregation level. Share uses 5,901 eligible forks. Group tests are unadjusted. Full results and eligibility rules appear in Tables 7 and B.1 .
Figure 1: Accuracy versus mean flips per trajectory. Point labels indicate selection thresholds. Full results are in Table 13 .
Flipped
Flips
Introduced Reflection Words
Length
Accuracy (%)
Student alone
0
2.5
1,345
28.4
random 6%
11
3.2
1,506
32.9
reflection words
14
8.4
2,156
34.9
μtT<10−5 ∗∗
4
3.1
1,594
38.4
Table 2: Reflection-word use, trajectory length, and accuracy under sparse intervention. Reflection counts exclude teacher-inserted tokens. Significance compares accuracy with random flips.
Figure 2: Fork changes on pre-training prefixes (left, % of original forks) and accuracy on each checkpoint’s own rollouts (right).
Figure 3: Fork resolution by pre-training teacher mass and reverse KL. Colors show within-group rates, right-hand values show final rates, and parentheses show shares of original forks.
Figure 4: Mass selection and random flips at three OPD checkpoints. Labels give mass thresholds or random probabilities. Gray lines mark standalone accuracy. Each point uses 800 trajectories.
Figure 5: Entropy on fixed pre-training prefixes (purple, left axes) and accuracy on checkpoint rollouts (gray, right axes). Teacher entropy is dotted. Post-answer repetition is excluded. All six settings appear in Section H.4 .
Fork Group
F/T
F/F
Acc. (%)
p
Random
96.6
20.4
57.13
–
Old
100.0
19.7
56.75
0.837
Unchanged
94.5
18.5
57.63
0.761
Random
28.9
5.9
51.00
–
New
31.4
6.8
44.88 ∗∗∗
<0.001
Random
52.2
10.8
50.88
–
Table 3: Fork-group interventions at step 16. F/T is flips per trajectory and F/F is the percentage of forks flipped. p compares accuracy with Random in each block ( Section B.2 ).
Appendix figures & tables35 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Reverse KL on 5.24M base-model positions. (a) Distributions where both argmaxes and the sampled token agree, and at forks where the sampled token differs from the teacher’s argmax. Bars use all positions as the denominator. Other sampling outcomes are omitted. (b) Cumulative loss over all positions ranked by reverse KL.
Figure 7: Token choices at 640k argmax forks where the sampled token also differs from the teacher’s choice. (a,b) Mean probabilities under student and teacher token rankings. (c,d) Rank of the other model’s argmax, with “out” denoting a token outside the top-16.
L
n
Gain (pp)
95% CI
pT
pP
1
5901
+1.41
[0.66, 2.15]
<0.001
0.015
16
5926
+3.25
[2.15, 4.36]
<0.001
0.004
128
5926
+16.51
[12.82, 20.20]
<0.001
<0.001
∞
5926
+79.68
[74.25, 85.10]
<0.001
<0.001
Appendix
Table 4: Overall paired gains over the complete resample baseline. Effects and Student- t 95% intervals are trajectory-weighted, in pp. pT tests trajectory means and pP tests problem means. n is the number of eligible positions ( Section B.1 ).
Group
nL=1
L=1
L=16
L=128
L=∞
0.00≤μtT<0.05
2774
+2.05
+3.84
+17.17
+78.67 ∗
0.05≤μtT<0.10
733
+0.60
+2.22
+14.42 ∗
+81.24
0.10≤μtT<0.20
1007
+0.03 ∗
+2.79
+17.63
+80.58
0.20≤μtT<0.30
706
+1.55
+2.79
+16.66
+80.83
0.30≤μtT<0.50
680
+0.93
+3.02
+14.16 ∗
+79.19
0.00≤μtS<0.05
3179
+2.06 ∗∗∗
+3.57
+16.85
+78.97
Appendix
Table 5: Trajectory-weighted gains over resampling by fork group and segment length, in pp. nL=1 counts eligible changed-token positions. Asterisks compare each group with remaining positions, not with zero, using the trajectory test.
Group
pT16
pP16
pT128
pP128
pT∞
pP∞
0.00≤μtT<0.05
0.198
0.262
0.058
0.006
0.029
0.202
0.05≤μtT<0.10
0.218
0.266
0.049
0.048
0.079
0.356
0.10≤μtT<0.20
0.368
0.693
0.186
0.336
0.237
0.029
0.20≤μtT<0.30
0.497
0.489
0.900
0.908
0.330
0.422
0.30≤μtT<0.50
0.723
0.334
0.030
0.048
0.655
0.828
0.00≤μtS<0.05
0.301
0.456
0.270
0.141
0.055
0.108
Appendix
Table 6: Unadjusted group-versus-rest p -values for longer teacher segments. Superscripts give segment length. Subscripts give trajectory ( T ) or problem ( P ) aggregation.
Flip − Resample
Net Gain − Rest
Trajectory
Problem
Trajectory
Problem
Fork Group
Share (%)
Gain
pT
Gain
pP
Δ
pT
Δ
pP
nT/nP
All Forks
100.0
+1.41 ∗∗∗
<0.001
+1.22 ∗
0.015
–
–
–
–
–
0.00≤μtT<0.05
47.0
+2.05 ∗∗∗
<0.001
+1.94 ∗
0.032
+1.17
0.057
+1.25
0.145
40/10
0.05≤μtT<0.10
12.4
+0.60
0.133
+0.69
0.059
-0.88
0.102
-0.60
0.244
40/10
0.10≤μtT<0.20
17.1
+0.03
0.961
+0.04
0.941
-1.56 ∗
0.021
-1.37
0.106
40/10
Appendix
Table 7: Complete single-flip comparisons, with effects in pp and shares of 5,901 eligible forks. Within-group gains use units with eligible positions. Group-versus-rest tests require at least three positions per side. nT/nP counts eligible trajectories/problems for these contrasts. Tests are unadjusted ( Section B.1 ).
Fork group
n
Flip (%)
Resample (%)
Gain [95% CI] (pp)
All eligible forks
5,901
11.31
9.91
+1.41 [0.66, 2.15]
μtS<0.05
3,179
11.22
9.16
+2.06 [1.13, 2.99]
μtS≥0.05
2,722
11.50
10.84
+0.66 [0.02, 1.30]
Blind
524
10.19
5.35
+4.84 [2.29, 7.39]
Not blind
5,146
9.56
8.80
+0.77 [0.19, 1.34]
Reflection/contrast
383
8.57
5.79
+2.78 [1.15, 4.41]
Appendix
Table 8: Single-token replacement and resampling success rates. Gain is flip minus resample success at the same positions. Each group pair uses trajectories with at least three eligible positions per side. Rates, gains, and 95% intervals use equal trajectory weights. Group contrasts are reported in Tables 1 and 7 .
Group
Feature
AUC [95% CI]
Token mass
μtT
0.512 [0.453, 0.537]
μtS
0.539 [0.503, 0.557]
Confidence
student
0.480 [0.435, 0.510]
teacher
0.488 [0.434, 0.517]
Entropy
student
0.486 [0.440, 0.510]
teacher
0.498 [0.444, 0.519]
Appendix
Table 9: Cross-validated AUC for predicting whether a single flip beats resampling, on 2,598 non-tied forks. Intervals use trajectory-bootstrap refits.
Comparison
Gain [95% CI] (pp)
pT
pP
True flip − keep
+1.55 [0.79, 2.32]
0.00020
0.01697
Resample − keep
+0.14 [-0.22, 0.50]
0.43082
0.31733
Appendix
Table 10: Keep-token baseline checks. Gains and Student- t 95% intervals are trajectory-weighted. pT and pP use trajectory and problem units, respectively, as in Section B.1 .
Figure 8: Success by fork position and teacher-segment length, averaged over positions in each fifth. The vertical axis omits 38–65%. L=1 retains changed-token samples. Longer segments use the original continuation runs ( App. A ).
Figure 9: Gain over the complete resample baseline by position decile and teacher-segment length. Paired position differences are averaged within each trajectory and then across trajectories. At L=1 only paths changing the original token are included.
Position
Resample
L=1
L=16
L=128
L=∞
All
0.103
0.113
0.134
0.270
0.887
0.0–0.2
0.163
0.173
0.200
0.332
0.944
0.2–0.4
0.153
0.158
0.171
0.305
0.949
0.4–0.6
0.096
0.114
0.135
0.302
0.920
0.6–0.8
0.073
0.080
0.101
0.238
0.908
0.8–1.0
0.047
0.060
0.081
0.196
0.748
Appendix
Table 11: Position-averaged success rates by trajectory fifth. Resample uses all positions, and L=1 uses positions with an actual token change. Paired gains and group tests are reported separately in Tables 5 and 7 .
Figure 10: Fork spacing on teacher-generated trajectories. (a) Within-trajectory distances between successive forks. (b) Lengths of maximal consecutive fork runs. Percentages use gaps and runs as their respective denominators.
Figure 11: Token-level diagnostics after a single flip. Each point averages four offsets, with up to 5,901 changed-token forks and 1,495 unchanged-token controls. (a) Student entropy after changed and unchanged tokens. (b) Both models’ entropy on the teacher’s continuation. (c) Their probabilities of the teacher’s tokens.
Trigger
Acc. (%)
F/T
F/F (%)
μtT<0.1 ∗∗∗
71.0
199
46
Random ( p=0.46 )
57.3
157
45
μtT<0.05 ∗∗∗
65.2
143
35
Random ( p=0.35 )
52.0
103
34
μtT<10−5 ∗∗
38.4
4
2
Random ( p=0.06 )
32.9
11
6
Appendix
Table 12: Selected dual-track controls. F/T is mean flips per trajectory and F/F is the percentage of forks flipped. Test details are in Section B.2 .
Trigger
Acc. (%)
F/T
μtT<10−5
38.38
4.11
μtT<10−4
42.88
10.27
μtT<10−3
51.50
25.75
μtT<0.05
65.25
142.50
μtT<0.1
71.00
199.01
KL>10
35.12
3.06
Appendix
Table 13: All points in Figure 1 . Each uses 200 problems with 4 trajectories each. F/T counts actual replacements per trajectory. Random controls are shaded.
Figure 12: Accuracy after teacher exit at fraction f (left axis) and gain over the student alone per preceding flip, in pp (right). Dotted lines mark standalone student and teacher accuracy.
Figure 13: Accuracy versus the percentage of forks flipped for three teacher–student pairs, with 200 problems and four trajectories per point. Dotted lines mark standalone baselines. MAX estimates the 100% count using each pair’s highest-coverage random run.
Figure 14: Strict prompt accuracy on official IFEval versus the percentage of forks flipped ( Section A.4.3 ). Dotted lines mark standalone baselines. MAX estimates the 100% count using the highest-coverage random run.
Figure 15: Dual-track accuracy on DAPO-Math for Llama 3.2 3B to 1B and Gemma 3 12B to 4B ( Sections A.4.4 and A.4.5 ). Each point uses 200 problems and four trajectories per problem. Dotted lines show Llama’s same-problem baselines with an 8,192-token context limit versus 12,288 for interventions. MAX estimates the 100% count using each pair’s highest-coverage random run.
n
Student
τ=0.05
τ=0.1
Teacher
S cannot, T can
308
0.065
0.575
0.666
0.821
Both can
388
0.505
0.871
0.905
0.933
Neither can
88
0.034
0.045
0.102
0.000
S can, T cannot
16
0.312
0.188
0.188
0.000
Appendix
Table 14: Accuracy by whether the student (S) and teacher (T) solve a problem on their own (pass@4 >0 ). n counts trajectories.
Trigger
Argmax (%)
Sample (%)
Gain (pp)
p
μtT<0.1
71.00
71.88
+0.88
0.5893
μtT<0.05
65.25
69.00
+3.75
0.0087
μtT<10−5
38.38
35.62
-2.75
0.1348
Appendix
Table 15: Teacher sampling versus argmax under the same mass trigger, on 200 problems with 4 trajectories each. Sampled replacements use temperature 0.7 and top- p 0.95. Gain is sample minus argmax, and p is from a two-sided test paired over problems ( Section B.2 ).
Step
Accuracy (%)
Solved (%)
Unchanged (%)
Moved (%)
New (%)
0
27.10
0.00
100.00
0.00
0.00
2
28.02
1.46
97.34
1.20
1.10
4
28.38
2.38
95.85
1.76
1.22
8
32.05
5.89
90.02
4.10
2.32
16
44.12
14.86
75.18
9.96
6.32
32
46.73
23.15
63.42
13.43
9.70
Appendix
Table 16: Accuracy on each checkpoint’s own rollouts and fork changes on fixed pre-training prefixes. Fork percentages use the original 729,574 forks as the denominator. New forks are additional positions outside the original set.
Figure 16: Fork changes on fixed pre-training prefixes (left axes) and accuracy on checkpoint-generated rollouts (right axes), at temperature 0.7. Fork percentages use each pair’s original fork count.
Figure 17: Cumulative shares below each pre-training mass or reverse-KL threshold. The gray curve uses all 729,574 original forks as its denominator, and the purple curve uses the 258,841 forks resolved at step 256.
Figure 18: Resolution by pre-training mass and reverse KL for the additional Qwen pairs. Colors show within-group rates, right-hand values show final rates, and parentheses show original fork shares.
Figure 19: Resolution by pre-training mass and reverse KL for Llama, using the same display and color scale as Figure 18 .
Figure 20: Resolution by pre-training mass and reverse KL on IFEval, using the same display and color scale as Figure 18 .
Figure 21: KL-based intervention at Final on DAPO-Math (800 trajectories) and IFEval (2,164 trajectories). Dashed lines show the corresponding student-alone accuracy. Error bars are 95% confidence intervals over problem means.
Fork Group
Step 16
Step 64
Final
F/T
F/F (%)
Acc. (%)
p
F/T
F/F (%)
Acc. (%)
p
F/T
F/F (%)
Acc. (%)
p
Random
96.6
20.4
57.13
–
99.0
17.7
55.63
–
100.4
19.3
58.25
–
Old
100.0
19.7
56.75
0.837
98.1
17.2
57.13
0.353
99.9
18.3
56.88
0.396
Unchanged
94.5
18.5
57.63
0.761
97.6
17.0
58.38
0.086
100.4
18.6
58.50
0.878
Random (Low)
28.9
5.9
51.00
–
–
–
–
–
–
–
–
–
New
31.4
6.8
44.88 ∗∗∗
<0.001
78.0
13.8
49.50 ∗∗∗
<0.001
82.5
15.9
54.63 ∗
0.024
Appendix
Table 17: Qwen fork-group interventions. F/T is flips per trajectory and F/F is the percentage of forks flipped. At step 16, New and Moved use Random (Low) and Random (Mid) as controls. Other comparisons use Random ( Section B.2 ).
Fork Group
Step 16
Step 64
Final
F/T
F/F (%)
Acc. (%)
p
F/T
F/F (%)
Acc. (%)
p
F/T
F/F (%)
Acc. (%)
p
Student Alone
0.0
0.0
26.75
–
0.0
0.0
27.63
–
0.0
0.0
32.00
–
New
15.2
14.6
29.00
0.297
16.6
16.7
29.00
0.531
17.0
18.8
32.50
0.849
Moved
7.2
6.7
29.00
0.333
8.4
8.3
28.50
0.711
8.8
9.1
30.50
0.509
Unchanged
74.0
74.3
44.50 ∗∗∗
<0.001
66.5
70.4
39.50 ∗∗∗
<0.001
60.5
67.8
43.00 ∗∗∗
<0.001
Old
78.7
80.1
41.00 ∗∗∗
<0.001
72.7
77.6
39.00 ∗∗∗
<0.001
66.8
77.0
43.00 ∗∗∗
<0.001
Appendix
Table 18: Gemma 12B-to-4B fork-group interventions on 200 problems. F/T and F/F follow Table 17 . p compares each policy’s one outcome per problem with the mean of four Student Alone outcomes using a two-sided paired t -test.
Fork Group
Step 16
Step 64
Final
F/T
F/F (%)
Acc. (%)
p
F/T
F/F (%)
Acc. (%)
p
F/T
F/F (%)
Acc. (%)
p
Student Alone
0.0
0.0
3.63
–
0.0
0.0
2.63
–
0.0
0.0
1.63
–
New
10.4
12.1
3.50
0.921
36.9
24.8
4.00
0.289
23.6
16.1
3.50
0.116
Moved
5.3
6.2
3.50
0.927
23.2
9.3
4.00
0.228
26.7
9.4
3.50
0.116
Unchanged
77.1
55.6
3.00
0.620
107.0
39.5
4.00
0.289
94.1
43.7
6.00 ∗∗
0.005
Old
84.1
58.7
6.00
0.149
98.3
41.7
5.00
0.117
87.5
66.0
4.50
0.051
Appendix
Table 19: Llama 3B-to-1B fork-group interventions, using the same columns and problem-paired comparison with Student Alone as Table 18 .
Rule
Step 16
Step 64
Final
F/T
Acc. (%)
F/T
Acc. (%)
F/T
Acc. (%)
Student Alone
0.00
67.38
0.00
68.99
0.00
65.48
New
0.48
67.51
0.47
68.67
0.50
66.13
Moved
0.44
67.47
0.49
67.93
0.50
65.57
Unchanged
0.55
67.98
0.58
69.13
0.56
65.53
Old
0.56
67.51
0.56
68.21
0.51
65.53
Appendix
Table 20: IFEval fork-group interventions, with 2,164 trajectories per condition. F/T is flips per trajectory. Accuracy uses strict prompt scoring.
Figure 22: Student entropy on fixed pre-training prefixes and accuracy on checkpoint-generated rollouts. Post-answer repetition is excluded from entropy scoring. Settings are in Section A.6.5 .
Pair / Dataset
Positions
N (M)
HBase
HFinal
ΔH
PBase
PFinal
ΔP (pp)
Qwen 4B–1.7B
Original Non-Forks
4.45
0.135
0.124
-0.011
95.49
95.32
-0.17
Still Agree
4.36
0.119
0.105
-0.014
96.13
96.80
+0.68
Qwen 4B–0.6B
Original Non-Forks
2.52
0.183
0.126
-0.058
94.17
95.35
+1.18
Still Agree
2.46
0.162
0.101
-0.061
94.96
97.21
+2.25
Qwen 8B–4B
Original Non-Forks
9.39
0.031
0.108
+0.077
99.17
97.90
-1.27
Still Agree
9.36
0.028
0.100
+0.072
99.28
98.19
-1.08
Appendix
Table 21: Paired distribution changes from Base to Final at originally agreed positions. H is entropy in nats, P is the probability of the teacher’s argmax in percent, and N is the number of positions in millions. Still Agree uses positions where all three argmax choices agree.
On-policy distillation (OPD) offers dense token-level supervision as an alternative to the sparse outcome-level advantages of reinforcement learning with verifiable rewards (RLVR). However, the teacher scores student-generated trajectories that are inherently off-policy for it, so the reliability of its supervision, and hence the source of the student's improvement, remains unclear. We quantitatively analyze teacher supervision during OPD training and find substantial noise whose prevalence increases with teacher scale. Surprisingly, the student policy is insensitive to such noise, converging to comparable performance regardless of whether noisy supervision is retained or removed. Does OPD distill at all? By analyzing what drives its gains, we find that learning concentrates on low log-probability tokens, and using a single fixed negative advantage matches the performance of teacher-provided ones. This suggests that OPD works largely by suppressing low log-probability tokens, which requires no teacher. These findings motivate On-Policy Self-Adaptation (OPSA), a supervision-free method using entropy-adaptive negative advantages. It assigns stronger learning signals to high-entropy positions, suppressing tail tokens, and evenly redistributing probability mass among head tokens. Compared with the base Qwen3-1.7B, OPSA improves Avg@32 by 35.41 points on AIME24, corresponding to a 263% relative gain, and more than doubles Pass@32 across all three benchmarks. It also outperforms OPD by 16.77 points in Avg@32 on AIME24. Extensive experiments and analyses across model families and tasks further demonstrate its effectiveness and generalizability.
Yi Ding, Ruqi Zhang
Department of Computer Science, Purdue University, USA
On-policy distillation (OPD) has become increasingly popular for transferring teacher capabilities to student models. In this work, we ask a critical research question: Is on-policy sampling always beneficial for distilling arbitrary teacher-student pairs? We show that a simple alternative, Semi-OPD, which distills from offline rollouts generated by the initial student, can often outperform OPD in both accuracy and training efficiency. Across 17 teacher-student pairs ranging from 1.5B to 235B parameters, Semi-OPD outperforms OPD in 14 cases, with up to +13.6% accuracy and 11.4x training speedup. We further find that the choice between OPD and Semi-OPD depends on the alignment between the initial teacher and student, quantified by an output-token overlap ratio: OPD is beneficial only when the two are highly aligned with high overlap ratios. Our deeper investigation suggests that effective distillation requires on-policyness w.r.t. both the student and the teacher. For misaligned pairs, student rollouts can become increasingly off-policy w.r.t. the teacher as context length grows, weakening the distillation signal. In contrast, Semi-OPD is often more stable, as it distills on shorter contexts while covering full trajectories and exposing the student to more teacher-preferred tokens. Beyond proposing Semi-OPD as an efficient alternative, our work motivates the community to rethink when to use OPD and to study stronger OPD variants with meaningful teacher-student pairs.
On-policy distillation (OPD) learns from teacher feedback on student-generated responses and has shown promise in reducing forgetting relative to supervised fine-tuning (SFT). However, its benefits and fragility remain incompletely understood. We study sequential distillation from multiple teachers, where the student minimizes its average divergence from the teachers. Forward Kullback--Leibler (KL) divergence yields a weighted arithmetic mixture, while reverse KL yields a normalized weighted geometric aggregate. We develop algorithms that learn these targets under off-policy and on-policy feedback, respectively, establishing logarithmic regret bounds in the tabular setting and extending the analysis to function approximation. By analyzing these aggregation targets, we identify mechanisms that help explain both the benefits and fragility of OPD. Relative to forward KL, reverse KL can better retain a confident expert's preferences under uninformative feedback, but is more sensitive to teachers that assign very low probabilities to correct responses. Its token-level conditionals also reveal a dependence on continuation distributions that can favor incorrect prefixes over long horizons.
Qiwei Di, Xuheng Li, Kaixuan Ji +3
Department of Computer Science, University of California, Los Angeles, CA 90095, USA