Organizations: Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University · Department of Applied Mathematics, The Hong Kong Polytechnic University
On-policy distillation (OPD) improves language models by distilling teacher predictions on student-generated trajectories, while on-policy self-distillation (OPSD) removes the need for an external teacher by using a reference-conditioned copy of the model as the teacher. Despite its great early improvements, we find that OPSD does not remain consistently effective under continued training and can eventually suffer substantial performance degradation. We study this training behavior and link the degradation to a growing mismatch between the evolving reference-free student and the frozen privileged teacher. In the early stage of OPSD, the privileged teacher remains closely aligned with the student and provides useful guidance for improving the student's generation trajectories. However, as training progresses, the growing distributional mismatch between the privileged teacher and reference-free student can cause the teacher to provide misleading supervision, ultimately degrading student performance. This phenomenon motivates us to first leverage privileged reference supervision to acquire useful knowledge and then transfer it to a fresh student. We introduce Bootstrapped On-Policy Distillation (BOPD), a two-stage framework that first uses OPSD to internalize task-relevant information from reference-conditioned supervision into an intermediate model, and then distills this reference-free intermediate model into a fresh copy of the initial model. Experiments across models show that BOPD improves mathematical reasoning and code generation beyond both the initial models and the constructed OPSD teachers. These results suggest that privileged reference supervision can be more effectively exploited by first transferring its task-relevant signal into a task-adapted model and then distilling from that model without reference conditioning.
Figures & tables
Figure 1: Performance of OPSD and BOPD during the training process
Figure 2: BOPD overview. In Stage I, a frozen copy of M0 receives a reference solution and supervises a student on its own rollouts, producing M1 . In Stage II, frozen M1 supervises a fresh student initialized from M0 and neither model receives a reference. Dashed arrows denote student initialization; solid arrows show rollout flow, supervision, and model transfer.
Figure 3: Continued OPSD training on Qwen3-4B. Top: Avg@6 on three mathematics benchmarks. Middle: overlap@16, overlap-token advantage@16, and absolute entropy gap between the OPSD student and its privileged teacher. Bottom: Fixed-prefix entropy, on-policy entropy and KL divergence between the student and teacher.
Avg@ 12↑
Pass@ 12↑
Size
Method
AIME24
AIME25
HMMT25
Mean
AIME24
AIME25
HMMT25
Mean
4B
Initial model
22.50
19.72
11.94
18.06
56.67
40.00
30.00
42.22
OPSD (Stage I)
25.83
20.00
13.06
19.63
60.00
46.67
33.33
46.67
BOPD (ours)
34.17
26.39
11.39
23.98
66.67
63.33
33.33
54.44
8B
Initial model
25.56
19.44
11.11
18.70
43.33
33.33
23.33
33.33
OPSD (Stage I)
26.67
21.67
12.22
20.19
46.67
36.67
26.67
36.67
Table 1: Mathematical reasoning results (%). Bold and underlining mark the best and second-best distinct scores, respectively, within each model size and metric.
Size
Method
HumanEval+
MBPP+
LCB v5-new
LCB v6-new
Mean
4B
Initial model
74.39
57.14
21.56
24.57
44.42
OPSD (Stage I)
75.61
60.32
22.75
23.43
45.53
BOPD (ours)
75.61
63.23
22.75
24.57
46.54
8B
Initial model
81.71
65.87
22.75
24.57
48.73
OPSD (Stage I)
79.27
62.96
22.16
25.14
47.38
BOPD (ours)
79.27
65.87
22.16
28.00
48.82
Table 2: Code-generation results. Mean equally averages the four benchmarks before rounding. Bold and underlining mark the best and second-best distinct scores within each model size and metric.
Figure 4: Capability retention after mathematical training. Points show accuracy changes relative to the initial model (zero line), in percentage points.
Figure 5: Accuracy and teacher–student alignment during 4B mathematical training. The top row shows checkpoint accuracy on each benchmark; the OPSD scores are also shown in Figure 3 . The bottom row reports alignment metrics on regenerated trajectories for the same 64 fixed training prompts. Curves begin at step 20; steps are relative to each method’s training stage.
Figure 6: Mean mathematical accuracy across AIME24, AIME25, and HMMT25. Each gray dashed line marks the corresponding initial-model accuracy.
Model
Student
Teacher
AIME24
AIME25
HMMT25
Avg@6
Pass@6
Initial M0
—
—
24.44
21.11
9.44
18.33
33.33
OPSD M1
M0
M0 + ref.
25.00
19.44
13.89
19.44
38.89
BOPD M2
M0
M1
32.78
25.56
11.11
23.15
44.44
M3
M0
M2
29.44
20.00
8.89
19.44
37.78
M4
M2
M2 + ref.
3.89
4.44
1.67
3.33
15.56
M5
M2
M4
8.89
6.11
3.33
6.11
21.11
Table 3: Further distillation at 4B. Each trained model in the first column is obtained by initializing from the model in the Student column and learning from the model in the Teacher column. “+ ref.” means the teacher also sees the reference solution.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Value
Training examples available
29,434
Optimizer steps per stage
100
GPUs / batch / accumulation (4B)
4 / 4 / 2
GPUs / batch / accumulation (8B)
2 / 2 / 8
Effective batch size
32
Student rollouts per stage
3,200
Appendix
Table 4: Recorded training settings for the main mathematical experiments.
Avg@ 12↑
Pass@ 12↑
Size
Method
AIME24
AIME25
HMMT25
Mean
AIME24
AIME25
HMMT25
Mean
4B
Initial model
73.61
67.50
44.72
61.94
83.33
86.67
66.67
78.89
OPSD (Stage I)
77.22
64.17
38.61
60.00
90.00
80.00
60.00
76.67
BOPD (ours)
75.28
69.44
45.28
63.33
83.33
83.33
66.67
77.78
8B
Initial model
75.83
68.89
46.67
63.80
83.33
83.33
66.67
77.78
OPSD (Stage I)
78.33
66.11
46.67
63.70
86.67
83.33
66.67
78.89
Appendix
Table 5: Thinking-enabled mathematical reasoning results (%). Each problem is sampled 12 times. Each benchmark cell reports Avg@12 / Pass@12. Bold and underlining mark the best and second-best distinct scores, respectively, within each model size and metric.
Size
Configuration
P strict
I strict
P loose
I loose
Trunc.
4B
Initial model
80.41
86.21
83.73
88.61
8
OPSD (Stage I)
81.33
87.05
84.29
89.21
12
Stage II OPSD
81.70
87.05
84.84
89.21
17
BOPD
81.70
86.81
85.21
89.33
11
OPSD (2 × steps)
79.30
85.73
82.81
88.25
9
8B
Initial model
82.62
88.37
86.14
90.77
11
Appendix
Table 6: IFEval accuracy (%) after mathematical training. P and I denote prompt and instruction accuracy. Trunc. counts responses reaching the output limit; they remain included in scores.
Size
Configuration
HumanEval
HumanEval+
Trunc.
4B
Initial model
83.54
74.39
2
OPSD (Stage I)
82.32
75.00
1
Stage II OPSD
84.15
78.66
1
BOPD
85.98
78.05
0
OPSD (2 × steps)
82.93
76.22
2
8B
Initial model
84.76
81.71
0
Appendix
Table 7: Code pass@1 (%) after mathematical training, with all 164 problems included. Trunc. counts output-limit truncations.
Size
Configuration
AIME24
AIME25
HMMT25
Mean
4B
OPSD (2 × steps)
12.22 / –
7.78 / –
7.22 / –
9.07 / 22.22
Solution SFT
10.56 / 20.00
12.78 / 23.33
6.67 / 16.67
10.00 / 20.00
SFT → OPD
20.56 / 43.33
15.00 / 36.67
7.78 / 23.33
14.44 / 34.44
Stage II OPSD
16.67 / 40.00
14.44 / 36.67
5.56 / 16.67
12.22 / 31.11
8B
OPSD (2 × steps)
20.00 / 36.67
16.67 / 43.33
5.00 / 16.67
13.89 / 32.22
Solution SFT
14.44 / 36.67
7.22 / 23.33
2.78 / 10.00
8.15 / 23.33
Appendix
Table 8: Ablation scores (%). Each benchmark cell lists Avg@6 / Pass@6. Mean lists the corresponding unweighted benchmark means. Dashes mark benchmark-level Pass@6 scores unavailable for the separate 4B 200-step run.
Avg@ 12↑
Pass@ 12↑
Method
AIME24
AIME25
HMMT25
Mean
AIME24
AIME25
HMMT25
Mean
Initial model
10.00
9.44
2.50
7.31
30.00
20.00
10.00
20.00
OPSD (Stage I)
16.39
13.89
6.67
12.31
50.00
30.00
20.00
33.33
BOPD (ours)
17.50
12.78
9.17
13.15
50.00
36.67
26.67
37.78
Appendix
Table 9: Mathematical reasoning results on SmolLM3-3B (%). Each problem is sampled 12 times. Bold and underlining mark the best and second-best distinct scores, respectively, within each metric.
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.
On-policy distillation (OPD) is a promising approach for transferring knowledge between language models, where a student receives dense token-level supervision along its own generated trajectories. However, teacher supervision can be unreliable when conditioned on incomplete or low-quality student prefixes. We identify Teacher Uncertainty Contraction (TUC), a systematic phenomenon whereby the teacher's predictive uncertainty decreases as it continues from a student-generated prefix. We theoretically characterize this trade-off through a variance-bias decomposition of teacher-branch gradients, showing that uncertainty contraction reduces variance while teacher-student path divergence increases bias, thereby favoring a finite continuation. Guided by this insight, we propose Adaptive-Continuations On-Policy Distillation (AC-OPD), which augments informative states along student rollouts with teacher continuations and adaptively selects their effective supervision horizons. Experiments on mathematical reasoning and code generation across model scales demonstrate that AC-OPD consistently improves over standard OPD. Controlled-continuations and matched-budget analyses further validate the adaptive-continuations design, highlighting adaptive teacher continuations as an effective principle for reliable on-policy distillation.The code will be made publicly available upon publication.
Jingang Zhou, Yuyi Zhou, Haiyang Guo +6
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Science · Ant Group
On-policy self-distillation (OPSD) improves large language models by letting a self-teacher with privileged information provide dense token-level supervision on the model's own trajectories. Yet existing methods typically construct the self-teacher from the current, initial, or slowly averaged policy state, leaving the quality of supervision constrained by the teacher's ability to exploit privileged information. We ask whether the model's own optimization progress can instead be recycled into a stronger self-teacher. In this paper, we introduce Bootstrapped On-Policy Self-Distillation (B-OPSD), which temporarily trains the policy ahead to obtain a future teacher, restores the student to the original policy state, and then uses the future teacher to supervise the restarted student. The future teacher improves supervision in two complementary ways, it can generate more reliable privileged trajectories and, conditioned on them, provide more informative token-level targets along the restarted student's on-policy trajectories. Experiments on mathematical reasoning with Qwen3-4B and Qwen3-8B show consistent improvements over standard OPSD in both settings, including gains from 27.50 to 41.30 and from 48.80 to 64.44 in the rollout-privileged setting. Our findings point to a broader principle for self-improving models that future learning progress can be distilled backward, preserving acquired knowledge while bootstrapping beyond the optimization state that produced it.
Zheng Zhang, Xinyue Tan, Lufei Li +3
School of Information Science and Technology, ShanghaiTech University · State Key Laboratory of General Artificial Intelligence, BIGAI · Hefei University of Technology
On-policy distillation (OPD) improves language-model post-training by letting the student learn from its own sampled trajectories under teacher supervision, aligning the training process with inference-time behavior. However, the rollout policy in OPD is usually treated as fixed: each round samples directly from the current student. We show that this choice can create a compounding teacher--student mismatch, because it requires the teacher to provide effective supervision on whichever trajectories the student samples, including ones less reachable by the teacher itself. On such trajectories, teacher supervision can become noisy or even misleading, leading to undesirable student patterns such as repetition. Moreover, since the updated student in turn defines the rollout policy of subsequent rounds, these patterns persist and potentially amplify over training. This failure mode suggests that improving OPD requires treating the rollout policy itself as an optimization target, rather than only modifying the distillation objective. We propose Mixture Proposal On-Policy Distillation (MPOPD), which derives a student-anchored, teacher-guided rollout distribution from a local trust-region objective based on relative teacher support. The resulting target is a geometric mixture of the student and teacher policies, and we introduce a proposal policy that approximates this mixture, enabling rollout generation in a single forward pass. Experiments on 16 math, knowledge, and medical reasoning benchmarks show consistent gains over OPD and recent variants. On math reasoning, our method improves average accuracy by 6.91 over OPD on Qwen3-4B-Base while reducing average generation length by 41%, suggesting that rollout policy design provides an effective way to improve OPD.
Sirui Chen, Yunzhe Qi, Haobo Xu +4
University of Illinois Urbana-Champaign · Stanford University