Organizations: Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University · Department of Applied Mathematics, The Hong Kong Polytechnic University
On-policy distillation (OPD) improves language models by distilling teacher predictions on student-generated trajectories, while on-policy self-distillation (OPSD) removes the need for an external teacher by using a reference-conditioned copy of the model as the teacher. Despite its great early improvements, we find that OPSD does not remain consistently effective under continued training and can eventually suffer substantial performance degradation. We study this training behavior and link the degradation to a growing mismatch between the evolving reference-free student and the frozen privileged teacher. In the early stage of OPSD, the privileged teacher remains closely aligned with the student and provides useful guidance for improving the student's generation trajectories. However, as training progresses, the growing distributional mismatch between the privileged teacher and reference-free student can cause the teacher to provide misleading supervision, ultimately degrading student performance. This phenomenon motivates us to first leverage privileged reference supervision to acquire useful knowledge and then transfer it to a fresh student. We introduce Bootstrapped On-Policy Distillation (BOPD), a two-stage framework that first uses OPSD to internalize task-relevant information from reference-conditioned supervision into an intermediate model, and then distills this reference-free intermediate model into a fresh copy of the initial model. Experiments across models show that BOPD improves mathematical reasoning and code generation beyond both the initial models and the constructed OPSD teachers. These results suggest that privileged reference supervision can be more effectively exploited by first transferring its task-relevant signal into a task-adapted model and then distilling from that model without reference conditioning.
Figures & tables
Figure 1: Performance of OPSD and BOPD during the training process
Figure 2: BOPD overview. In Stage I, a frozen copy of M0 receives a reference solution and supervises a student on its own rollouts, producing M1 . In Stage II, frozen M1 supervises a fresh student initialized from M0 and neither model receives a reference. Dashed arrows denote student initialization; solid arrows show rollout flow, supervision, and model transfer.
Figure 3: Continued OPSD training on Qwen3-4B. Top: Avg@6 on three mathematics benchmarks. Middle: overlap@16, overlap-token advantage@16, and absolute entropy gap between the OPSD student and its privileged teacher. Bottom: Fixed-prefix entropy, on-policy entropy and KL divergence between the student and teacher.
Avg@ 12↑
Pass@ 12↑
Size
Method
AIME24
AIME25
HMMT25
Mean
AIME24
AIME25
HMMT25
Mean
4B
Initial model
22.50
19.72
11.94
18.06
56.67
40.00
30.00
42.22
OPSD (Stage I)
25.83
20.00
13.06
19.63
60.00
46.67
33.33
46.67
BOPD (ours)
34.17
26.39
11.39
23.98
66.67
63.33
33.33
54.44
8B
Initial model
25.56
19.44
11.11
18.70
43.33
33.33
23.33
33.33
OPSD (Stage I)
26.67
21.67
12.22
20.19
46.67
36.67
26.67
36.67
Table 1: Mathematical reasoning results (%). Bold and underlining mark the best and second-best distinct scores, respectively, within each model size and metric.
Size
Method
HumanEval+
MBPP+
LCB v5-new
LCB v6-new
Mean
4B
Initial model
74.39
57.14
21.56
24.57
44.42
OPSD (Stage I)
75.61
60.32
22.75
23.43
45.53
BOPD (ours)
75.61
63.23
22.75
24.57
46.54
8B
Initial model
81.71
65.87
22.75
24.57
48.73
OPSD (Stage I)
79.27
62.96
22.16
25.14
47.38
BOPD (ours)
79.27
65.87
22.16
28.00
48.82
Table 2: Code-generation results. Mean equally averages the four benchmarks before rounding. Bold and underlining mark the best and second-best distinct scores within each model size and metric.
Figure 4: Capability retention after mathematical training. Points show accuracy changes relative to the initial model (zero line), in percentage points.
Figure 5: Accuracy and teacher–student alignment during 4B mathematical training. The top row shows checkpoint accuracy on each benchmark; the OPSD scores are also shown in Figure 3 . The bottom row reports alignment metrics on regenerated trajectories for the same 64 fixed training prompts. Curves begin at step 20; steps are relative to each method’s training stage.
Figure 6: Mean mathematical accuracy across AIME24, AIME25, and HMMT25. Each gray dashed line marks the corresponding initial-model accuracy.
Model
Student
Teacher
AIME24
AIME25
HMMT25
Avg@6
Pass@6
Initial M0
—
—
24.44
21.11
9.44
18.33
33.33
OPSD M1
M0
M0 + ref.
25.00
19.44
13.89
19.44
38.89
BOPD M2
M0
M1
32.78
25.56
11.11
23.15
44.44
M3
M0
M2
29.44
20.00
8.89
19.44
37.78
M4
M2
M2 + ref.
3.89
4.44
1.67
3.33
15.56
M5
M2
M4
8.89
6.11
3.33
6.11
21.11
Table 3: Further distillation at 4B. Each trained model in the first column is obtained by initializing from the model in the Student column and learning from the model in the Teacher column. “+ ref.” means the teacher also sees the reference solution.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Value
Training examples available
29,434
Optimizer steps per stage
100
GPUs / batch / accumulation (4B)
4 / 4 / 2
GPUs / batch / accumulation (8B)
2 / 2 / 8
Effective batch size
32
Student rollouts per stage
3,200
Appendix
Table 4: Recorded training settings for the main mathematical experiments.
Avg@ 12↑
Pass@ 12↑
Size
Method
AIME24
AIME25
HMMT25
Mean
AIME24
AIME25
HMMT25
Mean
4B
Initial model
73.61
67.50
44.72
61.94
83.33
86.67
66.67
78.89
OPSD (Stage I)
77.22
64.17
38.61
60.00
90.00
80.00
60.00
76.67
BOPD (ours)
75.28
69.44
45.28
63.33
83.33
83.33
66.67
77.78
8B
Initial model
75.83
68.89
46.67
63.80
83.33
83.33
66.67
77.78
OPSD (Stage I)
78.33
66.11
46.67
63.70
86.67
83.33
66.67
78.89
Appendix
Table 5: Thinking-enabled mathematical reasoning results (%). Each problem is sampled 12 times. Each benchmark cell reports Avg@12 / Pass@12. Bold and underlining mark the best and second-best distinct scores, respectively, within each model size and metric.
Size
Configuration
P strict
I strict
P loose
I loose
Trunc.
4B
Initial model
80.41
86.21
83.73
88.61
8
OPSD (Stage I)
81.33
87.05
84.29
89.21
12
Stage II OPSD
81.70
87.05
84.84
89.21
17
BOPD
81.70
86.81
85.21
89.33
11
OPSD (2 × steps)
79.30
85.73
82.81
88.25
9
8B
Initial model
82.62
88.37
86.14
90.77
11
Appendix
Table 6: IFEval accuracy (%) after mathematical training. P and I denote prompt and instruction accuracy. Trunc. counts responses reaching the output limit; they remain included in scores.
Size
Configuration
HumanEval
HumanEval+
Trunc.
4B
Initial model
83.54
74.39
2
OPSD (Stage I)
82.32
75.00
1
Stage II OPSD
84.15
78.66
1
BOPD
85.98
78.05
0
OPSD (2 × steps)
82.93
76.22
2
8B
Initial model
84.76
81.71
0
Appendix
Table 7: Code pass@1 (%) after mathematical training, with all 164 problems included. Trunc. counts output-limit truncations.
Size
Configuration
AIME24
AIME25
HMMT25
Mean
4B
OPSD (2 × steps)
12.22 / –
7.78 / –
7.22 / –
9.07 / 22.22
Solution SFT
10.56 / 20.00
12.78 / 23.33
6.67 / 16.67
10.00 / 20.00
SFT → OPD
20.56 / 43.33
15.00 / 36.67
7.78 / 23.33
14.44 / 34.44
Stage II OPSD
16.67 / 40.00
14.44 / 36.67
5.56 / 16.67
12.22 / 31.11
8B
OPSD (2 × steps)
20.00 / 36.67
16.67 / 43.33
5.00 / 16.67
13.89 / 32.22
Solution SFT
14.44 / 36.67
7.22 / 23.33
2.78 / 10.00
8.15 / 23.33
Appendix
Table 8: Ablation scores (%). Each benchmark cell lists Avg@6 / Pass@6. Mean lists the corresponding unweighted benchmark means. Dashes mark benchmark-level Pass@6 scores unavailable for the separate 4B 200-step run.
Avg@ 12↑
Pass@ 12↑
Method
AIME24
AIME25
HMMT25
Mean
AIME24
AIME25
HMMT25
Mean
Initial model
10.00
9.44
2.50
7.31
30.00
20.00
10.00
20.00
OPSD (Stage I)
16.39
13.89
6.67
12.31
50.00
30.00
20.00
33.33
BOPD (ours)
17.50
12.78
9.17
13.15
50.00
36.67
26.67
37.78
Appendix
Table 9: Mathematical reasoning results on SmolLM3-3B (%). Each problem is sampled 12 times. Bold and underlining mark the best and second-best distinct scores, respectively, within each metric.
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Science · Ant Group
School of Information Science and Technology, ShanghaiTech University · State Key Laboratory of General Artificial Intelligence, BIGAI · Hefei University of Technology