Organizations: Department of Computer Science, Rice University, Houston, USA · Department of Computer Science, University of North Carolina at Charlotte, Charlotte, USA · Department of Computer Science, Johns Hopkins University, Baltimore, USA · Department of Computer and Data Sciences, Case Western Reserve University.
On-policy distillation (OPD) trains student models under their own induced distribution while leveraging supervision from stronger teachers. We identify a failure mode of OPD: as training progresses, on-policy rollouts can undergo abrupt length inflation, causing truncated trajectories to dominate the training data. This truncation collapse coincides with abrupt repetition saturation and induces biased gradient signals, leading to severe training instability and sharp degradation in validation performance. We attribute this problem to the interaction between student-induced data collection and the distillation objective, which implicitly favors long and repetitive rollouts. To address this issue, we propose StableOPD, a stabilized OPD framework that combines a reference-based divergence constraint with rollout mixture distillation. These together mitigate repetition-induced length inflation and further stabilize OPD training. Across multiple math reasoning datasets, our approach prevents truncation collapse, stabilizes training dynamics, and improves performance by 7.2% on average.
Figures & tables
Figure 1: Abrupt length inflation within OPD.
Figure 2: Training dynamics of OPD on three groups. Training starts in a stable regime with low truncation and repetition, followed by a sharp phase transition where truncation and repetition increase and remain high while validation accuracy collapses, illustrating a robust truncation-repetition inflation failure mode of OPD.
Figure 3: Rollout-level evidence of abrupt repetition inflation for three student-teacher groups. Around the step where rollout length abruptly inflates, both student and teacher log_prob become much less negative, with the teacher’s increase being larger, which induces a sudden jump in the reveser KL advantage.
Figure 4: Reverse-KL advantage for regular and repetitive tokens during OPD training. Repetitive tokens receive larger advantages than regular tokens throughout training.
Model
Avg
MATH-500
Minerva
Olympiad
AMC
AIME24
AIME25
Qwen2.5-Math-7B
19.1
43.6
7.4
15.6
31.3
11.5
4.9
Qwen2.5-Math-7B-Instruct
37.6
80.4
32.7
41.0
48.5
12.5
10.2
SimpleRL-Zero
37.4
76.0
25.0
34.7
54.9
27.0
6.8
OpenReasoner-Zero
41.0
82.4
33.1
47.1
52.1
16.5
15.0
PRIME-Zero
40.8
81.4
39.0
40.3
54.0
17.0
12.8
Oat-Zero
43.8
78.0
34.6
43.4
61.2
33.4
11.9
Table 1: Accuracy on six mathematical benchmarks for Qwen2.5-Math-7B. The best performance on each benchmark is highlighted in bold . For OPD-based methods we use OpenThinkerV3 as teacher models.
Figure 5: Training dynamics of OPD vs. Stable-OPD. Student: Qwen2.5-Math-1.5B; Teacher: OpenThinker3-7B.
Method
Avg. Acc (%)
Qwen2.5-MATH-1.5B
16.0
OPD
28.0
OPD + KL
29.7
OPD + KL + Mixture Distillation
35.7
Table 2: Ablation studies of KL regularization and mixture distillation on average accuracy.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Avg
MATH-500
Minerva
Olympiad
AMC
AIME24
AIME25
Qwen2.5-Math-1.5B
16.0
28.0
9.6
21.2
26.4
7.2
3.6
Qwen2.5-Math-1.5B-Instruct
35.7
77.4
28.7
39.1
48.1
12.1
8.9
SFT
31.9
70.6
26.8
31.3
37.8
11.7
13.2
GRPO
30.1
61.8
26.8
32.0
40.2
11.8
7.7
OPD
28.9
56.7
23.4
31.0
35.9
11.1
15.0
Stable-OPD
36.1
73.9
32.6
37.4
43.0
13.8
16.0
Appendix
Table 3: Accuracy on six mathematical benchmarks for Qwen2.5-Math-1.5B. The best performance on each benchmark is highlighted in bold . For OPD-based methods we use OpenThinkerV3 as teacher models.
Model
Avg
MATH-500
Minerva
Olympiad
AMC
AIME24
AIME25
Qwen2.5-Math-1.5B
16.0
28.0
9.6
21.2
26.4
7.2
3.6
Qwen2.5-Math-1.5B-Instruct
35.7
77.4
28.7
39.1
48.1
12.1
8.9
SFT
31.9
70.6
26.8
31.3
37.8
11.7
13.2
GRPO
30.1
61.8
26.8
32.0
40.2
11.8
7.7
OPD †
28.0
58.4
22.4
26.9
36.2
10.9
13.1
Stable-OPD †
35.7
72.0
32.7
34.9
43.0
14.6
17.2
Appendix
Table 4: Accuracy on six mathematical benchmarks. The best performance on each benchmark is highlighted in bold . † denotes models distilled from DeepSeek-R1-Distill-7B, and ‡ denotes models distilled from OpenThinkerV3.
Figure 6: Training dynamics of OPD vs. Stable-OPD. Student: Qwen2.5-Math-1.5B; Teacher: DeepSeek-R1-Distill-7B.
Figure 7: Dynamics of OPD across training. Each panel shows truncation/repetition for both rollout and evaluation versus accuracy. Sudden accuracy changes often align with abrupt shifts in teacher log-probabilities and advantage estimates.