On-policy distillation (OPD) has become an important approach to language model post-training. However, despite its performance gains, OPD can also collapse into excessively long and repetitive generation, and the mechanism underlying these divergent outcomes remains poorly understood. We explain these outcomes through a reinforcement learning perspective: the teacher implicitly rewards student behaviors, even those it rarely exhibits itself. From this perspective, our experiments show that OPD improves performance without expanding the student's capabilities. When the implicit reward model is reliable, OPD makes correct responses easier to sample. In contrast, when the preference misaligns with quality, reward hacking happens: the implicit reward model amplifies overlong, repetitive student rollouts, even though it rarely generates such text itself. Guided by this diagnosis, we find that masking unhealthy responses during training and using SFT initialization can each effectively mitigate the collapse. Together, these findings show that OPD amplifies student behaviors favored by the teacher's implicit feedback, shifting the focus from how well the teacher generates to how reliably it evaluates student rollouts. Our code is available at https://github.com/HancCui/opd_hacking.
Figures & tables
Teacher
Student initialization
Training data
JustRL-1.5B
DeepSeek-R1-Distill-Qwen-1.5B
DeepMath-103K ( ≥6 )
DeepScaleR-1.5B-Preview
DeepSeek-R1-Distill-Qwen-1.5B
DeepMath-103K ( ≥6 )
Qwen3-4B
Qwen3-1.7B-Base
OpenThoughts3
Table 1: OPD settings used in our analysis. Each setting is named after its teacher.
Figure 2: pass @k curves of SINIT and SOPD . Across AMC23 and AIME24–26, OPD gains are largest at small k and shrink as the sampling budget grows. AIME curves extend to k=256 , while AMC23 extends to k=1024 .
Figure 3: Change in observed coverage from SINIT to SOPD . No problem is solvable by SOPD but not by SINIT in our observation. Counts come from the audited pools with conditional extension to 1,024 responses.
Figure 4: Per-problem mean@256 of SINIT ( x -axis) and SOPD ( y -axis). OPD raises success rates on problems already solvable by SINIT . The dashed line marks equal score.
Figure 5: Training process of the JustRL (top) and Qwen3-4B (bottom) settings. Optimization converges normally in both settings, and the trained student’s responses remain highly likely under SINIT . Columns: the advantage and training loss; the Min-10NN distance over six checkpoints; the NLL under SINIT of responses from SINIT and SOPD .
Model
Acc. (%)
Len. (k)
Trunc. (%)
Rep. (%)
JustRL-1.5B → DS-Distill-1.5B
Teacher
39.3
9.30
13.8
0.0
SINIT
20.9
11.19
12.5
0.0
SOPD
37.1 (+16.2)
9.72
16.5
0.0
Qwen3-4B → Qwen3-1.7B-Base
Teacher
19.6
3.09
2.1
0.0
Table 2: Endpoint generation characteristics on AIME24–26. Model pairs denote teacher → student. Bold values mark length-limit saturation and severe repetition.
Figure 6: Selection gap before and after training. Both students learn to discriminate teacher-preferred responses, with the Qwen3-4B setting showing overfitting.
Figure 7: Teacher advantage over SINIT responses from A1 (lowest advantage) to A10 (highest advantage). The JustRL teacher (top) assigns higher advantage to correct and shorter responses; the Qwen3-4B teacher (bottom) assigns higher advantage to overlong, repetitive ones. Dashed lines mark the teacher’s own mean length.
Figure 8: Student-rollout characteristics during training. All the masking settings recover from the collapse, whereas Qwen3-4B-warmup avoids it from the beginning.
Setting
AMC
AIME
Avg.
23
24
25
26
Base (pre-OPD)
2.41
1.67
0.00
1.67
1.44
4B → Base
29.82
3.33
3.33
4.17
10.16
+ mask
34.64
8.33
5.83
4.17
13.24
8B → Base
28.31
5.00
3.33
5.00
10.41
+ mask
32.83
5.00
2.50
3.33
10.92
Table 3: Masking improves average accuracy across teachers. Accuracy (%) and the four-benchmark mean; teachers are Qwen3 models, and Base/SFT denote Qwen3-1.7B initializations.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Intermediate teacher-scoring temperatures give the strongest single-run student trajectories in the pilot. Benchmark-equal student mean@4 accuracy is plotted against completed rollout iterations for the five teacher-scoring temperatures.
On-policy distillation (OPD) is increasingly used in LLM post-training because it can leverage a teacher model to provide dense supervision on student rollouts. The standard implementation, however, usually reduces distribution matching to a sampled-token log-ratio, which can make the learning signal fragile on long rollouts whose prefixes drift away from the teacher's typical support. We revisit this formulation from both theoretical and implementation perspectives. Theoretically, token-level OPD is biased relative to sequence-level reverse-KL minimization, but admits a substantially tighter worst-case variance bound; a controlled synthetic study further shows that stronger future-reward coupling increases gradient variance and destabilizes training. Empirically, we identify three failure modes of sampled-token OPD: imbalanced token-level supervision, unreliable teacher guidance on student-generated prefixes, and tokenizer or special-token mismatch. These findings motivate teacher top-K local support matching, a truncated reverse-KL objective that compares teacher and student distributions over a teacher-supported token set at each prefix, together with top-p rollout sampling and special-token masking. Across single-task reasoning and multi-task benchmarks spanning agentic and reasoning settings, this objective improves optimization stability and yields a +19.8% performance gain over standard sampled-token OPD baselines, providing a practical recipe for more stable on-policy distillation.
Yuqian Fu, Haohuan Huang, Kaiwen Jiang +4
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA · School of Artificial Intelligence, UCAS · Fudan University +1
On-policy distillation (OPD) has emerged as an effective framework for post-training language models by pairing student-generated trajectories with dense token-level supervision from a teacher. However, OPD implicitly assumes that teacher-derived rewards are an appropriate proxy for reasoning progress, and therefore treats all teacher feedback equally during policy optimization. While in practice, this assumption does not always hold. We observe that teacher-derived rewards often conflict with genuine reasoning progress, as reasoning steps with clear reasoning advancement may still receive lower distillation rewards, simply due to deviation from teacher's outputs. To address this mismatch, we propose Reasoning-Progress-Aware Reward Filtering for On-Policy Distillation (R2-OPD), which constructs two within-trajectory rankings of reasoning spans, one from teacher-derived rewards and the other from independently estimated progress reward. Distillation rewards are selectively suppressed whenever the two rankings disagree, reducing supervision that conflicts with reasoning progress while preserving effective teacher guidance. Our approach shows consistent improvement over standard OPD especially regarding reasoning performances.
Chen Yang, Haiyuan Wan, Rengrong Xiong +2
The Hong Kong University of Science and Technology (Guangzhou) · Tsinghua University · Zhejiang University +1
On-policy distillation (OPD) offers dense token-level supervision as an alternative to the sparse outcome-level advantages of reinforcement learning with verifiable rewards (RLVR). However, the teacher scores student-generated trajectories that are inherently off-policy for it, so the reliability of its supervision, and hence the source of the student's improvement, remains unclear. We quantitatively analyze teacher supervision during OPD training and find substantial noise whose prevalence increases with teacher scale. Surprisingly, the student policy is insensitive to such noise, converging to comparable performance regardless of whether noisy supervision is retained or removed. Does OPD distill at all? By analyzing what drives its gains, we find that learning concentrates on low log-probability tokens, and using a single fixed negative advantage matches the performance of teacher-provided ones. This suggests that OPD works largely by suppressing low log-probability tokens, which requires no teacher. These findings motivate On-Policy Self-Adaptation (OPSA), a supervision-free method using entropy-adaptive negative advantages. It assigns stronger learning signals to high-entropy positions, suppressing tail tokens, and evenly redistributing probability mass among head tokens. Compared with the base Qwen3-1.7B, OPSA improves Avg@32 by 35.41 points on AIME24, corresponding to a 263% relative gain, and more than doubles Pass@32 across all three benchmarks. It also outperforms OPD by 16.77 points in Avg@32 on AIME24. Extensive experiments and analyses across model families and tasks further demonstrate its effectiveness and generalizability.
Yi Ding, Ruqi Zhang
Department of Computer Science, Purdue University, USA