Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe

Yaxuan Li; Yuxin Zuo; Bingxiang He; Jinqian Zhang; Chaojun Xiao; Cheng Qian; Tianyu Yu; Huan-ang Gao

cs.LGApr 14, 2026

Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe

Authors: Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan-ang Gao,

Organizations: Tsinghua University · ShanghaiTech University · University of Illinois Urbana-Champaign · Renmin University of China

Abstract

On-policy distillation (OPD) has become a core technique in the post-training of large language models, yet its training dynamics remain poorly understood. This paper provides a systematic investigation of OPD dynamics and mechanisms. We first identify that two conditions govern whether OPD succeeds or fails: (i) the student and teacher should share compatible thinking patterns; and (ii) even with consistent thinking patterns and higher scores, the teacher must offer genuinely new capabilities beyond what the student has seen during training. We validate these findings through weak-to-strong reverse distillation, showing that same-family 1.5B and 7B teachers are distributionally indistinguishable from the student's perspective. Probing into the token-level mechanism, we show that successful OPD is characterized by progressive alignment on high-probability tokens at student-visited states, a small shared token set that concentrates most of the probability mass (97%-99%). We further propose two practical strategies to recover failing OPD: off-policy cold start and teacher-aligned prompt selection. Finally, we show that OPD's apparent free lunch of dense token-level reward comes at a cost, raising the question of whether OPD can scale to long-horizon distillation.

Figures & tables

Figure 2: OPD from two teachers with different thinking patterns into the same student (Qwen3-1.7B-Base). The GRPO teacher achieves stronger performance (left) and higher initial overlap ratio (right), suggesting that thinking pattern compatibility governs OPD effectiveness.

Figure 3: Validation performance of the two teachers (Qwen3-4B Non-thinking vs. Qwen3-4B-Base-GRPO) on AIME 2024, AIME 2025, and AMC 2023.

Figure 4: Comparison of OPD performance with and without additional teacher RL post-training across two model families. Left: DeepSeek family. Right: Qwen family. Post-trained teachers yield substantially stronger gains and higher teacher-student gap recovery rate .

Figure 5: Reverse distillation with JustRL-1.5B as the student and two same-family teachers (R1-Distill-1.5B and R1-Distill-7B). Both runs cause the student to regress to approximately the same level despite R1-Distill-7B scoring higher than JustRL-1.5B, indicating that OPD training dynamics are governed by thinking pattern rather than benchmark performance.

Figure 6: Successful vs. failing OPD with the same student (R1-Distill-1.5B) and two teachers. Top: avg@16 accuracy on three benchmarks. Dashed lines indicate teacher performance. Bottom: three dynamics over training. Successful distillation (JustRL-1.5B) shows rising overlap and narrowing entropy gap, and these trends are absent in the failing run (R1-Distill-7B).

Figure 7: Ablation on the optimization support in Top-

k

OPD. Overlap Top-

k

matches Student Top-

k

, while Non-Overlap Top-

k

is substantially weaker.

Figure 8: Effect of off-policy cold start before OPD, using fixed teacher Qwen3-4B (Non-thinking). The two curves correspond to OPD from Qwen3-1.7B-SFT and Qwen3-1.7B-Base.

Figure 9: Effect of prompt template alignment. The teacher-aligned template yields higher accuracy and overlap growth throughout training.

Figure 10: Effect of prompt content alignment. The teacher-aligned prompt contents yield stronger performance, with higher mass concentration on shared tokens and notably lower entropy.

Figure 11: (a) Validation accuracy on three benchmarks under different response lengths. (b) Accuracy gain from teacher continuation under different student prefix truncation lengths.

Figure 12: Training dynamics under different maximum response lengths for OPD.

Figure 13: Average student entropy across decoding positions during OPD training with 15K max response length, measured on student-generated trajectories from Step 180 to Step 250.

Figure 14: Sequence mean reward distributions for correct and incorrect student rollouts. Both teachers assign higher reward to correct rollouts with comparable AUROC (0.73 and 0.75).

Figure 15: Effect of the support size

k

in Top-

k

OPD. All numbers are reported as avg@16.

Figure 16: Training dynamics under different support sizes

k

for Top-

k

OPD.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Hyper-parameter	Value
Base model	Qwen3-4B-Base
RL algorithm	GRPO
Training epochs	1
Train batch size	64
Micro batch size	64
Rollout $n$	8

Appendix

Table 1: Training hyperparameters of GRPO for Qwen3-4B-Base-GRPO.

Item	Value
Training temperature	1.0
Global batch size	64
Mini batch size	64
Rollout number	4
LogProb top- $K$	16
Top- $K$ strategy	Student Top- $K$

Appendix

Table 2: Default hyperparameters for OPD.

Figure 17: Benchmark-wise breakdown of the average validation accuracy shown in Figure 2 . We report results on AIME 2024, AIME 2025, and AMC 2023 separately. Distillation from Qwen3-4B-Base-GRPO consistently matches or outperforms distillation from Qwen3-4B (Non-thinking) across the three benchmarks.

Figure 18: Probability mass assigned to overlap tokens during training. For both the student and teacher distributions, the overlap tokens consistently account for roughly 97%–99% of the total probability mass, indicating that the overlap is not only increasing at the set level but also dominates the probability distribution.

Figure 19: Auxiliary optimization diagnostics for the contrastive OPD setting in Section 4.1 , using R1-Distill-1.5B as the student and comparing JustRL-1.5B against R1-Distill-7B as the teacher. Left: batch-averaged OPD training loss ( PG Loss ) over training. Middle: gradient norm over training. Right: probability difference

p_{t}(v)-q_{t}(v)

measured on the token with the largest absolute advantage. The successful run exhibits a large reduction in optimization loss, sustained gradient magnitude, and a steady decrease in extreme-token probability mismatch. By contrast, the failing run starts with and maintains much weaker gradients, and its extreme-token probability discrepancy remains noticeably larger throughout training.

Figure 20: Cross-model validation with a fixed student (R1-Distill-7B) and two teachers. Top: avg@16 accuracy on AIME 2024, AIME 2025, and AMC 2023. Bottom: overlap ratio, overlap-token advantage, and absolute entropy gap over training. The successful run is again accompanied by increasing high-probability-token alignment, while the stagnating run is not.

Hyper-parameter	Value
Student model	Qwen3-1.7B-Base
Training objective	Full-parameter SFT
Template	qwen3
Training epochs	1
Sequence length	14,336
Per-device batch size	8

Appendix

Table 3: SFT hyperparameters for cold-start distillation from Qwen3-4B (Non-thinking) to Qwen3-1.7B-Base.

Figure 21: Student overlap mass and teacher overlap mass during training for SFT-initialized and base-initialized students.

Figure 22: Benchmark-wise breakdown of the average validation accuracy shown in Figure 9 . Using the teacher-aligned template consistently matches or outperforms the original DAPO template across the three benchmarks.

Figure 23: Average teacher entropy across decoding positions during OPD training with 15K max response length, measured on student-generated trajectories from Step 180 to Step 250. Elevated entropy first emerges in the suffix and gradually propagates toward earlier output positions over training.

Language Model Distillation

Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.

See all 489 papers

The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes
May 11, 2026Siqi Zhu, Xuyan Ye, Hongyu Lu +2On-Policy Self-Distillation On-Policy Distillation
1UIUC · 2Renmin University of China · †Work done during an internship at UIUC. +1
Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes
Mar 26, 2026Yuqian Fu, Haohuan Huang, Kaiwen Jiang +4Language Model Distillation On-Policy Distillation
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA · School of Artificial Intelligence, UCAS · Fudan University +1
Dr. OPD: Learning What to Follow for Optimal On-Policy Distillation of Large Language Models
Sep 29, 2026Zhenyu Wang, Tianze Wang, Linjun Zhang +1Language Model Distillation Bilevel Optimization
Department of Statistics, Rutgers University

May 11, 2026J/K move · Enter open · S save

The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes

Siqi Zhu, Xuyan Ye, Hongyu Lu, Weiye Shi+1

1UIUC · 2Renmin University of China · †Work done during an internship at UIUC.+1

On-Policy Self-Distillation On-Policy Distillation Learning Using Privileged Information Teacher-Student Learning

Open paper AskPDF arXiv

On-policy distillation (OPD) and on-policy self-distillation (OPSD) have emerged as promising post-training methods for large language models, offering dense token-level supervision on trajectories sampled from the model's own policy. However, existing results on their effectiveness remain mixed: while OP(S)D has shown promise in system prompt and knowledge internalization, recent studies also report instability and degradation. In this work, we present a comprehensive empirical study of when OPD and OPSD work, when they fail, and why. We find that OPD on mathematical reasoning is highly sensitive to teacher choice and loss formulation, whereas OPSD fails in our tested settings due to test-time absence of instance-specific privileged information (PI). In contrast, OPSD is effective when PI represents a shared latent rule, such as a system prompt or alignment preference. We identify three failure mechanisms: (1) distribution mismatch between teacher and student caused by conditioning on student-generated prefixes, (2) optimization instability from biased TopK reverse-KL gradients, and (3) an OPSD-specific limitation where the student learns a PI-free policy that aggregates PI-conditioned teachers, which is insufficient when PI is instance-specific. We further show that stop-gradient TopK objectives, RLVR-adapted teachers, and SFT-stabilized students mitigate these failures.

Abstract

Language Model Distillation

Explore similar work

The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes

Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes

Dr. OPD: Learning What to Follow for Optimal On-Policy Distillation of Large Language Models