A Survey of On-Policy Distillation for Large Language Models

Mingyang Song; Mao Zheng

cs.LGApr 1, 2026

A Survey of On-Policy Distillation for Large Language Models

Authors: Mingyang Song, Mao Zheng

Organizations: Large Language Model Department, Tencent, China

Abstract

As Large Language Models continue to grow in both capability and cost, transferring frontier capabilities into smaller, deployable students has become an important engineering problem, and knowledge distillation remains a common technique for this transfer. The prevailing recipe in industrial pipelines, static imitation of teacher-generated text, carries a structural weakness that grows more severe as tasks become longer and more reasoning-intensive. Because the student is trained on flawless teacher prefixes but generates its own at inference, small errors tend to accumulate into trajectories it has rarely been trained to recover from, and the resulting exposure bias has been shown to scale roughly with the square of sequence length. On-Policy Distillation reorganizes the training loop around this observation by having the teacher provide feedback on what the student actually produces, with the goal of reducing the compounding term toward linear and reframing distillation as an iterative correction process rather than single-pass imitation. The resulting literature has expanded along divergence design, reward-guided optimization, and self-play, yet contributions remain scattered across the knowledge distillation, RLHF, and imitation learning communities without a unified treatment. This survey provides such a treatment. We formalize OPD as f-divergence minimization over student-sampled trajectories, organize the field along three design axes (what to optimize, where the signal comes from, and how to stabilize training in practice), and consolidate success conditions, recurring failure modes, and the connection between OPD and KL-constrained reinforcement learning. We close with open problems that emerge from this synthesis, including distillation scaling laws, uncertainty-aware feedback, agent-level distillation, and the growing overlap between knowledge distillation and RL.

Figures & tables

Method	Category	Core Contribution	Signal Source	Loss Granularity
GKD ( Agarwal et al., 2024 )	Objective (Fixed)	DAgger for LLMs, interpolation	White-Box Logits	Token
DistiLLM ( Ko et al., 2024 )		Skewed KL mixtures	White-Box Logits	Token
MiniLLM ( Gu et al., 2024 )		Sequence-level Reverse KL via RL	White-Box Logits	Sequence
EOPD ( Jin et al., 2026b )	Objective (Adaptive)	Entropy-aware FKL+RKL blend	White-Box Logits	Token
G-OPD ( Yang et al., 2026e )	Objective (RL)	OPD equivalence to KL-RL	White-Box Logits	Sequence/Token
RLKD ( Xu et al., 2025b )	Objective (RL)	KD augmented with reward model	Reward Model	Sequence

Table 1: Comparison of key OPD methods categorized by their primary contribution.

Method	Teacher	Student	Training Signal	Key Benchmarks
Mathematical Reasoning
G-OPD ( Yang et al., 2026e )	Qwen3-30B-A3B-Instruct	Qwen3-1.7B/4B	Dense logit + reward extrap.	AIME, HMMT, Code
OPSD ( Zhao et al., 2026b )	Self (GT-conditioned)	Qwen3-1.7B/4B/8B-Inst	GT-PI self-distillation	AIME, HMMT
PACED ( Xu et al., 2026d )	Qwen3-14B	Qwen3-8B	Beta-kernel curriculum	MATH-500, AIME
KDRL ( Xu et al., 2025a )	Skywork-OR1-Math-7B	R1-Distill-Qwen-1.5B/DeepScaleR	Joint KD + RL	MATH500, Minerva, AIME24
LUFFY ( Yan et al., 2025 )	DeepSeek-R1 (off-policy traces)	Qwen2.5-Math-7B/1.5B	Mixed-policy GRPO	AIME, AMC, MATH-500

Table 2: Representative experimental configurations of on-policy distillation methods across different application domains.

Method	Year	Category	Divergence/Objective	Signal	Granularity	Key Innovation
GKD ( Agarwal et al., 2024 )	2023	Objective	FKL / RKL / JSD	White-box	Token	Unified OPD framework, DAgger for LLMs
MiniLLM ( Gu et al., 2024 )	2023	Objective	Reverse KL	White-box	Sequence	REINFORCE with teacher as reward
DistiLLM ( Ko et al., 2024 )	2024	Objective	Skew KL (SKL+SRKL)	White-box	Token	$\alpha$ -smoothing for KL stability
DistiLLM-2 ( Ko et al., 2025 )	2025	Objective	Contrastive SKL/SRKL	White-box	Token	Asymmetric loss per data source
KETCHUP ( Fan et al., 2025 )	2025	Objective	Reverse KL	White-box	Sequence	$K$ -step Bellman return estimation
vOPD ( Oh et al., 2026 )	2026	Objective	Reverse KL + CV baseline	White-box	Token	Control variate variance reduction

Table 3: OPD methods: Fixed divergence objectives.

Panel of Figure 2 — Figure 2: Forward KL vs. Reverse KL divergence for fitting a student distribution $P_{S}$ to a bimodal teacher $P_{T}$ . (a) The teacher places equal probability mass on two valid modes (e.g., two correct answers or two stylistic variants). (b) Forward KL minimizes $D_{\mathrm{KL}}(P_{T}\parallel P_{S})$ and is mode-covering (zero-avoiding): the student spreads a single broad Gaussian over both modes and places substantial probability in the inter-mode hallucination zone (red dashed boundary), corresponding to outputs that are a blend of the two valid answers and therefore neither. (c) Reverse KL minimizes $D_{\mathrm{KL}}(P_{S}\parallel P_{T})$ and is mode-seeking (zero-forcing): the student concentrates sharply onto a single peak (Mode 2), dropping Mode 1 entirely but producing crisp, coherent generations within the chosen mode. This trade-off motivates the adaptive-divergence methods in § 4 (EOPD, AOPD), which interpolate between the two regimes based on per-token teacher entropy or log-ratio statistics.

Method	Year	Category	Divergence/Objective	Signal	Granularity	Key Innovation
EOPD ( Jin et al., 2026b )	2026	Objective	Continuous FKL+RKL blend	White-box	Token	Entropy-aware on-policy distillation
AOPD ( Jia et al., 2026 )	2026	Objective	Asymmetric PG/FKL switch	White-box	Token	Advantage-sign token-level switching
CREDIT ( Shen et al., 2026b )	2026	Self (EF)	Contrastive pMI decomposition	Self (contrastive)	Token	Input-specific credit for OPSD reward
OGLS-SD ( Yang et al., 2026f )	2026	Self (EF)	Outcome-guided logit steering	Self (outcome)	Token	Outcome-based teacher calibration
Position-Weighted OPSD ( Liu et al., 2026d )	2026	Objective	Position-aware reliability	Self (PI)	Token	Per-position reliability score
Direction-Adaptive SD ( Zhang et al., 2026c )	2026	Objective	Entropy-routed polarity	Self (PI)	Token	High-entropy teacher reversal

Table 4: OPD methods: Adaptive divergence objectives.

Method	Year	Category	Divergence/Objective	Signal	Granularity	Key Innovation
G-OPD ( Yang et al., 2026e )	2026	Objective	KL-Constrained RL	White-box	Token/Seq	OPD $\equiv$ dense KL-RL, reward extrapolation
RLKD ( Xu et al., 2025b )	2025	Objective	KD + Reward Model	Reward Model	Sequence	Generative Structure Reward Model (GSRM)
KDRL ( Xu et al., 2025a )	2025	Objective	Joint KD + RL	White-box	Token/Seq	On-policy KL regularizer during RL
RLAD ( Zhang et al., 2026o )	2026	Objective	Trust Region Ratio	White-box	Token/Seq	PPO-style selective teacher following
AlignDistil ( Zhang et al., 2025a )	2025	Objective	DPO + Reverse DPO	White-box	Token	Token-level RLHF via contrastive DPO reward
SuperCorrect ( Yang et al., 2025b )	2024	Objective	Template + Cross-DPO	White-box	Sequence	Hierarchical thought scaffolding

Table 5: OPD methods: RL-augmented objectives.

Method	Year	Category	Divergence/Objective	Signal	Granularity	Key Innovation
DSKD ( Zhang et al., 2025b )	2025	Signal	Dual-Space KL	White-box	Token	Cross-vocabulary projection
PromptKD ( Kim et al., 2024 )	2024	Signal	Prompt-based elicitation	White-box	Token	Input-side distillation
MAD-OPD ( Wang et al., 2026c )	2026	Signal	Multi-agent debate ensemble	White-box	Token	Confidence-weighted multi-teacher
Veto ( Jang et al., 2026 )	2026	Signal	Geometric bridge KL	White-box	Token	Adaptive target reformulation
SimCT ( Sun et al., 2026 )	2026	Signal	Multi-token continuation KL	White-box	Token	Cross-tokenizer supervision recovery
CSD ( Kim et al., 2026c )	2025	Signal	Continuous score matching	White-box	Token	Bypasses discrete vocabulary mismatch

Table 6: OPD methods: White-box logit supervision innovations.

Method	Year	Category	Divergence/Objective	Signal	Granularity	Key Innovation
GAD ( Ye et al., 2025 )	2025	Signal	Adversarial matching	Black-box	Sequence	Discriminator as proxy reward
Lion ( Jiang et al., 2023 )	2023	Signal	Verbal feedback	Black-box	Sequence	NL critique curriculum
OVD ( Xiong et al., 2026 )	2026	Signal	Verbal scores (0–9)	Black-box	Sequence	Score-based trajectory matching
LUFFY ( Yan et al., 2025 )	2025	Signal	Mixed-policy GRPO	Off-policy + RL	Sequence	Importance-weighted off-policy
PRISM ( Wang et al., 2026g )	2026	Signal	Black-box pre-alignment	Black-box (API)	Sequence	Multimodal OPD before RLVR
ROPD ( Fang et al., 2026 )	2026	Signal	Rubric-based scoring	Black-box (API)	Sequence	Structured semantic rubrics, 10 $\times$ sample efficiency

Table 7: OPD methods: Black-box and API-constrained distillation.

Method	Year	Category	Divergence/Objective	Signal	Granularity	Key Innovation
OPSD ( Zhao et al., 2026b )	2026	Self (PI)	KL on GT-conditioned	Self	Token	Ground-truth as privileged info
CRISP ( Sang et al., 2026 )	2026	Self (PI)	Conciseness PI	Self	Token	57% token reduction, +9% accuracy
GATES ( Stein et al., 2026 )	2026	Self (PI)	Gated consensus KL	Self	Token	Document as PI, gated supervision
OPSDL ( Zhang et al., 2026g )	2026	Self (PI)	Short-ctx $\to$ long-ctx	Self	Token	Long-context self-distillation
SD-ZERO ( He et al., 2026 )	2026	Self (EF)	Self-revision preference	Self + Verifier	Sequence	Binary reward $\to$ dense self-supervision
$\pi$ -Play ( Zhang et al., 2026l )	2026	Self (EF)	Multi-agent co-evolution	Self	Token	Internally generated PI from co-evolution

Table 8: OPD methods: Self-distillation.

Method	Year	Category	Divergence/Objective	Signal	Granularity	Key Innovation
TIP ( Xu et al., 2026c )	2026	Dynamics	Soft-OR token weighting	White-box	Token	Entropy $\times$ divergence quadrant scoring
SCOPE ( Zheng et al., 2026a )	2026	Dynamics	Rollout routing	White-box	Token	Redirect on flawed prefix
SelecTKD ( Huang et al., 2025 )	2025	Dynamics	Confidence filtering	White-box	Token	Top-1 probability threshold
AdaSwitch ( Peng et al., 2025 )	2025	Dynamics	Exploration/guidance switch	White-box	Token	Binary mode switching
PACED ( Xu et al., 2026d )	2026	Dynamics	Beta-kernel curriculum	White-box	Sample	Frontier difficulty sampling
FOPD ( Zhang et al., 2026b )	2026	Dynamics	Prefix truncation	White-box	Hybrid	Early-token focus

Table 9: OPD methods: Training efficiency and stabilization.

Exposure Bias

See all 10 papers

Apr 14, 2026

Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe

On-policy distillation (OPD) has become a core technique in the post-training of large language models, yet its training dynamics remain poorly understood. This paper provides a systematic investigation of OPD dynamics and mechanisms. We first identify that two conditions govern whether OPD succeeds or fails: (i) the student and teacher should share compatible thinking patterns; and (ii) even with consistent thinking patterns and higher scores, the teacher must offer genuinely new capabilities beyond what the student has seen during training. We validate these findings through weak-to-strong reverse distillation, showing that same-family 1.5B and 7B teachers are distributionally indistinguishable from the student's perspective. Probing into the token-level mechanism, we show that successful OPD is characterized by progressive alignment on high-probability tokens at student-visited states, a small shared token set that concentrates most of the probability mass (97%-99%). We further propose two practical strategies to recover failing OPD: off-policy cold start and teacher-aligned prompt selection. Finally, we show that OPD's apparent free lunch of dense token-level reward comes at a cost, raising the question of whether OPD can scale to long-horizon distillation.

Yaxuan Li, Yuxin Zuo, Bingxiang He +8

Tsinghua University · ShanghaiTech University · University of Illinois Urbana-Champaign +1

Language Model Distillation On-Policy Distillation Knowledge Distillation Teacher-Student Learning

Ask PDF

May 11, 2026

The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes

On-policy distillation (OPD) and on-policy self-distillation (OPSD) have emerged as promising post-training methods for large language models, offering dense token-level supervision on trajectories sampled from the model's own policy. However, existing results on their effectiveness remain mixed: while OP(S)D has shown promise in system prompt and knowledge internalization, recent studies also report instability and degradation. In this work, we present a comprehensive empirical study of when OPD and OPSD work, when they fail, and why. We find that OPD on mathematical reasoning is highly sensitive to teacher choice and loss formulation, whereas OPSD fails in our tested settings due to test-time absence of instance-specific privileged information (PI). In contrast, OPSD is effective when PI represents a shared latent rule, such as a system prompt or alignment preference. We identify three failure mechanisms: (1) distribution mismatch between teacher and student caused by conditioning on student-generated prefixes, (2) optimization instability from biased TopK reverse-KL gradients, and (3) an OPSD-specific limitation where the student learns a PI-free policy that aggregates PI-conditioned teachers, which is insufficient when PI is instance-specific. We further show that stop-gradient TopK objectives, RLVR-adapted teachers, and SFT-stabilized students mitigate these failures.

Siqi Zhu, Xuyan Ye, Hongyu Lu +2

1UIUC · 2Renmin University of China · †Work done during an internship at UIUC. +1

On-Policy Self-Distillation On-Policy Distillation Learning Using Privileged Information Teacher-Student Learning

Ask PDF

Mar 26, 2026

Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes

On-policy distillation (OPD) is increasingly used in LLM post-training because it can leverage a teacher model to provide dense supervision on student rollouts. The standard implementation, however, usually reduces distribution matching to a sampled-token log-ratio, which can make the learning signal fragile on long rollouts whose prefixes drift away from the teacher's typical support. We revisit this formulation from both theoretical and implementation perspectives. Theoretically, token-level OPD is biased relative to sequence-level reverse-KL minimization, but admits a substantially tighter worst-case variance bound; a controlled synthetic study further shows that stronger future-reward coupling increases gradient variance and destabilizes training. Empirically, we identify three failure modes of sampled-token OPD: imbalanced token-level supervision, unreliable teacher guidance on student-generated prefixes, and tokenizer or special-token mismatch. These findings motivate teacher top-K local support matching, a truncated reverse-KL objective that compares teacher and student distributions over a teacher-supported token set at each prefix, together with top-p rollout sampling and special-token masking. Across single-task reasoning and multi-task benchmarks spanning agentic and reasoning settings, this objective improves optimization stability and yields a +19.8% performance gain over standard sampled-token OPD baselines, providing a practical recipe for more stable on-policy distillation.

Yuqian Fu, Haohuan Huang, Kaiwen Jiang +4

State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA · School of Artificial Intelligence, UCAS · Fudan University +1

Language Model Distillation On-Policy Distillation

Ask PDF

Abstract

Exposure Bias

Explore similar work

Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe

The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes

Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes