cs.LGAug 26, 2026

One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillation

Authors: Justin Robert, Raheel Qader

Organizations: OVHai LLM

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

On-policy distillation trains a language model on its own generations while a teacher scores them token by token. It combines the dense supervision of imitation learning with the on-policy sampling of reinforcement learning. But it requires a second, larger model to act as teacher. On-Policy Self-Distillation (OPSD) removes that cost. The teacher is the model itself, conditioned on privileged information the student will not have at test time, such as a reference solution, a plan, or environment feedback. The teacher is no stronger than the student, only better informed. Early results were promising, with accuracy comparable to reinforcement learning at a fraction of the generated tokens. But the same asymmetry that produces the signal also biases it. One failure mode now dominates the field: collapse, the progressive narrowing of the set of reasoning paths the model can produce. Collapse is not specific to OPSD, though privileged information aggravates it. This review treats collapse as a symptom governed by three levers: (i) where the signal is applied, that is, how tokens are weighted; (ii) what the teacher is shown, that is, the nature of the privileged information; and (iii) when the signal changes, that is, the teacher's dynamics and the decay of guidance. We restrict our scope to mathematical reasoning, where the method originated and where its failure modes are best documented. We report no new experiments. The contribution is structural: a shared vocabulary for phenomena named differently across papers, and a clear line between what is settled and what is still disputed.

Figures & tables

Figure 1: Genealogy of the training methods leading to OPSD.
Figure 2: Overview of On-Policy Self-Distillation (OPSD).
Figure 3: Forward KL versus reverse KL.
Nature of the PIReconstructible at test timeMeasured effectRegime testedRequires
Final answer only [ 56 , 24 ]noharmful or inert (a)strong →\to weak; self-dist.annotated solutions
Solution: reasoning + answer (OPSD) [ 61 ]noharmful on thinking (b)self-dist.annotated solution
Partial trace, anchor (AR-OPD) [ 60 ]partialpositive (c)self-dist.annotated traces
Plan, hints (DOPD) [ 56 ]yespositive (d)strong →\to weakannotated solutions + generation
Error-aligned critique [ 23 ]yespositive (e)self-dist.critic model
Rubric [ 15 ]yesdiversity yes, accuracy little (f)self-dist.manual authoring
Table 1: Not all privileged information is equal, and the kind easiest to obtain is the least useful. Entries are ordered by broadly decreasing presupposition of the solution; the first two presuppose the answer equally and differ in whether the derivation is supplied. The Regime column distinguishes strict self-distillation (the OPSD framework) from distillation of a strong model into a weak one (the OPD framework). The Requires column indicates the resources needed to construct the privileged information.
Figure 4: The three levers, mapped onto the loop of Figure 2 . Lever C appears at two points: the teacher’s exposure to the privileged information and the update of its weights are two independent mechanisms.
More in this topic

On-Policy Self-Distillation

On-policy self-distillation trains a model on its own generated trajectories using a teacher view to provide dense supervision.

See all 156 papers

Explore similar work

CardsList
May 11, 2026

The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes

On-policy distillation (OPD) and on-policy self-distillation (OPSD) have emerged as promising post-training methods for large language models, offering dense token-level supervision on trajectories sampled from the model's own policy. However, existing results on their effectiveness remain mixed: while OP(S)D has shown promise in system prompt and knowledge internalization, recent studies also report instability and degradation. In this work, we present a comprehensive empirical study of when OPD and OPSD work, when they fail, and why. We find that OPD on mathematical reasoning is highly sensitive to teacher choice and loss formulation, whereas OPSD fails in our tested settings due to test-time absence of instance-specific privileged information (PI). In contrast, OPSD is effective when PI represents a shared latent rule, such as a system prompt or alignment preference. We identify three failure mechanisms: (1) distribution mismatch between teacher and student caused by conditioning on student-generated prefixes, (2) optimization instability from biased TopK reverse-KL gradients, and (3) an OPSD-specific limitation where the student learns a PI-free policy that aggregates PI-conditioned teachers, which is insufficient when PI is instance-specific. We further show that stop-gradient TopK objectives, RLVR-adapted teachers, and SFT-stabilized students mitigate these failures.

Siqi Zhu, Xuyan Ye, Hongyu Lu +2

1UIUC · 2Renmin University of China · †Work done during an internship at UIUC. +1

On-Policy Self-DistillationOn-Policy DistillationLearning Using Privileged InformationTeacher-Student Learning
AskPDF
Sep 30, 2026

Diagnosing On-Policy Self-Distillation for Reasoning Language Models

On-policy self-distillation (OPSD) has attracted growing interest as a promising approach to improve the reasoning ability of language models. Without external rewards nor a separate stronger teacher, the self-teacher with privileged information could provide dense signals on student's trajectories. However, its behavior in language reasoning remains unclear, with reported outcomes ranging from modest gains to behavioral collapse. In this work, we diagnose OPSD for mathematical reasoning across models spanning 0.6B--8B parameters. We conduct controlled experiments and token-level analyses to fully delve into OPSD. We point out that teacher's signal is shaped by reasoning-mode alignment and the complete teacher prefix, rather than by privileged semantics alone. OPSD improves reasoning only in narrow compatibility regimes. Otherwise, it produces ineffective length growth, stable degradation, or behavioral collapse. Token-level analysis shows that teacher's signal is not stable and does not predict downstream performance. Based on these results, we argue that OPSD is a sensitive algorithm rather than a generally reliable reasoning-improvement post-training method.

Yang Li, Gongle Xue, Yuheng Yuan +4

Peking University

Language Model DistillationOn-Policy DistillationLLM Reasoning
AskPDF
Aug 8, 2026

Matching Supervision to the Student's Learning Capacity: A Unified Framework for On-Policy Self-Distillation

On-policy self-distillation (OPSD) improves the reasoning abilities of LLMs by internalizing privileged context into model parameters through self-distillation. Two recent research lines promote vanilla OPSD by choosing which tokens to learn from and by controlling how much privileged information the teacher receives, respectively. However, we show that each line optimizes one variable while holding the other fixed, which leads to a suboptimal solution. We argue that the two variables are coupled through the student's learning capacity: the privileged information sets the per-token divergence the teacher prescribes, while token weighting selects which of these the student must absorb. We formalize the two lines of work into a unified optimization framework, which maximizes the aggregate teacher--student divergence, subject to a budget on the aggregate learning difficulty the student can absorb. Under this modelling, we propose Unified On-Policy Self-Distillation (USD), a lightweight online algorithm to solve the Lagrangian. USD reveals that a single dual variable governs both decisions: at one price for learning difficulty, it simultaneously sets the token-selection threshold and the direction of privileged-information adjustment, keeping supervision matched to the student's evolving capacity. Through extensive experiments, USD consistently demonstrates superior performance over OPSD and token- and PI-side baselines across various model scales on various reasoning benchmarks. Code is available at https://github.com/lauvlalala/USD.

Yongkang Yang, Zhezheng Hao, Hong Zhang +8

Tencent, Workbuddy Team · Zhejiang University · Independent Researcher

Adaptive Knowledge DistillationOn-Policy Self-DistillationLanguage Model DistillationKnowledge DistillationLLM Reasoning
AskPDF