On-Policy Distillation
Overview
On-policy distillation studies how to transfer a teacher’s capabilities by training a student on trajectories the student generates, rather than only on fixed teacher-written examples. Common approaches match teacher and student token distributions with KL-style objectives, often combining this dense feedback with verifier rewards, reinforcement learning, or feedback from successful peer attempts. Work spans language-model reasoning and tool-using agents as well as multimodal, speech, diffusion, and embodied models; methods increasingly select useful tokens, turns, or rollout lengths to reduce compute. A central tension is that student-generated states can make supervision more relevant, but teacher signals may be unreliable or mismatched, and repeated rollouts can be costly or destabilize training. 4 sentences total.
Latest papers
557 papers70 papers in the last four weeks, down 63% on the four weeks before. 0.5% of all new papers.
SpecOPD: Branch-Parallel On-Policy Distillation for Speculative Decoding
OmniOPD: Stable On-Policy Distillation for Joint Audio-Video Generation
Order-Only On-Policy Distillation
Bernoulli On-Policy Distillation: Don't Ignore High-Confidence Student Tokens
Better Answers, Weaker Evidence Control: Evidence Bypass in On-Policy Distillation
OC-OPD: Observation-coordinated Supervision for On-Policy Distillation
Search-Aware On-Policy Self-Distillation with Sibling Alignment for Generative Recommendation
Adaptive-Policy Distillation: Knowledge Injection under Information Asymmetry
DecisionRank: On-Policy Distillation for Decision-Model Rerankers
Turing LLM Post-Training: Calibrated Distillation and Efficient Dynamic-MoE Execution
Distilling the Shape of Uncertainty: What On-Policy Distillation Learns and How to Predict It
How Much Do Teacher Bridges Matter in Interleaved On-Policy Distillation?
ATI-OPD: Adaptive Teacher Intervention for Stable Long-Sequence On-Policy Distillation
Multi-Draft On-Policy Distillation for Diffusion Drafters
Live-OPSD: On-Policy Self-Distillation for Proactive Streaming Large Video Models
Proposing Better Rollouts for On-Policy Distillation
How Much Does the Hint Matter? Scope Conditions for Hindsight Self-Distillation in Financial Reasoning
Timing Matters Twice: When to Query and When to Skip in Flow-Matching On-Policy Distillation
Self-Evolution Reasoner: Continuous Optimization via Policy-Intrinsic Exploration and Exploitation
A Distribution-Matching Perspective on LLM Training Algorithms
TAD: Distilling Vision-Language-Action Policies from Teacher Trajectories Alone
PutOnPlateInScene25Main, \gad{} improves OpenVLA-OFT (7B) from to \successonce{} ( percentage points) and from to \successend{}. It trails white-box on-policy distillation (OPD) at / by / percentage points ( evaluation trajectories per checkpoint). The tested four-epoch continuation achieves the highest \gad{} result without additional teacher data. This best run uses OPD's student interaction budget and its complete-rollout batch-passes. The comparison demonstrates online VLA improvement without teacher scoring while quantifying unequal interaction and optimization budgets.