On-Policy Distillation

Follow this topic Ask about this topic

On-policy distillation studies how to transfer a teacher’s capabilities by training a student on trajectories the student generates, rather than only on fixed teacher-written examples. Common approaches match teacher and student token distributions with KL-style objectives, often combining this dense feedback with verifier rewards, reinforcement learning, or feedback from successful peer attempts. Work spans language-model reasoning and tool-using agents as well as multimodal, speech, diffusion, and embodied models; methods increasingly select useful tokens, turns, or rollout lengths to reduce compute. A central tension is that student-generated states can make supervision more relevant, but teacher signals may be unreliable or mismatched, and repeated rollouts can be costly or destabilize training. 4 sentences total.

Latest papers

511 papers178 papers in the last four weeks, up 141% on the four weeks before. 1.0% of all new papers.

SAGE-OPD: Selective Agent-Guided Intervention for Multi-Turn On-Policy Distillation
Jun 17, 2026Yuhang Zhou, Lizhu Zhang, Yifan Wu +5Selective Knowledge Distillation LLM Agent Training
Meta AI
PowerOPD: Stabilizing On-Policy Distillation with Bounded Power Transformation
Jun 15, 2026Anhao Zhao, Junlong Tong, Yingqi Fan +3On-Policy Distillation Teacher-Student Learning
Eastern Institute of Technology, Ningbo · Shanghai Jiao Tong University · University of Waterloo +1
On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents
Jun 14, 2026Gengsheng Li, Mao Zheng, Mingyang Song +8Language Model Distillation LLM Agent Training
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Large Language Model Department, Tencent +4
Localizing Credit at the Divergence: Path-Conditioned Self-Distillation for LLM Reasoning
Jun 14, 2026Yu Li, Shu Hong, Tian LanRL for Language Model Reasoning Credit Assignment in RL
Department of Electrical and Computer Engineering, George Washington University
Be My Tutor: On-Policy Co-Distillation for Mutual LLM Improvement via Peer Feedback
Jun 12, 2026Woohyeon Byeon, Jiwon Jeon, Jeonghye Kim +1Language Model Distillation On-Policy Distillation
KAIST
Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillation
Jun 11, 2026Guo Yu, Wenlin Liu, Yulan Hu +3Sparse Fine-Tuning On-Policy Distillation
School of Artificial Intelligence, Nanjing University · National Key Laboratory for Novel Software Technology, Nanjing University · Amap, Alibaba Group
RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation
Jun 10, 2026Leyi Pan, Shuchang Tao, Yunpeng Zhai +5RL for Language Model Reasoning Language Model Distillation
1Tsinghua University · 2Tongyi Lab , Alibaba Group · 3Peking University
When Context Returns: Toward Robust Internalization in On-Policy Distillation
Jun 10, 2026Xun Wang, Ruishuo Chen, Zhuoran Li +2On-Policy Distillation Context Distillation
1IIIS, Tsinghua University
HERO: Hindsight-Enhanced Reflection from Environment Observations for Agentic Self-Distillation
Jun 10, 2026Haoran Liu, Yuwei Zhang, Xiyao Li +2Credit Assignment in RL On-Policy Distillation
University of California, San Diego · Independent Researcher · University of California, Berkeley
Kwai Keye-VL-2.0 Technical Report
Jun 9, 2026Kwai Keye Team, Bin Wen, Changyi Liu +50Temporal Video Grounding Long-Context Modeling
Beyond Absolute Imitation: Anchored Residual Guidance for Privileged On-Policy Distillation
Jun 9, 2026Wenhao ZhangCoT Distillation Language Model Distillation
South China University of Technology
Escaping the KL Agreement Trap in On-Policy Distillation
Jun 8, 2026Haoran Xin, Anhao Zhao, Ying Sun +3On-Policy Distillation Teacher-Student Learning
Thrust of Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong Polytechnic University · Eastern Institute of Technology, Ningbo +1
Breaking the Tokenizer Barrier: On-Policy Distillation across Model Families
Jun 8, 2026Yifan Niu, Han Xiao, Dongyi Liu +4Cross-Tokenizer Knowledge Distillation Language Model Distillation
The Hong Kong University of Science and Technology (Guangzhou) · Tencent · The Hong Kong University of Science and Technology
SG-OPD: Sign-Gated On-Policy Distillation via Sign-Consistency Gating and Phased Teacher Sampling
Jun 8, 2026Haoran Xu, Hongyu Wang, Yifei Gao +3Language Model Distillation On-Policy Distillation
Zhejiang University · Hunan University · Tianjin University +2
Stabilizing On-Policy Distillation for MLLM Reasoning with Global Normalization
Jun 8, 2026Dongze Hao, Zhiwei Jin, Chen Chen +1Multimodal Reasoning On-Policy Distillation
OPPO AI Center
Stage-1 Controls the Entropy Regime, Not the Outcome
Jun 8, 2026Jianxiong ShenSupervised Fine-Tuning VLM Reasoning
Trajectory-Refined Distillation
Jun 7, 2026Li Jiang, Haoran Xu, Yichuan Ding +1On-Policy Distillation LLM Post-Training
McGill University · Mila Quebec AI Institute · UT Austin
On the Geometry of On-Policy Distillation
Jun 5, 2026Zhennan Shen, Yanshu Li, Qingyu Yin +6Language Model Distillation On-Policy Distillation
1HKUST · 2Brown University · 3Zhejiang University +5
Teaching the Way, Not the Answer: Privileged Tutoring Distillation for Multimodal Policy Optimization
Jun 5, 2026Shizhe Xiang, Ke An, Wenlong Yu +5Large Vision-Language Models Multimodal Reasoning
Tianjin University · Beijing Institute of Technology · Singapore Management University +2
Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation
Jun 4, 2026Xingyu Su, Jacob Helwig, Shubham Parashar +6Language Model Distillation Diffusion Language Models
Department of Computer Science and Engineering, Texas A&M University · Department of Bioinformatics and Systems Medicine,2026 University of Texas Health Science Center at Houston · Department of Electrical and Computer Engineering, Texas A&M UniversityJun 4
OPRD: On-Policy Representation Distillation
Jun 4, 2026Shenzhi Yang, Guangcheng Zhu, Bowen Song +8Cross-Architecture Knowledge Distillation Language Model Distillation
Zhejiang University · Ant Group
ViCuR: Visual Cues as Recoverable Privilege for Multimodal On-Policy Distillation
Jun 4, 2026Kanghui Tian, Siyuan Liu, Ziang Yan +3VLM Distillation Multimodal Reasoning
1Shanghai AI Laboratory · 3Nanjing University · 2Fudan University
Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillation
Jun 1, 2026Yuying Li, Leqi Zheng, Yongzi Yu +6Selective Knowledge Distillation Language Model Distillation
THU · Meituan · HKUST +2
SafeSteer: Localized On-Policy Distillation for Efficient Safety Alignment
Jun 1, 2026Hao Li, Jingkun An, Zijun Song +8LLM Safety Alignment On-Policy Distillation
Beihang University · Beijing Institute of Technology · Beijing University of Posts and Telecommunications +4
OmniOPD: Logit-Free On-Policy Distillation via Speculative Verification
May 31, 2026Yuhang Zhou, Lizhu Zhang, Yifan Wu +5RL for Language Model Reasoning Language Model Distillation
Meta AI
Trust Region On-Policy Distillation
May 31, 2026Xingrun Xing, Haoqing Wang, Boyan Gao +2Language Model Distillation Trust-Region Optimization
Samsung Research, Beijing, China · University of Oxford · Peking University
OPD+: Rethinking the Advantage Design for On-Policy Distillation
May 31, 2026Hanyang Zhao, Haoxian Chen, Han Lin +3Language Model Distillation Policy Gradient Methods
Columbia University · Amazon · Meta +1
Decomposed On-Policy Distillation for Vision-Language Reasoning: Steering Gradients for Visual Grounding
May 30, 2026Hee Suk Yoon, Eunseop Yoon, Jaehyun Jang +6Cross-Modal Knowledge Distillation VLM Distillation
Korea Advanced Institute of Science and Technology (KAIST) · University of Illinois Urbana-Champaign (UIUC) · Microsoft Research Asia (MSRA)
Weak Critics Make Strong Learners: On-Policy Critique Distillation for Scalable Oversight
May 29, 2026Can Jin, Jiakang Li, Rui Wu +2Scalable Oversight Weak-to-Strong Generalization
Department of Computer Science, Rutgers University, Piscataway, United States.
Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidance
May 29, 2026Yuxuan Jiang, Francis FerraroCoT Distillation Language Model Distillation
University of Maryland, Baltimore County

Jun 17, 2026J/K move · Enter open · S save

SAGE-OPD: Selective Agent-Guided Intervention for Multi-Turn On-Policy Distillation

Yuhang Zhou, Lizhu Zhang, Yifan Wu, Mingyi Wang+4

Meta AI

Selective Knowledge Distillation LLM Agent Training On-Policy Distillation Knowledge Distillation

Open paper AskPDF arXiv

On-policy distillation (OPD) improves student models by training them on trajectories induced by their own policy, making it a promising approach for mitigating exposure bias in agent training. However, most OPD studies focus on single-turn settings, while realistic LLM agents interact with environments over multiple turns. In this regime, early errors can alter future observations and compound across the trajectory, and standard dense token-level OPD becomes brittle, as it may over-penalize semantically valid alternatives, reinforce local degeneracies such as repeated actions, and propagate unreliable teacher supervision on off-distribution histories. We propose SAGE-OPD, a verifier-free selective intervention framework specifically designed for multi-turn OPD. Instead of applying teacher supervision uniformly across all turns, SAGE-OPD first observes environment feedback and uses teacher judgment to decide whether each student response should be skipped or intervened on. To further address compounding errors, SAGE-OPD weights token-level distillation by teacher confidence, reducing the influence of uncertain teacher distributions on corrupted or ambiguous histories. Finally, SAGE-OPD applies loss normalization to preserve the overall loss scale of standard OPD while retaining selective turn-level weighting. Experiments on agent tasks show that SAGE-OPD consistently improves over baselines, achieving up to a 13.3% relative improvement in ALFWorld unseen success rate over standard OPD. Ablation studies further demonstrate that turn-level intervention, teacher confidence weighting, and loss normalization provide complementary benefits. Our results suggest that effective multi-turn OPD should remain on-policy, but teacher supervision should be selectively allocated to turns where intervention is necessary and reliable.