On-Policy Distillation

Follow this topic Ask about this topic

On-policy distillation studies how to transfer a teacher’s capabilities by training a student on trajectories the student generates, rather than only on fixed teacher-written examples. Common approaches match teacher and student token distributions with KL-style objectives, often combining this dense feedback with verifier rewards, reinforcement learning, or feedback from successful peer attempts. Work spans language-model reasoning and tool-using agents as well as multimodal, speech, diffusion, and embodied models; methods increasingly select useful tokens, turns, or rollout lengths to reduce compute. A central tension is that student-generated states can make supervision more relevant, but teacher signals may be unreliable or mismatched, and repeated rollouts can be costly or destabilize training. 4 sentences total.

Latest papers

507 papers177 papers in the last four weeks, up 139% on the four weeks before. 1.0% of all new papers.

Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation
May 13, 2026Kaiyuan Liu, Ziyuan Zhuang, Yang Bai +3Selective Knowledge Distillation Language Model Distillation
College of Computer Science and Technology, Zhejiang University · Meituan LongCat Team, China · College of Computer Science and Technology, Jilin University
Reward-Weighted On-Policy Distillation with an Open Property-Equivalence Verifier for NL-to-SVA Generation
May 13, 2026Qingyun Zou, Yingze Li, Tianen Liu +2Formal Hardware Verification Formal Verification
National University of Singapore
Teacher-Guided Policy Optimization for On-Policy Reasoning Distillation under Large Policy Divergence
May 13, 2026Xinyu Liu, Kechen Jiao, Chunyang Xiao +10RL for Language Model Reasoning Policy Optimization
School of Computer Science and Engineering, Northeastern University, China · Tsinghua University · Meituan +2
Multi-Rollout On-Policy Distillation via Peer Successes and Failures
May 12, 2026Weichen Yu, Xiaomin Li, Yizhou Zhao +8RL for Language Model Reasoning Language Model Distillation
1Carnegie Mellon University · 2Microsoft · 3Purdue University
Beyond GRPO and On-Policy Distillation: An Empirical Sparse-to-Dense Reward Principle for Language-Model Post-Training
May 12, 2026Yuanda Xu, Hejian Sang, Zhengze Zhou +3Reinforcement Learning RL for Language Model Reasoning
OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoning
May 12, 2026Yuxiao Yang, Xiaoyun Wang, Weitong ZhangLanguage Model Calibration Language Model Distillation
UNC Chapel Hill · NVIDIA
A Recipe for Long-Context Reasoning in Large Language Models via On-Policy Optimization and Distillation
May 12, 2026Miguel Moura Ramos, Duarte M. Alves, André F. T. MartinsLLM Alignment RL for Language Model Reasoning
Instituto Superior Técnico, Universidade de Lisboa · Instituto de Telecomunicações · TransPerfect +1
Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation
May 12, 2026Yuchen Cai, Ding Cao, Liang Lin +9Efficient Language Model Training On-Policy Distillation
1USTC · 2Tencent · 4HKUST(GZ) +3
The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes
May 11, 2026Siqi Zhu, Xuyan Ye, Hongyu Lu +2On-Policy Self-Distillation On-Policy Distillation
1UIUC · 2Renmin University of China · †Work done during an internship at UIUC. +1
Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why
May 11, 2026Mohammadreza Armandpour, Fatih Ilhan, David Harrison +6On-Policy Self-Distillation Language Model Distillation
Apple
ProteinOPD: Towards Effective and Efficient Preference Alignment for Protein Design
May 11, 2026Yulin Zhang, He Cao, Zihao Jiang +6Protein Language Models Multi-Teacher Knowledge Distillation
Tsinghua University · International Digital Economy Academy · Hong Kong University of2026 Science and Technology (Guangzhou) +1
On-Policy Distillation with Best-of-N Teacher Rollout Selection
May 10, 2026Ke Zhang, Yunjie Tian, Dongdi Zhao +4CoT Distillation Best-of-N Sampling
Johns Hopkins University · TikTok · University of California, San Diego +1
Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillation
May 10, 2026Yuxuan Jiang, Runchao Li, Shubhashis Roy Dipta +2Language Model Distillation On-Policy Distillation
University of Maryland, Baltimore County · Case Western Reserve University · Arizona State University +1
CoDistill-GRPO: A Co-Distillation Recipe for Efficient Group Relative Policy Optimization
May 9, 2026Soo Min Kwon, Ziteng Sun, Ananda Theertha Suresh +2RL for Language Model Reasoning Group Relative Policy Optimization
University of Michigan, Ann Arbor · *Work done during an internship at Google · Google Research +1
ErrOPD: When Does On-Policy Distillation Transfer Teacher Errors?
May 9, 2026Xin Li, Hao Jiang, Annan Wang +2On-Policy Distillation Structured Output Generation
Nanyang Technological University, Singapore
Flow-OPD: On-Policy Distillation for Flow Matching Models
May 8, 2026Zhen Fang, Wenxuan Huang, Yu Zeng +8Diffusion Model Alignment Flow Matching
University of Science and Technology of China · University of California, Los Angeles · 3The Chinese University of Hong Kong +1
KL for a KL: On-Policy Distillation with Control Variate Baseline
May 8, 2026Minjae Oh, Sangjun Song, Gyubin Choi +2Language Model Distillation Policy Gradient Methods
Graduate School of Data Science Seoul National University
Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning
May 8, 2026Zhicheng Yang, Zhijiang Guo, Yifan Song +5RL for Language Model Reasoning Language Model Distillation
1The Hong Kong University of Science and Technology (Guangzhou) · 2The Hong Kong University of Science and Technology · 3MBZUAI +2
SOD: Step-wise On-policy Distillation for Small Language Model Agents
May 8, 2026Qiyong Zhong, Mao Zheng, Mingyang Song +5Tool-Integrated Reasoning LLM Agent Training
1Zhejiang University · 2Large Language Model Department, Tencent · University of Science and Technology of China +1
SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation
May 8, 2026Jie Sun, Mao Zheng, Mingyang Song +6Cross-Tokenizer Knowledge Distillation Language Model Distillation
University of Science and Technology of China · Shanghai Innovation Institute · Large Language Model Department, Tencent +1
Rubric-based On-policy Distillation
May 8, 2026Junfeng Fang, Zhepei Hong, Mao Zheng +7LLM Alignment Language Model Distillation
National University of Singapore · University of Science and Technology of China · Tencent
Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Level
May 7, 2026Nan Jia, Haojin Yang, Xing Ma +6RL for Language Model Reasoning Language Model Distillation
Huazhong University of Science and Technology · Peking University · Meituan
OPSD Compresses What RLVR Teaches: A Post-RL Compaction Stage for Reasoning Models
May 7, 2026Jaehoon Kim, Dongha LeeReinforcement Learning with Verifiable Rewards On-Policy Distillation
Yonsei University
Near-Policy: Accelerating On-Policy Distillation via Asynchronous Generation and Selective Packing
May 7, 2026Miao Rang, Zhenni Bi, Hang Zhou +6Language Model Distillation On-Policy Distillation
1Huawei Technologies · 2Tianjin University
Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularization
May 6, 2026Xin Yu, Liuchen Liao, Yiwen Zhang +3On-Policy Self-Distillation Language Model Distillation
1The Pennsylvania State University · 2TikTok
Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe
May 5, 2026Wenjin Hou, Shangpin Peng, Weinong Wang +13Language Model Distillation Multi-Teacher Knowledge Distillation
1Zhejiang University · 2Shenzhen Loop Area Institute · 3LLM Department, Tencent
MAD-OPD: Breaking the Ceiling in On-Policy Distillation via Multi-Agent Debate
May 2, 2026Jianze Wang, Ying Liu, Jinlong Chen +7Multi-Agent Debate Multi-Teacher Knowledge Distillation
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology · Alibaba Group
Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Grounding
May 1, 2026Yan Zhang, Daiqing Wu, Huawen Shen +2GUI Grounding On-Policy Distillation
Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · VCIP & TMCC & DISSec, College of Computer Science, Nankai University
Healthcare AI GYM for Medical Agents
May 1, 2026Minbyul JeongMedical LLMs RL for Tool Use
Upstage AI
Beyond SFT-to-RL: Pre-alignment via Black-Box On-Policy Distillation for Multimodal RL
Apr 30, 2026Sudong Wang, Weiquan Huang, Xiaomin Yu +9Multimodal Large Language Models Multimodal Reasoning
Hong Kong University of Science and Technology (Guangzhou) · Nanyang Technological University · Tsinghua University +3

May 13, 2026J/K move · Enter open · S save

Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation

Kaiyuan Liu, Ziyuan Zhuang, Yang Bai, Bing Wang+2

College of Computer Science and Technology, Zhejiang University · Meituan LongCat Team, China · College of Computer Science and Technology, Jilin University

Selective Knowledge Distillation Language Model Distillation On-Policy Distillation Teacher-Student Learning Knowledge Distillation

Open paper AskPDF arXiv

On-policy distillation (OPD) trains a student model on its own rollouts using dense feedback from a stronger teacher. Prior literature suggests that, provided teacher feedback is available, supervising the full sequence of response tokens should monotonically improve performance. However, we demonstrate that this assumption sometimes fails to hold in strong-to-weak OPD settings. While later segments of a generated trajectory may still exhibit a non-zero teacher-student advantage, they frequently lack the local contrast that makes dense feedback effective for prioritizing student learning. We term this failure mode local teachability collapse. The resulting principle is straightforward: supervision should concentrate on trajectory regions where the teacher's feedback remains discriminative, rather than uniformly covering the entire response. We operationalize this principle through a trajectory-specific release rule. This rule measures the teacher's margin over the student's top-

K

candidate set, aggregates this margin across NLTK-tokenized sentence segments, and truncates dense OPD supervision upon detecting a BIC-style downward change point. Experimental results across strong-to-weak distillation tasks using the Qwen3 model family indicate that this release rule consistently outperforms standard full-trajectory OPD across five in-domain benchmarks at various student scales. Furthermore, compared to baseline distillation methods, our approach better preserves model capabilities on out-of-domain task. These results suggest that effective strong-to-weak OPD requires evaluating not only the availability of teacher guidance but also its local utility, ensuring that the generated feedback remains teachable.