On-Policy Distillation

Follow this topic Ask about this topic

On-policy distillation studies how to transfer a teacher’s capabilities by training a student on trajectories the student generates, rather than only on fixed teacher-written examples. Common approaches match teacher and student token distributions with KL-style objectives, often combining this dense feedback with verifier rewards, reinforcement learning, or feedback from successful peer attempts. Work spans language-model reasoning and tool-using agents as well as multimodal, speech, diffusion, and embodied models; methods increasingly select useful tokens, turns, or rollout lengths to reduce compute. A central tension is that student-generated states can make supervision more relevant, but teacher signals may be unreliable or mismatched, and repeated rollouts can be costly or destabilize training. 4 sentences total.

Latest papers

507 papers177 papers in the last four weeks, up 139% on the four weeks before. 1.0% of all new papers.

Co-Evolving Policy Distillation
Apr 29, 2026Naibin Gu, Chenxu Yang, Qingyi Si +7Reinforcement Learning with Verifiable Rewards Multi-Teacher Knowledge Distillation
Institute of Information Engineering, CAS, 2School of Cyber Security, UCAS, 3JD.COM
TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents
Apr 27, 2026Jiaqi Wang, Wenhao Zhang, Weijie Shi +2Language Model Distillation LLM Agents
Tongyi Lab , Alibaba Group · † The Chinese University of Hong Kong.
Learning from the Near Future: Temporal Self-Distillation for RLVR
Apr 22, 2026Chuanyu Qin, Chenxu Yang, Qingyi Si +6Reinforcement Learning with Verifiable Rewards Self-Distillation
Institute of Information Engineering, CAS · School of Cyber Security, UCAS
Hybrid Policy Distillation for LLMs
Apr 22, 2026Wenhong Zhu, Ruobing Xie, Rui Wang +1Language Model Distillation On-Policy Distillation
Department of Computer Science and Engineering, Shanghai Jiao Tong University · Shanghai Innovation Institute · Large Language Department, Tencent
The Illusion of Certainty: Decoupling Capability and Calibration in On-Policy Distillation
Apr 18, 2026Jiaxin Zhang, Xiangyu Peng, Qinglin Chen +3On-Policy Self-Distillation Language Model Calibration
Salesforce AI Research
TIP: Token Importance in On-Policy Distillation
Apr 15, 2026Yuanda Xu, Hejian Sang, Zhengze Zhou +3Knowledge Distillation On-Policy Distillation
Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
Apr 14, 2026Yaxuan Li, Yuxin Zuo, Bingxiang He +8Language Model Distillation On-Policy Distillation
Tsinghua University · ShanghaiTech University · University of Illinois Urbana-Champaign +1
Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation
Apr 14, 2026Yecheng Wu, Song Han, Han CaiOn-Policy Distillation LLM Post-Training
NVIDIA
SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting
Apr 12, 2026Binbin Zheng, Xing Ma, Yiheng Liang +6On-Policy Distillation RL for Language Model Reasoning
University of Science and Technology of China · Meituan LongCat Interaction Team · Nanjing University +2
Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models
Apr 9, 2026Feng Luo, Yu-Neng Chuang, Guanchu Wang +4Repetition in Language Generation Language Model Distillation
Department of Computer Science, Rice University, Houston, USA · Department of Computer Science, University of North Carolina at Charlotte, Charlotte, USA · Department of Computer Science, Johns Hopkins University, Baltimore, USA +1
On-Policy Distillation of Language Models for Autonomous Vehicle Motion Planning
Apr 9, 2026Amirhossein Afsharrad, Amirhesam Abedsoltan, Ahmadreza Moradipari +1Language Model Distillation Autonomous Driving Planning
Stanford University · University of California, San Diego · University of California, Santa Barbara
DP-OPD: Differentially Private On-Policy Distillation for Language Models
Apr 6, 2026Fatemeh Khadem, Sajad Mousavi, Yi Fang +1Privacy-Preserving Language Models Privacy-Preserving ML
Santa Clara University, Santa Clara, California, USA · Independent Researcher
SODA: Semi On-Policy Black-Box Distillation for Large Language Models
Apr 4, 2026Xiwen Chen, Jingjing Wang, Wenhui Zhu +7Language Model Distillation Knowledge Distillation
Clemson University · LinkedIn · Washington University in St. Louis +2
Self-Distilled RLVR
Apr 3, 2026Chenxu Yang, Chuanyu Qin, Qingyi Si +7On-Policy Self-Distillation Reinforcement Learning with Verifiable Rewards
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China
A Survey of On-Policy Distillation for Large Language Models
Apr 1, 2026Mingyang Song, Mao ZhengExposure Bias Language Model Distillation
Large Language Model Department Tencent, China
VLA-OPD: Bridging Offline SFT and Online RL for Vision-Language-Action Models via On-Policy Distillation
Mar 27, 2026Zhide Zhong, Haodong Yan, Junfeng Li +3Vision-Language-Action Models On-Policy Distillation
HKUST (GZ)
Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes
Mar 26, 2026Yuqian Fu, Haohuan Huang, Kaiwen Jiang +4Language Model Distillation On-Policy Distillation
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA · School of Artificial Intelligence, UCAS · Fudan University +1
HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation
Mar 25, 2026Ken DingRL for Language Model Reasoning LLM Reasoning
NVIDIA
Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation
Mar 19, 2026Zhuolin Yang, Zihan Liu, Yang Chen +14RL for Language Model Reasoning RL for LLM Agents
Scaling Reasoning Efficiently via Relaxed On-Policy Distillation
Mar 11, 2026Jongwoo Ko, Sara Abdali, Young Jin Kim +2On-Policy Distillation RL for Language Model Reasoning
Entropy-Aware On-Policy Distillation of Language Models
Mar 7, 2026Woogyeol Jin, Taywon Min, Yongjin Yang +5Language Model Distillation On-Policy Distillation
KAIST AI, Seoul, South Korea · University of Toronto, Ontario, Canada · Vector Institute, Ontario, Canada +1
X-OPD: Cross-Modal On-Policy Distillation for Capability Alignment in Speech LLMs
Mar 6, 2026Di Cao, Dongjie Fu, Hai Yu +3Speech Language Models LLM Alignment
Tencent Hunyuan · Zhejiang University
CRISP: Compressed Reasoning via Iterative Self-Policy Distillation
Mar 5, 2026Hejian Sang, Yuanda Xu, Zhengze Zhou +3CoT Distillation Efficient Language Model Reasoning
Surgical Post-Training: Proximal On-Policy Distillation for Reasoning with Knowledge Retention
Mar 2, 2026Wenye Lin, Kai HanLLM Post-Training LLM Reasoning
The University of Hong Kong
Reinforcement-aware Knowledge Distillation for LLM Reasoning
Feb 26, 2026Zhaoyang Zhang, Shuli Jiang, Yantao Shen +6Knowledge Distillation RL for Knowledge Distillation
AWS Agentic AI
Fast and Effective On-policy Distillation from Reasoning Prefixes
Feb 16, 2026Dongxu Zhang, Zhichao Yang, Sepehr Janghorbani +6Language Model Distillation On-Policy Distillation
Optum AI
OVD: On-policy Verbal Distillation
Jan 29, 2026Jing Xiong, Hui Shen, Shansan Gong +7Language Model Distillation On-Policy Distillation
The University of Hong Kong, Hong Kong, China · Nanjing University, Nanjing, China · Huawei Technologies, China

Apr 29, 2026J/K move · Enter open · S save

Co-Evolving Policy Distillation

Naibin Gu, Chenxu Yang, Qingyi Si, Chuanyu Qin+6

Institute of Information Engineering, CAS, 2School of Cyber Security, UCAS, 3JD.COM

Reinforcement Learning with Verifiable Rewards Multi-Teacher Knowledge Distillation On-Policy Distillation RL Post-Training

Open paper AskPDF arXiv

RLVR and OPD have become standard paradigms for post-training. We provide a unified analysis of these two paradigms in consolidating multiple expert capabilities into a single model, identifying capability loss in different ways: mixed RLVR suffers from inter-capability divergence cost, while the pipeline of first training experts and then performing OPD, though avoiding divergence, fails to fully absorb teacher capabilities due to large behavioral pattern gaps between teacher and student. We propose Co-Evolving Policy Distillation (CoPD), which encourages parallel training of experts and introduces OPD during each expert's ongoing RLVR training rather than after complete expert training, with experts serving as mutual teachers (making OPD bidirectional) to co-evolve. This enables more consistent behavioral patterns among experts while maintaining sufficient complementary knowledge throughout. Experiments validate that CoPD achieves all-in-one integration of text, image, and video reasoning capabilities, significantly outperforming strong baselines such as mixed RLVR and MOPD, and even surpassing domain-specific experts. The model parallel training pattern offered by CoPD may inspire a novel training scaling paradigm.