On-Policy Distillation

Follow this topic Ask about this topic

On-policy distillation studies how to transfer a teacher’s capabilities by training a student on trajectories the student generates, rather than only on fixed teacher-written examples. Common approaches match teacher and student token distributions with KL-style objectives, often combining this dense feedback with verifier rewards, reinforcement learning, or feedback from successful peer attempts. Work spans language-model reasoning and tool-using agents as well as multimodal, speech, diffusion, and embodied models; methods increasingly select useful tokens, turns, or rollout lengths to reduce compute. A central tension is that student-generated states can make supervision more relevant, but teacher signals may be unreliable or mismatched, and repeated rollouts can be costly or destabilize training. 4 sentences total.

Latest papers

511 papers178 papers in the last four weeks, up 141% on the four weeks before. 1.0% of all new papers.

Contrastive Reinforced Policy Optimization via Privileged Self-Distillation
Jul 30, 2026Xingjian Wu, Junlin Liu, Xingchen Liu +6Agentic RL RL for Language Model Reasoning
Meituan
Flux-OPD: On-Policy Distillation with Evolving Contexts
Jul 30, 2026Yuran Wang, Zekun Wang, Bohan Zeng +10Language Model Distillation On-Policy Distillation
Peking University · Kling Team · Tsinghua University +3
From Scoring to Acting: Outcome-Verified Comparative Self-Distillation for LLM Agents
Jul 30, 2026Xu Xia, Jinghua Piao, Min Yang +3LLM Agent Skill Learning Self-Distillation
Southeast University · Zhongguancun Academy · Tsinghua University +2
Beyond the Best Teacher: Expanding and Compressing the Reasoning Solution Manifold
Jul 30, 2026Songshuo Lu, Zhi Chen, Yaohua TangRL for Language Model Reasoning Multi-Teacher Knowledge Distillation
Moore Threads AI
Veritas++: Value-aware On-Policy Distillation for Perception-Enhanced AIGI Detection
Jul 29, 2026Hao Tan, Jun Lan, Zichang Tan +7VLM Distillation Large Vision-Language Models
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences (UCAS), Beijing 101408, China, and also with the State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation Chinese Academy of Sciences (CASIA), Beijing 100190, China · State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences (CASIA), Beijing 100190, China · Ant Group +2
On-Policy Distillation for LLM Safety: A Routing Approach to Template-Robust Realignment
Jul 29, 2026Yongjian Guo, Wanlun Ma, Lingyu Shen +2LLM Safety Alignment Adversarial Robustness of Language Models
Tsinghua University · Swinburne University of Technology · EPFL
Weak-to-Strong On-Policy Distillation
Jul 28, 2026Fangxu Yu, Weijia Xu, Michael Xu +2Language Model Distillation On-Policy Distillation
University of Maryland, College Park · Microsoft Research · MBZUAI
Pass the Baton: Trajectory-Relayed On-Policy Distillation
Jul 28, 2026Haolei Xu, Xiaowen Xu, Haiwen Hong +5Language Model Distillation On-Policy Distillation
1Zhejiang University · 2Yuvion Team, Alibaba Group
Rethinking Classifier-Free Guidance in On-Policy Diffusion Distillation
Jul 27, 2026Bingnan Li, Haozhe Wang, Haozhong Xiong +5Diffusion Model Guidance Classifier-Free Guidance
Qwen Business Unit of Alibaba
The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillation
Jul 27, 2026Tianyi Men, Zhuoran Jin, Kang Liu +1LLM Planning Long-Horizon Planning
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
FlowCTS: On-policy Continuous Trajectory Supervision of Flow Models
Jul 27, 2026Kaiyang Ye, Yuan Ge, Junxiang Zhang +8T2I Generation Flow-Based Generative Modeling
Northeastern University · NiuTrans Research · Kunming University of Science and Technology
RP-OPSD: Resolution-Privileged On-Policy Self-Distillation for Multimodal Large Language Models
Jul 27, 2026Qihui Zhu, Yuchen Wang, Zijian Wen +7Multimodal Large Language Models Self-Distillation
University of Science and Technology of China · 2ChangXin Memory Technologies, Inc.
Outcome-Confounded Local Supervision in On-Policy Distillation
Jul 26, 2026Guoqing MaCredit Assignment in RL Language Model Distillation
The Chinese University of Hong Kong, Shenzhen
Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization
Jul 24, 2026Hao Wang, Kun Yuan, Wenlin Zhong +4Cross-Tokenizer Knowledge Distillation LLM Compression
University of Chinese Academy of Sciences · KwaiKAT Team · Zhejiang University +1
X $^3$ -OPD: Distilling Reasoning into Large Audio-Language Models via On-Policy Alignment
Jul 23, 2026Dongjie Fu, Di Cao, Xize Cheng +6Cross-Modal Knowledge Distillation Audio Reasoning
1Tencent Hunyuan · 2Zhejiang University
OPOD: On-Policy Omni Distillation
Jul 23, 2026Tong Zhao, Yuyang Hu, Yutao Zhu +5Cross-Modal Knowledge Distillation Unified Multimodal Models
1Renmin University of China · 2Tencent
SOPD-SocialNav: Selective On-Policy Distillation for Vision-Language Social Navigation
Jul 22, 2026Xinyu Zhang, Zishuo Wang, Ling XiaoEfficient VLM Inference VLM Distillation
Graduate School of Information Science and Technology, Hokkaido University, Sapporo, Japan
AdaFlash: Adaptive Speculative Decoding via On-Policy Distilled Diffusion Drafters
Jul 21, 2026Yu-Yang Qian, Hao-Cong Wu, Chen Chen +4Speculative Decoding for Diffusion Language Models Speculative Decoding
State Key Laboratory for Novel Software Technology, Nanjing University · School of Artificial Intelligence, Nanjing University · Huawei Foundation Model Dept
Contrastive On-Policy Distillation
Jul 21, 2026Jiacheng Ruan, Jun Tang, Wenzhen Yuan +5Language Model Distillation On-Policy Distillation
Shanghai Jiao Tong University · Alibaba Group
MOPDA: Mixed-Trajectory On-Policy Distillation for Language-Guided Industrial Anomaly Detection
Jul 21, 2026Shuimu Chen, Jing Jin, Nan Su +5Industrial Anomaly Detection VLM Distillation
Tsinghua University · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) · Shenzhen University
ABOPD: Antibody CDR Design via On-Policy Distillation
Jul 21, 2026Zhuo Yang, Jiaying He, Jiaqing Xie +5Diffusion Model Distillation On-Policy Distillation
Shanghai Innovation Institute · Southeast University · Zhejiang University +4
CriPO: Enhancing Rubric-based RL via Self-Distillation
Jul 20, 2026Mingxuan Xia, Yuhang Yang, Chao Ye +7Credit Assignment in RL Rubric-Based RL
Zhejiang University · ByteDance
Distilled Reinforcement Learning for LLM Post-training
Jul 19, 2026Chen Wang, Zhaochun Li, Jionghao Bai +4RL for Language Model Reasoning Cross-Architecture Knowledge Distillation
College of Elite Engineers, Nankai University · Zhongguancun Academy · Beijing Institute of Technology +4
CADENCE: Closing the Reasoning Gap via Coverage-Adaptive On-Policy Distillation
Jul 18, 2026Satyam Kumar, Saurabh JhaCoT Distillation On-Policy Distillation
Better Starts, Better Ends: Bootstrapped Iterative Self-Reasoning Distillation for Compressed Reasoning
Jul 17, 2026Leichao Dong, Dongxu Zhang, Yiding Sun +4CoT Distillation Efficient Language Model Reasoning
1Xi’an Jiaotong University · 2Peking University
On-Policy Delta Distillation
Jul 16, 2026Byeongho Heo, Jaehui Hwang, Sangdoo Yun +1RL for Language Model Reasoning Language Model Distillation
NAVER AI Lab
SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning
Jul 16, 2026Jinyang Wu, Shuo Yang, Zhengxi Lu +8Agentic RL On-Policy Distillation
1Tsinghua University · 2Zhejiang University · 3The Chinese University of Hong Kong +2
Post-Training Shifts Confidence: A Three-Stage Analysis of How SFT, RL, and OPD Shape Pre-, Intra-, and Post-CoT Calibration
Jul 15, 2026Shuhao Li, Guodong Du, Anhao Zhao +3Confidence Estimation in Language Models Language Model Calibration
Eastern Institute of Technology, Ningbo · The Hong Kong Polytechnic University
Demystifying On-Policy Distillation: Roles, Pathologies, and Regulations
Jul 15, 2026Rui Wang, Hongru Wang, Yi Chen +4RL for Language Model Reasoning Language Model Distillation
†The Chinese University of Hong Kong · ‡Tencent AI Lab
ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation
Jul 14, 2026Qingyu Zhang, Qianhao Yuan, Hongyu Lin +7Open-Ended Generation LLM Pruning
2Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · 1ByteDance

Jul 30, 2026J/K move · Enter open · S save

Contrastive Reinforced Policy Optimization via Privileged Self-Distillation

Xingjian Wu, Junlin Liu, Xingchen Liu, Xuhang Zhu+5

Meituan

Agentic RL RL for Language Model Reasoning On-Policy Distillation RL Post-Training Contrastive RL

Open paper AskPDF arXiv

Recent advances in post-training Large Language Models (LLMs) increasingly rely on Reinforcement Learning with Verifiable Rewards (RLVR) or On-Policy Self-Distillation (OPSD). While OPSD provides dense, logit-level supervision, it inherently suffers from exposure bias due to the privileged information of the self-teacher. In multi-turn agentic settings, this leads to reasoning route convergence and the loss of clear optimization directions. To tackle these challenges, we introduce Contrastive Reinforced Policy Optimization (CRPO), which reformulates agentic OPSD from a contrastive learning perspective. By leveraging predictive entropy to distinguish between positive positions (reflective exploration) and negative positions (exposure bias), CRPO conducts group-wise contrast to preserve reliable, fine-grained optimization signals. Extensive evaluations across 13 challenging reasoning and deep-search benchmarks demonstrate that CRPO consistently outperforms existing reinforcement learning and self-distillation baselines, significantly enhancing training stability and generalization in long-horizon interactions.