On-Policy Self-Distillation

Follow this topic Ask about this topic

On-policy self-distillation studies how a model can improve from its own generated responses, using a teacher version of the model to provide token-level guidance along the student’s actual trajectories. Common approaches give the teacher extra information, such as solutions, critiques, visual evidence, or task feedback, and use filtering, confidence weighting, or reference anchoring to make its guidance more useful. Methods are being applied to language-model reasoning, tool-using agents, multimodal tasks, and generative models, often alongside verifiers or reinforcement-learning methods such as GRPO and RLVR. A central tension is that privileged guidance can correct errors but also encourage shortcuts, suppress exploration, or harm out-of-distribution generalization, so reliable and transferable supervision remains an open challenge.

Latest papers

162 papers58 papers in the last four weeks, up 383% on the four weeks before. 0.3% of all new papers.

Visual-OPSD: Cross-Modal On-Policy Self-Distillation for Efficient Unified Multimodal Reasoning
Jun 17, 2026Pengyu Li, Zhitao Gao, Lingling Zhang +5Cross-Modal Knowledge Distillation Efficient Multimodal Inference
Xi’an Jiaotong University · Shaanxi Province Key Laboratory of Big Data Knowledge Engineering · MOE KLINNS Lab, Xi’an Jiaotong University +1
Learning from Your Own Mistakes: Constructing Learnable Micro-Reflective Trajectories for Self-Distillation
Jun 17, 2026Zhilin Huang, Hang Gao, Ziqiang Dong +6LLM Self-Correction CoT Distillation
Qwen Business Unit of Alibaba · Tsinghua University · Peking University
Learning from the Self-future: On-policy Self-distillation for dLLMs
Jun 16, 2026Yifu Luo, Zeyu Chen, Haoyu Wang +4On-Policy Self-Distillation Diffusion Language Model Training
Tsinghua University · Technical University of Munich · Nanyang Technological University +5
Trust the Right Teacher: Quality-Aware Self-Distillation for GUI Grounding
Jun 16, 2026Jingyuan Huang, Zuming Huang, Yucheng Shi +4On-Policy Self-Distillation GUI Grounding
University of Georgia · INFLY Tech · Tencent AI Lab +1
OmniOPSD: Rationale-Privileged On-Policy Self-Distillation for Affective Computing
Jun 14, 2026Zebang Cheng, Shuimu Chen, Boxue Yang +7On-Policy Self-Distillation Multimodal Large Language Models
Shenzhen University · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) · Tsinghua University +6
ParaBridge: Bridging Paralinguistic Perception and Dialogue Behavior in Speech Language Models
Jun 9, 2026Yuxiang Wang, Qinke Ni, Shengbo Cai +3On-Policy Self-Distillation Spoken Dialogue Systems
1The Chinese University of Hong Kong, Shenzhen · 2Tencent Hunyuan · 5Tsinghua University +2
Thinking Without Images: Internalizing Visual Manipulation with On-Policy Self-Distillation
Jun 7, 2026Yishuo Cai, Jiahui Liu, Yuanxin Liu +9Tool Use in VLMs Efficient VLM Inference
1State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · 2Central South University · University of Science and Technology of China +2
Physics-Guided Policy Optimization with Self-Distillation
Jun 2, 2026Ke Wang, Yuning Wu, Haoran Liu +3On-Policy Self-Distillation Policy Optimization
Amazon, Arlington, USA · Amazon, New York, USA · Amazon, San Jose, USA +1
World Models Meet Language Models: On the Complementarity of Concrete and Abstract Reasoning
Jun 2, 2026Yucheng Zhou, Wei Tao, Yiwen Guo +1On-Policy Self-Distillation World Models
University of Macau · LIGHTSPEED · Independent Researcher
When Should the Teacher Move? Temporal Coupling and Stability in Self On-Policy Distillation
Jun 2, 2026Haowei Guo, Baolong Bi, Ruicheng Zhang +3On-Policy Self-Distillation Language Model Distillation
Peking University · University of Chinese Academy of Sciences · Tsinghua University
Constitutional On-Policy Safe Distillation
Jun 2, 2026Ming Wen, Yuxuan Liu, Kun Yang +8On-Policy Self-Distillation LLM Safety Alignment
Institute of Trustworthy Embodied AI, Fudan University · 2Shanghai Innovation Institute · 3Ant Group +3
Self-Distilled Policy Gradient
Jun 2, 2026Yifeng Liu, Shiyuan Zhang, Yifan Zhang +1Policy Gradient On-Policy Self-Distillation
Department of Computer Science, University of California, Los Angeles, CA, USA · Princeton AI Laboratory, Princeton University, Princeton, NJ, USA
Internalize the Temperature: On-Policy Self-Distillation as Policy Reheater for Reinforcement Learning
May 30, 2026Xuewei Yang, Jiachen Yu, Jie Wu +3On-Policy Self-Distillation RL for Language Model Reasoning
Tsinghua University
A Predictive Law for On-Policy Self-Distillation From World Feedback
May 28, 2026Tommy He, Jerome Sieber, Matteo SaponatiOn-Policy Self-Distillation RL Post-Training
Tufa Labs, Zürich, Switzerland
Beyond Imitation: Reflective On-Policy Self-Distillation for LLM Reasoning
May 27, 2026Ziqi Zhao, Xinyu Ma, Liu Yang +6On-Policy Self-Distillation Language Model Distillation
The Hong Kong Polytechnic University · Baidu Inc. · Shandong University +1
Restoring the Sweet Spot: Pass-Rate Weighted Self-Distillation for LLM Reasoning
May 26, 2026Zehao Liu, Yuanpu Cao, Jinghui Chen +1Reinforcement Learning On-Policy Self-Distillation
College of Information Sciences and Technology Pennsylvania State University
EDGE-OPD: Internalizing Privileged Context with Evidence Guided On-Policy Distillation
May 22, 2026Aristotelis Lazaridis, Dylan Bates, Aman Sharma +3On-Policy Self-Distillation On-Policy Distillation
EdgeRunner AI
Search-E1: Self-Distillation Drives Self-Evolution in Search-Augmented Reasoning
May 21, 2026Zihan Liang, Yufei Ma, Ben Chen +4Tool-Augmented Language Model Agents On-Policy Self-Distillation
Tailoring Teaching to Aptitude: Direction-Adaptive Self-Distillation for LLM Reasoning
May 21, 2026Hongbin Zhang, Chaozheng Wang, Kehai Chen +4On-Policy Self-Distillation Mathematical Reasoning
Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China · Keeta AI, Meituan, Beijing, China · Peng Cheng Laboratory, Shenzhen, China
When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning
May 20, 2026Xiaogeng Liu, Xinyan Wang, Yingzi Ma +2CoT Distillation On-Policy Self-Distillation
1Johns Hopkins University · University of Wisconsin–Madison · 3Nanyang Technological University
AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signals
May 20, 2026Duy Nguyen, Hanqi Xiao, Archiki Prasad +7On-Policy Self-Distillation Language Model Distillation
UNC Chapel Hill · Capital One · The University of Texas at Austin
SD-Search: On-Policy Hindsight Self-Distillation for Search-Augmented Reasoning
May 18, 2026Yufei Ma, Zihan Liang, Ben Chen +6On-Policy Self-Distillation RL for Language Model Reasoning
Kuaishou Technology
Learning from Language Feedback via Variational Policy Distillation
May 14, 2026Yang Li, Erik Nijkamp, Semih Yavuz +1On-Policy Self-Distillation RL for Language Model Reasoning
Salesforce AI Research
Reducing the Safety Tax in LLM Safety Alignment with On-Policy Self-Distillation
May 14, 2026Yu Fu, Longxuan Yu, Haz Sameen Shahgir +4LLM Alignment On-Policy Self-Distillation
1UC Riverside · 2International Computer Science Institute · 3Microsoft,∗ +1
Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoning
May 13, 2026Junlong Ke, Zichen Wen, Weijia Li +2On-Policy Self-Distillation RL for Language Model Reasoning
2Tsinghua University · 1Shanghai Jiao Tong University · 3Shanghai AI Laboratory
From Generic Correlation to Input-Specific Credit in On-Policy Self Distillation
May 12, 2026Guobin Shen, Lei Huang, Xiang Cheng +4Reinforcement Learning On-Policy Self-Distillation
Xiaohongshu Inc. · Institute of Automation, Chinese Academy of Sciences
Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information
May 12, 2026Guobin Shen, Xiang Cheng, Chenxiao Zhao +4On-Policy Self-Distillation RL for Language Model Reasoning
1Xiaohongshu Inc. · Institute of Automation, Chinese Academy of Sciences
Adaptive Teacher Exposure for Self-Distillation in LLM Reasoning
May 12, 2026Zihao Han, Tiangang Zhang, Huaibin Wang +1Adaptive Knowledge Distillation On-Policy Self-Distillation
ByteDance Douyin
The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes
May 11, 2026Siqi Zhu, Xuyan Ye, Hongyu Lu +2On-Policy Self-Distillation On-Policy Distillation
1UIUC · 2Renmin University of China · †Work done during an internship at UIUC. +1
Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why
May 11, 2026Mohammadreza Armandpour, Fatih Ilhan, David Harrison +6On-Policy Self-Distillation Language Model Distillation
Apple

Jun 17, 2026J/K move · Enter open · S save

Visual-OPSD: Cross-Modal On-Policy Self-Distillation for Efficient Unified Multimodal Reasoning

Pengyu Li, Zhitao Gao, Lingling Zhang, Muye Huang+4

Xi’an Jiaotong University · Shaanxi Province Key Laboratory of Big Data Knowledge Engineering · MOE KLINNS Lab, Xi’an Jiaotong University+1

Cross-Modal Knowledge Distillation Efficient Multimodal Inference Unified Multimodal Models On-Policy Self-Distillation Multimodal Reasoning Multimodal Foundation Models+1

Open paper AskPDF arXiv

Unified multimodal models (UMMs) interleave generated ''visual thoughts'' (VTs) with text reasoning to improve spatial tasks. This incurs roughly an order-of-magnitude inference cost from multi-step diffusion. We find this cost yields limited direct benefit. On ThinkMorph, removing or noising VTs barely changes accuracy across nine benchmarks. Once rendered, attention concentrates on the VT regardless of content. Yet a KL diagnostic shows that conditioning on a privileged VT trace shifts the model's completion distribution. This suggests the generation pathway encodes useful reasoning beyond the rendered pixels. Motivated by this gap, we propose Visual On-Policy Self-Distillation(Visual-OPSD). Teacher and student share identical weights but differ in context: the teacher sees privileged VTs while the student sees only the question. Token-level JSD distillation on on-policy student trajectories transfers the teacher's reasoning to a text-only student. Across nine benchmarks, Visual-OPSD improves over its generative teacher by

+3.40

pp with

14.3\times

speedup (10.0s vs. 142.8s per sample) and outperforms same-scale VLMs by

+63.83

pp on VSP. A Gaussian-noise control (

+0.40

pp vs.

+10.28

pp for real VTs) and

58.4\%

closure of the KL gap confirm that gains come from the semantic content of the generation pathway.