On-Policy Self-Distillation

Follow this topic Ask about this topic

On-policy self-distillation studies how a model can improve from its own generated responses, using a teacher version of the model to provide token-level guidance along the student’s actual trajectories. Common approaches give the teacher extra information, such as solutions, critiques, visual evidence, or task feedback, and use filtering, confidence weighting, or reference anchoring to make its guidance more useful. Methods are being applied to language-model reasoning, tool-using agents, multimodal tasks, and generative models, often alongside verifiers or reinforcement-learning methods such as GRPO and RLVR. A central tension is that privileged guidance can correct errors but also encourage shortcuts, suppress exploration, or harm out-of-distribution generalization, so reliable and transferable supervision remains an open challenge.

Latest papers

161 papers58 papers in the last four weeks, up 383% on the four weeks before. 0.3% of all new papers.

Weaken to Strengthen: Weakening-Induced On-Policy Self-Distillation for Large Reasoning Models
Oct 28, 2026Yecheng Wu, Song Han, Han CaiOn-Policy Self-Distillation Self-Supervised Knowledge Distillation
Massachusetts Institute of Technology · NVIDIA
On-Policy Self-Distillation with Teacher Evolution
Oct 28, 2026Shufan Shen, Zhongni Hou, Chao Bi +3On-Policy Self-Distillation Language Model Distillation
State Key Lab. of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Meituan
SCOPE: Selective Critical-Step On-Policy Self-Distillation for Agentic Reinforcement Learning
Oct 28, 2026Qirui Liu, Wei Cao, Yu Mi +5Agentic RL On-Policy Self-Distillation
Zhejiang University · Ant Group
MASS: Multi-mask Selective Self-Distillation for Leakage-Resilient Reasoning
Oct 28, 2026Xiuqi Liu, Oveys Delafrooz Noroozi, Upamanyu MadhowLanguage Model Distillation Selective Knowledge Distillation
Chongqing Bashu Secondary School · University of California, Santa Barbara
RPSD: Retrieval-Privileged On-Policy Self-Distillation with Optimal Transport
Oct 27, 2026Dongyue Li, Zebang Cheng, Zhongyi Deng +3Sequential Recommendation RL for Recommendation
Nagoya University · Shenzhen University · Sichuan Agricultural University +2
Steering the Teacher: Improving On-Policy Self-Distillation for Thinking Models
Oct 25, 2026Tiejin Chen, Xiaoou Liu, Hanyang Chen +1On-Policy Self-Distillation Citation Hallucination
Arizona State University
SWE-Ladder: Learning to Localize, Reproduce, and Patch via Progressive Privileged Self-Distillation
Oct 24, 2026Xudong Cai, Zhengxi Lu, Yuchen Yan +4Software Engineering Agents Repository-Level Program Repair
Zhejiang University
On-Policy Self-Training: Data-Free Self-Improvement in Large Language Models
Oct 23, 2026Leon Sick, Lukas Hoyer, Yannick Strümpler +3Language Model Self-Improvement Self-Training for Language Models
Google · Ulm University
Beyond On-Policy vs. Off-Policy: At What Level Should LLM Agent Distillation Follow the Student?
Oct 22, 2026Ya Gao, Zhengtao Zou, Pekka MarttinenOn-Policy Distillation Knowledge Distillation
Aalto University
OP-Print: Tracing Membership Imprints in On-Policy Self-Distillation
Oct 22, 2026Qi Li, Xinchao WangMembership Inference Attacks On-Policy Self-Distillation
National University of Singapore
When to Follow the Teacher: Advantage-Based On-Policy Distillation
Oct 22, 2026Mahdi Sabbaghi, George Pappas, Hamed Hassani +1Language Model Distillation On-Policy Distillation
University of Pennsylvania · University of Southern California
One Teacher, Many Views: Reducing Reference Sensitivity in On-Policy Self-Distillation
Oct 22, 2026Khouloud Saadi, Di WangOn-Policy Self-Distillation Knowledge Distillation
Department of Computer Science, King Abdullah University of Science and Technology, KAUST, Thuwal, Saudi Arabia
TeachBack: Learning to Teach via Bidirectional Preference Optimization for On-Policy Self-Distillation
Oct 21, 2026Linfei Ji, Jiayun Li, Haozhao Wang +5Language Model Distillation On-Policy Self-Distillation
Huazhong University of Science and Technology · Tsinghua University · Harbin Institute of Technology +1
Reinforcement Learning with Visually Attributable On-Policy Self-Distillation
Oct 19, 2026Shiao Xie, Siyu Chen, Jianwei Lv +2RL for VLMs On-Policy Self-Distillation
Baidu Inc., China
Bridged Self-Distillation for On-Policy Learning
Oct 19, 2026Runsheng Chen, Yihong Liu, Raoyuan Zhao +3Reasoning Distillation On-Policy Distillation
Technical University of Munich, Germany · Technical University of Munich · LMU Munich, Germany +3
Correctness-Dependent Asymmetric Target Distributions for Self-Distillation Policy Optimization
Oct 19, 2026Suwan Yoon, Seongryong Jung, Changhee LeeLanguage Model Post-Training On-Policy Self-Distillation
Korea University, Seoul, Korea · Chung-Ang University, Seoul, Korea
Asynchronous On-Policy Self-Distillation
Oct 19, 2026Xidong Wu, Qi Qi, Yuqiong Ji +1On-Policy Self-Distillation Language Model Distillation
University of Pittsburgh · University of Iowa · University of Pennsylvania +1
Not Every Rollout Needs Distillation: Correctness-Filtered On-Policy Self-Distillation
Oct 19, 2026Harrison Zhao, Lei Huang, Fei Liu +2On-Policy Self-Distillation Language Model Distillation
Meta
Predicting Privileged Information Effectiveness for On-Policy Self-Distillation
Oct 15, 2026Kaijian Zou, Emiliano Penaloza, MohammadHossein Rezaei +5On-Policy Self-Distillation Learning Using Privileged Information
University of Michigan – Ann Arbor · Mila · Scale AI
Learning from Unverified Peers via Shared Residual Distillation
Oct 15, 2026Feiyang Chen, Runze Ma, Zichen Wang +4Self-Training for Language Models Language Model Distillation
Shanghai Jiao Tong University
VASPO: Self-Distillation with Privileged Value Estimation
Oct 14, 2026Fei Wu, Zhenrong Zhang, Jianshu Zhang +1RL for Language Model Reasoning On-Policy Self-Distillation
University of Science and Technology of China · IFLYTEK Research
Does self-distillation improve reasoning?
Oct 13, 2026Jason Ken Adhinarta, Raina Wu, Ashutosh Sharma +5Self-Distillation On-Policy Self-Distillation
Massachusetts Institute of Technology · MIT-IBM Computing Research Lab
LoOPSD: On-Policy Self-Distillation with Looped Teachers
Oct 13, 2026Wei Fan, Qingyue Yang, Chao-Hong Tan +6On-Policy Self-Distillation LLM Reasoning
Anhui Province Key Laboratory of Digital Security · Independent Researcher · Token Foundry, Alibaba Group
Beyond Fixed Targets: On-Policy One-Shot Speaker Personalization for Autoregressive Text-To-Speech Models
Oct 13, 2026Sangkwon Park, Che Hyun Lee, Heeseung Kim +1TTS Synthesis On-Policy Self-Distillation
ECE, Seoul National University · University of Seoul · IPAI, AIIS, ASRI, INMC, and ISRC, Seoul National University
Self-Distillation with Verbal Regularization
Oct 12, 2026Jiazheng Li, Yawei Li, Junru Lu +7On-Policy Self-Distillation Language Model Self-Improvement
King’s College London · LMU Munich · Munich Center for Machine Learning +1
RSIPO: Recursive Self-Improving Policy Optimization for Agentic Reinforcement Learning via Skill-Policy Co-Evolution
Oct 11, 2026Fan Zhang, Hao Wu, Jinyang Wu +7Agentic RL LLM Agent Skill Learning
CUHK · THU · NUS +5
A Privileged Teacher That Only Scores Cannot Search: Critique-Conditioned On-Policy Self-Distillation for Reasoning
Oct 9, 2026Mayug Maniparambil, Ayush Rai, Ekansh Verma +2On-Policy Self-Distillation RL for Language Model Reasoning
Dublin City University
Reflect-VLA: Internalizing Reflective Guidance via Dual-Path On-Policy Self-Distillation for Autonomous Driving
Oct 9, 2026Yuechen Luo, He Chong, Fang Li +7Autonomous Driving Planning VLMs for Autonomous Driving
Tsinghua University · University of Macau · University of Science and Technology of China +2
JoyInside-VLM-Agent: Product-Conditioned Multimodal Policy Learning for Embodied Agents
Oct 9, 2026Gangyi Li, LuYao Wang, Jingfu Sun +7Multimodal Agents Robot Policy Learning
Mitigating Capability Degradation in LLM Self-Improvement via Representation Geometry Regularization
Oct 9, 2026Jiaqi Tian, Ting LongLanguage Model Self-Improvement On-Policy Self-Distillation
School of Artificial Intelligence Jilin University China

Oct 28, 2026J/K move · Enter open · S save

Weaken to Strengthen: Weakening-Induced On-Policy Self-Distillation for Large Reasoning Models

Yecheng Wu, Song Han, Han Cai

Massachusetts Institute of Technology · NVIDIA

On-Policy Self-Distillation Self-Supervised Knowledge Distillation Online Knowledge Distillation

Open paper AskPDF arXiv

Self-distillation offers a way to improve reasoning models when a suitable external teacher is unavailable. However, common approaches rely on privileged information to construct a stronger self-teacher, limiting their use when only unlabeled prompts are available. In this paper, we introduce Weakening-Induced On-Policy Self-Distillation (WISD), which instead constructs a weaker counterpart of the original model and uses the contrast between the two as a training signal without requiring privileged information. We explore two ways to construct the weaker model: low-bit quantization and magnitude pruning. We evaluate WISD across diverse architectures and scales on mathematical and coding reasoning tasks. Across these benchmarks, quantization and pruning improve average accuracy over the original models by 4.0 and 5.5 percentage points, respectively, achieving state-of-the-art performance even without privileged information. Notably, pruning raises Qwen3.5-4B's accuracy from 57.7% to 70.5% on HMMT 2025 and from 39.1% to 56.7% on LiveCodeBench v6. Quantization raises Qwen3.6-35B-A3B's accuracy from 68.5% to 74.6% and from 61.8% to 69.5% on the same benchmarks, respectively. These results establish WISD as a novel and effective self-distillation framework for large reasoning models.