On-Policy Self-Distillation

Follow this topic Ask about this topic

Momentum

30 papers in the last four weeks, up 233% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 108

All topics

A Good Self-Teacher Meets the Student Where They Are: Joint On-Policy Learning and Teaching
Oct 7, 2026Randy Ardywibowo, Arnav Dalal, Jiantao JiaoOn-Policy Self-Distillation RL for Language Models
Perplexity · NVIDIA
MIRROR: From Imitation to Internalization in LLM Personalization
Oct 7, 2026Huayi Lai, Jicheng Yang, Min Yi +1Personalized Language Models Personalized Language Model Alignment
Baidu Inc.
MIMESIS: Learning User Simulators as Training Environments for Interactive Agents
Oct 7, 2026Hoang Phan, Dat Huynh, Andrey Zhmoginov +7User Simulation LLM Agent Training
Meta Superintelligence Labs · New York University · University of Wisconsin - Madison
Privileged Context as Drift in On-Policy Self-Distillation
Oct 6, 2026Ravenor Davion, Nick RuiOn-Policy Self-Distillation Context Distillation
Stanford University
MeSD: Multi-Evidence Self-Distillation for VideoLLM
Oct 5, 2026Weijie Zhu, Han Fang, Hanyu Fu +13On-Policy Self-Distillation Video-Language Models
UCAS · Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd · PKU +2
Learning without Overwriting: A Theory of Self-Distillation and Supervised Fine-Tuning in Continual Reasoning
Oct 4, 2026Shinichi Uemura, Taiji SuzukiSupervised Fine-Tuning Continual Learning for LLMs
Department of Mathematics Informatics, The University of Tokyo · Center for Advanced Intelligence Project, RIKEN, Tokyo, Japan
Outcome-Guided On-Policy Self-Distillation
Oct 4, 2026ZheXu Wang, Mao-Lin Luo, Yankun Hong +6On-Policy Self-Distillation Uncertainty-Aware Knowledge Distillation
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China · Huawei Noah’s Ark Lab +2
E $^2$ -OPSD: Taming Entropy Overshoot in On-Policy Self-Distillation
Oct 4, 2026Yifei Liu, Minghao Fang, Xinyu Gu +6On-Policy Self-Distillation Mathematical Reasoning
The Chinese University of Hong Kong · Zhejiang University · Shanghai Artificial Intelligence Laboratory +5
ComputerSD: Online Self-Distillation from Real-Time Feedback for Computer-Use Agents
Sep 30, 2026Yong Du, Tongbo Chen, Zhengxi Lu +5On-Policy Self-Distillation Computer-Use Agents
Zhejiang University · Ant Group
Better Supervision Is Nearby: Neighborhood On-Policy Self-Distillation
Sep 30, 2026Xincheng Wei, Yifan Ding, Yoshua Li +6On-Policy Self-Distillation Mathematical Reasoning
The Chinese University of Hong Kong, Shenzhen · Meituan, LongCat Team · University of Toronto +1
Disentangling Self-Distillation: Measuring and Modeling Acquisition and Retention
Sep 30, 2026Luis Zuin, Alexis Huet, Dario Rossi +1On-Policy Self-Distillation Self-Distillation
Huawei Technologies Co., Ltd., Paris, France
OmniReasoning: Pushing the Limits of Audio-Visual Joint Reasoning
Sep 30, 2026Junming Lin, Yuxuan Wang, Zhenxin Lei +11On-Policy Self-Distillation Multimodal Reasoning
School of Intelligence Science and Technology, Peking University · State Key Laboratory of General Artificial Intelligence, Peking University · Alibaba Token Hub, Alibaba Group
ReSAIL: Mitigating Collapse in Iterative Agent Self-Distillation
Sep 30, 2026Shengjie Jin, Hengbo Xu, Zelong Sun +2On-Policy Self-Distillation LLM Agent Self-Improvement
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China
Frame Differential On-Policy Self-Distillation for Video Reasoning
Sep 30, 2026Haiying He, Xin Zheng, Shaoli Hu +4On-Policy Self-Distillation Video Reasoning
HKUST · NKU · SEU +1
UniEvo-VL: An On-policy Self-Distillation Training Recipe for Multimodal Model Self-improvement
Sep 30, 2026Fang Wu, Da Xing, Yanjie Huang +16Unified Multimodal Models On-Policy Self-Distillation
Stanford University · Johns Hopkins University · Independent Researcher +7
Activation-Conditioned Self-Distillation
Sep 29, 2026Zhexi Lu, Subhajit Chaudhury, Tejaswini Pedapati +2Numerical Reasoning in Language Models On-Policy Self-Distillation
Rensselaer Polytechnic Institute · IBM Research
RS-OPSD: Reliable Privileged On-Policy-Self-Distillation for Ultra-High-Resolution Remote Sensing VQA
Sep 29, 2026Chengjie Jiang, Yunqi Zhou, Jiafeng Yan +3Remote Sensing Image Understanding Visual Question Answering
Tsinghua University · Zhejiang University · Central University of Finance and Economics +2
Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning
Sep 29, 2026Md. Ismail Hossain, Humaira Kousar, Isidora Chara TourniOn-Policy Self-Distillation Mathematical Reasoning
North South University · KAIST · Andria Labs
Can Vision-Language Models Stay Helpful When Facing Implicit Risks? Intent-Privilege OPSD for Efficient Safety-Helpfulness Alignment
Sep 29, 2026Haotian Deng, Wenbin Xing, Gang Xu +5VLM Robustness Vision-Language Models
Southern University of Science and Technology · Sun Yat-sen University · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) +5
Train Ahead, Distill Back: Bootstrapping On-Policy Self-Distillation for Large Language Models
Sep 29, 2026Zheng Zhang, Xinyue Tan, Lufei Li +3On-Policy Self-Distillation Language Model Distillation
School of Information Science and Technology, ShanghaiTech University · State Key Laboratory of General Artificial Intelligence, BIGAI · Hefei University of Technology
Know Thyself, Teach Thyself: Internal Information Flow for Selective Self-Distillation
Sep 29, 2026Rui Wang, Ruijie Wang, Bo Chen +2On-Policy Self-Distillation Selective Knowledge Distillation
Peking University · University of Oxford · The University of Hong Kong
PE-OPSD: Internalizing Prompt Enhancement into Flow-matching Models via On-Policy Self-Distillation
Sep 29, 2026Mingfeng Lin, Chengfei Cai, Lin Xu +3On-Policy Self-Distillation T2I Generation
Harbin Institute of Technology (Shenzhen) · Zhejiang University · Peking University +1
On-Policy Self-Distillation for Multi-Turn Image Editing
Sep 28, 2026Liangbing Zhao, Le Zhuo, Mohamed ElhoseinyText-Guided Image Editing Diffusion Model Distillation
KAUST · Krea AI
PIVOT: Pivot-Aware On Policy Self Distillation for Multi-Turn VLM Agents
Sep 28, 2026Jiazhou Zhou, Hu Zhou, Yucheng Chen +3On-Policy Self-Distillation Reinforcement Learning with Verifiable Rewards
AI Thrust, The Hong Kong University of Science and Technology (Guangzhou) · International Digital Economy Academy (IDEA) · The Hong Kong Polytechnic University +2
Rubric-Aware On-Policy Self-Distillation for LLM Personalization
Sep 28, 2026Yilun Qiu, Xiaoyan Zhao, Chengbing Wang +6LLM Personalization On-Policy Self-Distillation
Xiaohongshu Inc. · National University of Singapore · University of Science and Technology of China +1
Evidence-Aligned Multimodal On-Policy Self-Distillation for Fine-Grained Visual Understanding
Sep 28, 2026Nanxing Hu, Qiwei Yan, Jinchao Zhang +1On-Policy Self-Distillation VLM Distillation
Beihang University · University of the Chinese Academy of Sciences
EOPSA: Efficient On-Policy Self-Distilled Safety Alignment
Sep 28, 2026Qirui Liu, Yichen Sun, Yan Wang +5On-Policy Self-Distillation Selective Knowledge Distillation
The State Key Laboratory of Blockchain and Data Security, Zhejiang University · Ant Group
K-OPSD: Verifiable On-Policy Self-Distillation for Post-Training Vision-Language Models on AEC Drawings
Sep 28, 2026Yunfei Bai, Enrico Chionna, Akash Amol +2On-Policy Self-Distillation VLM Distillation
Amazon Web Services
SCOPD: Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language Models
Sep 28, 2026Ahmadreza Jeddi, Enming Zhang, Jasper Gerigk +12Efficient VLM Inference On-Policy Self-Distillation
University of Toronto · Vector Institute · Samsung AI Center Toronto +7
Fisher-Informed Recalibration for Feedback-Based On-Policy Self-Distillation of LLMs
Sep 27, 2026Seohyun Lee, Dong-Jun Han, Seyyedali Hosseinalipour +1On-Policy Self-Distillation Knowledge Distillation
Purdue University · Yonsei University · University at Buffalo-SUNY

Oct 7, 2026, cs.LGJ/K move · Enter open · S save

A Good Self-Teacher Meets the Student Where They Are: Joint On-Policy Learning and Teaching

Randy Ardywibowo, Arnav Dalal, Jiantao Jiao

Perplexity · NVIDIA

On-Policy Self-Distillation RL for Language Models On-Policy Distillation Knowledge Distillation

Open paper AskPDF arXiv

Reinforcement Learning (RL) from outcome rewards suffers from sparse supervision, particularly on difficult, long-horizon tasks where successful trajectories are rare and costly to generate. On-Policy Distillation (OPD) offers an attractive alternative by providing dense token-level supervision from a stronger teacher along the student's own generations. Self-distillation methods further remove the need for a separate teacher model by conditioning the same policy on privileged information to serve as its own teacher. However, privileged conditioning alone does not guarantee that the resulting distillation update improves the student. Indeed, privileged information can lead the teacher to solve tasks through shortcuts unavailable to the student, producing supervision poorly matched to the student's current behavior. Consequently, even a higher-performing teacher can provide guidance that degrades student performance. To address this, we analyze how the choice of privileged teacher affects the student's update. We derive a necessary and sufficient condition for the teacher's local distillation update to be a positive multiple of the student's reward gradient. Our analysis suggests that the teacher should not only perform well on the task, but also provide guidance suited to the student's current capabilities. This characterization motivates a practical teacher-training surrogate that combines outcome rewards with token-level Kullback-Leibler (KL) regularization toward the student. Based on this result, we propose Joint On-Policy Learning and Teaching (JOLT), which jointly trains a single policy in two roles: a privileged teacher using a KL-regularized objective, and an unprivileged student using dense on-policy distillation. Across mathematical reasoning, coding, tool use, and terminal use, JOLT improves training efficiency and performance, with further gains from student rewards.

Momentum

Related topics

Latest papers 108