Self-Distillation

Follow this topic Ask about this topic

Self-distillation uses a model—or a related version of it—as a teacher, transferring knowledge through output probabilities, hidden representations, or generated experience. Common approaches include teacher–student consistency with exponential-moving-average or earlier-checkpoint teachers, on-policy learning from the model’s own trajectories, and privileged or hindsight feedback that supplies extra context during training. The methods are used for language-model reasoning and tool use, as well as vision, audio, video, time series, and faster generative inference; evaluation often spans math and code reasoning, agent tasks, and domain-specific perception benchmarks. A central challenge is that teacher signals can be biased or misaligned with task success, reinforcing errors, reducing exploration, or causing forgetting, so generalization, capability retention, training stability, and compute cost remain important open concerns.

Latest papers

137 papers40 papers in the last four weeks, up 208% on the four weeks before. 0.2% of all new papers.

Self-Distillation Policy Optimization via Visual Feedback: Bridging Code and Visual Artifacts
Jun 9, 2026Haoyu DongRL for Code Generation Code Generation
Microsoft
PBSD: Privileged Bayesian Self-Distillation for Long-Horizon Credit Assignment
Jun 8, 2026Yang Tian, Rui Wang, Xumeng Wen +5Step-Level Credit Assignment in RL Reinforcement Learning with Verifiable Rewards
School of AI, Shanghai Jiao Tong University · 2XYZ AI Lab
Learning Visual Spatial Planning from Symbolic State via Modality-Gap-Aware Self-Distillation
Jun 4, 2026Haocheng Luo, Jiahui Liu, Ruicheng Zhang +6Cross-Modal Knowledge Distillation Visual Spatial Reasoning
1Tsinghua University · 2The Hong Kong University of Science and Technology
CoralBay: A Self-Supervised CT Foundation Model
Jun 2, 2026Ioannis Gatopoulos, Nicolas Känzig, Sebastian Otálora +1Medical Imaging Foundation Models 3D Representation Learning
What Do Students Learn? A Feature-Level Analysis of Dark Knowledge
Jun 2, 2026Seungu Kang, Songkuk KimRepresentation Learning Self-Distillation
Yonsei University, Seoul 03722, Republic of Korea
Echo-POSED: Geometric Self-Distillation for Echocardiography Guidance
May 30, 2026Elias Stenhede, Edvart Grüner Bjerke, Joanna Sulkowska +4Self-Supervised Learning Echocardiography
Medical Technology & E-Health, Akershus University Hospital, Norway · Department of Technology Systems, University of Oslo, Norway · K.G. Jebsen Center for Cardiac Biomarkers, University of Oslo, Norway +3
Distilling LLM Feedback for Lean Theorem Proving
May 29, 2026Gaetan Narozniak, Gérard Biau, Rémi Munos +2RL for Language Model Reasoning Mathematical Reasoning
FAIR at Meta, Inria · Sorbonne Université, Institut universitaire de France · FAIR at Meta +2
TRACER: Persistent Regularization for Robust Multimodal Finetuning
May 28, 2026Hesam Asadollahzadeh, Feng Liu, Christopher Leckie +1VLM Robustness Neural Network Robustness
School of Computing and Information Systems (CIS), Faculty of Engineering and IT (FEIT), University of Melbourne, Australia.
OISD: On-Policy Internal Self-Distillation of Language Models
May 27, 2026Xinyu Liu, Darryl Cherian Jacob, Yang Zhou +2RL for Language Model Reasoning Self-Distillation
Auburn University · William & Mary
Skill-Conditioned Gated Self-Distillation for LLM Reasoning
May 27, 2026Jiazhen Huang, Xiao Chen, Xiao Luo +3Mathematical Reasoning Self-Distillation
Tsinghua University · University of Wisconsin-Madison · X-Humanoid +2
Tailoring Teaching to Aptitude: Direction-Adaptive Self-Distillation for LLM Reasoning
May 21, 2026Hongbin Zhang, Chaozheng Wang, Kehai Chen +4On-Policy Self-Distillation Mathematical Reasoning
Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China · Keeta AI, Meituan, Beijing, China · Peng Cheng Laboratory, Shenzhen, China
It Takes Two: Complementary Self-Distillation for Contextual Integrity in LLMs
May 18, 2026Sangwoo Park, Woongyeong Yeo, Seanie Lee +6Contextual Integrity LLM Alignment
KAIST
Self-Distillation is Optimal Among Spectral Shrinkage Estimators in Spiked Covariance Models
May 18, 2026Radu Lecoiu, Debarghya Mukherjee, Pragya SurSelf-Distillation Spectral Methods
Department of Statistics, Harvard University · Department of Mathematics & Statistics, Boston University
MixSD: Mixed Contextual Self-Distillation for Knowledge Injection
May 16, 2026Jiarui Liu, Lechen Zhang, Yongjin Yang +5Knowledge Augmentation for Language Models Factual Knowledge in Language Models
Carnegie Mellon University · Jinesis Lab, University of Toronto & Vector Institute · University of Illinois Urbana-Champaign +6
Self-Distilled Agentic Reinforcement Learning
May 14, 2026Zhengxi Lu, Zhiyuan Yao, Zhuowen Han +8Agentic RL Self-Distillation
Zhejiang University · Meituan · Tsinghua University
Learning with Rare Success but Rich Feedback via Reflection-Enhanced Self-Distillation
May 12, 2026Yuwei Zhang, Sha Li, Changlong Yu +9Continual Learning for LLMs Self-Distillation
UC San Diego1 · Amazon2 · Georgia Institute of Technology3
Self-Distilled Trajectory-Aware Boltzmann Modeling: Bridging the Training-Inference Discrepancy in Diffusion Language Models
May 12, 2026Kecheng Chen, Ziru Liu, Xijia Tao +9Diffusion Model Distillation Diffusion Model Fine-Tuning
1City University of Hong Kong · 2Huawei Research · 3The University of Hong Kong
Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information
May 12, 2026Guobin Shen, Xiang Cheng, Chenxiao Zhao +4On-Policy Self-Distillation RL for Language Model Reasoning
1Xiaohongshu Inc. · Institute of Automation, Chinese Academy of Sciences
Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVR
May 11, 2026Jeonghye Kim, Jiwon Jeon, Dongsheng Li +1RL for Language Model Reasoning Language Model Distillation
Microsoft Research · KAIST
EchoDistill: Robust Large Audio Language Models via Noisy-to-Clean Self-Distillation
May 11, 2026Kaiwen Luo, Chunxi Luo, Liang Lin +5Audio Understanding Audio-Language Models
NTU · SHU · UCAS +2
UniSD: Towards a Unified Self-Distillation Framework for Large Language Models
May 7, 2026Yiqiao Jin, Yiyang Wang, Lucheng Fu +7Language Model Distillation Self-Distillation
Georgia Institute of Technology · University of California, Los Angeles · Carnegie Mellon University +1
VISD: Enhancing Video Reasoning via Structured Self-Distillation
May 7, 2026Hao Lin, Kunyang Lv, Xu Jiang +5Video-Language Models Video Reasoning
1HUST · 2Wuhan University · 3Peking University +1
Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularization
May 6, 2026Xin Yu, Liuchen Liao, Yiwen Zhang +3On-Policy Self-Distillation Language Model Distillation
1The Pennsylvania State University · 2TikTok
Power Distribution Bridges Sampling, Self-Reward RL, and Self-Distillation
May 6, 2026Akiyoshi Tomihari, Issei SatoReinforcement Learning KL-Regularized RL
Department of Computer Science, The University of Tokyo
Mutual Forcing: Dual-Mode Self-Evolution for Fast Autoregressive Audio-Video Character Generation
Apr 28, 2026Yupeng Zhou, Lianghua Huang, Zhifan Wu +7Audio-Video Generation Long-Horizon Video Generation
1VCIP, School of Computer Science, Nankai University · 2Tongyi Lab · 2Tongyi Lab,3Peking University +1
Learning from the Near Future: Temporal Self-Distillation for RLVR
Apr 22, 2026Chuanyu Qin, Chenxu Yang, Qingyi Si +6Reinforcement Learning with Verifiable Rewards Self-Distillation
Institute of Information Engineering, CAS · School of Cyber Security, UCAS
Randomly Initialized Networks Can Learn from Peer-to-Peer Consensus
Apr 20, 2026Esteban Rodríguez-Betancourt, Edgar Casasola-MurilloRepresentation Learning Self-Supervised Learning
Posgrado en Computación e Informática Universidad de Costa Rica · Escuela de Ciencias de la Computación Universidad de Costa Rica
Self-Distillation as a Performance Recovery Mechanism for LLMs: Counteracting Compression and Catastrophic Forgetting
Apr 17, 2026Chi Liu, Xin Chen, Xu Zhou +2LLM Compression Language Model Distillation
PayPal AI
Why Fine-Tuning Encourages Hallucinations and How to Fix It
Apr 16, 2026Guy Kaplan, Zorik Gekhman, Zhen Zhu +5Supervised Fine-Tuning Continual Learning for LLMs
♡Hebrew University of Jerusalem · ♣Technion – Israel Institute of Technology · ♢University of Illinois Urbana-Champaign +1
Shuffle the Context: RoPE-Perturbed Self-Distillation for Long-Context Adaptation
Apr 15, 2026Zichong Li, Chen Liang, Liliang Ren +3Long-Context Language Modeling Language Model Robustness
Work is done during internship at Microsoft. · Georgia Institute of Technology · Microsoft.

Jun 9, 2026J/K move · Enter open · S save

Self-Distillation Policy Optimization via Visual Feedback: Bridging Code and Visual Artifacts

Haoyu Dong

Microsoft

RL for Code Generation Code Generation Language Model Distillation Self-Distillation LLM Code Generation Chart-to-Code Generation

Open paper AskPDF arXiv

Code-generating large language models (LLMs) increasingly produce visual artifacts such as charts, web pages, and slides by writing programs that are executed by non-differentiable renderers, committing to code before observing the render. As a result, otherwise executable code often yields artifacts with visually salient defects, including overlapping elements, clipped text, broken alignment, low contrast, and overflow. We study visual-feedback self-distillation for code-generated visual artifacts. We propose Visual-SDPO, a self-distillation policy-optimization framework that treats rendered visual feedback as privileged context for a weight-sharing teacher and distills this feedback into a coding student. To make supervision spatially targeted rather than uniform, we introduce Visual-Grounded Code Credit Weighting, which traces each detected defect back to the code statements responsible for the affected elements and amplifies the distillation signal on those statements. A sequence-level GRPO (Group Relative Policy Optimization) term complements the dense token-level objective by rewarding executable, visually high-quality rollouts, while failed executions remain learnable through the self-distillation path by passing execution errors as privileged context to the teacher. We instantiate Visual-SDPO for chart, web/UI, and slide generation with a unified Qwen3-VL-8B-Instruct backbone. Across chart-to-code, UI-to-code, and slide-generation benchmarks (ChartMimic, Design2Code, and AeSlides), Visual-SDPO improves over the zero-shot base by more than 10 absolute points in the primary metric and over GRPO by at least 2.4 points, with fewer training steps and no added inference-time cost.