Overview

Self-distillation uses a model—or a related version of it—as a teacher, transferring knowledge through output probabilities, hidden representations, or generated experience. Common approaches include teacher–student consistency with exponential-moving-average or earlier-checkpoint teachers, on-policy learning from the model’s own trajectories, and privileged or hindsight feedback that supplies extra context during training. The methods are used for language-model reasoning and tool use, as well as vision, audio, video, time series, and faster generative inference; evaluation often spans math and code reasoning, agent tasks, and domain-specific perception benchmarks. A central challenge is that teacher signals can be biased or misaligned with task success, reinforcing errors, reducing exploration, or causing forgetting, so generalization, capability retention, training stability, and compute cost remain important open concerns.

Latest papers

CardsList

151 papers20 papers in the last four weeks, down 46% on the four weeks before. 0.2% of all new papers.

  1. Decoupling Knowledge Forgetting and Refusal Learning in MLLM Unlearning

    Oct 10, 2026Haodong Chen, Shengtao Wen, Dong Liang +2MLLM UnlearningRefusal Behavior in Language Models

  2. Continually Post-training Small Models via Synthetic Self-distillation

    Oct 9, 2026Woosung Koh, Hyunsoo Lee, Kyungjae Lee +4LLM Post-TrainingContinual Learning for LLMs

  3. Rubric-CEPR: Self-Evolving Image Editing via Reward-Verified Self-Distillation

    Oct 8, 2026Ritesh Thawkar, Shubham Patle, Shravan Venkatraman +1Image EditingText-Guided Image Editing

  4. Environmental Feedback Modeling Matters: Rethinking Feedback Treatment in Agentic Hindsight Self-Distillation

    Oct 8, 2026Hangxi Guo, Fengyuan Liu, Yue Wang +4LLM Agent TrainingSelf-Distillation

  5. SAPD: Step-Aligned Privileged Distillation

    Oct 7, 2026Tianle Wang, Jiayu Liu, Ruizhi Zhao +1LLM Post-TrainingLanguage Model Distillation

  6. How Much Evidence Should a Coding Agent's Self-Correction Carry? Adaptive Dirichlet Evidence for Self-Distillation

    Oct 6, 2026Yunbo Long, Guangya Hao, Yuhan Liu +4LLM Self-CorrectionCoding Agents

  7. Self-Retrospection Distillation: Turning Post-hoc Experiences into Prior Foresight

    Oct 6, 2026Haoxiang Zhang, Qinglin Chen, Hiroaki Hayashi +9Agentic RLRL for Language Model Reasoning

  8. SPECTRUM: Proximal Spectral Modulation for Looped Self-Distillation

    Oct 5, 2026Yunbo Long, WenJie Chen, Jiaquan Zhang +4Self-DistillationCode Generation

  9. RAISED: Self-Distillation for Robustness to Prompt Injection in LLM Agents

    Oct 5, 2026Mohamed Dhouib, Clement Elliker, Alexi Canesse +5Prompt Injection DefenseLLM Agents

  10. Pivot-SD: Efficient Self-Distillation for Masked Diffusion Language Models

    Oct 2, 2026Seo Hyun Kim, Sunwoo Hong, Younwoo Choi +3Diffusion Language Model DistillationMasked Diffusion Models

  11. The Missing Primitive: Diagnosing and Repairing Mathematical Reasoning in Large Language Models

    Oct 1, 2026Shuo Xing, Zilin Dai, Chengyuan Qian +7Mathematical Reasoning BenchmarksSelf-Distillation

  12. RISED: RubrIcs for agentic multi-environment Selection and sElf-Distillation

    Oct 1, 2026Jingtan Wang, Sirajul Salekin, Young mok Jung +3Reinforcement LearningRubric-Based RL

  13. OPSRD: On-Policy Self-Role Distillation

    Sep 30, 2026Weijie Ren, Yanwen Zhang, Hao Li +3LLM PromptingSelf-Distillation

  14. Disentangling Self-Distillation: Measuring and Modeling Acquisition and Retention

    Sep 30, 2026Luis Zuin, Alexis Huet, Dario Rossi +1On-Policy Self-DistillationSelf-Distillation

  15. ReSAIL: Mitigating Collapse in Iterative Agent Self-Distillation

    Sep 30, 2026Shengjie Jin, Hengbo Xu, Zelong Sun +2On-Policy Self-DistillationLLM Agent Self-Improvement

  16. OP-CAD: On-Policy Clean-Audio Distillation for Robust Audio-Visual Reasoning

    Sep 30, 2026Xingming Shui, Dapeng Chen, Bowei Liu +5Audio-Visual UnderstandingMultimodal Robustness

  17. AdviSD: Learning to Advise Frontier LLMs via Targeted Multi-Turn Self-Distillation

    Sep 29, 2026Rishabh Agrawal, Hejie Cui, Shasha Li +2Language Model SteeringLLM Agent Self-Improvement

  18. Spatial-OPSD: Self-Improving Spatial Reasoning via Label-Free Self-Distillation

    Sep 29, 2026Zhenyu Liu, Zhangquan Chen, Keyi Chen +4Visual Spatial Reasoning3D Spatial Reasoning

  19. Beam Search as Test-Time Self-Distillation via Counterfactual Contexts

    Sep 29, 2026Su Ee Tan, Xiaotong Ji, Rasul Tutunov +2Language Model DistillationSelf-Distillation

  20. EGSD: Event-Grounded Self-Distillation for Streaming Video Understanding

    Sep 29, 2026Yuwei Miao, Xuesheng Zhang, Wenhao Zou +5Streaming Video UnderstandingSelf-Distillation

  21. Can We Trust the Teacher? Decoupled Credit Direction-Magnitude for Self-Distillation

    Sep 28, 2026Yugu Li, Zehong Cao, Peizhen Li +3Token-Level Credit AssignmentRL for Language Model Reasoning

  22. IGSD: Environment-Verified Hindsight Self-Distillation for Search Agents

    Sep 26, 2026Angqing Jiang, Gaoming Zhang, Chaoqun Zhang +5Self-DistillationAgentic Search

  23. Recursive Self-Improvement via On-Policy Distillation for Reasoning

    Sep 25, 2026Shangjian Yin, Zehao Zhao, Kavosh Asadi +7Language Model Self-ImprovementOn-Policy Self-Distillation

  24. Two Global Crops Suffice: Locating Semantic Emergence in DINO-Style Self-Supervised Learning

    Sep 23, 2026Basavaraj Sunagad, Artur Jesslen, Adam KortylewskiMulti-View LearningSelf-Distillation

  25. Self-Distilled Pronunciation and Accent Control for Neural Text-to-Speech

    Sep 15, 2026Shuhei KatoTTS SynthesisSpeech Prosody

  26. Temporal Self-Distillation: Faster Inference in Discrete Diffusion Language Models

    Sep 14, 2026Shijian Xu, Andrea Miele, Metod Jazbec +3Diffusion Model DistillationDiffusion Language Model Inference