Overview

Self-distillation uses a model—or a related version of it—as a teacher, transferring knowledge through output probabilities, hidden representations, or generated experience. Common approaches include teacher–student consistency with exponential-moving-average or earlier-checkpoint teachers, on-policy learning from the model’s own trajectories, and privileged or hindsight feedback that supplies extra context during training. The methods are used for language-model reasoning and tool use, as well as vision, audio, video, time series, and faster generative inference; evaluation often spans math and code reasoning, agent tasks, and domain-specific perception benchmarks. A central challenge is that teacher signals can be biased or misaligned with task success, reinforcing errors, reducing exploration, or causing forgetting, so generalization, capability retention, training stability, and compute cost remain important open concerns.

Latest papers

CardsList

151 papers23 papers in the last four weeks, down 23% on the four weeks before. 0.2% of all new papers.

  1. Model Annealing: Data-Efficient Learning through Review and Consolidation

    Nov 12, 2026Ali Mohammadi Ruzbahani, Rezvan AbouzadehData-Efficient LearningSelf-Distillation

  2. Hint-Informed Self-Distillation for Reasoning Segmentation

    Nov 11, 2026Kunze Jiang, Yuchen Wang, Kang Li +1Reasoning SegmentationReferring Image Segmentation

  3. ECOS: Environment-Calibrated On-Policy Self-Distillation for Language Agents

    Nov 9, 2026Chenhao Zhang, Zhongan Wang, Mingxuan Li +3Self-DistillationOn-Policy Distillation

  4. Speculative Self-Distillation enables Efficient Knowledge Internalization

    Nov 9, 2026Shayan Talaei, Agam Bhatia, Arshia Soltani Moakhar +3Language Model DistillationSelf-Distillation

  5. Search-Aware On-Policy Self-Distillation with Sibling Alignment for Generative Recommendation

    Nov 1, 2026Rongfeng Guo, Zhuang Zhuang, Ji Dai +6Generative RecommendationSelf-Distillation

  6. Steer, Don't Tell: Breaking Reasoning Plateaus with Reachability-Aligned Self-Distillation

    Oct 31, 2026Zijian Wu, Lingkai Kong, Lin Yang +5LLM ReasoningLarge Reasoning Models

  7. Knowledge-structured and evidence-aware semantic self-distillation for glomerular pathology foundation model

    Oct 28, 2026Qiming He, Jing Li, Zihan Wang +24Pathology Foundation ModelsMedical VLMs

  8. What Can Firms Do in the Face of Model Distillation?

    Oct 28, 2026Diptangshu Sen, Jibang Wu, Juba ZianiModel Extraction AttacksKnowledge Distillation

  9. Uncovering Shortcut Amplification in Video RLVR through the Lens of Perceptual Uncertainty Collapse

    Oct 27, 2026Jiafeng Liang, Xianhao Ou, Tao Ren +6RL for Video ReasoningReinforcement Learning with Verifiable Rewards

  10. Self-Regularized Flow Language Models

    Oct 27, 2026Theodoros Kouzelis, Spyros Gidaris, Nikos KomodakisDiffusion Language Model TrainingContinuous Diffusion Language Models

  11. WithinVAE: Compact Visual Autoencoding with Self-Distilled Semantics

    Oct 27, 2026Yuzhou Ji, Jiepeng Wang, Tianyu Jin +4Variational AutoencodersVisual Token Compression

  12. How Much Does the Hint Matter? Scope Conditions for Hindsight Self-Distillation in Financial Reasoning

    Oct 26, 2026Johnny Yu, Cyril GorllaSelf-DistillationKnowledge Distillation

  13. Spatial Audio Generation and Editing with Executable Plans and Self-Distilling Transfusion

    Oct 25, 2026Kam Man Wu, Yu Zhang, Pengjun Fang +4Spatial AudioAudio Editing

  14. Distilling Dense Trajectory Teachers For Large-Motion Video World Models

    Oct 24, 2026Shuofeng Sun, Linqing Zhao, Haibin YanVideo World ModelsCamera-Controlled Video Generation

  15. CORD: Conflict-Aware Optimization for Joint RL and Distillation in Agents

    Oct 21, 2026Wenhao Yu, Zhengcheng Wang, Jiahong Liu +7Gradient InterferenceSelf-Distillation

  16. FLMNet: Self-Ensemble Distillation from Masks to Seeds for Single-Tower CTR Prediction

    Oct 19, 2026Yifan Wang, Yifei DuCTR PredictionSelf-Distillation

  17. Self-Distillation for Supervised Fine-Tuning

    Oct 18, 2026Wenxuan Zhu, Bang An, Shuming Liu +6Supervised Fine-TuningLanguage Model Distillation

  18. DiPLO: Self-distilled Diffusion Planning for Loco-manipulation

    Oct 17, 2026Shafeef Omar, Sebastian Sanokowski, Elham Daneshmand +4Robotic Loco-ManipulationHumanoid Loco-Manipulation

  19. AtomSD: Procedural Atom-Guided Self-Distillation for LLM Agents

    Oct 17, 2026Bohan Yu, Pengfei Cao, Kai Tian +8LLM Agent Skill LearningRL for LLM Agents

  20. More Methods, Better Agents? A Reality Check on Self-Distillation in Agentic Reinforcement Learning

    Oct 14, 2026Linhao Zhong, Bozhen Fang, Xuehao Wang +7Agentic RLSelf-Distillation

  21. Function-Space Temporal Self-Distillation for Policy Optimization

    Oct 14, 2026Aryan Das, Arnab Bhattacharya, Vinay Kumar VermaRL for Language Model ReasoningSelf-Distillation

  22. Does self-distillation improve reasoning?

    Oct 13, 2026Jason Ken Adhinarta, Raina Wu, Ashutosh Sharma +5Self-DistillationOn-Policy Self-Distillation

  23. Decoupling Knowledge Forgetting and Refusal Learning in MLLM Unlearning

    Oct 10, 2026Haodong Chen, Shengtao Wen, Dong Liang +2MLLM UnlearningRefusal Behavior in Language Models

  24. Continually Post-training Small Models via Synthetic Self-distillation

    Oct 9, 2026Woosung Koh, Hyunsoo Lee, Kyungjae Lee +4LLM Post-TrainingContinual Learning for LLMs