Overview

Self-distillation uses a model—or a related version of it—as a teacher, transferring knowledge through output probabilities, hidden representations, or generated experience. Common approaches include teacher–student consistency with exponential-moving-average or earlier-checkpoint teachers, on-policy learning from the model’s own trajectories, and privileged or hindsight feedback that supplies extra context during training. The methods are used for language-model reasoning and tool use, as well as vision, audio, video, time series, and faster generative inference; evaluation often spans math and code reasoning, agent tasks, and domain-specific perception benchmarks. A central challenge is that teacher signals can be biased or misaligned with task success, reinforcing errors, reducing exploration, or causing forgetting, so generalization, capability retention, training stability, and compute cost remain important open concerns.

Latest papers

CardsList

151 papers23 papers in the last four weeks, down 23% on the four weeks before. 0.2% of all new papers.

  1. Rubric-CEPR: Self-Evolving Image Editing via Reward-Verified Self-Distillation

    Oct 8, 2026Ritesh Thawkar, Shubham Patle, Shravan Venkatraman +1Image EditingText-Guided Image Editing

  2. Environmental Feedback Modeling Matters: Rethinking Feedback Treatment in Agentic Hindsight Self-Distillation

    Oct 8, 2026Hangxi Guo, Fengyuan Liu, Yue Wang +4LLM Agent TrainingSelf-Distillation

  3. SAPD: Step-Aligned Privileged Distillation

    Oct 7, 2026Tianle Wang, Jiayu Liu, Ruizhi Zhao +1LLM Post-TrainingLanguage Model Distillation

  4. How Much Evidence Should a Coding Agent's Self-Correction Carry? Adaptive Dirichlet Evidence for Self-Distillation

    Oct 6, 2026Yunbo Long, Guangya Hao, Yuhan Liu +4LLM Self-CorrectionCoding Agents

  5. Self-Retrospection Distillation: Turning Post-hoc Experiences into Prior Foresight

    Oct 6, 2026Haoxiang Zhang, Qinglin Chen, Hiroaki Hayashi +9Agentic RLRL for Language Model Reasoning

  6. SPECTRUM: Proximal Spectral Modulation for Looped Self-Distillation

    Oct 5, 2026Yunbo Long, WenJie Chen, Jiaquan Zhang +4Self-DistillationCode Generation

  7. RAISED: Self-Distillation for Robustness to Prompt Injection in LLM Agents

    Oct 5, 2026Mohamed Dhouib, Clement Elliker, Alexi Canesse +5Prompt Injection DefenseLLM Agents

  8. Pivot-SD: Efficient Self-Distillation for Masked Diffusion Language Models

    Oct 2, 2026Seo Hyun Kim, Sunwoo Hong, Younwoo Choi +3Diffusion Language Model DistillationMasked Diffusion Models

  9. The Missing Primitive: Diagnosing and Repairing Mathematical Reasoning in Large Language Models

    Oct 1, 2026Shuo Xing, Zilin Dai, Chengyuan Qian +7Mathematical Reasoning BenchmarksSelf-Distillation

  10. RISED: RubrIcs for agentic multi-environment Selection and sElf-Distillation

    Oct 1, 2026Jingtan Wang, Sirajul Salekin, Young mok Jung +3Reinforcement LearningRubric-Based RL

  11. OPSRD: On-Policy Self-Role Distillation

    Sep 30, 2026Weijie Ren, Yanwen Zhang, Hao Li +3LLM PromptingSelf-Distillation

  12. Disentangling Self-Distillation: Measuring and Modeling Acquisition and Retention

    Sep 30, 2026Luis Zuin, Alexis Huet, Dario Rossi +1On-Policy Self-DistillationSelf-Distillation

  13. ReSAIL: Mitigating Collapse in Iterative Agent Self-Distillation

    Sep 30, 2026Shengjie Jin, Hengbo Xu, Zelong Sun +2On-Policy Self-DistillationLLM Agent Self-Improvement

  14. OP-CAD: On-Policy Clean-Audio Distillation for Robust Audio-Visual Reasoning

    Sep 30, 2026Xingming Shui, Dapeng Chen, Bowei Liu +5Audio-Visual UnderstandingMultimodal Robustness

  15. AdviSD: Learning to Advise Frontier LLMs via Targeted Multi-Turn Self-Distillation

    Sep 29, 2026Rishabh Agrawal, Hejie Cui, Shasha Li +2Language Model SteeringLLM Agent Self-Improvement

  16. Spatial-OPSD: Self-Improving Spatial Reasoning via Label-Free Self-Distillation

    Sep 29, 2026Zhenyu Liu, Zhangquan Chen, Keyi Chen +4Visual Spatial Reasoning3D Spatial Reasoning

  17. Beam Search as Test-Time Self-Distillation via Counterfactual Contexts

    Sep 29, 2026Su Ee Tan, Xiaotong Ji, Rasul Tutunov +2Language Model DistillationSelf-Distillation

  18. EGSD: Event-Grounded Self-Distillation for Streaming Video Understanding

    Sep 29, 2026Yuwei Miao, Xuesheng Zhang, Wenhao Zou +5Streaming Video UnderstandingSelf-Distillation

  19. Can We Trust the Teacher? Decoupled Credit Direction-Magnitude for Self-Distillation

    Sep 28, 2026Yugu Li, Zehong Cao, Peizhen Li +3Token-Level Credit AssignmentRL for Language Model Reasoning

  20. IGSD: Environment-Verified Hindsight Self-Distillation for Search Agents

    Sep 26, 2026Angqing Jiang, Gaoming Zhang, Chaoqun Zhang +5Self-DistillationAgentic Search

  21. Recursive Self-Improvement via On-Policy Distillation for Reasoning

    Sep 25, 2026Shangjian Yin, Zehao Zhao, Kavosh Asadi +7Language Model Self-ImprovementOn-Policy Self-Distillation

  22. Two Global Crops Suffice: Locating Semantic Emergence in DINO-Style Self-Supervised Learning

    Sep 23, 2026Basavaraj Sunagad, Artur Jesslen, Adam KortylewskiMulti-View LearningSelf-Distillation

  23. Self-Distilled Pronunciation and Accent Control for Neural Text-to-Speech

    Sep 15, 2026Shuhei KatoTTS SynthesisSpeech Prosody

  24. Temporal Self-Distillation: Faster Inference in Discrete Diffusion Language Models

    Sep 14, 2026Shijian Xu, Andrea Miele, Metod Jazbec +3Diffusion Model DistillationDiffusion Language Model Inference

  25. Negative Self-Distillation: Learning to Reason by Avoiding Flaws

    Sep 10, 2026Rongcan Pei, Zhepei Wei, Shuyao Xu +3Language Model DistillationSelf-Distillation

  26. Decoupled Self-Forcing Distillation for Streaming Talking Head Generation

    Sep 9, 2026Yanru An, Ruiyan Wang, Wenwu Wei +7Audio-Video GenerationDiffusion Model Distillation

  27. Beyond Verified Answers: Solver-Informed Self-Distillation for Bootstrapping Operations Research Language Models

    Sep 9, 2026Rui Zhu, Minglong Cao, Chenyu Zhou +2RL for Language ModelsLarge Language Model-Guided Optimization