Self-Distillation
Overview
Self-distillation uses a model—or a related version of it—as a teacher, transferring knowledge through output probabilities, hidden representations, or generated experience. Common approaches include teacher–student consistency with exponential-moving-average or earlier-checkpoint teachers, on-policy learning from the model’s own trajectories, and privileged or hindsight feedback that supplies extra context during training. The methods are used for language-model reasoning and tool use, as well as vision, audio, video, time series, and faster generative inference; evaluation often spans math and code reasoning, agent tasks, and domain-specific perception benchmarks. A central challenge is that teacher signals can be biased or misaligned with task success, reinforcing errors, reducing exploration, or causing forgetting, so generalization, capability retention, training stability, and compute cost remain important open concerns.
Latest papers
151 papers23 papers in the last four weeks, down 23% on the four weeks before. 0.2% of all new papers.
Hint-Informed Self-Distillation for Reasoning Segmentation
ECOS: Environment-Calibrated On-Policy Self-Distillation for Language Agents
Speculative Self-Distillation enables Efficient Knowledge Internalization
Search-Aware On-Policy Self-Distillation with Sibling Alignment for Generative Recommendation
Steer, Don't Tell: Breaking Reasoning Plateaus with Reachability-Aligned Self-Distillation
Knowledge-structured and evidence-aware semantic self-distillation for glomerular pathology foundation model
What Can Firms Do in the Face of Model Distillation?
Uncovering Shortcut Amplification in Video RLVR through the Lens of Perceptual Uncertainty Collapse
Self-Regularized Flow Language Models
WithinVAE: Compact Visual Autoencoding with Self-Distilled Semantics
How Much Does the Hint Matter? Scope Conditions for Hindsight Self-Distillation in Financial Reasoning
Spatial Audio Generation and Editing with Executable Plans and Self-Distilling Transfusion
Distilling Dense Trajectory Teachers For Large-Motion Video World Models
Self-Distilling Future Video Representations for World Action Models
CORD: Conflict-Aware Optimization for Joint RL and Distillation in Agents
Self-Evolving Agents through Privileged Retrospective Distillation
FLMNet: Self-Ensemble Distillation from Masks to Seeds for Single-Tower CTR Prediction
Self-Distillation for Supervised Fine-Tuning
DiPLO: Self-distilled Diffusion Planning for Loco-manipulation
AtomSD: Procedural Atom-Guided Self-Distillation for LLM Agents
Progressive Misalignment Can Emerge from Benign Recursive Self-Distillation
More Methods, Better Agents? A Reality Check on Self-Distillation in Agentic Reinforcement Learning
Function-Space Temporal Self-Distillation for Policy Optimization
Does self-distillation improve reasoning?
Teach Me After You Learn It: Lookahead Self-Distillation for RLVR
Training Generalist Navigation Policies via Action Self-Distillation
Decoupling Knowledge Forgetting and Refusal Learning in MLLM Unlearning
A Privileged Teacher That Only Scores Cannot Search: Critique-Conditioned On-Policy Self-Distillation for Reasoning
Wait'', But'', and ``Actually''. We further show that the critique is useful both at test time, where resuming a wrong rollout with the critique in context converts math problems the model cannot solve by sampling, and in cold-start GRPO, where it provides an advantage signal on prompts for which verifiable rewards alone provide none.