Overview

Self-distillation uses a model—or a related version of it—as a teacher, transferring knowledge through output probabilities, hidden representations, or generated experience. Common approaches include teacher–student consistency with exponential-moving-average or earlier-checkpoint teachers, on-policy learning from the model’s own trajectories, and privileged or hindsight feedback that supplies extra context during training. The methods are used for language-model reasoning and tool use, as well as vision, audio, video, time series, and faster generative inference; evaluation often spans math and code reasoning, agent tasks, and domain-specific perception benchmarks. A central challenge is that teacher signals can be biased or misaligned with task success, reinforcing errors, reducing exploration, or causing forgetting, so generalization, capability retention, training stability, and compute cost remain important open concerns.

Latest papers

CardsList

154 papers20 papers in the last four weeks, down 46% on the four weeks before. 0.2% of all new papers.

  1. Optimal Unconstrained Self-Distillation in Ridge Regression: Strict Improvements, Precise Asymptotics, and One-Shot Tuning

    Feb 19, 2026Hien Dang, Pratik Patil, Alessandro RinaldoSelf-DistillationStatistical Learning Theory

  2. Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Models

    Feb 2, 2026Hao Wang, Hao Gu, Hongming Piao +6Supervised Fine-TuningLanguage Model Distillation

  3. ThinkSafe: Self-Generated Safety Alignment for Reasoning Models

    Jan 30, 2026Seanie Lee, Sangwoo Park, Yumin Choi +6Safety-Preserving Fine-Tuning of Language ModelsLLM Safety

  4. Stabilizing Consistency Training: A Flow Map Analysis and Self-Distillation

    Jan 30, 2026Youngjoong Kim, Duhoe Kim, Woosung Kim +1Consistency ModelsConsistency Distillation