Overview

Self-distillation uses a model—or a related version of it—as a teacher, transferring knowledge through output probabilities, hidden representations, or generated experience. Common approaches include teacher–student consistency with exponential-moving-average or earlier-checkpoint teachers, on-policy learning from the model’s own trajectories, and privileged or hindsight feedback that supplies extra context during training. The methods are used for language-model reasoning and tool use, as well as vision, audio, video, time series, and faster generative inference; evaluation often spans math and code reasoning, agent tasks, and domain-specific perception benchmarks. A central challenge is that teacher signals can be biased or misaligned with task success, reinforcing errors, reducing exploration, or causing forgetting, so generalization, capability retention, training stability, and compute cost remain important open concerns.

Latest papers

CardsList

151 papers23 papers in the last four weeks, down 23% on the four weeks before. 0.2% of all new papers.

  1. Stabilizing Consistency Training: A Flow Map Analysis and Self-Distillation

    Jan 30, 2026Youngjoong Kim, Duhoe Kim, Woosung Kim +1Consistency ModelsConsistency Distillation