On-Policy Self-Distillation

On-policy self-distillation studies how a model can improve from its own generated responses, using a teacher version of the model to provide token-level guidance along the student’s actual trajectories. Common approaches give the teacher extra information, such as solutions, critiques, visual evidence, or task feedback, and use filtering, confidence weighting, or reference anchoring to make its guidance more useful. Methods are being applied to language-model reasoning, tool-using agents, multimodal tasks, and generative models, often alongside verifiers or reinforcement-learning methods such as GRPO and RLVR. A central tension is that privileged guidance can correct errors but also encourage shortcuts, suppress exploration, or harm out-of-distribution generalization, so reliable and transferable supervision remains an open challenge.