cs.LGNov 10, 2026

Bootstrapped On-Policy Distillation

Authors: Yue Wu, Xueyu Zhou, Yangrong Hu, Jian Huang

Organizations: Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University · Department of Applied Mathematics, The Hong Kong Polytechnic University

Abstract

On-policy distillation (OPD) improves language models by distilling teacher predictions on student-generated trajectories, while on-policy self-distillation (OPSD) removes the need for an external teacher by using a reference-conditioned copy of the model as the teacher. Despite its great early improvements, we find that OPSD does not remain consistently effective under continued training and can eventually suffer substantial performance degradation. We study this training behavior and link the degradation to a growing mismatch between the evolving reference-free student and the frozen privileged teacher. In the early stage of OPSD, the privileged teacher remains closely aligned with the student and provides useful guidance for improving the student's generation trajectories. However, as training progresses, the growing distributional mismatch between the privileged teacher and reference-free student can cause the teacher to provide misleading supervision, ultimately degrading student performance. This phenomenon motivates us to first leverage privileged reference supervision to acquire useful knowledge and then transfer it to a fresh student. We introduce Bootstrapped On-Policy Distillation (BOPD), a two-stage framework that first uses OPSD to internalize task-relevant information from reference-conditioned supervision into an intermediate model, and then distills this reference-free intermediate model into a fresh copy of the initial model. Experiments across models show that BOPD improves mathematical reasoning and code generation beyond both the initial models and the constructed OPSD teachers. These results suggest that privileged reference supervision can be more effectively exploited by first transferring its task-relevant signal into a task-adapted model and then distilling from that model without reference conditioning.

Figures & tables

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Teacher Should Think Ahead: Adaptive Continuations for Reliable On-Policy Distillation

    Sep 6, 2026Jingang Zhou, Yuyi Zhou, Haiyang Guo +6Language Model DistillationOn-Policy Distillation

  2. Train Ahead, Distill Back: Bootstrapping On-Policy Self-Distillation for Large Language Models

    Sep 29, 2026Zheng Zhang, Xinyue Tan, Lufei Li +3On-Policy Self-DistillationLanguage Model Distillation

  3. Proposing Better Rollouts for On-Policy Distillation

    Oct 27, 2026Sirui Chen, Yunzhe Qi, Haobo Xu +4Language Model DistillationOn-Policy Distillation