cs.LGNov 16, 2026

Rethinking Supervision Scaling in On-Policy Distillation: From Dense Supervision to Stochastic Horizons

Authors: Xinhao Yao, Zhenxing Dou, Yong Liu

Organizations: Renmin University of China

Abstract

On-policy distillation (OPD) provides dense token-level supervision along student-generated trajectories, yet the learning benefits of longer trajectories may not scale proportionally with their computational cost. We investigate how supervision utility evolves with trajectory length. Controlled experiments reveal diminishing returns from additional supervised tokens, while showing that later positions remain learnable and exhibit substantial cross-position supervision transfer. Gradient analysis further identifies shared update components across response positions and diverse position-specific residuals whose cross-rollout consistency decreases at greater depths. Motivated by these findings, we propose Stochastic Horizons On-Policy Distillation (SH-OPD), which reduces redundant supervision by randomly varying rollout horizons while preserving access to deeper response states. SH-OPD samples a shared horizon for each training batch, enabling efficient parallel generation without per-sequence horizon scheduling overhead. We characterize the resulting reweighting bias and derive a compute-reweighting trade-off under standard optimization assumptions. Experiments on mathematical reasoning and code generation demonstrate that SH-OPD achieves slightly higher mathematical reasoning accuracy at approximately 45% of the per-update cost of standard OPD, while maintaining competitive code generation performance at around 60% of the baseline cost. These results highlight the benefits of efficient supervision allocation across response depths.

Figures & tables

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation

    May 29, 2026Yanjiang Liu, Jie Lou, Xinyan Guan +7CoT DistillationLanguage Model Distillation

  2. Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning

    May 8, 2026Zhicheng Yang, Zhijiang Guo, Yifan Song +5RL for Language Model ReasoningLanguage Model Distillation

  3. Step-Level On-Policy Distillation: Interpolating Between On-Policy Distillation and Supervised Fine-Tuning

    Aug 17, 2026Changhui Sun, Lanbo Liu, Hang Lei +11Language Model DistillationOn-Policy Distillation