cs.LGNov 1, 2026

Bernoulli On-Policy Distillation: Don't Ignore High-Confidence Student Tokens

Authors: Mingyu Cao, Yaokai Jin, Shiwei Liu, Lu Yin

Organizations: University of Surrey · University College London · ELLIS Institute Tübingen · Max Planck Institute for Intelligent Systems · 

Abstract

On-policy distillation (OPD) trains a student language model on its own trajectories using token-level supervision from a stronger teacher. We show that standard sampled reverse-KL OPD systematically suppresses updates to high-confidence tokens, such that their weak gradient contribution does not necessarily imply weak or uninformative teacher supervision. We introduce Bernoulli On-Policy Distillation (Bernoulli OPD), which directly matches the teacher and student probabilities for that token. This yields a bounded probability space correction that preserves teacher-student disagreement while avoiding the additional confidence-dependent suppression induced by vanilla OPD. Fixed-rollout analyses show that Bernoulli OPD substantially restores optimization signal to high-confidence tokens while remaining sensitive to teacher-student disagreement. We further analyze the resulting model behavior through confidence distributions, answer calibration, policy movement, and training dynamics, showing that Bernoulli OPD's gains cannot be explained simply by longer reasoning, larger optimization steps, or more uniform imitation of the teacher. Code is available at https://github.com/duterscmy/Bernoulli-OPD.

Figures & tables

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes

    Mar 26, 2026Yuqian Fu, Haohuan Huang, Kaiwen Jiang +4Language Model DistillationOn-Policy Distillation

  2. Trust Region On-Policy Distillation

    May 31, 2026Xingrun Xing, Haoqing Wang, Boyan Gao +2Language Model DistillationTrust-Region Optimization