cs.LGNov 1, 2026

OC-OPD: Observation-coordinated Supervision for On-Policy Distillation

Authors: Ruoran Xu, Yilun Zhao, Ziyu Chen, Arman Cohan

Organizations: Yale NLP Lab

Abstract

Reverse and forward Kullback--Leibler (KL) divergences provide complementary supervision in on-policy distillation: reverse KL refines the student's active token rankings, and forward KL allocates mass to teacher-supported candidates. At student-generated prefixes, we form a four-observation matrix by treating the student and teacher as both candidate sources and evaluators. The diagonal self-observations measure uncertainty, while source-centered cross-observations recover the two KL objectives. We establish an exact identity equating the Fisher inverse-metric coupling of their logit gradients with the sum of these contrasts, yielding a common local descent direction for distinct positive distributions. Observation-coordinated on-policy distillation (OC-OPD) uses observations to set preferences, closed-form bargaining to coordinate gradients, and distributional discrepancy to scale the update. Across three heterogeneous student settings (non-thinking generation, thinking generation, and base-model initialization), OC-OPD achieves strong gains in mathematical reasoning and transfers effectively to out-of-distribution scientific question answering and code generation. Our code is publicly available at: https://github.com/RuoranXu/OC-OPD.

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Reinforcing What the Student Has Learned: Teacher-Guided On-Policy Distillation

    Nov 5, 2026Bozhen Fang, Linhao Zhong, Xuehao Wang +3Language Model DistillationOn-Policy Distillation

  2. A Token-Level Analysis of Sampled-Token Reverse-KL On-Policy Distillation

    Aug 26, 2026Bing Shao, Jiazheng Zhang, Long Ma +13Language Model DistillationOn-Policy Distillation

  3. Calibrating Teacher--Student Discrepancy for On-Policy Distillation

    Sep 18, 2026Qiangqiang He, Jin Li, MingCai ChenOn-Policy DistillationLanguage Model Distillation