Teacher Should Think Ahead: Adaptive Continuations for Reliable On-Policy Distillation

Jingang Zhou; Yuyi Zhou; Haiyang Guo; Xukai Wang; Shuai Feng; Sirui Gao; Jian Xu; Qingpei Guo

cs.LGSep 6, 2026

Teacher Should Think Ahead: Adaptive Continuations for Reliable On-Policy Distillation

Authors: Jingang Zhou, Yuyi Zhou, Haiyang Guo, Xukai Wang, Shuai Feng, Sirui Gao, Jian Xu, Qingpei Guo,

Organizations: School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Science · Ant Group

Abstract

On-policy distillation (OPD) is a promising approach for transferring knowledge between language models, where a student receives dense token-level supervision along its own generated trajectories. However, teacher supervision can be unreliable when conditioned on incomplete or low-quality student prefixes. We identify Teacher Uncertainty Contraction (TUC), a systematic phenomenon whereby the teacher's predictive uncertainty decreases as it continues from a student-generated prefix. We theoretically characterize this trade-off through a variance-bias decomposition of teacher-branch gradients, showing that uncertainty contraction reduces variance while teacher-student path divergence increases bias, thereby favoring a finite continuation. Guided by this insight, we propose Adaptive-Continuations On-Policy Distillation (AC-OPD), which augments informative states along student rollouts with teacher continuations and adaptively selects their effective supervision horizons. Experiments on mathematical reasoning and code generation across model scales demonstrate that AC-OPD consistently improves over standard OPD. Controlled-continuations and matched-budget analyses further validate the adaptive-continuations design, highlighting adaptive teacher continuations as an effective principle for reliable on-policy distillation.The code will be made publicly available upon publication.

Figures & tables

Figure 1: Teacher Uncertainty Contraction along teacher continuations. Mean teacher entropy generally decreases with the teacher continuations across different anchor positions in student-generated responses. Height and color both encode entropy.

Figure 2: Teacher Uncertainty Contraction. Both curves measure teacher uncertainty: the teacher-continuation curve evaluates

\pi_{T}

C_{j}^{T}

, whereas the student-continuation curve evaluates

\pi_{T}

C_{j}^{S}

. Entropy (a) and Gini impurity (b) contract only on teacher-generated contexts. Shaded regions are prompt-clustered

95\%

confidence intervals.

Figure 3: Overview of AC-OPD. (a) The student generates an on-policy trajectory, from which $K$ anchors are selected using the squared log-ratio score. (b) From each anchor, the teacher generates a continuation of at most $m_{\max}$ tokens. The fixed variant retains the full continuation for supervision, whereas the adaptive variant selects an effective prefix of length $h_{i}$ and masks the remaining tokens. (c) The resulting branch loss is combined with the standard OPD trunk loss to update the student.

	Mathematical Reasoning: pass@16 $\uparrow$					Code Generation: pass@1 $\uparrow$
Method	AMC23	AIME24	AIME25	Minerva	Math Avg.	HE+	MBPP+	Code Avg.
Setting 1: Qwen3-4B $\rightarrow$ Qwen3-1.7B-Base
SFT	61.5 $\pm$ 0.20	23.0 $\pm$ 0.34	16.3 $\pm$ 0.46	44.5 $\pm$ 0.30	36.33 $\pm$ 0.27	58.5 $\pm$ 0.16	57.5 $\pm$ 0.18	58.00 $\pm$ 0.22
OPD	61.0 $\pm$ 0.39	13.3 $\pm$ 0.18	15.6 $\pm$ 0.33	44.7 $\pm$ 0.29	33.65 $\pm$ 0.21	59.8 $\pm$ 0.25	55.6 $\pm$ 0.31	57.70 $\pm$ 0.27
ExOPD	57.7 $\pm$ 0.29	21.1 $\pm$ 0.44	17.4 $\pm$ 0.40	44.7 $\pm$ 0.14	35.23 $\pm$ 0.19	60.4 $\pm$ 0.22	56.9 $\pm$ 0.12	58.65 $\pm$ 0.18
EOPD	57.8 $\pm$ 0.21	20.0 $\pm$ 0.41	16.7 $\pm$ 0.32	45.2 $\pm$ 0.12	34.93 $\pm$ 0.21	60.2 $\pm$ 0.25	55.8 $\pm$ 0.15	58.00 $\pm$ 0.23

Table 1: Main results under two Qwen3 teacher-student configurations. Each entry reports the mean

\pm

standard deviation over five independent training runs. All results are reported as percentages, and standard deviations are measured in percentage points. Per-benchmark results are rounded to one decimal place, whereas domain averages are computed from unrounded scores and reported to two decimal places. Bold denotes the best mean result within each setting.

Anchor policy	HE+	MBPP+	Code Avg.
Even	61.6	56.3	58.97
Random	61.6	55.6	58.57
Squared log-ratio	64.8	57.6	61.20

Table 2: Effect of anchor selection. Comparison of three anchor policies under the Setting 1 adaptive-continuation configuration. Results are means over five independent runs, with Code Avg. computed from unrounded benchmark scores.

Figure 4: Effect of the continuation length. We vary the fixed continuation length

h

while keeping all other settings unchanged. Performance is non-monotonic and peaks at

h=8

on both benchmarks; dashed lines denote the corresponding AC-OPD results.

Policy	Mean $h_{i}$	HE+	MBPP+	Code Avg.
Fixed-6	6.00	59.8	57.1	58.45
Fixed-8	8.00	64.4	57.3	60.85
Shuffled Adaptive	5.74	63.4	56.8	60.10
Adaptive	5.74	64.8	57.6	61.20

Table 3: Comparison of fixed and adaptive continuation policies. Fixed-6 is mean-continuation matched, while Shuffled Adaptive preserves the continuation distribution but breaks context-continuation correspondence. Results are averaged over five independent runs.

Figure 5: Early training progress. Comparison of OPD and AC-OPD with matched supervised tokens on code task.

Language Model Distillation

Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.

See all 512 papers

On the Off-Policy Teacher in On-Policy Distillation
Sep 29, 2026Langlin Huang, Hao Liu, Mononito Goswami +5Language Model Distillation On-Policy Distillation
Washington University in St. Louis · AWS AI Labs · Carnegie Mellon University
Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation
May 29, 2026Yanjiang Liu, Jie Lou, Xinyan Guan +7CoT Distillation Language Model Distillation
University of Chinese Academy of Sciences · 2Chinese Information Processing Laboratory Institute of Software, Chinese Academy of Sciences 2University of Chinese Academy of Sciences, Beijing, China · 3Xiaohongshu
Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillation
Sep 2, 2026Zhiwei Zhang, Zechen Sun, Fei Zhao +6On-Policy Distillation RL Post-Training
AllSpark Team

Sep 29, 2026J/K move · Enter open · S save

On the Off-Policy Teacher in On-Policy Distillation

Langlin Huang, Hao Liu, Mononito Goswami, Xinyu Li+4

Washington University in St. Louis · AWS AI Labs · Carnegie Mellon University

Language Model Distillation On-Policy Distillation Knowledge Distillation

Open paper AskPDF arXiv

On-policy distillation (OPD) has recently emerged as a promising post-training paradigm in which the student learns from trajectories generated by its own policy under dense teacher supervision. However, OPD introduces a fundamental asymmetry: although the sampled trajectories are on-policy for the student, they are off-policy for the teacher. The teacher is typically optimized to continue from prefixes generated by its own policy, but during OPD it must instead supervise prefixes generated by the student. Empirically, we find that its continuation performance degrades as these prefixes grow longer. To address this issue, we propose Student-COnditioned Updates of the Teacher (SCOUT), a co-training framework that adapts the teacher to student-generated prefixes. Alongside standard OPD updates, SCOUT periodically optimizes the teacher's conditional ability using reinforcement learning with verifiable rewards, where the teacher generates continuations from student prefixes and learns from outcome rewards. Controlled experiments show that SCOUT improves the teacher's ability to continue from student-generated prefixes, supporting the intended mechanism of student-conditioned teacher adaptation. Across multiple teacher--student configurations, model scales, and reasoning domains, SCOUT also consistently improves the effectiveness of on-policy distillation.

Abstract

Language Model Distillation

Explore similar work

On the Off-Policy Teacher in On-Policy Distillation

Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation

Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillation