Learning from a Thoughtful Teacher: Adaptive On-Policy Self-Distillation for Mathematical Reasoning

Jiacheng Du; Weiwei Xie; Tianyi Du; Shaoxiong Guo; Qibing Ren; Jiaheng Zhang

cs.AISep 26, 2026

Learning from a Thoughtful Teacher: Adaptive On-Policy Self-Distillation for Mathematical Reasoning

Authors: Jiacheng Du, Weiwei Xie, Tianyi Du, Shaoxiong Guo, Qibing Ren, Jiaheng Zhang

Organizations: National University of Singapore · Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory

Abstract

On-policy self-distillation (OPSD) trains a question-only student with token-level feedback from a teacher given training-only privileged information (PI). OPSD therefore provides dense, on-policy supervision, and is free of a larger external teacher, but its effectiveness rests on how PI is designed and utilized. Our preliminary diagnostics suggest a significant gap between teacher utility and student learnability, where a small fraction of high-disagreement tokens dominate the distillation signal, and short teacher continuations at these positions further expose more explicit PI leakage than transferable correction cues, indicating a strong intent on injecting PI-conditioned shortcuts. We propose Adaptive On-Policy Self-Distillation (AOPSD), which adapts what information the teacher receives and how strongly its feedback influences learning. AOPSD encodes each solution as a reasoning DAG, orders problems by the student's evolving capability, and reveals only the affordable subgraph and its next frontier as PI. For high-disagreement tokens, AOPSD utilizes short teacher continuations as probes to encourage useful guidance while mitigating PI-conditioned shortcuts among teacher supervisions. On HMMT25, AIME24, AIME25, and BRUMo25, AOPSD achieves 72.5% Pass@8, which is 6.7 percentage points above OPSD and 4.2 above the strongest competing baseline while reducing 15 percentage points of training time at lower cost.

Figures & tables

Figure 1: Preliminary diagnostics of PI utility and transfer. (a) Change relative to the baseline model: every PI makes the privileged teacher substantially stronger (amber) while the resulting question-only student gets weaker (blue). (b) Token share and share of total absolute sampled-token log gap, grouped by

|g_{t}|

$k$	Correction cues		Explicit PI references
(tokens)	Count	Representative excerpt	Count	Representative excerpt
2	3	an error	8	original DAG
4	7	Not quite—	70	checking the private reasoning DAG
6	22	the density argument must be flawed	153	go back to the private reasoning DAG
8	31	Or maybe we made a mistake?	238	This matches the reasoning in the DAG
16	64	Wait—not necessarily.	507	earlier in the DAG, it says …

Table 1: A short continuation makes high-disagreement intent legible. Cumulative counts of recognizable correction cues and explicit PI leakage within a total continuation of

k

tokens.

Figure 2: AOPSD : adapt what to reveal and how strongly to learn. (a) Teacher and student share an initialization, and the teacher scores the student’s question-only trajectory using privileged information. (b) Based on the student’s current capability, the DAG is adaptively disclosed, and all questions are presented through curriculum learning in order of difficulty. (c) High-gap tokens trigger teacher continuations scored by the question-only student. Band-pass weighting suppresses redundant and potentially PI-dependent guidance while retaining the learnable and useful guidance.

Figure 3: A reasoning DAG reveals one complete dependency frontier at a time. (a) A verified route is compressed into route-defining mathematical checkpoints; edges encode immediate prerequisites. (b) For a question-only rollout, reached states

R

are blue, immediately enabled states

F(R)

are orange, and later states are gray. As more prerequisites are established, disclosure advances by one dependency layer rather than by an arbitrary text prefix.

Figure 4: Continuation surprisal controls the token’s weight. (a) For a large gap, the PI-conditioned teacher generates a continuation after the targeted token. The student then scores it via mean NLL

L_{t}

. (b) The band-pass rule favors intermediate surprisal relative to the two extremes.

	Across benchmarks				Avg. by epoch
Method	HMMT25	AIME24	AIME25	BRUMo25	E1	E2	E3
Base	46.7	76.7	66.7	73.3	65.9	65.9	65.9
SFT	46.7	80.0	63.3	76.7	65.0	66.7	66.7
GRPO	46.7	83.3	66.7	76.7	70.8	70.8	68.4
OPSD	43.3	76.7	66.7	76.7	63.4	68.4	65.9
OP 2 SD	46.6	80.0	70.0	76.7	69.2	68.3	68.3

Table 2: Overall performance. Pass@8 (%,

\uparrow

). The left block compares the final reported checkpoint; the right block shows the four-benchmark macro-average trajectory.

Configuration		Relative time
Sampled-token PG	24	$1.000\times$
PG + continuation	27	$1.125\times$
Top-16 forward KL	30	$1.250\times$
Top-32 forward KL	32	$1.333\times$

Table 3: Empirical test on training time

Figure 5: Ablation study. Continuation and PI + curriculum add one adaptation to OPSD; Full AOPSD combines both. The DAG variants change context under the OPSD objective. Table gives all scores.

Figure 6: Hyperparameter analysis. Each panel varies one factor around

(\delta,k,\beta_{+},\beta_{-})=(2,8,1,2.5)

, and Full benchmark scores appear in Table .

On-Policy Self-Distillation

On-policy self-distillation trains a model on its own generated trajectories using a teacher view to provide dense supervision.

See all 156 papers

E $^2$ -OPSD: Taming Entropy Overshoot in On-Policy Self-Distillation
Oct 4, 2026Yifei Liu, Minghao Fang, Xinyu Gu +6On-Policy Self-Distillation Mathematical Reasoning
The Chinese University of Hong Kong · Zhejiang University · Shanghai Artificial Intelligence Laboratory +5
Teach Yourself Where to Look: On-Policy Attention Self-Distillation for Reasoning
Sep 27, 2026Safaeid Hossain Arib, Rabeya Akter, Ismam Nur Swapnil +3Numerical Reasoning in Language Models Attention-Guided Distillation
ACI PLC, Bangladesh · University of Dhaka, Bangladesh · BRAC University, Bangladesh +2
Privileged Solutions or Context-Induced Teacher Behavior? Dissecting On-Policy Self-Distillation
Aug 10, 2026Yuki Ichihara, Naoto Iwase, Mohammad Atif Quamar +1On-Policy Self-Distillation Language Model Distillation
Mohamed bin Zayed University of Artificial Intelligence · Nagoya University · RIKEN AIP

Oct 4, 2026J/K move · Enter open · S save

E $^2$ -OPSD: Taming Entropy Overshoot in On-Policy Self-Distillation

Yifei Liu, Minghao Fang, Xinyu Gu, Chengkai Yao+5

The Chinese University of Hong Kong · Zhejiang University · Shanghai Artificial Intelligence Laboratory+5

On-Policy Self-Distillation Mathematical Reasoning Uncertainty-Aware Knowledge Distillation Knowledge Distillation

Open paper AskPDF arXiv

On-policy self-distillation (OPSD) provides dense token-level supervision without a second model: one network acts as teacher with the reference solution and as student with only the problem. We identify a specific failure mode of this recipe. During training, student token entropy rises past the teacher's and remains elevated, a pattern we call entropy overshoot. We trace it to both sides of distillation. The reference-conditioned teacher is confident along its answer-directed reasoning path, but this confidence transfers poorly to student-generated prefixes, making its supervision overly tied to answer-specific cues rather than reusable reasoning patterns; meanwhile, the forward KL used by OPSD continually diffuses the student's predictive distribution without pulling it back. We introduce E

^2

-OPSD to address both causes. Exemplar-guided teaching replaces the current answer with a retrieved solved neighboring problem, providing transferable reasoning guidance without revealing the destination and better matching student-reachable states. Entropy-aware distillation uses the student-teacher entropy gap to determine the direction and strength of each token's correction. E

^2

-OPSD improves math reasoning by up to 4.3 points in mean@16 over OPSD, while out-of-domain evaluations show gains over the corresponding base models of up to 4.9 points in mean@16 and 5.5 points in pass@8. Despite these gains, E

^2

-OPSD remains simple, requiring no additional forward passes or networks. Code repo: https://github.com/orionfei/E2-OPSD.

Abstract

On-Policy Self-Distillation

Explore similar work

E2^22-OPSD: Taming Entropy Overshoot in On-Policy Self-Distillation

Teach Yourself Where to Look: On-Policy Attention Self-Distillation for Reasoning

Privileged Solutions or Context-Induced Teacher Behavior? Dissecting On-Policy Self-Distillation

E $^2$ -OPSD: Taming Entropy Overshoot in On-Policy Self-Distillation