On-policy self-distillation (OPSD) trains a question-only student with token-level feedback from a teacher given training-only privileged information (PI). OPSD therefore provides dense, on-policy supervision, and is free of a larger external teacher, but its effectiveness rests on how PI is designed and utilized. Our preliminary diagnostics suggest a significant gap between teacher utility and student learnability, where a small fraction of high-disagreement tokens dominate the distillation signal, and short teacher continuations at these positions further expose more explicit PI leakage than transferable correction cues, indicating a strong intent on injecting PI-conditioned shortcuts. We propose Adaptive On-Policy Self-Distillation (AOPSD), which adapts what information the teacher receives and how strongly its feedback influences learning. AOPSD encodes each solution as a reasoning DAG, orders problems by the student's evolving capability, and reveals only the affordable subgraph and its next frontier as PI. For high-disagreement tokens, AOPSD utilizes short teacher continuations as probes to encourage useful guidance while mitigating PI-conditioned shortcuts among teacher supervisions. On HMMT25, AIME24, AIME25, and BRUMo25, AOPSD achieves 72.5% Pass@8, which is 6.7 percentage points above OPSD and 4.2 above the strongest competing baseline while reducing 15 percentage points of training time at lower cost.
Figures & tables
Figure 1: Preliminary diagnostics of PI utility and transfer. (a) Change relative to the baseline model: every PI makes the privileged teacher substantially stronger (amber) while the resulting question-only student gets weaker (blue). (b) Token share and share of total absolute sampled-token log gap, grouped by ∣gt∣ .
k
Correction cues
Explicit PI references
(tokens)
Count
Representative excerpt
Count
Representative excerpt
2
3
an error
8
original DAG
4
7
Not quite—
70
checking the private reasoning DAG
6
22
the density argument must be flawed
153
go back to the private reasoning DAG
8
31
Or maybe we made a mistake?
238
This matches the reasoning in the DAG
16
64
Wait—not necessarily.
507
earlier in the DAG, it says …
Table 1: A short continuation makes high-disagreement intent legible. Cumulative counts of recognizable correction cues and explicit PI leakage within a total continuation of k tokens.
Figure 2: AOPSD : adapt what to reveal and how strongly to learn. (a) Teacher and student share an initialization, and the teacher scores the student’s question-only trajectory using privileged information. (b) Based on the student’s current capability, the DAG is adaptively disclosed, and all questions are presented through curriculum learning in order of difficulty. (c) High-gap tokens trigger teacher continuations scored by the question-only student. Band-pass weighting suppresses redundant and potentially PI-dependent guidance while retaining the learnable and useful guidance.
Figure 3: A reasoning DAG reveals one complete dependency frontier at a time. (a) A verified route is compressed into route-defining mathematical checkpoints; edges encode immediate prerequisites. (b) For a question-only rollout, reached states R are blue, immediately enabled states F(R) are orange, and later states are gray. As more prerequisites are established, disclosure advances by one dependency layer rather than by an arbitrary text prefix.
Figure 4: Continuation surprisal controls the token’s weight. (a) For a large gap, the PI-conditioned teacher generates a continuation after the targeted token. The student then scores it via mean NLL Lt . (b) The band-pass rule favors intermediate surprisal relative to the two extremes.
Across benchmarks
Avg. by epoch
Method
HMMT25
AIME24
AIME25
BRUMo25
E1
E2
E3
Base
46.7
76.7
66.7
73.3
65.9
65.9
65.9
SFT
46.7
80.0
63.3
76.7
65.0
66.7
66.7
GRPO
46.7
83.3
66.7
76.7
70.8
70.8
68.4
OPSD
43.3
76.7
66.7
76.7
63.4
68.4
65.9
OP 2 SD
46.6
80.0
70.0
76.7
69.2
68.3
68.3
Table 2: Overall performance. Pass@8 (%, ↑ ). The left block compares the final reported checkpoint; the right block shows the four-benchmark macro-average trajectory.
Configuration
Time ↓ (min/epoch)
Relative time
Sampled-token PG
24
1.000×
PG + continuation
27
1.125×
Top-16 forward KL
30
1.250×
Top-32 forward KL
32
1.333×
Table 3: Empirical test on training time
Figure 5: Ablation study. Continuation and PI + curriculum add one adaptation to OPSD; Full AOPSD combines both. The DAG variants change context under the OPSD objective. Table gives all scores.
Figure 6: Hyperparameter analysis. Each panel varies one factor around (δ,k,β+,β−)=(2,8,1,2.5) , and Full benchmark scores appear in Table .
On-policy self-distillation (OPSD) provides dense token-level supervision without a second model: one network acts as teacher with the reference solution and as student with only the problem. We identify a specific failure mode of this recipe. During training, student token entropy rises past the teacher's and remains elevated, a pattern we call entropy overshoot. We trace it to both sides of distillation. The reference-conditioned teacher is confident along its answer-directed reasoning path, but this confidence transfers poorly to student-generated prefixes, making its supervision overly tied to answer-specific cues rather than reusable reasoning patterns; meanwhile, the forward KL used by OPSD continually diffuses the student's predictive distribution without pulling it back. We introduce E2-OPSD to address both causes. Exemplar-guided teaching replaces the current answer with a retrieved solved neighboring problem, providing transferable reasoning guidance without revealing the destination and better matching student-reachable states. Entropy-aware distillation uses the student-teacher entropy gap to determine the direction and strength of each token's correction. E2-OPSD improves math reasoning by up to 4.3 points in mean@16 over OPSD, while out-of-domain evaluations show gains over the corresponding base models of up to 4.9 points in mean@16 and 5.5 points in pass@8. Despite these gains, E2-OPSD remains simple, requiring no additional forward passes or networks. Code repo: https://github.com/orionfei/E2-OPSD.
Yifei Liu, Minghao Fang, Xinyu Gu +6
The Chinese University of Hong Kong · Zhejiang University · Shanghai Artificial Intelligence Laboratory +5
On-policy self-distillation trains reasoning models on their own trajectories using dense token distribution guidance from a privileged teacher with access to a verified solution. This supervision transfers what the teacher predicts without directly transferring where it attends within the preceding context. We introduce On-Policy Attention Self-Distillation (OPASD), which complements token-level supervision with solution-conditioned attention distillation. Because the privileged teacher can attend to verified solution tokens unavailable to the student, OPASD projects teacher attention onto student-visible positions and renormalizes the resulting distribution before alignment. Across three model sizes and four competition-level mathematics benchmarks, OPASD consistently outperforms token-only OPSD, improving average accuracy by 4.98 to 8.40 percentage points. OPASD also avoids the response-length inflation and performance degradation observed with token-only distillation, reducing generated rollout tokens by 73.9% and estimated model compute by 72.6% while training 1.53x faster. These results show that solution-conditioned attention provides a complementary supervision signal that makes on-policy self-distillation more accurate, stable, and compute-efficient.
Safaeid Hossain Arib, Rabeya Akter, Ismam Nur Swapnil +3
ACI PLC, Bangladesh · University of Dhaka, Bangladesh · BRAC University, Bangladesh +2
On-Policy Self-Distillation (OPSD) is commonly interpreted as the transfer of privileged information: a teacher observes the verified solution to the target problem and supervises the student's trajectory. However, this interpretation conflates two effects. The reference solution not only reveals the answer to the current instance but also changes the context under which the teacher provides token-level supervision. We investigate the role of target-specific privilege with OP2SD (On-Policy Self-Distillation from Other Problems), which replaces the paired reference with a problem and solution from a different example, while preserving the student rollout, teacher, and distillation objective. Across three models and three mathematics benchmarks, OP2SD improves over the base model, remains competitive with OPSD. The success of OP2SD implies that OPSD gains do not necessarily come from access to the reference solution, and that the teacher's context-induced behavior is an important factor.
Yuki Ichihara, Naoto Iwase, Mohammad Atif Quamar +1
Mohamed bin Zayed University of Artificial Intelligence · Nagoya University · RIKEN AIP