E$^2$-OPSD: Taming Entropy Overshoot in On-Policy Self-Distillation

Yifei Liu; Minghao Fang; Xinyu Gu; Chengkai Yao; Mengdi Liu; Tengfei Ma; Jiangbin Zheng; Chang Yu

cs.LGOct 4, 2026

E $^2$ -OPSD: Taming Entropy Overshoot in On-Policy Self-Distillation

Authors: Yifei Liu, Minghao Fang, Xinyu Gu, Chengkai Yao, Mengdi Liu, Tengfei Ma, Jiangbin Zheng, Chang Yu, +1 more

Organizations: The Chinese University of Hong Kong · Zhejiang University · Shanghai Artificial Intelligence Laboratory · Shanghai Jiaotong University · Visual Information Processing and Learning, ICT, CAS · Hunan University · Westlake University · Nanjing University

Abstract

On-policy self-distillation (OPSD) provides dense token-level supervision without a second model: one network acts as teacher with the reference solution and as student with only the problem. We identify a specific failure mode of this recipe. During training, student token entropy rises past the teacher's and remains elevated, a pattern we call entropy overshoot. We trace it to both sides of distillation. The reference-conditioned teacher is confident along its answer-directed reasoning path, but this confidence transfers poorly to student-generated prefixes, making its supervision overly tied to answer-specific cues rather than reusable reasoning patterns; meanwhile, the forward KL used by OPSD continually diffuses the student's predictive distribution without pulling it back. We introduce E $^2$ -OPSD to address both causes. Exemplar-guided teaching replaces the current answer with a retrieved solved neighboring problem, providing transferable reasoning guidance without revealing the destination and better matching student-reachable states. Entropy-aware distillation uses the student-teacher entropy gap to determine the direction and strength of each token's correction. E $^2$ -OPSD improves math reasoning by up to 4.3 points in mean@16 over OPSD, while out-of-domain evaluations show gains over the corresponding base models of up to 4.9 points in mean@16 and 5.5 points in pass@8. Despite these gains, E $^2$ -OPSD remains simple, requiring no additional forward passes or networks.

Figures & tables

Figure 1: Left: Entropy overshoot in Qwen3-1.7B. Right: The reference-conditioned teacher is less certain on student prefixes, suggesting a reasoning-path mismatch.

Panel of Figure 2 — Figure 2: (a) Exemplar-Guided Teaching. Standard OPSD gives the teacher the current reference solution, letting it jump from the student prefix toward the answer. E 2 -OPSD instead uses a solved neighbor as a strategy guide, forcing the teacher to follow the student’s intermediate step and solve the current problem itself, turning answer leakage into strategy transfer. (b) Forward and reverse KL play complementary roles. Forward KL promotes exploration over teacher-supported alternatives, while reverse KL removes unsupported probability mass to concentrate the policy. Entropy-aware distillation switches between them.

		Math Reasoning (mean@16)
Model	Method	AMC23	AIME24	AIME25	HMMT25	Avg.
Qwen3-0.6B (Thinking)	Base	$45.8$	$9.8$	$14.2$	$8.3$	$19.5$
	OPSD	$45.9$	$\underline{10.9}$	$14.2$	$\mathbf{9.2}$	$20.0$
	SDPO	$45.5$	$9.4$	$14.6$	$7.8$	$19.3$
	GRPO	$46.1$	$9.8$	$\mathbf{16.4}$	$8.4$	$\underline{20.2}$
	RLSD	$\mathbf{46.8}$	$\mathbf{11.0}$	$15.1$	$7.8$	$\underline{20.2}$

Table 1: Main results across model scales and types on math reasoning, reported as mean@16 over three seeds. Green subscripts show our gains over Base, computed from the displayed values. Best values are bold and second-best distinct values underlined. Full evaluation results are in Appendix B .

		Logic: ZebraLogic		Coding: HumanEval
Model	Method	mean@16	pass@8	mean@16	pass@8
	Base	$28.8$	$56.5$	$52.6$	$78.4$
Qwen3-0.6B (Thinking)	E 2 -OPSD	$\mathbf{29.0}$ +0.2	$56.5$ +0.0	$\mathbf{53.1}$ +0.5	$\mathbf{79.0}$ +0.6
	Base	$62.5$	$78.3$	$83.8$	$93.2$
Qwen3-1.7B (Thinking)	E 2 -OPSD	$\mathbf{67.4}$ +4.9	$\mathbf{83.8}$ +5.5	$\mathbf{84.5}$ +0.7	$\mathbf{93.8}$ +0.6
	Base	$79.1$	$92.4$	$84.0$	$89.9$

Table 2: Out-of-domain evaluation on logical reasoning and code generation. E 2 -OPSD consistently improves both mean@16 and pass@8 across ZebraLogic and HumanEval, showing that its gains extend beyond in-domain mathematical reasoning.

Panel of Figure 4 — Figure 4: Training dynamics. (a,b) Student and teacher token entropy under OPSD (dashed) and E 2 -OPSD (solid); the dotted line marks the first student–teacher entropy crossing under OPSD. (c) On Qwen3-1.7B Thinking, routing share denotes the fraction of response tokens assigned to forward or reverse KL by BER. The routing naturally shifts from forward-dominated exploration toward increased reverse-KL concentration as training proceeds.

Panel of Figure 5 — Figure 5: Component ablation and mechanism probes. (a) Performance contribution of each design component; “+” denotes the component added on top of OPSD. (b) EGT improves teacher–student compatibility. The x-axis denotes teacher-to-student forward KL evaluated on student prefixes; the y-axis shows response length from each setting’s own rollouts (log scale). (c) AGD sharpens entropy-aware routing. HL denotes high-student/low-teacher entropy and LH the reverse.

Figure 6: State-dependent KL routing avoids fixed directional bias. KL-direction dynamics on Qwen3-1.7B Thinking with the teacher context fixed and AGD disabled. The x-axis shows optimizer steps; the left y-axis reports student and teacher token entropy, while the right y-axis of panel (d) reports validation mean@8. Panels compare (a) forward KL, (b) reverse KL, (c) an equal-weight forward/reverse KL mixture, and (d) BER. The trajectories reveal distinct distributional dynamics under fixed versus state-dependent KL routing.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Backbone	Reasoning mode	Difficulty levels
Qwen3-0.6B	Thinking	1–3
Qwen3-1.7B	Thinking	5–7
Qwen3-4B-Instruct-2507	Instruct	6–8
Olmo-3-7B-Instruct	Instruct	7–10

Appendix

Table 3: Backbone configurations and DeepMath training difficulty bands.

Parameter	Value
Optimizer	AdamW
Learning rate	$1\times 10^{-6}$
Warmup	100 steps
Weight decay	$0.01$
Gradient-norm clipping	$0.1$
Prompts per rollout round	256

Appendix

Table 4: Shared training parameters.

Group	Parameter	Value
Evaluation	Responses per problem	16
	Evaluation seeds	17, 42, 1234
	Temperature	$0.6$
	Top- $p$ / top- $k$	$0.95$ / 20
	Maximum prompt length	2,048 tokens
Maximum response length	Math reasoning	38,912 tokens

Appendix

Table 5: Evaluation parameters. Batch sizes count prompts.

Method	Setting	Value	Description
OPSD	Teacher	Frozen initial weights	Current reference solution as context.
	Objective	Forward KL	Teacher-to-student distillation.
	Support / clipping	Top-100 + tail; $0.05$	Teacher-selected support; per-element cap.
SDPO	Teacher / EMA decay	EMA; $0.95$	Updated after each optimizer step.
	Teacher context	Successful sibling	Excludes the rollout being scored.
	JSD mixing $\alpha$	$0.5$	Equal student–teacher mixture.

Appendix

Table 6: Method-specific objectives and training settings.

		Math Reasoning (mean@16)
Model	Method	AMC23	AIME24	AIME25	HMMT25	Avg.
Qwen3-0.6B (Thinking)	Base	$45.81{\pm}0.98$	$9.79{\pm}0.29$	$14.17{\pm}1.12$	$8.26{\pm}0.43$	$19.51{\pm}0.19$
	OPSD	$45.88{\pm}0.86$	$10.90{\pm}0.86$	$14.17{\pm}0.59$	$\mathbf{9.17}{\pm}0.51$	$20.03{\pm}0.49$
	SDPO	$45.48{\pm}0.27$	$9.38{\pm}0.34$	$14.58{\pm}1.39$	$7.78{\pm}1.70$	$19.30{\pm}0.62$
	GRPO	$46.11{\pm}0.98$	$9.79{\pm}0.17$	$\mathbf{16.39}{\pm}0.26$	$8.40{\pm}0.77$	$20.17{\pm}0.03$
	RLSD	$\mathbf{46.84}{\pm}1.95$	$\underline{10.97}{\pm}0.49$	$15.14{\pm}0.64$	$7.78{\pm}0.26$	$\underline{20.18}{\pm}0.53$

Appendix

Table 7: Mathematical reasoning mean@16 (%). Avg. equally weights the four benchmarks.

		Math Reasoning (pass@2)
Model	Method	AMC23	AIME24	AIME25	HMMT25	Avg.
Qwen3-0.6B (Thinking)	Base	$56.41{\pm}0.97$	$14.05{\pm}1.10$	$19.74{\pm}1.63$	$11.48{\pm}0.17$	$25.42{\pm}0.35$
	OPSD	$56.63{\pm}1.46$	$16.39{\pm}1.23$	$19.95{\pm}1.05$	$\mathbf{12.79}{\pm}0.80$	$26.44{\pm}0.76$
	SDPO	$56.11{\pm}0.10$	$14.44{\pm}0.80$	$20.19{\pm}1.88$	$10.79{\pm}2.17$	$25.38{\pm}0.70$
	GRPO	$56.75{\pm}0.96$	$14.69{\pm}0.93$	$\mathbf{22.94}{\pm}0.16$	$11.56{\pm}1.07$	$26.49{\pm}0.22$
	RLSD	$\mathbf{57.57}{\pm}2.43$	$\underline{16.79}{\pm}0.73$	$21.13{\pm}0.36$	$10.72{\pm}0.40$	$\underline{26.55}{\pm}0.39$

Appendix

Table 8: Mathematical reasoning unbiased pass@2 (%).

		Math Reasoning (pass@4)
Model	Method	AMC23	AIME24	AIME25	HMMT25	Avg.
Qwen3-0.6B (Thinking)	Base	$66.30{\pm}0.67$	$19.70{\pm}2.15$	$24.67{\pm}1.98$	$15.02{\pm}0.31$	$31.42{\pm}0.71$
	OPSD	$66.28{\pm}1.94$	$23.39{\pm}2.33$	$25.01{\pm}1.61$	$\mathbf{16.84}{\pm}1.19$	$32.88{\pm}1.12$
	SDPO	$65.01{\pm}0.57$	$21.34{\pm}1.46$	$25.35{\pm}2.85$	$14.25{\pm}2.30$	$31.49{\pm}0.91$
	GRPO	$\underline{66.38}{\pm}0.94$	$20.94{\pm}2.31$	$\mathbf{29.06}{\pm}0.48$	$14.94{\pm}1.58$	$32.83{\pm}0.58$
	RLSD	$\mathbf{66.82}{\pm}2.26$	$\underline{23.79}{\pm}1.18$	$27.00{\pm}0.90$	$14.42{\pm}0.77$	$\underline{33.01}{\pm}0.01$

Appendix

Table 9: Mathematical reasoning unbiased pass@4 (%).

Model	Method	mean@16	pass@2	pass@4	pass@8
Logic: ZebraLogic
	Base	$28.83{\pm}0.30$	$38.93{\pm}0.35$	$48.54{\pm}0.23$	$56.47{\pm}0.63$
Qwen3-0.6B (Thinking)	E 2 -OPSD (Ours)	$\mathbf{28.96}{\pm}0.21$	$\mathbf{39.27}{\pm}0.32$	$\mathbf{48.86}{\pm}0.53$	$\mathbf{56.54}{\pm}0.71$
	Base	$62.54{\pm}1.05$	$69.14{\pm}0.99$	$74.19{\pm}1.21$	$78.26{\pm}1.32$
Qwen3-1.7B (Thinking)	E 2 -OPSD (Ours)	$\mathbf{67.35}{\pm}0.43$	$\mathbf{73.57}{\pm}0.62$	$\mathbf{78.75}{\pm}0.84$	$\mathbf{83.77}{\pm}0.98$
	Base	$79.15{\pm}0.16$	$85.65{\pm}0.23$	$89.61{\pm}0.41$	$92.35{\pm}0.64$

Appendix

Table 10: Out-of-domain evaluation on logical reasoning and code generation. ZebraLogic uses whole-puzzle correctness; best means are bold within each model and dataset.

Method	AMC23	AIME24	AIME25	HMMT25	Avg.
Base	$75.50$ $\pm 0.28$	$46.04$ $\pm 1.78$	$34.58$ $\pm 0.29$	$21.88$ $\pm 0.78$	$44.50$ $\pm 0.27$
OPSD	$74.87$ $\pm 0.46$	$45.97$ $\pm 1.58$	$35.00$ $\pm 1.36$	$22.71$ $\pm 0.61$	$44.64$ $\pm 0.41$
+ EGT	$76.46$ $\pm 0.99$	$48.26$ $\pm 1.21$	$36.81$ $\pm 1.49$	$22.57$ $\pm 0.20$	$46.02$ $\pm 0.40$
+ BER	$77.26$ $\pm 0.38$	$52.01$ $\pm 1.13$	$37.15$ $\pm 0.94$	$24.03$ $\pm 0.35$	$47.61$ $\pm 0.64$
+ BER + AGD	$\underline{77.66}$ $\pm 0.68$	$\underline{52.15}$ $\pm 1.58$	$\mathbf{39.44}$ $\pm 0.69$	$\underline{24.31}$ $\pm 1.83$	$\underline{48.39}$ $\pm 0.55$
E 2 -OPSD (Ours)	$\mathbf{79.02}$ $\pm 0.41$	$\mathbf{52.29}$ $\pm 0.45$	$\underline{39.24}$ $\pm 0.26$	$\mathbf{25.00}$ $\pm 0.90$	$\mathbf{48.89}$ $\pm 0.32$

Appendix

Table 11: Per-benchmark mean@16 with standard deviations for the component ablation on Qwen3-1.7B Thinking (Figure 5 (a)). Avg. equally weights the four benchmarks. Best means are bold and second-best distinct means underlined. AGD denotes adaptive gap damping.

Parameter	Value
Problems / responses per problem	256 / 8
Sampling temperature	1.0
Top- $p$ / top- $k$	0.95 / 20
Maximum response length	16,384 tokens
Maximum total context length	40,960 tokens
Student prompt length limit	2,048 tokens

Appendix

Table 12: Diagnostic rollout settings. Indices

i

and

g

start at zero. The two teacher conditions share sampling seeds. Teacher generation budgets are also limited by the remaining context window.

Metric	Student	Reference teacher	EGT teacher
Mean response length (tokens)	9,139.7	835.6	2,089.5
Own-prefix entropy (nats)	0.2679	0.1900	0.2605
Forward KL to student (nats)	0	0.1549	0.1148

Appendix

Table 13: Teacher-context probe underlying Figure 5 (b). Lengths use 2,048 responses per condition; entropy and KL use 2,047 matched slots. Student zeros denote identical-prefix or self comparisons.

Bucket	Token share (%)	AGD off (%)	AGD on (%)	Change (pp)
HH	14.40	45.57	43.26	$-2.31$
HL	2.33	8.99	12.42	$+3.43$
LH	5.34	11.19	16.08	$+4.89$
LL	77.93	34.25	28.24	$-6.01$

Appendix

Table 14: AGD gradient probe on 2,048 fixed student responses. Token shares pool all response tokens; gradient shares use the response- and problem-level normalization described in Appendix C.1 . Changes are computed from the displayed gradient shares.

Subset	Positions	Mean overlap	$O_{t}<0.5$ (%)
All strongly damped positions	12,708,962	0.9848	0.380
with $H(p_{t})\geq\ln 2$	766,682	0.8343	3.662

Appendix

Table 15: Overlap at strongly damped positions. Percentages are computed within each row.

Explore similar work

Sep 27, 2026cs.LG

Teach Yourself Where to Look: On-Policy Attention Self-Distillation for Reasoning

On-policy self-distillation trains reasoning models on their own trajectories using dense token distribution guidance from a privileged teacher with access to a verified solution. This supervision transfers what the teacher predicts without directly transferring where it attends within the preceding context. We introduce On-Policy Attention Self-Distillation (OPASD), which complements token-level supervision with solution-conditioned attention distillation. Because the privileged teacher can attend to verified solution tokens unavailable to the student, OPASD projects teacher attention onto student-visible positions and renormalizes the resulting distribution before alignment. Across three model sizes and four competition-level mathematics benchmarks, OPASD consistently outperforms token-only OPSD, improving average accuracy by 4.98 to 8.40 percentage points. OPASD also avoids the response-length inflation and performance degradation observed with token-only distillation, reducing generated rollout tokens by 73.9% and estimated model compute by 72.6% while training 1.53x faster. These results show that solution-conditioned attention provides a complementary supervision signal that makes on-policy self-distillation more accurate, stable, and compute-efficient.

Safaeid Hossain Arib, Rabeya Akter, Ismam Nur Swapnil +3

ACI PLC, Bangladesh · University of Dhaka, Bangladesh · BRAC University, Bangladesh +2

Unsupervised On-Policy Self-Distillation Efficient On-Policy Distillation On-Policy

Ask PDF

Aug 10, 2026cs.LG

Privileged Solutions or Context-Induced Teacher Behavior? Dissecting On-Policy Self-Distillation

On-Policy Self-Distillation (OPSD) is commonly interpreted as the transfer of privileged information: a teacher observes the verified solution to the target problem and supervises the student's trajectory. However, this interpretation conflates two effects. The reference solution not only reveals the answer to the current instance but also changes the context under which the teacher provides token-level supervision. We investigate the role of target-specific privilege with

\mathrm{OP}^{2}\mathrm{SD}

(On-Policy Self-Distillation from Other Problems), which replaces the paired reference with a problem and solution from a different example, while preserving the student rollout, teacher, and distillation objective. Across three models and three mathematics benchmarks,

\mathrm{OP}^{2}\mathrm{SD}

improves over the base model, remains competitive with OPSD. The success of

\mathrm{OP}^{2}\mathrm{SD}

implies that OPSD gains do not necessarily come from access to the reference solution, and that the teacher's context-induced behavior is an important factor.

Yuki Ichihara, Naoto Iwase, Mohammad Atif Quamar +1

Mohamed bin Zayed University of Artificial Intelligence · Nagoya University · RIKEN AIP

Teacher Privileged Context Mathematical Reasoning Benchmarks Agent Behavior Modeling

Ask PDF

Oct 4, 2026cs.LG

Outcome-Guided On-Policy Self-Distillation

On-policy self-distillation (OPSD) provides denser token-level supervision and better computational efficiency than Reinforcement Learning with Verifiable Rewards (RLVR). However, this denser supervision may introduce substantial noise and training instability. Existing improvements often rely on high-variance per-token statistics and introduce extra hyperparameters and trade-offs. Based on the advantage formulation in RLVR, we analyze the OPSD objective from the same perspective, incorporating outcome correctness signals. We find that vanilla OPSD imposes insufficient penalties and excessive rewards on incorrect trajectories because it applies a fixed divergence objective regardless of outcome correctness. Furthermore, the reliability of teacher supervision is associated with both trajectory outcome and the cumulative average teacher entropy along the rollout. Based on these observations, we propose Outcome-Guided On-Policy Self-Distillation (OG-OPSD), which dynamically adapts both the divergence objective and distillation position according to binary outcome rewards and the cumulative average teacher entropy. Extensive experiments show that OG-OPSD consistently improves the performance of vanilla OPSD and multiple strong baselines in mathematical reasoning, multimodal reasoning, and out-of-distribution tasks across Qwen3 models at 1.7B, 4B, and 8B scales, as well as Qwen3-VL-2B.

ZheXu Wang, Mao-Lin Luo, Yankun Hong +6

School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China · Huawei Noah’s Ark Lab +2

Unsupervised On-Policy Self-Distillation

Ask PDF