Diagnosing On-Policy Self-Distillation for Reasoning Language Models

Yang Li; Gongle Xue; Yuheng Yuan; Yijia Guo; Shizhe Zhang; Liwen Hu; Lei Ma

cs.CLSep 30, 2026

Diagnosing On-Policy Self-Distillation for Reasoning Language Models

Authors: Yang Li, Gongle Xue, Yuheng Yuan, Yijia Guo, Shizhe Zhang, Liwen Hu, Lei Ma

Organizations: Peking University

Abstract

On-policy self-distillation (OPSD) has attracted growing interest as a promising approach to improve the reasoning ability of language models. Without external rewards nor a separate stronger teacher, the self-teacher with privileged information could provide dense signals on student's trajectories. However, its behavior in language reasoning remains unclear, with reported outcomes ranging from modest gains to behavioral collapse. In this work, we diagnose OPSD for mathematical reasoning across models spanning 0.6B--8B parameters. We conduct controlled experiments and token-level analyses to fully delve into OPSD. We point out that teacher's signal is shaped by reasoning-mode alignment and the complete teacher prefix, rather than by privileged semantics alone. OPSD improves reasoning only in narrow compatibility regimes. Otherwise, it produces ineffective length growth, stable degradation, or behavioral collapse. Token-level analysis shows that teacher's signal is not stable and does not predict downstream performance. Based on these results, we argue that OPSD is a sensitive algorithm rather than a generally reliable reasoning-improvement post-training method.

Figures & tables

Figure 1: Overview of our diagnosis. We design a comprehensive diagnostic pipeline to study how OPSD behaves in different settings and what can token-level dynamics reveal during training.

Model	S	T	Eval	Avg@8			Pass@8			Length
Model	S	T	Eval	Base	OPSD	$\Delta$	Base	OPSD	$\Delta$	Base	OPSD	$\Delta$
Qwen3 mean	Th	Th	Th	53.5	50.7	$-2.8$	72.2	71.1	$-1.1$	18,013	21,250	$+17.9\%$
	Th	NTh	Th	53.5	36.6	$-16.9$	72.2	59.5	$-12.7$	18,013	13,291	$-26.0\%$
	NTh	Th	NTh	15.8	19.6	$+3.8$	32.2	39.7	$+7.5$	5,510	14,093	$+154.8\%$
	NTh	NTh	NTh	15.8	15.2	$-0.6$	32.2	28.6	$-3.6$	5,510	13,634	$+144.3\%$
	NTh	Th	Th	53.5	56.2	$+2.7$	72.2	73.1	$+0.9$	18,013	19,527	$+8.4\%$

Table 1: Controls for establishing a valid comparison setting. (a) Reasoning-mode results averaged over Qwen3-1.7B/4B/8B. For Avg@8 and Pass@8,

\Delta

is the absolute OPSD–Base change in percentage points. For Length, it is the relative change. Eval is the inference mode of both Base and OPSD, and S/T denote student/teacher modes. (b) Qwen3-4B feature rates. Self-correction includes Hmm/Actually/No/But. (c) No-privilege same-prefix control for mismatched NTh / Th , evaluated in Th mode. Parentheses show change from the corresponding full-solution checkpoint. Per-model results are in Table 14 . Per-dataset results are in Table 49 .

Figure 2: Stability over training steps. Per-dataset results are in Table 52 .

Positions	Avg@8	Pass@8	Length
First-256	49.5 (+0.7)	71.3 (+0.8)	21.4k (+14.7%)
Uni-256	47.1 (-1.7)	72.1 (+1.6)	22.0k (+17.9%)
Final-256	47.8 (-1.0)	69.2 (-1.3)	22.4k (+20.1%)

Table 2: Results of position loss. Parentheses: change from Base. Per-dataset results are in Table 51 .

Figure 3: OPSD vs. rollout cap for four models. Solid: OPSD. Dashed: Base. Left axes: Avg@8/Pass@8. Right: mean eval output length. Per-dataset results are in Table 50 and Table 51 .

Teacher context	Rollout length: 256			Rollout length: 1,024
Teacher context	Avg@8	Pass@8	Length	Avg@8	Pass@8	Length
Full solution	50.5 ( $+0.9$ )	71.5 ( $+1.0$ )	16.2k ( $+12.9\%$ )	48.6 ( $-1.0$ )	69.3 ( $-1.2$ )	16.4k ( $+12.2\%$ )
Unrelated solution	48.9 ( $-0.7$ )	69.7 ( $-0.8$ )	15.4k ( $+6.0\%$ )	47.8 ( $-1.8$ )	67.3 ( $-3.2$ )	18.0k ( $+25.3\%$ )
Answer only	45.9 ( $-3.7$ )	65.7 ( $-4.8$ )	16.3k ( $+6.4\%$ )	44.3 ( $-5.3$ )	66.3 ( $-4.2$ )	17.4k ( $+12.9\%$ )

Table 3: Macro-average effect of privileged information across five model settings. Parentheses report change from Base in percentage points for Avg@8 and Pass@8 and relative percent for Length. Per-model and per-dataset results are in Table 53 .

	Qwen3-1.7B			OLMo-3-7B-Think
Teacher context	Avg@8	Pass@8	Length	Avg@8	Pass@8	Length
None (Base)	34.0	59.2	18.6k	63.5	81.7	18.7k
Answer only	35.6	59.2	25.8k	56.1	75.8	24.1k
Unrelated solution	37.1	60.0	22.0k	57.5	76.7	23.4k
Full solution	35.3	60.8	22.4k	58.8	77.5	21.7k
CoT + solution	30.8	55.8	18.6k	58.6	77.5	20.8k

Table 4: Results at a 1,024-token rollout cap. Per-dataset results are in Table 53 .

Figure 4: OPSD results across baseline performance levels. Each model cluster reports Base, OPSD-256, and OPSD-1024. Dark foreground bars show Avg@8. Light background bars show Pass@8. Models are ordered by Base Pass@8. Per-dataset results are in Table 58 .

	Fwd. KL		JSD		Rev. KL
Model	Avg	Pass	Avg	Pass	Avg	Pass
Qwen3-1.7B	$+1.3$	$+1.6$	$-2.7$	$-4.2$	$-7.5$	$-11.7$
Qwen3-4B	$-4.4$	$-3.4$	$-5.7$	$-3.4$	$-30.3$	$-22.5$
Qwen3-4B-Thinking	$-9.6$	$-3.4$	$-7.4$	$-4.2$	$-1.2$	$-1.7$
OLMo-3-7B-Think	$-4.7$	$-4.2$	$-1.4$	$0.0$	$-1.1$	$+0.8$
OLMo-3-7B-Instruct	$+1.3$	$+1.7$	$-2.3$	$-2.5$	$+3.0$	$+0.9$

Table 5: Optimization sensitivity. (a) Avg@8 and Pass@8 changes after 100 steps of OPSD-1024 under forward KL, symmetric JSD, and reverse KL. Bold marks the best change for each model. Per-model and per-dataset results are in Table 16 and Table 55 . (b) Post-training pipeline results. Format is the boxed-answer rate (%) and Length is mean output length. Per-checkpoint results are in Table 22 . Per-dataset results are in Table 61 and Table 64 .

Model	Base	OPSD 50	GRPO 50
Qwen3-1.7B	34.0/59.2/–	38.0 / 63.3 /0.66	37.1/ 63.3 /114.27
Qwen3-4B	62.3/79.2/–	60.1/ 80.0 /1.13	62.8 /78.3/171.81

Table 6: Cells report Avg@8 / Pass@8 / GPU hours. Bold marks the best Avg@8 or Pass@8. Per-dataset results are in Table 62 and Table 63 .

S	T	Fwd. KL	$D>.05$	Agree.	Enc.	Disc.
NTh	NTh	0.100	17.8	94.3	32.3	37.9
NTh	Th	0.193	24.4	93.3	14.0	70.9
Th	NTh	0.468	42.9	87.5	15.3	63.0
Th	Th	0.144	20.6	92.9	34.5	36.0

Table 7: Qwen3-1.7B grid. S/T are student/teacher modes, and

D>.05

, Agree., Enc., and Disc. are percentages.

Panel of Figure 5 — Figure 5: Privileged-context effects on the initial teacher–student signal. (a) PCA of teacher’s signal on student rollout across teacher-context conditions. (b) Full-solution of OpenThoughts signal distributions across models with different downstream OPSD performances.

Metric	Inst.	Reason.
Mean Fwd. KL	0.151	0.139
Discouraged (%)	44.1	61.0
HE20 KL (%)	57.2	47.9
HE20 $\|$ adv. $\|$ (%)	70.5	64.6

Table 8: Model-type signal analysis. HE20: highest-entropy 20%.

Context	Fwd. KL	$D>.05$	Abs. adv.	Agree.
Answer only	0.016	2.9	0.039	98.0
Unrelated solution	0.054	14.9	0.100	95.0
Full solution	0.108	20.3	0.151	93.3
CoT + solution	0.179	30.8	0.224	90.6

Table 9: Privileged-context effects. (a) Ten-model OpenThoughts signal averages.

D>.05

, Abs. adv., and Agree. are the high-KL-position rate, mean absolute sampled-token log advantage, and top-1 agreement. (b) First-batch forward KL and Avg@8 under a 1,024-token rollout cap. Only teacher context and its prefix cap change. Full per-model signal results are split across Table 26 and Table 27 ; downstream per-dataset results are in Table 53 .

Model	KL First10	KL Final10	$\Delta$ Avg	$\Delta$ Pass	KL First10	KL Final10	$\Delta$ Avg	$\Delta$ Pass
	256-token rollout cap				1,024-token rollout cap
DS-Qwen-1.5B	0.062	$0.073\uparrow$	$+0.7$	$+0.9$	0.032	$0.040\uparrow$	$+0.6$	$-0.8$
Falcon-H1R-7B	0.112	$0.087\downarrow$	$-0.6$	$0.0$	0.077	$0.054\downarrow$	$-0.4$	$0.0$
Qwen3-0.6B	0.125	$0.334\uparrow$	$+0.2$	$+0.8$	0.073	$0.212\uparrow$	$-2.6$	$-6.7$
MiMo-7B-RL	0.175	$0.340\uparrow$	$-0.3$	$+0.8$	0.098	$0.243\uparrow$	$-11.3$	$-10.0$
OLMo-3-7B-Think	0.193	$0.377\uparrow$	$-2.7$	$-1.7$	0.108	$0.251\uparrow$	$-4.7$	$-4.2$

Table 10: Forward KL and downstream change under full-solution OPSD. KL First10 and Final10 average KL loss over the first and final ten steps.

\Delta

Avg and

\Delta

Pass are Avg@8 and Pass@8 changes from Base. DS-Qwen-1.5B refers to DeepSeek-R1-Distill-Qwen-1.5B.

Response position	$[0,128)$	$[128,256)$	$[256,512)$	$[512,1\mathrm{k})$	$[1\mathrm{k},2\mathrm{k})$	$[2\mathrm{k},4\mathrm{k})$	$[4\mathrm{k},6\mathrm{k})$
Mean/SNR	0.287 / 0.248	0.190 / 0.217	0.126 / 0.185	0.087 / 0.160	0.067 / 0.138	0.056 / 0.121	0.048 / 0.103

Table 11: Forward KL by response-position window. Each entry reports mean/SNR, where

\mathrm{SNR}=\mu/\sigma

and

\sigma

is the token-level standard deviation. Per-model results are in Table 29 .

Figure 6: OPSD training dynamics across (a) initialization, (b) Instruct/Think checkpoints, and (c) Qwen3 variants. Per-window results for the initialization comparison are in Table 31 .

Appendix figures & tables72 assets

Supplementary material from the paper’s appendix.

Appendix

Student model (teacher if different)	Engine	GPUs	Micro-batch	Accumulation	Global batch
Qwen3-0.6B	vLLM	2	16	4	128
Qwen3-1.7B	vLLM	2	8	4	64
Qwen3-4B / Qwen3-4B-Instruct	vLLM	2	4	4	32
Qwen3-4B-Instruct (Qwen3-4B)	vLLM	2	4	4	32
Qwen3-4B-Thinking	vLLM	2	4	4	32
Qwen3-8B	vLLM	4	4	2	32

Appendix

Table 12: Distributed configurations at the default 1,024-token rollout cap. Global batch equals micro-batch

\times

accumulation

\times

GPUs. The length sweep rebalances micro-batch and accumulation while preserving global batch.

Model and eval mode	Engine	Max new	Context	Temp.	Top- $p$	Top- $k$	Pres. pen.
Qwen3-0.6/1.7/4/8B, Th	vLLM	38,912	40,960	0.6	0.95	20	0.0
Qwen3-1.7/4/8B, NTh	vLLM	32,768	40,960	0.7	0.80	20	0.0
Qwen3-4B-Instruct, NTh	vLLM	32,768	40,960	0.7	0.80	20	0.0
Qwen3-4B-Thinking	vLLM	81,920	90,112	0.6	0.95	20	0.0
Qwen3.5-4B, Th or NTh	SGLang	81,920	131,072	1.0	0.95	20	1.5
DeepSeek-R1-Distill-Qwen-1.5B	vLLM	32,768	40,960	0.6	0.95	off	0.0

Appendix

Table 13: Inference settings. Max new and Context are token counts. † For MiMo-7B-RL, 32,768 is the nominal cap. The effective cap is the context window minus the rendered prompt length and one guard token.

Model	S	T	Eval	Avg@8			Pass@8			Length
Model	S	T	Eval	Base	OPSD	$\Delta$	Base	OPSD	$\Delta$	Base	OPSD	$\Delta$
Qwen3-1.7B	Th	Th	Th	34.0	35.3	$+1.3$	59.2	60.8	$+1.6$	18,624	22,380	$+20.2\%$
Qwen3-1.7B	Th	NTh	Th	34.0	19.2	$-14.8$	59.2	39.2	$-20.0$	18,624	11,109	$-40.4\%$
Qwen3-1.7B	NTh	Th	NTh	9.8	14.1	$+4.3$	20.8	28.3	$+7.5$	3,923	9,731	$+148.1\%$
Qwen3-1.7B	NTh	NTh	NTh	9.8	9.2	$-0.6$	20.8	20.8	$0.0$	3,923	8,463	$+115.7\%$
Qwen3-1.7B	NTh	Th	Th	34.0	40.8	$+6.8$	59.2	63.3	$+4.1$	18,624	19,996	$+7.4\%$

Appendix

Table 14: Model-level reasoning-mode results underlying Table 1 , with the complete Qwen3.5-4B grid. Rows are macro-averaged over four benchmarks. Notation follows Table 1 . Per-dataset results are in Table 49 .

Model	Subset	Avg@8			Pass@8			Length
Model	Subset	Base	OPSD	$\Delta$	Base	OPSD	$\Delta$	Base	OPSD	$\Delta$
Qwen3-1.7B	Full	34.0	38.1	$+4.1$	59.2	62.5	$+3.3$	18.6k	21.4k	$+15.2\%$
	Short-prefix	34.0	37.1	$+3.1$	59.2	62.5	$+3.3$	18.6k	20.9k	$+12.3\%$
	Long-prefix	34.0	36.7	$+2.7$	59.2	62.5	$+3.3$	18.6k	20.8k	$+11.5\%$
Qwen3-4B	Full	62.3	60.9	$-1.4$	79.2	75.8	$-3.4$	17.8k	19.1k	$+7.4\%$
	Short-prefix	62.3	61.0	$-1.3$	79.2	79.2	$+0.0$	17.8k	19.0k	$+6.9\%$

Appendix

Table 15: Teacher-prefix-length stratification at a 256-token rollout length. Short-prefix and Long-prefix retain gold-trace length deciles 0–4 and 7–9.

\Delta

is the OPSD–Base change (percentage points for accuracy, relative for Length). Bold marks the best Avg@8 and Pass@8 among Full, Short-prefix, and Long-prefix within each model (ties are all bolded). Per-dataset results are in Table 72 .

Model	Objective	Avg@8		Pass@8		Length
		Value	$\Delta$	Value	$\Delta$	Value	$\Delta$
Qwen3-1.7B	Fwd. KL ( $\beta=0$ )	35.31	$+1.3$	60.83	$+1.6$	22.4k	$+20.3\%$
	JSD ( $\beta=0.5$ )	31.25	$-2.7$	55.00	$-4.2$	15.2k	$-18.4\%$
	Rev. KL ( $\beta=1$ )	26.46	$-7.5$	47.50	$-11.7$	24.4k	$+31.0\%$
Qwen3-4B	Fwd. KL ( $\beta=0$ )	57.92	$-4.4$	75.83	$-3.4$	19.9k	$+12.0\%$
	JSD ( $\beta=0.5$ )	56.56	$-5.7$	75.83	$-3.4$	14.1k	$-20.7\%$

Appendix

Table 16: Macro-average divergence-ablation results over four benchmarks.

\Delta

is the absolute change from Base for Avg@8/Pass@8 and the relative change for Length. Bold marks the best Avg@8 and Pass@8 within each model. Decoding settings are fixed across objectives. Per-dataset results are in Table 55 .

Model	Subset	Avg@8	Pass@8	Length
Qwen3-1.7B	Full	38.1 ( $+4.1$ )	62.5 ( $+3.3$ )	21.4k ( $+15.2\%$ )
	HE20	36.9 ( $+2.9$ )	65.8 ( $+6.6$ )	21.7k ( $+16.7\%$ )
	HE80	37.8 ( $+3.8$ )	61.7 ( $+2.5$ )	20.8k ( $+11.6\%$ )
	LE20	37.6 ( $+3.6$ )	64.2 ( $+5.0$ )	18.0k ( $-3.5\%$ )
	LE80	33.2 ( $-0.8$ )	59.2 ( $+0.0$ )	17.7k ( $-5.2\%$ )
Qwen3-4B-Thinking	Full	70.3 ( $-5.1$ )	83.3 ( $-3.4$ )	30.5k ( $+34.0\%$ )

Appendix

Table 17: Selected macro-average results for student-entropy loss masking at a 256-token rollout length. Parentheses show changes from Base (percentage points for Avg@8/Pass@8, relative percentage for Length). HE/LE retain the highest/lowest-entropy 20%/80%. Bold marks the best accuracy per model. Complete results are in Table 56 .

Model	Loss support	Avg@8		Pass@8		Length
		Value	$\Delta$	Value	$\Delta$	Value	$\Delta$
Qwen3-1.7B	Full vocab	35.31	$+1.3$	60.83	$+1.6$	22.4k	$+20.3\%$
	$k=16$	33.02	$-1.0$	62.50	$+3.3$	22.6k	$+21.4\%$
	$k=4$	25.83	$-8.2$	47.50	$-11.7$	28.8k	$+54.6\%$
Qwen3-4B-Instruct	Full vocab	50.00	$+1.5$	70.00	$-1.7$	10.6k	$+15.2\%$
	$k=16$	51.98	$+3.5$	70.83	$-0.9$	10.6k	$+15.4\%$

Appendix

Table 19: Macro-average vocabulary top-

k

loss results. The Qwen3 rows use forward KL with teacher Top-

k

. OLMo-3-7B-Think uses reverse KL with student Top-

k

. Full-vocab rows are the matched controls from Table 16 .

\Delta

is the absolute change from Base for Avg@8/Pass@8 and the relative change for Length. Bold marks the best Avg@8 and Pass@8 within each model. Ties are all bolded. Per-dataset results are in Table 57 .

Model	Stage	Avg@8	Pass@8	Length
Qwen3-1.7B	Base	34.0	59.2	18.6k
	c256	38.1	62.5	21.4k
	AdvT4	34.7	59.2	19.8k
	AdvT16	37.2	61.7	19.5k
Qwen3-4B	Base	62.3	79.2	17.8k
	c256	60.9	75.8	19.1k

Appendix

Table 20: Teacher-favored position selection (AdvT4 / AdvT16) versus the matched 256-token full-loss control. Bold marks the best Avg@8 / Pass@8 within each model. Values are unweighted averages over AIME 2024–2026 and HMMT25.

Model	Mode	Highest 20%		Lowest 20%		HE20 share
		KL	$D>.05$	KL	$D>.05$	KL	Adv
Qwen3-1.7B	Th / Th	0.371	66.4	0.004	0.1	51.5	69.8
Qwen3-1.7B	NTh / NTh	0.277	61.5	0.003	0.1	55.3	72.9
Qwen3-4B	Th / Th	0.349	68.4	0.002	0.1	51.2	68.4
Qwen3-4B	NTh / NTh	0.221	57.2	0.002	0.1	60.1	74.4
Qwen3-4B-Instruct	NTh / NTh	0.522	66.0	0.003	0.1	59.7	76.3

Appendix

Table 21: Token-level entropy-tail signals for Table 17 . HE20 KL/Adv are the highest-entropy 20% shares of total forward KL and absolute advantage, with each total computed from the same entropy-analysis rollout pool.

Figure 7: Training traces for the SFT-initialization sweep. Top: SFT loss and token entropy for Qwen3-1.7B-Base and Qwen3-4B-Base. Bottom: loss and student entropy during 100-step OPSD runs from the 15k SFT checkpoints.

Model	SFT steps	Direct SFT			After 100-step OPSD
		Avg@8	Pass@8	Format	Avg@8	Pass@8	Format	Length
Qwen3-1.7B-Base	Base	2.08	8.33	80.63	1.15	7.50	63.23	8.0k
	5k	7.08	21.67	51.98	0.31	2.50	0.83	34.6k
	10k	9.59	20.83	64.58	3.12	11.67	6.56	35.2k
	15k	11.46	29.17	63.54	1.56	7.50	3.85	36.2k
Qwen3-4B-Base	Base	7.29	20.00	87.19	4.79	17.50	74.59	10.3k

Appendix

Table 22: Macro-average direct-SFT and downstream OPSD results. Format is the boxed-answer rate (%). Downstream OPSD initializes both student and teacher from the SFT checkpoint. Bold marks the best Avg@8 and Pass@8 within each model and stage. Per-checkpoint and per-dataset results, including the additional 7.5k and 12.5k direct-SFT evaluations, are in Table 59 and Table 60 .

Checkpoint	Cond.	$N$	Tokens	$\geq$ 32k	$\geq$ 38k	Boxed	Avg@8
1.7B SFT-5k	direct	960	23.8k	39.8%	19.5%	52.0%	7.1%
1.7B SFT-5k	OPSD	960	34.7k	48.0%	32.4%	0.8%	0.3%
1.7B SFT-10k	OPSD	960	35.2k	71.2%	55.4%	6.6%	3.1%
4B SFT-5k	OPSD	960	33.1k	80.0%	0.0%	2.1%	0.8%

Appendix

Table 23: Generation-level diagnostics pooled over AIME 2024–2026 and HMMT25. Boxed is the fraction of samples with a boxed final answer. Avg@8 is the fraction of correct samples. OPSD denotes evaluation after 100-step OPSD from that SFT checkpoint.

Initialization	Stage	Avg@8	Pass@8	Format	Length
Qwen3-4B ( Th / Th )	Base	62.3	79.2	97.1	17.8k
Qwen3-4B ( Th / Th )	+ 100-step OPSD	57.9	75.8	86.4	19.9k
4B SFT-15k+GRPO	GRPO-100	33.65	60.83	94.79	14.2k
4B SFT-15k+GRPO	+ 100-step OPSD	3.54	14.17	5.62	37.6k

Appendix

Table 24: Matched Th / Th OPSD from released Qwen3-4B and the SFT-then-GRPO checkpoint. Per-dataset results and training traces are in Table 64 and Table 31 .

Model	S	T	Fwd. KL	$D>.05$	Agree.	Enc.	Disc.
Qwen3-0.6B	Th	Th	0.074	21.7	92.6	36.8	62.3
DeepSeek-R1-Distill-Qwen-1.5B	Th	Th	0.033	12.5	95.1	42.7	56.9
OLMo-3-7B-Instruct	NTh	NTh	0.126	24.1	92.3	38.9	51.4
Qwen3-4B-Instruct	NTh	NTh	0.175	18.9	93.5	21.8	36.8
MiMo-7B-RL	Th	Th	0.104	23.3	93.4	20.0	75.8
OLMo-3-7B-Think	Th	Th	0.115	23.7	92.6	37.1	60.9

Appendix

Table 25: Additional matched-mode signals (%). Ordered by Base Pass@8. Definitions follow Table 7 .

Figure 8: Per-model teacher-context PCA for the ten checkpoints underlying the aggregate view in Figure 5 . Points are rollout-level signal summaries and large markers are centroids for the seven dataset–context conditions. The panels use a common PCA basis and shared axes. Features are standardized within each model–mode setting before the joint PCA fit.

Model	Mode	Context	Fwd. KL	$D>.05$	Abs. adv.	Agree.
Qwen3-0.6B	Th / Th	Answer	0.010	1.8	0.036	97.9
		Unrelated	0.046	17.5	0.112	93.8
		Full	0.074	21.5	0.139	92.7
		CoT+Sol.	0.134	36.9	0.214	89.1
		OMR Answer	0.008	1.3	0.034	98.0
		OMR Full	0.089	24.2	0.157	91.7

Appendix

Table 26: Teacher-side context signal by checkpoint (models 1–5). Unqualified conditions use OpenThoughts; OMR and DAPO denote OpenMathReasoning and DAPO-Math-17k. CoT+Sol. denotes CoT+Solution.

D>.05

and Agree. are percentages, and Abs. adv. is the mean absolute sampled-token log advantage.

Model	Mode	Context	Fwd. KL	$D>.05$	Abs. adv.	Agree.
Qwen3-4B-Thinking-2507	Th / Th	Answer	0.018	3.2	0.039	98.1
		Unrelated	0.105	19.2	0.128	94.1
		Full	0.159	24.1	0.181	92.4
		CoT+Sol.	0.275	34.8	0.270	89.0
		OMR Answer	0.018	2.8	0.038	98.1
		OMR Full	0.176	25.3	0.208	91.5

Appendix

Table 27: Teacher-side context signal by checkpoint (models 6–10 and ten-model macro average). Condition and metric definitions follow Table 26 .

Model	Mode	Prefix	Fwd KL	$D>.05$	Abs. adv.	Agree.
Qwen3-1.7B	Th / Th	Sol	0.144	20.2	0.184	92.9
Qwen3-1.7B	Th / Th	Sol-long	0.147	20.4	0.188	92.9
Qwen3-1.7B	NTh / NTh	Sol	0.090	16.7	0.125	94.7
Qwen3-1.7B	NTh / NTh	Sol-long	0.094	17.2	0.129	94.6
Qwen3-4B	Th / Th	Sol	0.135	22.0	0.176	92.8
Qwen3-4B	Th / Th	Sol-long	0.137	22.3	0.178	92.7

Appendix

Table 28: Length-control extension of the full-solution signal in Table 9 : short vs. long full-solution teacher prefixes on shared student rollouts. Sol uses a 1,024-token teacher prompt cap. Sol-long raises it to 12,288.

D>.05

and Agree. are percentages. Abs. adv. is the mean absolute sampled-token log advantage.

Model	Mode	$[0,128)$	$[128,256)$	$[256,512)$	$[512,1\mathrm{k})$	$[1\mathrm{k},2\mathrm{k})$	$[2\mathrm{k},4\mathrm{k})$	$[4\mathrm{k},6\mathrm{k})$
Qwen3-1.7B	Th / Th	0.291/0.238	0.224/0.214	0.138/0.186	0.091/0.153	0.066/0.124	0.048/0.104	0.041/0.090
Qwen3-4B	Th / Th	0.249/0.233	0.197/0.230	0.132/0.203	0.094/0.173	0.073/0.143	0.059/0.124	0.048/0.113
Qwen3-4B-Instruct	NTh / NTh	0.339/0.224	0.230/0.202	0.157/0.166	0.100/0.143	0.064/0.125	0.046/0.114	0.046/0.104
OLMo-3-7B-Think	Th / Th	0.258/0.278	0.158/0.255	0.106/0.221	0.077/0.173	0.062/0.132	0.050/0.106	0.039/0.088
OLMo-3-7B-Instruct	NTh / NTh	0.299/0.312	0.139/0.254	0.095/0.218	0.071/0.196	0.071/0.182	0.077/0.159	0.068/0.120
Macro average	–	0.287/0.248	0.190/0.217	0.126/0.185	0.087/0.160	0.067/0.138	0.056/0.121	0.048/0.103

Appendix

Table 29: Forward KL by response-position window (rollout cap 6,144). Each cell reports mean/SNR, where

\mathrm{SNR}=\mu/\sigma

and

\sigma

is the token-level standard deviation. The macro row assigns equal weight to each checkpoint, matching Table 11 .

	256-token rollout cap				1,024-token rollout cap
Model	KL First10	KL Final10	$\Delta$ Avg	$\Delta$ Pass	KL First10	KL Final10	$\Delta$ Avg	$\Delta$ Pass
Qwen3-4B	0.226	$0.412\uparrow$	$-1.4$	$-3.4$	0.137	$0.329\uparrow$	$-4.4$	$-3.4$

Appendix

Table 30: Qwen3-4B row supplementing the training-time forward-KL analysis in Table 10 . KL First10 and Final10 average the first and final ten optimizer-step records.

\Delta

Avg and

\Delta

Pass are changes from Base in percentage points.

Initialization	Steps	$H_{S}$	$D_{\mathrm{KL}}(\pi^{T}\\|\pi^{S})$	$\\|\nabla\\|$	Loss
Qwen3-4B ( Th / Th )	1–10	0.24	0.137	0.39	$-0.009$
Qwen3-4B ( Th / Th )	91–100	0.22	0.329	0.18	$-0.020$
SFT+GRPO then OPSD	1–10	0.41	0.047	0.35	$-0.001$
SFT+GRPO then OPSD	91–100	1.29	0.215	0.86	$-0.029$

Appendix

Table 31: Mean diagnostics over early (steps 1–10) and late (steps 91–100) windows in the initialization block of Figure 6 . Both runs use matched Th / Th and

\tau_{\mathrm{clip}}=0.05

H_{S}

is student entropy and

\|\nabla\|

is the pre-clipping gradient norm.

Model	Base	Full solution	No priv.: C $\to$ D	No priv.: D $\to$ C
Qwen3-1.7B	34.0 / 59.2 / 18.6k	35.3 / 60.8 / 22.4k	23.1 / 46.7 / 12.6k	36.3 / 62.5 / 25.0k
OLMo-3-7B-Think	63.5 / 81.7 / 18.7k	58.8 / 77.5 / 21.7k	60.7 / 80.8 / 16.5k	56.4 / 75.0 / 24.1k

Appendix

Table 32: Instruction-only prefix controls after 100 OPSD steps. Each cell reports Avg@8 / Pass@8 / mean output length. Per-dataset results are in Table 77 .

Model	$\tau_{\mathrm{clip}}$	Avg@8	Pass@8	Length
Qwen3-1.7B	Base	34.0	59.2	18.6k
	$0.01$	31.4	56.7	26.2k
	$0.05$	35.3	60.8	22.4k
	$0.1$	32.9	60.0	16.6k
	$0.2$	30.6	53.3	15.5k
OLMo-3-7B-Think	Base	63.5	81.7	18.7k

Appendix

Table 33: Vocabulary-contribution clip threshold sweep after 100 OPSD steps. Bold marks the best Avg@8 / Pass@8 within each model (ties both bolded). Per-dataset results are in Table 79 .

Model	SFT data	Stage	Avg@8	Pass@8	Length
Qwen3-1.7B	OT	Base	34.0	59.2	18.6k
		SFT-10	5.1	17.5	6.9k
		+ OPSD-100	5.2	16.7	6.6k
Qwen3-1.7B	OMR	Base	34.0	59.2	18.6k
		SFT-10	28.0	54.2	16.9k
		+ OPSD-100	27.1	49.2	16.9k

Appendix

Table 34: Ten-step SFT followed by 100-step no-privilege OPSD. OT/OMR denote the SFT corpus. OPSD uses OpenThoughts throughout. Bold marks the best Avg@8 / Pass@8 within each model–corpus block. Per-dataset results are in Table 80 .

Model	Base	Full OPSD	LoRA OPSD
Qwen3-1.7B	34.0 / 59.2 / 18.6k	35.3 / 60.8 / 22.4k	35.8 / 61.7 / 23.3k
Qwen3-4B	62.3 / 79.2 / 17.8k	57.9 / 75.8 / 19.9k	59.9 / 80.0 / 20.2k
Qwen3-4B-Instruct	48.5 / 71.7 / 9.1k	50.0 / 70.0 / 10.6k	49.2 / 70.8 / 11.2k
Qwen3-4B-Thinking	75.4 / 86.7 / 22.8k	65.8 / 83.3 / 43.4k	68.0 / 82.5 / 40.5k
OLMo-3-7B-Think	63.5 / 81.7 / 18.7k	58.8 / 77.5 / 21.7k	62.5 / 77.5 / 21.3k
OLMo-3-7B-Instruct	39.5 / 60.8 / 7.6k	40.8 / 62.5 / 7.4k	41.3 / 62.5 / 7.9k

Appendix

Table 35: Matched-mode full-parameter and LoRA OPSD after 100 updates. Each cell reports Avg@8 / Pass@8 / mean output length. Bold marks the best Avg@8 and Pass@8 within each model. Ties are all bolded.

Eval mode	Evaluated state	Avg@8	Pass@8	Format	Length
NTh	Base	52.71	77.50	97.71	10.2k
	Full OPSD ( NTh / NTh )	8.02	35.00	14.90	70.7k
	LoRA OPSD ( NTh / NTh )	45.73	68.33	99.48	8.4k
Th	Base	80.62	91.67	90.52	38.1k
	Full OPSD ( Th / Th )	0.00	0.00	5.42	81.9k
	LoRA OPSD ( Th / Th )	8.54	20.83	10.62	77.4k

Appendix

Table 36: Qwen3.5-4B full-parameter and LoRA results after 100 matched-mode OPSD updates.

Figure 9: Matched-mode training dynamics for full-parameter and LoRA Qwen3.5-4B versus three Qwen3 full-parameter controls. Curves are five-step moving averages from step 2 onward. Step 1 contains a one-off startup batch and is omitted.

S	T	Fwd. KL	$D>.05$	Agree.	Enc.	Disc.
NTh	NTh	0.087	17.7	94.7	34.1	63.4
NTh	Th	0.208	31.2	92.7	14.9	84.6
Th	NTh	0.140	27.2	92.0	56.0	43.4
Th	Th	0.066	16.5	95.0	40.2	58.7

Appendix

Table 37: Initial Qwen3.5-4B token-level signals. Definitions follow Table 7 . Per-setting and per-dataset results for the downstream mode grid are in Table 49 .

Variant	Model	Privileged field	Avg@8	$\Delta_{\mathrm{B/O}}$	Pass@8	$\Delta_{\mathrm{B/O}}$	Length	$\Delta_{\mathrm{B/O}}$
Purified OPSD	Qwen3-1.7B	Answer	39.9	$+5.9/+4.6$	63.3	$+4.1/+2.5$	24.2k	$+29.7/+7.9$
	Qwen3-1.7B	Solution	37.5	$+3.5/+2.2$	65.0	$+5.8/+4.2$	20.8k	$+11.5/-7.2$
	Qwen3-4B	Answer	63.3	$+1.0/+5.4$	79.2	$+0.0/+3.3$	19.6k	$+10.1/-2.0$
	Qwen3-4B	Solution	61.3	$-1.0/+3.4$	77.5	$-1.7/+1.7$	18.5k	$+4.3/-7.1$
	Qwen3-4B-Thinking	Answer	75.1	$-0.3/+9.3$	84.2	$-2.5/+0.8$	24.8k	$+8.8/-42.9$
	Qwen3-4B-Thinking	Solution	75.3	$-0.1/+9.5$	86.7	$+0.0/+3.3$	24.2k	$+6.2/-44.2$

Appendix

Table 38: Paper-configuration replication results, macro-averaged over AIME 2024–2026 and HMMT25. Each

\Delta_{\mathrm{B/O}}

entry gives the change relative to Base / standard OPSD-1024 in the same native matched reasoning mode. Avg@8 and Pass@8 changes are absolute percentage points; Length changes are relative percentages. Per-dataset results are in Table 75 and Table 76 .

Experiment family	Conditions	Training seeds	Evaluation seeds
Base checkpoint	Untrained Base	–	42 / 1024 / 65536
Rollout length	256 / 1024	42 / 1024 / 65536	42 / 1024 / 65536
Loss-token position	First / Uniform / Final-256	42 / 1024 / 65536	42 / 1024 / 65536
Teacher context	Solution / Answer / Unrelated	42 / 1024 / 65536	42 / 1024 / 65536

Appendix

Table 39: Seed repetitions for Qwen3-1.7B and OLMo-3-7B-Think. “Train” denotes independent training runs. “Eval” denotes decoding seeds applied to the seed-42-trained checkpoint.

Model	Evaluation seed
	$42$			$1024$			$65536$
	Avg@8	Pass@8	Length	Avg@8	Pass@8	Length	Avg@8	Pass@8	Length
Qwen3-1.7B	33.96	59.17	18.6k	35.10	55.00	18.6k	35.31	61.67	18.6k
OLMo-3-7B-Think	63.54	81.67	18.7k	62.19	80.83	19.0k	63.12	79.17	18.9k

Appendix

Table 40: Base checkpoints under evaluation seeds

42

1024

65536

. Training-seed tables use Base@eval-

42

. Evaluation-seed tables use Base at the matched evaluation seed. Per-dataset results are in Table 65 .

Model	Condition	Training seed
		$42$			$1024$			$65536$
		Avg@8	Pass@8	Length	Avg@8	Pass@8	Length	Avg@8	Pass@8	Length
Qwen3-1.7B	OPSD-256	38.12 ↑	62.50 ↑	21.4k	39.17 ↑	65.00 ↑	21.1k	37.08 ↑	65.00 ↑	21.3k
	OPSD-1024	35.31 ↑	60.83 ↑	22.4k	35.94 ↑	56.67 ↓	22.0k	34.58 ↑	62.50 ↑	22.5k
OLMo-3-7B-Think	OPSD-256	60.83 ↓	80.00 ↓	21.4k	61.46 ↓	79.17 ↓	21.5k	61.98 ↓	81.67	21.5k
	OPSD-1024	58.75 ↓	77.50 ↓	21.7k	59.17 ↓	78.33 ↓	21.8k	59.69 ↓	81.67	21.4k

Appendix

Table 41: Rollout-length results across training seeds

42

1024

65536

, evaluated with seed

42

. OPSD-256/1024 use full-solution context and matched Th / Th modes ( subsection 5.2 ). Per-dataset results are in Table 66 .

Model	Condition	Evaluation seed
		$42$			$1024$			$65536$
		Avg@8	Pass@8	Length	Avg@8	Pass@8	Length	Avg@8	Pass@8	Length
Qwen3-1.7B	OPSD-256	38.12 ↑	62.50 ↑	21.4k	36.98 ↑	59.17 ↑	21.0k	36.67 ↑	62.50 ↑	21.3k
	OPSD-1024	35.31 ↑	60.83 ↑	22.4k	36.77 ↑	65.00 ↑	21.9k	36.56 ↑	61.67	22.2k
OLMo-3-7B-Think	OPSD-256	60.83 ↓	80.00 ↓	21.4k	61.98 ↓	80.83	21.7k	61.35 ↓	77.50 ↓	21.6k
	OPSD-1024	58.75 ↓	77.50 ↓	21.7k	58.85 ↓	78.33 ↓	21.5k	60.31 ↓	79.17	21.6k

Appendix

Table 42: Rollout-length results across evaluation seeds

42

1024

65536

using the seed-42-trained checkpoints. Per-dataset results are in Table 67 .

Model	Condition	Training seed
		$42$			$1024$			$65536$
		Avg@8	Pass@8	Length	Avg@8	Pass@8	Length	Avg@8	Pass@8	Length
Qwen3-1.7B	First-256 / OPSD-256	38.12 ↑	62.50 ↑	21.4k	39.17 ↑	65.00 ↑	21.1k	37.08 ↑	65.00 ↑	21.3k
	Uni-256	35.21 ↑	61.67 ↑	22.6k	34.90 ↑	59.17	22.4k	36.56 ↑	62.50 ↑	22.8k
	Final-256	34.27 ↑	59.17	23.7k	33.23 ↓	55.00 ↓	23.8k	34.17 ↑	58.33 ↓	23.7k
OLMo-3-7B-Think	First-256 / OPSD-256	60.83 ↓	80.00 ↓	21.4k	61.46 ↓	79.17 ↓	21.5k	61.98 ↓	81.67	21.5k

Appendix

Table 43: Loss-token position results across training seeds. First-256 reuses the corresponding OPSD-256 run; Uni-256 and Final-256 use 1,024-token rollouts and retain at most 256 uniformly sampled or final loss positions. Per-dataset results are in Table 68 .

Model	Condition	Evaluation seed
		$42$			$1024$			$65536$
		Avg@8	Pass@8	Length	Avg@8	Pass@8	Length	Avg@8	Pass@8	Length
Qwen3-1.7B	First-256 / OPSD-256	38.12 ↑	62.50 ↑	21.4k	36.98 ↑	59.17 ↑	21.0k	36.67 ↑	62.50 ↑	21.3k
	Uni-256	35.21 ↑	61.67 ↑	22.6k	35.52 ↑	59.17 ↑	22.3k	34.27 ↓	58.33 ↓	22.8k
	Final-256	34.27 ↑	59.17	23.7k	35.00 ↓	61.67 ↑	23.5k	33.85 ↓	58.33 ↓	23.3k
OLMo-3-7B-Think	First-256 / OPSD-256	60.83 ↓	80.00 ↓	21.4k	61.98 ↓	80.83	21.7k	61.35 ↓	77.50 ↓	21.6k

Appendix

Table 44: Loss-token position results across evaluation seeds using the seed-42-trained checkpoints. First-256 reuses OPSD-256. Per-dataset results are in Table 69 .

Model	Condition	Training seed
		$42$			$1024$			$65536$
		Avg@8	Pass@8	Length	Avg@8	Pass@8	Length	Avg@8	Pass@8	Length
Qwen3-1.7B	Solution	35.31 ↑	60.83 ↑	22.4k	35.94 ↑	56.67 ↓	22.0k	34.58 ↑	62.50 ↑	22.5k
	Answer	35.62 ↑	59.17	25.8k	36.67 ↑	64.17 ↑	25.6k	34.69 ↑	59.17	25.5k
	Unrelated	37.08 ↑	60.00 ↑	22.0k	38.96 ↑	62.50 ↑	21.9k	37.29 ↑	62.50 ↑	21.6k
OLMo-3-7B-Think	Solution	58.75 ↓	77.50 ↓	21.7k	59.17 ↓	78.33 ↓	21.8k	59.69 ↓	81.67	21.4k

Appendix

Table 45: Teacher-context results across training seeds at a 1,024-token rollout cap (Full solution / Answer only / Unrelated solution, subsection 5.3 ). Per-dataset results are in Table 70 .

Model	Condition	Evaluation seed
		$42$			$1024$			$65536$
		Avg@8	Pass@8	Length	Avg@8	Pass@8	Length	Avg@8	Pass@8	Length
Qwen3-1.7B	Solution	35.31 ↑	60.83 ↑	22.4k	36.77 ↑	65.00 ↑	21.9k	36.56 ↑	61.67	22.2k
	Answer	35.62 ↑	59.17	25.8k	35.83 ↑	55.83 ↑	25.8k	35.94 ↑	60.00 ↓	25.9k
	Unrelated	37.08 ↑	60.00 ↑	22.0k	35.31 ↑	61.67 ↑	22.2k	36.67 ↑	59.17 ↓	22.0k
OLMo-3-7B-Think	Solution	58.75 ↓	77.50 ↓	21.7k	58.85 ↓	78.33 ↓	21.5k	60.31 ↓	79.17	21.6k

Appendix

Table 46: Teacher-context results across evaluation seeds using the seed-42-trained checkpoints and a 1,024-token rollout cap. Per-dataset results are in Table 71 .

Model	Decode $\tau$	Base			100-step OPSD			$\Delta$ (OPSD – Base)
		Avg@8	Pass@8	Length	Avg@8	Pass@8	Length	Avg@8	Pass@8	Length
Qwen3-1.7B	0.6	33.96	59.17	18.6k	35.31	60.83	22.4k	$+1.35$	$+1.66$	$+3.8$ k
Qwen3-1.7B	1.0	38.12	62.50	18.0k	36.67	65.83	21.4k	$-1.45$	$+3.33$	$+3.4$ k
Qwen3-4B-Instruct	0.7	48.54	71.67	9.2k	50.00	70.00	10.6k	$+1.46$	$-1.67$	$+1.4$ k
Qwen3-4B-Instruct	1.0	49.17	70.83	9.2k	48.75	66.67	10.7k	$-0.42$	$-4.16$	$+1.5$ k
Qwen3-4B-Thinking	0.6	75.42	86.67	22.8k	65.83	83.33	43.4k	$-9.59$	$-3.34$	$+20.6$ k

Appendix

Table 47: Decoding-temperature robustness check, grouped by identical decode temperature within each model. The Base and OPSD values within each row are directly comparable.

\Delta

is OPSD minus Base. Each OPSD row uses the checkpoint trained under the default configuration. Results are unweighted averages over four benchmarks. Per-dataset results are in Table 73 .

$\tau_{S}$	$\tau_{T}$	Eval. $\tau$	Avg@8	Pass@8	Length
1.1	1.1	0.6	35.31	60.83	22.4k
0.6	0.6	0.6	35.42	60.83	20.0k
0.6	1.1	0.6	35.21	56.67	22.3k
1.1	0.6	0.6	32.29	57.50	18.3k

Appendix

Table 48: Training temperature check for Qwen3-1.7B OPSD.

\tau_{S}

is the student rollout-sampling temperature and

\tau_{T}

is the teacher scoring temperature. All other settings are fixed. Values are unweighted averages over four benchmarks. Bold marks the best Avg@8 and Pass@8. Ties are all bolded. Per-dataset results are in Table 74 .

Model	S	T	Eval	Checkpoint	AIME 2024			AIME 2025			AIME 2026			HMMT25			Dataset Avg.
Model	S	T	Eval	Checkpoint	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length
Qwen3-1.7B	Th	–	Th	Base	43.8	76.7	18647	35.4	60.0	18266	35.0	53.3	18161	21.7	46.7	19421	34.0	59.2	18624
	Th	Th	Th	OPSD	45.8	73.3	22175	36.7	66.7	22143	36.7	63.3	21288	22.1	40.0	23913	35.3	60.8	22380
	Th	NTh	Th	OPSD	28.8	53.3	12469	19.2	40.0	9756	17.5	36.7	11343	11.3	26.7	10867	19.2	39.2	11109
	NTh	–	NTh	Base	14.2	30.0	4958	10.4	23.3	2808	8.8	20.0	4977	5.8	10.0	2948	9.8	20.8	3923
	NTh	Th	NTh	OPSD	20.0	36.7	10349	15.0	33.3	8806	11.7	23.3	11657	9.6	20.0	8112	14.1	28.3	9731

Appendix

Table 49: Per-dataset reasoning-mode results for Table 1 and Table 14 , including the no-privilege control. Each dataset reports Avg@8, Pass@8, and mean output length. Dataset Avg. is the unweighted four-benchmark mean. Bold marks the best Dataset Avg. for Avg@8 and Pass@8 within each model. Ties are all bolded.

Model	Training rollout	AIME 2024			AIME 2025			AIME 2026			HMMT25			Dataset Avg.
Model	Training rollout	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length
Qwen3-1.7B NTh / NTh	Base	14.2	30.0	4958	10.4	23.3	2808	8.8	20.0	4977	5.8	10.0	2948	9.8	20.8	3923
	128	12.1	26.7	5258	10.0	23.3	3116	6.7	16.7	5283	4.2	13.3	3335	8.2	20.0	4248
	256	14.2	33.3	6162	12.9	30.0	5492	9.6	20.0	7519	5.0	16.7	3751	10.4	25.0	5731
	512	13.3	36.7	9160	10.4	30.0	8869	13.3	26.7	9046	4.2	13.3	5936	10.3	26.7	8253
	1024	10.8	23.3	10015	10.8	30.0	7387	9.6	20.0	10503	5.4	10.0	5946	9.2	20.8	8463

Appendix

Table 50: Per-dataset rollout-length results for the matched NTh / NTh Qwen3 sweeps in Figure 3 . Each dataset reports Avg@8, Pass@8, and mean output length. Dataset Avg. gives the plotted mean.

Model	Training rollout	AIME 2024			AIME 2025			AIME 2026			HMMT25			Dataset Avg.
Model	Training rollout	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length
Qwen3-1.7B Th / Th	Base	43.8	76.7	18647	35.4	60.0	18266	35.0	53.3	18161	21.7	46.7	19421	34.0	59.2	18624
	128	47.5	70.0	19086	37.1	56.7	20269	31.7	60.0	20086	21.7	46.7	21820	34.5	58.3	20315
	256	46.3	70.0	20783	36.7	66.7	21697	45.0	73.3	20995	24.6	40.0	22306	38.1	62.5	21445
	512	50.0	76.7	20560	31.7	56.7	21037	38.3	63.3	20711	22.9	50.0	22648	35.7	61.7	21239
	1024	45.8	73.3	22175	36.7	66.7	22143	36.7	63.3	21288	22.1	40.0	23913	35.3	60.8	22380

Appendix

Table 51: Per-dataset rollout-length results for the remaining settings in Figure 3 and 256-position controls for Qwen3-1.7B and OLMo-3-7B-Think. First-256 reuses the 256-token rollout row; Uni-256 and Final-256 use a 1,024-token rollout. Each dataset reports Avg@8, Pass@8, and mean output length. Dataset Avg. is the unweighted mean.

Model	Training rollout	AIME 2024			AIME 2025			AIME 2026			HMMT25			Dataset Avg.
Model	Training rollout	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length
Qwen3-4B- Instruct NTh / NTh	Base	62.5	86.7	9412	45.8	70.0	8558	55.0	80.0	8434	30.8	50.0	10254	48.5	71.7	9165
	128	60.0	83.3	9790	47.1	70.0	8749	58.8	80.0	9095	32.1	50.0	11587	49.5	70.8	9805
	256	63.8	86.7	10378	47.5	73.3	10414	55.8	73.3	10143	31.2	50.0	12315	49.6	70.8	10813
	512	62.5	86.7	10748	47.1	73.3	10873	55.0	73.3	11005	30.4	53.3	11977	48.8	71.7	11151
	1024	65.0	83.3	10141	45.8	73.3	10539	58.3	76.7	10130	30.8	46.7	11409	50.0	70.0	10555

Appendix

Table 60

Steps	AIME 2024			AIME 2025			AIME 2026			HMMT25			Dataset Avg.
Steps	Avg	Pass	Len	Avg	Pass	Len	Avg	Pass	Len	Avg	Pass	Len	Avg	Pass	Len
Qwen3-1.7B
Base	43.8	76.7	18647	35.4	60.0	18266	35.0	53.3	18161	21.7	46.7	19421	34.0	59.2	18624
50	47.9	80.0	23206	35.4	60.0	23071	38.3	63.3	23010	22.1	43.3	25737	35.9	61.7	23756
100	45.8	73.3	22175	36.7	66.7	22143	36.7	63.3	21288	22.1	40.0	23913	35.3	60.8	22380
150	43.3	73.3	19383	31.7	56.7	19359	36.3	60.0	18536	20.4	43.3	21521	32.9	58.3	19700

Appendix

Table 52: Per-dataset extended-training OPSD results for Figure 2 .

Model	Rollout	Privileged context	AIME 2024			AIME 2025			AIME 2026			HMMT25			Dataset Avg.
Model	Rollout	Privileged context	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length
Qwen3-1.7B	–	None	43.8	76.7	18647	35.4	60.0	18266	35.0	53.3	18161	21.7	46.7	19421	34.0	59.2	18624
	256	Full solution	46.3	70.0	20783	36.7	66.7	21697	45.0	73.3	20995	24.6	40.0	22306	38.1	62.5	21445
	256	Answer only	49.6	70.0	23223	38.8	60.0	24084	38.3	66.7	22631	25.0	40.0	25702	37.9	59.2	23910
	256	Unrelated	47.9	73.3	19074	40.0	63.3	19943	36.3	66.7	20098	23.3	46.7	20165	36.9	62.5	19820
	1024	Full solution	45.8	73.3	22175	36.7	66.7	22143	36.7	63.3	21288	22.1	40.0	23913	35.3	60.8	22380

Appendix

Table 53: Per-dataset teacher-context results for Table 3 and Table 4 . Each dataset reports Avg@8, Pass@8, and mean output length. Dataset Avg. is the unweighted four-benchmark mean.

Post-thinking subset	Privileged context	Teacher cap	AIME 2024			AIME 2025			AIME 2026			HMMT25			Dataset Avg.
Post-thinking subset	Privileged context	Teacher cap	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length
$\leq$ 1k	Solution	1,024	45.8	73.3	22865	35.0	60.0	22109	34.6	63.3	21651	22.1	40.0	24262	34.4	59.2	22722
$\leq$ 1k	CoT + solution	12,288	37.9	76.7	16997	27.9	53.3	16101	28.3	56.7	16444	17.1	36.7	18137	27.8	55.8	16920
2–4k	Solution	4,096	45.4	66.7	21721	37.1	66.7	22442	32.5	56.7	22517	23.8	46.7	24077	34.7	59.2	22689
2–4k	CoT + solution	12,288	38.3	80.0	17884	27.9	43.3	16633	27.1	53.3	16678	22.5	40.0	17620	29.0	54.2	17204

Appendix

Table 54: Qwen3-1.7B teacher-context replication on the OpenMathReasoning cot split with a 1,024-token student rollout cap. Each pair uses identical examples. Subset labels give the rendered post-thinking solution length before CoT is added. CoT + solution uses a 12,288-token teacher cap.

Model	Objective	AIME 2024			AIME 2025			AIME 2026			HMMT25			Dataset Avg.
Model	Objective	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length
Qwen3-1.7B	Fwd. KL ( $\beta=0$ )	45.8	73.3	22175	36.7	66.7	22143	36.7	63.3	21288	22.1	40.0	23913	35.3	60.8	22380
	JSD ( $\beta=0.5$ )	40.0	70.0	14950	32.9	53.3	15222	33.3	60.0	14764	18.8	36.7	15931	31.3	55.0	15217
	Rev. KL ( $\beta=1$ )	37.5	63.3	23381	31.7	46.7	22391	23.8	46.7	23906	12.9	33.3	27968	26.5	47.5	24411
Qwen3-4B	Fwd. KL ( $\beta=0$ )	66.3	80.0	18395	62.9	80.0	19655	62.9	83.3	18119	39.6	60.0	23625	57.9	75.8	19949
	JSD ( $\beta=0.5$ )	67.1	83.3	12485	59.6	76.7	14881	60.8	83.3	13550	38.8	60.0	15634	56.6	75.8	14137

Appendix

Table 55: Per-dataset divergence-ablation results for subsection 5.5 and Table 16 . Each dataset reports Avg@8, Pass@8, and mean output length. Dataset Avg. is the unweighted four-benchmark mean.

Model	Subset	AIME 2024			AIME 2025			AIME 2026			HMMT25			Dataset Avg.
Model	Subset	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length
Qwen3-1.7B	Base	43.8	76.7	18647	35.4	60.0	18266	35.0	53.3	18161	21.7	46.7	19421	34.0	59.2	18624
	Full	46.3	70.0	20783	36.7	66.7	21697	45.0	73.3	20995	24.6	40.0	22306	38.1	62.5	21445
	HE20	44.6	76.7	21009	36.7	63.3	22166	39.6	73.3	21696	26.7	50.0	22074	36.9	65.8	21736
	HE80	52.5	76.7	20083	34.6	56.7	21365	40.0	70.0	20013	24.2	43.3	21676	37.8	61.7	20784
	LE20	49.6	73.3	17001	37.9	63.3	17766	39.2	73.3	18538	23.8	46.7	18619	37.6	64.2	17981

Appendix

Table 56: Per-dataset student-entropy loss-masking results for Table 17 . Full uses all response tokens. HE20/HE80 and LE20/LE80 retain the highest- or lowest-entropy fraction of each completion.

Model	Support	AIME 2024			AIME 2025			AIME 2026			HMMT25			Dataset Avg.
Model	Support	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length
Qwen3-1.7B	Full vocab	45.8	73.3	22175	36.7	66.7	22143	36.7	63.3	21288	22.1	40.0	23913	35.3	60.8	22380
	$k=16$	42.5	80.0	22299	31.7	56.7	22440	34.6	63.3	21396	23.3	50.0	24293	33.0	62.5	22607
	$k=4$	28.3	60.0	28891	28.8	46.7	28108	31.3	53.3	26761	15.0	30.0	31383	25.8	47.5	28786
Qwen3-4B-Instruct	Full vocab	65.0	83.3	10141	45.8	73.3	10539	58.3	76.7	10130	30.8	46.7	11409	50.0	70.0	10555
	$k=16$	63.8	80.0	10156	50.8	70.0	10317	59.6	80.0	9726	33.8	53.3	12112	52.0	70.8	10578

Appendix

Table 57: Per-dataset vocabulary top-

k

results for Table 19 . Each dataset reports Avg@8, Pass@8, and mean output length. Dataset Avg. is the unweighted four-benchmark mean.

Model	Checkpoint	AIME 2024			AIME 2025			AIME 2026			HMMT25			Dataset Avg.
Model	Checkpoint	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length
Qwen3-0.6B	Base	8.8	30.0	16815	17.5	40.0	15188	12.9	26.7	16334	8.8	23.3	15532	12.0	30.0	15967
	256	10.8	30.0	21812	18.3	36.7	19027	10.4	30.0	21177	9.2	26.7	19708	12.2	30.8	20431
	1024	9.2	30.0	20924	12.5	30.0	18855	8.3	16.7	19339	7.5	16.7	20224	9.4	23.3	19835
DeepSeek-R1-Distill-Qwen-1.5B	Base	28.3	53.3	17211	24.6	46.7	15944	19.6	43.3	17906	14.2	40.0	18742	21.7	45.8	17451
	256	28.8	66.7	17814	23.3	43.3	17795	22.5	50.0	18387	15.0	26.7	18842	22.4	46.7	18209

Appendix

Table 58: Per-dataset results for Figure 4 . 256 and 1024 are full-solution OPSD conditions. Dataset Avg. is the unweighted four-benchmark mean.

SFT steps	Evaluation	AIME 2024			AIME 2025			AIME 2026			HMMT25			Dataset Avg.
SFT steps	Evaluation	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length
Base	Direct	3.8	13.3	6629	1.2	10.0	5214	3.3	10.0	5654	0.0	0.0	4805	2.1	8.3	5576
5k	Direct	8.8	26.7	24373	9.6	23.3	23444	7.5	23.3	23689	2.5	13.3	23591	7.1	21.7	23774
7.5k	Direct	9.2	26.7	24317	10.0	26.7	23141	9.6	26.7	25677	4.2	20.0	23953	8.2	25.0	24272
10k	Direct	10.4	20.0	22854	12.9	23.3	18654	10.4	23.3	20890	4.6	16.7	20950	9.6	20.8	20837
12.5k	Direct	12.9	33.3	20910	16.2	36.7	19194	10.8	30.0	21869	4.2	13.3	20361	11.0	28.3	20584

Appendix

Table 59: Per-dataset Qwen3-1.7B-Base SFT-initialization results for Table 22 . Thinking-mode evaluation uses a 38,912-token maximum output length.

SFT steps	Evaluation	AIME 2024			AIME 2025			AIME 2026			HMMT25			Dataset Avg.
SFT steps	Evaluation	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length
Base	Direct	12.1	26.7	3692	7.5	26.7	2707	8.3	20.0	2889	1.2	6.7	2863	7.3	20.0	3038
5k	Direct	30.4	66.7	16251	30.0	53.3	16427	25.4	50.0	16250	16.7	33.3	18485	25.6	50.8	16853
7.5k	Direct	35.0	63.3	14817	33.8	56.7	14477	31.2	56.7	15043	20.0	33.3	15893	30.0	52.5	15057
10k	Direct	37.1	70.0	13999	34.6	56.7	13907	36.7	66.7	14128	20.8	50.0	14470	32.3	60.8	14126
12.5k	Direct	42.5	73.3	13632	32.5	56.7	14781	33.3	66.7	14865	20.0	33.3	15521	32.1	57.5	14700

Appendix

Table 60: Per-dataset Qwen3-4B-Base SFT-initialization results for Table 22 .

Init. / method	AIME 2024				AIME 2025				AIME 2026				HMMT25
Init. / method	Avg	Pass	Fmt	Len	Avg	Pass	Fmt	Len	Avg	Pass	Fmt	Len	Avg	Pass	Fmt	Len
Qwen3-4B-Base (direct)	12.1	26.7	86.2	3692	7.5	26.7	89.2	2707	8.3	20.0	86.7	2889	1.2	6.7	86.7	2863
+ 100-step OPSD	7.9	26.7	74.2	9927	4.6	23.3	76.2	10265	4.2	10.0	73.8	10319	2.5	10.0	74.2	10718
SFT-15k (direct)	45.8	73.3	89.6	13843	34.2	63.3	90.4	13968	36.2	60.0	88.8	13625	22.5	43.3	88.3	15708
+ 100-step OPSD	2.1	13.3	3.3	33095	3.3	10.0	5.0	32466	2.5	10.0	3.3	32770	0.0	0.0	2.1	33220
+ GRPO	43.8	73.3	93.3	14430	34.6	56.7	96.2	13451	35.8	70.0	93.3	13860	20.4	43.3	96.2	14990

Appendix

Table 61: Per-dataset post-training-pipeline results for the first six rows of subsection 5.5 . Fmt is the boxed-answer rate. Table 64 compares GRPO-then-OPSD with matched Qwen3-4B Th / Th .

Checkpoint	AIME 2024				AIME 2025				AIME 2026				HMMT25
Checkpoint	Avg	Pass	Fmt	Len	Avg	Pass	Fmt	Len	Avg	Pass	Fmt	Len	Avg	Pass	Fmt	Len
Qwen3-1.7B + GRPO-50	47.1	76.7	95.8	19761	37.1	63.3	97.5	19985	38.8	66.7	97.1	20334	25.4	46.7	95.8	21751
Qwen3-4B + GRPO-50	72.5	83.3	97.5	16342	67.9	86.7	92.9	19731	67.1	83.3	92.5	18507	43.8	60.0	91.3	22044

Appendix

Table 62: Per-dataset 50-step GRPO results on released Qwen3 thinking checkpoints for Table 6 . Fmt is the boxed-answer rate.

Checkpoint	AIME 2024				AIME 2025				AIME 2026				HMMT25
Checkpoint	Avg	Pass	Fmt	Len	Avg	Pass	Fmt	Len	Avg	Pass	Fmt	Len	Avg	Pass	Fmt	Len
Qwen3-1.7B + OPSD-50	48.3	76.7	87.9	21217	40.4	66.7	93.3	21169	40.0	63.3	88.3	21876	23.3	46.7	91.3	23860
Qwen3-4B + OPSD-50	70.4	83.3	91.7	17829	63.8	86.7	88.3	20209	61.3	83.3	88.8	18498	45.0	66.7	89.2	22175

Appendix

Table 63: Per-dataset 50-step OPSD results on the same OpenMathReasoning examples and in the same order as GRPO, corresponding to Table 6 . OPSD uses the post-thinking solution as privileged teacher context. Fmt is the boxed-answer rate.

Checkpoint	Stage	AIME 2024				AIME 2025				AIME 2026				HMMT25
Checkpoint	Stage	Avg	Pass	Fmt	Len	Avg	Pass	Fmt	Len	Avg	Pass	Fmt	Len	Avg	Pass	Fmt	Len
Qwen3-4B ( Th / Th )	Base	74.6	90.0	98.8	15495	65.4	80.0	96.2	18696	65.8	80.0	95.0	17249	43.3	66.7	98.3	19636
	+ OPSD	66.3	80.0	86.7	18395	62.9	80.0	86.2	19655	62.9	83.3	89.6	18119	39.6	60.0	82.9	23625
4B SFT-15k + GRPO	GRPO-100	43.8	73.3	93.3	14430	34.6	56.7	96.2	13451	35.8	70.0	93.3	13860	20.4	43.3	96.2	14990
	+ OPSD	3.3	10.0	7.1	37973	5.0	13.3	5.4	37043	3.3	20.0	5.4	37485	2.5	13.3	4.6	37812

Appendix

Table 64: Per-dataset results for Table 24 . Fmt is the boxed-answer rate.

Model	Eval seed	AIME 2024			AIME 2025			AIME 2026			HMMT25			Dataset Avg.
Model	Eval seed	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length
Qwen3-1.7B	42	43.8	76.7	18647	35.4	60.0	18266	35.0	53.3	18161	21.7	46.7	19421	34.0	59.2	18624
	1024	47.5	76.7	17907	36.7	53.3	18615	35.8	56.7	17919	20.4	33.3	19812	35.1	55.0	18563
	65536	42.9	73.3	18102	37.9	56.7	18542	33.3	66.7	18614	27.1	50.0	19196	35.3	61.7	18613
OLMo-3-7B-Think	42	71.7	86.7	17296	66.7	80.0	19137	71.2	90.0	17560	44.6	70.0	20950	63.5	81.7	18736
	1024	67.9	83.3	17617	64.2	86.7	19148	72.1	86.7	17955	44.6	66.7	21317	62.2	80.8	19009

Appendix

Table 65: Per-dataset Base results for Table 40 . Each dataset reports Avg@8, Pass@8, and mean output length. Dataset Avg. is the unweighted four-benchmark mean.

Model	Condition	Train seed	AIME 2024			AIME 2025			AIME 2026			HMMT25			Dataset Avg.
Model	Condition	Train seed	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length
Qwen3-1.7B	OPSD-256	42	46.2 ↑	70.0 ↓	20783	36.7 ↑	66.7 ↑	21697	45.0 ↑	73.3 ↑	20995	24.6 ↑	40.0 ↓	22306	38.1 ↑	62.5 ↑	21445
		1024	51.2 ↑	83.3 ↑	20253	36.2 ↑	63.3 ↑	21118	42.1 ↑	66.7 ↑	21295	27.1 ↑	46.7	21828	39.2 ↑	65.0 ↑	21124
		65536	50.4 ↑	76.7	20253	37.1 ↑	63.3 ↑	20962	38.3 ↑	70.0 ↑	21160	22.5 ↑	50.0 ↑	22975	37.1 ↑	65.0 ↑	21338
Qwen3-1.7B	OPSD-1024	42	45.8 ↑	73.3 ↓	22175	36.7 ↑	66.7 ↑	22143	36.7 ↑	63.3 ↑	21288	22.1 ↑	40.0 ↓	23913	35.3 ↑	60.8 ↑	22380
		1024	49.2 ↑	73.3 ↓	21091	36.7 ↑	63.3 ↑	21896	37.5 ↑	60.0 ↑	21149	20.4 ↓	30.0 ↓	23892	35.9 ↑	56.7 ↓	22007

Appendix

Table 66: Per-dataset training-seed results for Table 41 . Each dataset reports Avg@8, Pass@8, and mean output length. Dataset Avg. is the unweighted four-benchmark mean. ↑ / ↓ mark accuracy changes relative to Base@eval-

42

. Length is unannotated.

Model	Condition	Eval seed	AIME 2024			AIME 2025			AIME 2026			HMMT25			Dataset Avg.
Model	Condition	Eval seed	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length
Qwen3-1.7B	OPSD-256	42	46.2 ↑	70.0 ↓	20783	36.7 ↑	66.7 ↑	21697	45.0 ↑	73.3 ↑	20995	24.6 ↑	40.0 ↓	22306	38.1 ↑	62.5 ↑	21445
		1024	47.9 ↑	70.0 ↓	20152	36.7	60.0 ↑	21390	37.9 ↑	56.7	20678	25.4 ↑	50.0 ↑	21863	37.0 ↑	59.2 ↑	21021
		65536	47.5 ↑	73.3	20297	34.2 ↓	63.3 ↑	21394	41.7 ↑	66.7	21027	23.3 ↓	46.7 ↓	22365	36.7 ↑	62.5 ↑	21271
Qwen3-1.7B	OPSD-1024	42	45.8 ↑	73.3 ↓	22175	36.7 ↑	66.7 ↑	22143	36.7 ↑	63.3 ↑	21288	22.1 ↑	40.0 ↓	23913	35.3 ↑	60.8 ↑	22380
		1024	49.2 ↑	76.7	21279	34.2 ↓	63.3 ↑	21384	38.8 ↑	66.7 ↑	21373	25.0 ↑	53.3 ↑	23458	36.8 ↑	65.0 ↑	21874

Appendix

Table 67: Per-dataset evaluation-seed results for Table 42 . Each dataset reports Avg@8, Pass@8, and mean output length. Dataset Avg. is the unweighted four-benchmark mean. ↑ / ↓ mark accuracy changes relative to Base at the matched evaluation seed. Length is unannotated.

Model	Condition	Train seed	AIME 2024			AIME 2025			AIME 2026			HMMT25			Dataset Avg.
Model	Condition	Train seed	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length
Qwen3-1.7B	First-256 / OPSD-256	42	46.2 ↑	70.0 ↓	20783	36.7 ↑	66.7 ↑	21697	45.0 ↑	73.3 ↑	20995	24.6 ↑	40.0 ↓	22306	38.1 ↑	62.5 ↑	21445
		1024	51.2 ↑	83.3 ↑	20253	36.2 ↑	63.3 ↑	21118	42.1 ↑	66.7 ↑	21295	27.1 ↑	46.7	21828	39.2 ↑	65.0 ↑	21124
		65536	50.4 ↑	76.7	20253	37.1 ↑	63.3 ↑	20962	38.3 ↑	70.0 ↑	21160	22.5 ↑	50.0 ↑	22975	37.1 ↑	65.0 ↑	21338
Qwen3-1.7B	Uni-256	42	48.8 ↑	76.7	21874	31.7 ↓	60.0	21873	36.7 ↑	63.3 ↑	22032	23.8 ↑	46.7	24462	35.2 ↑	61.7 ↑	22560
		1024	47.5 ↑	73.3 ↓	21809	34.2 ↓	63.3 ↑	22245	34.6 ↓	60.0 ↑	22053	23.3 ↑	40.0 ↓	23502	34.9 ↑	59.2	22402

Appendix

Table 68: Per-dataset training-seed results for Table 43 . First-256 reuses the corresponding OPSD-256 run. Each dataset reports Avg@8, Pass@8, and mean output length. Dataset Avg. is the unweighted four-benchmark mean. ↑ / ↓ mark accuracy changes relative to Base@eval-

42

. Length is unannotated.

Model	Condition	Eval seed	AIME 2024			AIME 2025			AIME 2026			HMMT25			Dataset Avg.
Model	Condition	Eval seed	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length
Qwen3-1.7B	First-256 / OPSD-256	42	46.2 ↑	70.0 ↓	20783	36.7 ↑	66.7 ↑	21697	45.0 ↑	73.3 ↑	20995	24.6 ↑	40.0 ↓	22306	38.1 ↑	62.5 ↑	21445
		1024	47.9 ↑	70.0 ↓	20152	36.7	60.0 ↑	21390	37.9 ↑	56.7	20678	25.4 ↑	50.0 ↑	21863	37.0 ↑	59.2 ↑	21021
		65536	47.5 ↑	73.3	20297	34.2 ↓	63.3 ↑	21394	41.7 ↑	66.7	21027	23.3 ↓	46.7 ↓	22365	36.7 ↑	62.5 ↑	21271
Qwen3-1.7B	Uni-256	42	48.8 ↑	76.7	21874	31.7 ↓	60.0	21873	36.7 ↑	63.3 ↑	22032	23.8 ↑	46.7	24462	35.2 ↑	61.7 ↑	22560
		1024	47.1 ↓	73.3 ↓	22073	34.6 ↓	63.3 ↑	21959	36.7 ↑	60.0 ↑	21197	23.8 ↑	40.0 ↑	24054	35.5 ↑	59.2 ↑	22321

Appendix

Table 69: Per-dataset evaluation-seed results for Table 44 . First-256 reuses OPSD-256. Each dataset reports Avg@8, Pass@8, and mean output length. Dataset Avg. is the unweighted four-benchmark mean. ↑ / ↓ mark accuracy changes relative to Base at the matched evaluation seed. Length is unannotated.

Model	Condition	Train seed	AIME 2024			AIME 2025			AIME 2026			HMMT25			Dataset Avg.
Model	Condition	Train seed	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length
Qwen3-1.7B	Solution	42	45.8 ↑	73.3 ↓	22175	36.7 ↑	66.7 ↑	22143	36.7 ↑	63.3 ↑	21288	22.1 ↑	40.0 ↓	23913	35.3 ↑	60.8 ↑	22380
		1024	49.2 ↑	73.3 ↓	21091	36.7 ↑	63.3 ↑	21896	37.5 ↑	60.0 ↑	21149	20.4 ↓	30.0 ↓	23892	35.9 ↑	56.7 ↓	22007
		65536	41.7 ↓	76.7	22332	35.8 ↑	53.3 ↓	21324	37.1 ↑	70.0 ↑	21537	23.8 ↑	50.0 ↑	24818	34.6 ↑	62.5 ↑	22503
Qwen3-1.7B	Answer	42	47.1 ↑	76.7	25291	34.2 ↓	56.7 ↓	25710	39.6 ↑	70.0 ↑	24298	21.7	33.3 ↓	27985	35.6 ↑	59.2	25821
		1024	44.2 ↑	73.3 ↓	25539	37.1 ↑	63.3 ↑	25758	40.8 ↑	70.0 ↑	23793	24.6 ↑	50.0 ↑	27365	36.7 ↑	64.2 ↑	25614

Appendix

Table 70: Per-dataset training-seed results for Table 45 . Each dataset reports Avg@8, Pass@8, and mean output length. Dataset Avg. is the unweighted four-benchmark mean. ↑ / ↓ mark accuracy changes relative to Base@eval-

42

. Length is unannotated.

Model	Condition	Eval seed	AIME 2024			AIME 2025			AIME 2026			HMMT25			Dataset Avg.
Model	Condition	Eval seed	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length
Qwen3-1.7B	Solution	42	45.8 ↑	73.3 ↓	22175	36.7 ↑	66.7 ↑	22143	36.7 ↑	63.3 ↑	21288	22.1 ↑	40.0 ↓	23913	35.3 ↑	60.8 ↑	22380
		1024	49.2 ↑	76.7	21279	34.2 ↓	63.3 ↑	21384	38.8 ↑	66.7 ↑	21373	25.0 ↑	53.3 ↑	23458	36.8 ↑	65.0 ↑	21874
		65536	49.6 ↑	73.3	21563	35.0 ↓	63.3 ↑	21611	38.8 ↑	66.7	21850	22.9 ↓	43.3 ↓	23902	36.6 ↑	61.7	22231
Qwen3-1.7B	Answer	42	47.1 ↑	76.7	25291	34.2 ↓	56.7 ↓	25710	39.6 ↑	70.0 ↑	24298	21.7	33.3 ↓	27985	35.6 ↑	59.2	25821
		1024	46.7 ↓	70.0 ↓	25361	39.2 ↑	66.7 ↑	25576	37.1 ↑	53.3 ↓	24341	20.4	33.3	27802	35.8 ↑	55.8 ↑	25770

Appendix

Table 71: Per-dataset evaluation-seed results for Table 46 . Each dataset reports Avg@8, Pass@8, and mean output length. Dataset Avg. is the unweighted four-benchmark mean. ↑ / ↓ mark accuracy changes relative to Base at the matched evaluation seed. Length is unannotated.

Model	Subset	AIME 2024			AIME 2025			AIME 2026			HMMT25			Dataset Avg.
Model	Subset	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length
Qwen3-1.7B	Base	43.8	76.7	18647	35.4	60.0	18266	35.0	53.3	18161	21.7	46.7	19421	34.0	59.2	18624
	Full	46.3	70.0	20783	36.7	66.7	21697	45.0	73.3	20995	24.6	40.0	22306	38.1	62.5	21445
	Short-prefix	49.2	80.0	20157	38.8	66.7	21312	36.3	60.0	20889	24.2	43.3	21264	37.1	62.5	20905
	Long-prefix	48.8	73.3	19618	35.4	63.3	21613	37.5	66.7	20203	25.0	46.7	21638	36.7	62.5	20768
Qwen3-4B	Base	74.6	90.0	15495	65.4	80.0	18696	65.8	80.0	17249	43.3	66.7	19636	62.3	79.2	17769

Appendix

Table 72: Per-dataset teacher-prefix-length results for Table 15 . Full uses the complete training set and a 256-token rollout. Short-prefix and Long-prefix are defined by gold-trace length and contain different problems. Each dataset reports Avg@8, Pass@8, and mean output length.

Model	Decode $\tau$ / ckpt	AIME 2024			AIME 2025			AIME 2026			HMMT25			Dataset Avg.
Model	Decode $\tau$ / ckpt	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length
Qwen3-1.7B	0.6 / Base	43.8	76.7	18647	35.4	60.0	18266	35.0	53.3	18161	21.7	46.7	19421	34.0	59.2	18624
	0.6 / OPSD	45.8	73.3	22175	36.7	66.7	22143	36.7	63.3	21288	22.1	40.0	23913	35.3	60.8	22380
	1.0 / Base	49.2	73.3	17458	39.2	63.3	17941	40.4	66.7	17523	23.8	46.7	19009	38.1	62.5	17983
	1.0 / OPSD	51.2	80.0	20477	35.4	73.3	21752	34.6	60.0	20409	25.4	50.0	22901	36.7	65.8	21385
Qwen3-4B-Instruct	0.7 / Base	62.5	86.7	9412	45.8	70.0	8558	55.0	80.0	8434	30.8	50.0	10254	48.5	71.7	9165

Appendix

Table 73: Per-dataset decoding-temperature results for Table 47 . Base and OPSD use the listed temperature.

$\tau_{S}$	$\tau_{T}$	AIME 2024			AIME 2025			AIME 2026			HMMT25			Dataset Avg.
$\tau_{S}$	$\tau_{T}$	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length	Avg	Pass	Length
1.1	1.1	45.8	73.3	22175	36.7	66.7	22143	36.7	63.3	21288	22.1	40.0	23913	35.3	60.8	22380
0.6	0.6	46.7	76.7	19889	37.1	63.3	19212	35.4	63.3	19985	22.5	40.0	21010	35.4	60.8	20024
0.6	1.1	46.2	76.7	21424	34.2	53.3	22346	38.3	56.7	21255	22.1	40.0	24204	35.2	56.7	22307
1.1	0.6	43.8	76.7	17488	34.2	60.0	18115	30.4	53.3	18741	20.8	40.0	19019	32.3	57.5	18341

Appendix

Table 74: Per-dataset Qwen3-1.7B training-temperature results for Table 48 . All rows use decoding temperature

0.6

Model	Priv.	AIME24	AIME25	AIME26	HMMT25	Macro Avg	Macro Pass	Length
Qwen3-1.7B	Answer	52.9 / 76.7	41.7 / 63.3	41.7 / 70.0	23.3 / 43.3	39.9	63.3	24.2k
Qwen3-1.7B	Solution	46.3 / 76.7	37.9 / 60.0	36.7 / 63.3	29.2 / 60.0	37.5	65.0	20.8k
Qwen3-4B	Answer	75.0 / 83.3	70.8 / 80.0	66.7 / 86.7	40.8 / 66.7	63.3	79.2	19.6k
Qwen3-4B	Solution	72.5 / 80.0	62.1 / 80.0	64.6 / 83.3	46.3 / 66.7	61.3	77.5	18.5k
Qwen3-4B-Thinking-2507	Answer	83.8 / 90.0	82.5 / 86.7	78.3 / 86.7	55.8 / 73.3	75.1	84.2	24.8k
Qwen3-4B-Thinking-2507	Solution	84.2 / 93.3	79.6 / 86.7	81.3 / 90.0	56.3 / 76.7	75.3	86.7	24.2k

Appendix

Table 75: Per-dataset Purified OPSD results for Table 38 . Priv. is the privileged teacher field used to construct the PMI target.

Model	AIME24	AIME25	AIME26	HMMT25	Macro Avg	Macro Pass	Length
Qwen3-1.7B	32.5 / 63.3	27.9 / 56.7	25.4 / 50.0	14.6 / 26.7	25.1	49.2	13.4k
Qwen3-4B-Thinking-2507	76.7 / 86.7	74.6 / 86.7	77.1 / 90.0	46.3 / 66.7	68.6	82.5	18.2k
Qwen3-4B-Instruct	59.2 / 76.7	43.3 / 60.0	57.5 / 83.3	28.8 / 43.3	47.2	65.8	11.0k
OLMo-3-7B-Think	69.2 / 80.0	69.2 / 86.7	72.1 / 90.0	45.8 / 70.0	64.1	81.7	18.8k
OLMo-3-7B-Instruct	55.0 / 83.3	45.0 / 60.0	45.0 / 66.7	24.6 / 50.0	42.4	65.0	8.3k

Appendix

Table 76: Per-dataset

\beta

-OPSD results for Table 38 . Each benchmark cell reports Avg@8 / Pass@8.

Model	Condition	AIME 2024			AIME 2025			AIME 2026			HMMT25			Dataset Avg.
Model	Condition	Avg	Pass	Len	Avg	Pass	Len	Avg	Pass	Len	Avg	Pass	Len	Avg	Pass	Len
Qwen3-1.7B	Base	43.8	76.7	18647	35.4	60.0	18266	35.0	53.3	18161	21.7	46.7	19421	34.0	59.2	18624
	Full solution	45.8	73.3	22175	36.7	66.7	22143	36.7	63.3	21288	22.1	40.0	23913	35.3	60.8	22380
	No priv.: C $\to$ D	30.0	53.3	12632	25.0	43.3	12005	22.5	53.3	12903	15.0	36.7	12791	23.1	46.7	12583
	No priv.: D $\to$ C	43.3	73.3	25307	37.5	66.7	25255	42.1	70.0	23577	22.1	40.0	25965	36.3	62.5	25026
OLMo-3-7B-Think	Base	71.7	86.7	17296	66.7	80.0	19137	71.2	90.0	17560	44.6	70.0	20950	63.5	81.7	18736

Appendix

Table 77: Per-dataset English instruction-prefix controls for Table 32 . C/D denote concise/detailed student-to-teacher instructions. Both instruction-only rows contain no privileged teacher context.

Model	Stage	AIME 2024			AIME 2025			AIME 2026			HMMT25			Dataset Avg.
Model	Stage	Avg	Pass	Len	Avg	Pass	Len	Avg	Pass	Len	Avg	Pass	Len	Avg	Pass	Len
Qwen3-1.7B	Base	43.8	76.7	18647	35.4	60.0	18266	35.0	53.3	18161	21.7	46.7	19421	34.0	59.2	18624
	c256	46.2	70.0	20783	36.7	66.7	21697	45.0	73.3	20995	24.6	40.0	22306	38.1	62.5	21445
	AdvT4	47.1	80.0	19206	34.6	50.0	20145	35.8	56.7	18732	21.2	50.0	20992	34.7	59.2	19769
	AdvT16	51.2	80.0	18716	37.1	60.0	19128	36.2	60.0	19273	24.2	46.7	20824	37.2	61.7	19485
Qwen3-4B	Base	74.6	90.0	15495	65.4	80.0	18696	65.8	80.0	17249	43.3	66.7	19636	62.3	79.2	17769

Appendix

Table 78: Per-dataset AdvT4 / AdvT16 results for Table 20 .

Model	$\tau_{\mathrm{clip}}$	AIME 2024			AIME 2025			AIME 2026			HMMT25			Dataset Avg.
Model	$\tau_{\mathrm{clip}}$	Avg	Pass	Len	Avg	Pass	Len	Avg	Pass	Len	Avg	Pass	Len	Avg	Pass	Len
Qwen3-1.7B	Base	43.8	76.7	18647	35.4	60.0	18266	35.0	53.3	18161	21.7	46.7	19421	34.0	59.2	18624
	$0.01$	40.4	66.7	25792	30.8	56.7	25552	32.1	63.3	24715	22.1	40.0	28918	31.4	56.7	26244
	$0.05$	45.8	73.3	22175	36.7	66.7	22143	36.7	63.3	21288	22.1	40.0	23913	35.3	60.8	22380
	$0.1$	44.2	76.7	16599	32.1	53.3	16320	32.9	60.0	16266	22.5	50.0	17404	32.9	60.0	16647
	$0.2$	34.6	63.3	14852	32.5	53.3	15336	31.7	53.3	15600	23.8	43.3	16392	30.6	53.3	15545

Appendix

Table 79: Per-dataset pointwise vocabulary-contribution-cap results for Table 33 .

Model	SFT data	Stage	AIME 2024			AIME 2025			AIME 2026			HMMT25			Dataset Avg.
Model	SFT data	Stage	Avg	Pass	Len	Avg	Pass	Len	Avg	Pass	Len	Avg	Pass	Len	Avg	Pass	Len
Qwen3-1.7B	OT	Base	43.8	76.7	18647	35.4	60.0	18266	35.0	53.3	18161	21.7	46.7	19421	34.0	59.2	18624
		SFT-10	7.1	23.3	9991	7.5	20.0	5382	5.8	26.7	6527	0.0	0.0	5897	5.1	17.5	6949
		+ OPSD-100	7.9	23.3	8030	5.8	20.0	6372	5.8	16.7	5708	1.3	6.7	6183	5.2	16.7	6573
Qwen3-1.7B	OMR	Base	43.8	76.7	18647	35.4	60.0	18266	35.0	53.3	18161	21.7	46.7	19421	34.0	59.2	18624
		SFT-10	35.4	66.7	17957	30.0	46.7	15927	29.6	63.3	17051	17.1	40.0	16562	28.0	54.2	16874

Appendix

Table 80: Per-dataset 10-step SFT followed by 100-step no-privilege OPSD for Table 34 . OT/OMR denote the SFT corpus. OPSD uses OpenThoughts.

Language Model Distillation

Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.

See all 512 papers

May 12, 2026

OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoning

We study on-policy self-distillation (OPSD), where a language model improves its reasoning ability by distilling privileged teacher distributions along its own on-policy trajectories. Despite its promise, OPSD can suffer from training instability due to a pattern mismatch between teacher and student responses. Self-reflected teacher responses may introduce reflection-induced biases and response templates that miscalibrate token-level supervision, ultimately harming the student's reasoning ability. To mitigate this issue, we propose OGLS-SD, an outcome-guided logit-steering framework that leverages verifiable outcome rewards to calibrate privileged teacher logits. Specifically, OGLS-SD contrasts teacher logits induced by successful and failed on-policy trajectories, constructing an outcome-discriminative steering direction for token-level guidance. Experiments on mathematical reasoning benchmarks show that OGLS-SD stabilizes self-distillation and improves performance over standard OPSD and other variants.

Yuxiao Yang, Xiaoyun Wang, Weitong Zhang

UNC Chapel Hill · NVIDIA

Language Model Calibration Language Model Distillation On-Policy Distillation Activation Steering LLM Reasoning

Ask PDF

Jul 30, 2026

$β$ -OPSD: Deriving with Policy Optimization, Training with Self-Distillation

On-policy self-distillation (OPSD) is a promising approach to improve reasoning language models, but it remains brittle in practice: making it work reliably often requires substantial engineering effort. We identify a structural source of this difficulty: vanilla OPSD is precisely the

β=1

member of a broader policy-optimization family, where

β

weights the KL penalty anchoring the student to a reference policy. This equivalence turns

β

from an implicit value fixed at one into a controllable regularization parameter, yielding a more general formulation that trades off proximity to a reference policy against privileged teacher guidance. We introduce

β

-OPSD and derive its optimal policy as a geometric interpolation between the reference policy and the privileged teacher. Directly optimizing this objective with reinforcement learning, however, would be costly and high-variance. Rather than optimize the RL objective directly, we turn its closed-form solution into a distillation target. Each value of

β

selects a target along the reference-to-teacher path, which we implement efficiently by mixing their token-level logits. In this way, inexpensive distillation approximates the solution of expensive policy optimization. Return-to-go credit assignment further aligns token updates with the sequence-level objective while retaining the simplicity of OPSD. Experiments on mathematical reasoning benchmarks show that

β

-OPSD consistently outperforms vanilla OPSD, improving optimization stability and downstream reasoning performance. Our results provide a principled route from self-distillation to policy optimization and back without sacrificing the efficiency that makes OPSD practical.

Jiawei Xu, Minghui Liu, Juzheng Zhang +2

University of Maryland, College Park

KL-Regularized RL On-Policy Self-Distillation RL for Language Model Reasoning Language Model Distillation Policy Learning

Ask PDF

Aug 26, 2026

One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillation

On-policy distillation trains a language model on its own generations while a teacher scores them token by token. It combines the dense supervision of imitation learning with the on-policy sampling of reinforcement learning. But it requires a second, larger model to act as teacher. On-Policy Self-Distillation (OPSD) removes that cost. The teacher is the model itself, conditioned on privileged information the student will not have at test time, such as a reference solution, a plan, or environment feedback. The teacher is no stronger than the student, only better informed. Early results were promising, with accuracy comparable to reinforcement learning at a fraction of the generated tokens. But the same asymmetry that produces the signal also biases it. One failure mode now dominates the field: collapse, the progressive narrowing of the set of reasoning paths the model can produce. Collapse is not specific to OPSD, though privileged information aggravates it. This review treats collapse as a symptom governed by three levers: (i) where the signal is applied, that is, how tokens are weighted; (ii) what the teacher is shown, that is, the nature of the privileged information; and (iii) when the signal changes, that is, the teacher's dynamics and the decay of guidance. We restrict our scope to mathematical reasoning, where the method originated and where its failure modes are best documented. We report no new experiments. The contribution is structural: a shared vocabulary for phenomena named differently across papers, and a clear line between what is settled and what is still disputed.

Justin Robert, Raheel Qader

OVHai LLM

On-Policy Self-Distillation On-Policy Distillation Mode Collapse

Ask PDF

Abstract

Language Model Distillation

Explore similar work

OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoning

βββ-OPSD: Deriving with Policy Optimization, Training with Self-Distillation

One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillation

$β$ -OPSD: Deriving with Policy Optimization, Training with Self-Distillation