Rethinking Privileged Information in On-Policy Self-Distillation

Samyak Shrestha; Alexander Tessier

cs.LGAug 18, 2026

Rethinking Privileged Information in On-Policy Self-Distillation

Authors: Samyak Shrestha, Alexander Tessier

Organizations: FirstPrinciples

Abstract

On-policy self-distillation (OPSD) trains a student on its own responses using token-level supervision from the same model conditioned on privileged reference information. We investigate whether performance gains from OPSD show that the student learned the information in the reference or instead reflect recovery of reasoning behavior already present in the base model. We perform OPSD experiments on science and mathematics datasets using Qwen3 models ranging from 1.7B to 8B. Our analysis framework separates the supervision induced by the reference from the supervision provided by the teacher without the reference and measures how each aligns with changes in the student's predictions. The correct reference does not provide a consistent performance benefit across teacher generation modes, model sizes, and training datasets. Students can improve without the correct reference, and a solution from another problem can outperform the correct solution on several mathematical reasoning benchmarks. The student's predictions align more strongly with the base model's thinking behavior than with the supervision induced by the reference, but controls constructed from other problems reproduce much of both alignments. Moreover, stronger alignment attributable to the correct reference does not reliably coincide with a greater performance benefit from the reference. Performance gains and distributional alignment alone therefore cannot determine how privileged reference information contributes to student learning in OPSD.

Figures & tables

Panel of Figure 1 — Figure 1: OPSD setup and experimental conditions. The student remains in non-thinking mode while we vary the teacher’s generation mode and reference context.

Method	NuminaMath
	MATH-500	AIME24	AIME25	HMMT25	Avg.
Base model	84.7	25.0	19.7	13.1	35.6
OPSD, non-thinking teacher, correct ref.	91.5	66.7	49.2	30.6	59.5
OPSD, non-thinking teacher, mismatched ref.	94.4	71.9	61.7	42.2	67.6
OPSD, thinking teacher, correct ref.	88.0	65.3	52.8	31.7	59.4
OPSD, thinking teacher, no ref.	89.5	63.6	51.1	36.9	60.3

Table 1: Main benchmark results for Qwen3-8B. The base model and all OPSD students are evaluated in non-thinking mode. Ref. denotes the reference solution. Avg. is the unweighted mean across benchmarks. AIME, HMMT, and GPQA-Diamond use Avg@12, MATH-500 uses Avg@4, and MMLU-Pro and UGPhysics use Avg@1.

Figure 2: Cosine similarity between the reference and thinking directions on NuminaMath under a non-thinking teacher, across five reference types. Error bars show 1.96 bootstrap standard errors across problems.

Figure 3: Alignment between changes in the Qwen3-8B student’s predictions and the teacher directions during training.

Figure 4: Changes in the Qwen3-8B student’s predictions measured against the teacher directions and their controls. (a) Projection coefficients onto

D_{\mathrm{ref}}

and

D_{\mathrm{rec}}

under a thinking teacher. (b) Cosine similarity with the same directions.

Teacher	Dataset	Evaluated problem	Control
Thinking	NuminaMath	0.94	0.96
Thinking	MegaScience	0.97	0.98
Non-thinking	NuminaMath	0.88	0.94
Non-thinking	MegaScience	0.94	0.96

Table 2: Fraction

R_{c}^{2}

of the squared change in the Qwen3-8B student’s log probabilities that lies in the span of the teacher directions at checkpoint 300.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Source or subject	Problems
NuminaMath	Olympiads	6,500
	AoPS forum	2,000
	Synthetic AMC	1,000
	MATH	500
MegaScience	Physics	4,000
	Mathematics	2,500

Appendix

Table 3: Composition of the training subsets.

Reference condition	Problems
Original dataset solution	4,953
Qwen3-8B reasoning trace	4,953
Qwen3-32B reasoning trace	4,953
Answer-only reference	4,953
Abstract hint	4,898

Appendix

Table 4: Training-set sizes for the reference-type comparison.

Condition	Information provided to the teacher
Original dataset solution	The complete solution supplied by the training dataset.
Answer-only	Only the final answer extracted from the last $\backslash$ boxed{} expression in the original solution.
Abstract hint	Method-level guidance generated from the problem and original solution without a worked derivation or final answer.
Qwen3-8B trace	A verified full reasoning trace generated by Qwen3-8B in thinking mode.
Qwen3-32B trace	A verified full reasoning trace generated by Qwen3-32B in thinking mode.
Mismatched reference	The original dataset solution from another problem.

Appendix

Table 5: Reference conditions used during training.

Setting	Value
Base model	Qwen3-8B
Student mode	Non-thinking
LoRA rank / alpha	64 / 128
LoRA target modules	$\mathtt{q,k,v,o,gate,up,down\_proj}$
Learning rate	$5\times 10^{-6}$
Optimizer	Fused AdamW

Appendix

Table 6: Shared OPSD training configuration.

Dataset	Teacher mode	Completion limit	KL clip per coordinate
NuminaMath	Non-thinking	4,096	$10^{-7}$
NuminaMath	Thinking	1,024	0.06
MegaScience	Non-thinking	2,048	$10^{-7}$
MegaScience	Thinking	1,024	0.06

Appendix

Table 7: Settings that vary by dataset and teacher mode.

Benchmark	Problems	Reported read	Grading
MATH-500	500	Avg@4	Final boxed answer with symbolic equivalence.
AIME 2024	30	Avg@12	Final boxed answer with symbolic equivalence.
AIME 2025	30	Avg@12	Final boxed answer with symbolic equivalence.
HMMT 2025	30	Avg@12	Final boxed answer with symbolic equivalence.
GPQA-Diamond	198	Avg@12	Parsed answer choice with exact letter matching.
MMLU-Pro	12,032	Avg@1	Parsed answer choice with exact letter matching.

Appendix

Table 8: Evaluation benchmarks and grading procedures.

Reference provided to the teacher	Avg@12	Difference from canonical	Paired 95% interval
Canonical solution	61.1	–	–
Answer only	66.7	$+5.6$	$[-1.4,+12.8]$
Abstract hint	31.1	$-30.0$	$[-39.4,-20.6]$
Qwen3-8B reasoning trace	69.4	$+8.3$	$[+3.3,+13.9]$
Qwen3-32B reasoning trace	63.3	$+2.2$	$[-2.5,+6.7]$

Appendix

Table 9: Reference-type comparison on AIME 2024 under a non-thinking teacher (Avg@12).

Benchmark	Correct solution	Solution from another problem	Difference	Paired 95% interval
AIME 2024	66.7	71.9	$+5.3$	$[-0.6,+11.1]$
AIME 2025	49.2	61.7	$+12.5$	$[+6.4,+18.9]$
HMMT 2025	30.6	42.2	$+11.7$	$[+6.1,+17.5]$

Appendix

Table 10: Correct and mismatched references under a non-thinking teacher (Avg@12).

Training data	Condition	Checkpoint
NuminaMath	Non-thinking teacher, correct reference	50
NuminaMath	Non-thinking teacher, mismatched reference	50
NuminaMath	Thinking teacher, correct reference	200
NuminaMath	Thinking teacher, no reference	200
MegaScience	Non-thinking teacher, correct reference	150
MegaScience	Non-thinking teacher, mismatched reference	150

Appendix

Table 11: Checkpoints used for the main benchmark comparison.

Benchmark	Completion limit (tokens)
MATH-500	38,912
AIME 2024, AIME 2025, HMMT 2025	32,768
GPQA-Diamond	32,768
MMLU-Pro	32,768
UGPhysics	32,768

Appendix

Table 12: Generation limits for the main reported evaluations.

Method	AIME24	AIME25	HMMT25	Avg.
Base model	74.72	67.78	45.00	62.50
OPSD, thinking teacher, correct ref.	75.28	70.56	47.50	64.45
OPSD, thinking teacher, no ref.	76.67	71.39	45.00	64.35

Appendix

Table 13: Thinking-mode evaluation of Qwen3-8B on mathematical reasoning benchmarks.

Package	Version	Package	Version
Python	3.10.12	PyTorch	2.8.0
Transformers	4.57.1	TRL	0.26.0
PEFT	0.17.1	DeepSpeed	0.18.2
Datasets	3.6.0	Accelerate	1.11.0
vLLM	0.11.0	FlashAttention	2.8.3
math-verify	0.8.0

Appendix

Table 14: Principal software versions.

Analysis	Comparison	NuminaMath	MegaScience
Figure 2	Canonical solution	$[-0.146,-0.133]$	–
Figure 2	Qwen3-8B trace	$[-0.069,-0.062]$	–
Figure 2	Qwen3-32B trace	$[-0.081,-0.068]$	–
Figure 2	Abstract hint	$[-0.144,-0.136]$	–
Figure 2	Answer only	$[-0.150,-0.145]$	–
Figure 3	Reference direction at step 150	$[0.054,0.055]$	$[0.066,0.079]$

Appendix

Table 15: Cosine-similarity differences across three control assignments. Bracketed entries show the minimum and maximum values, not confidence intervals.

Figure 5: Magnitude of the Qwen3-8B student’s log-probability change and its KL divergence from the reference-free teacher during training under a thinking teacher.

Figure 6: Policy entropy of the Qwen3-8B student during training on NuminaMath under thinking and non-thinking teachers.

Figure 7: Cosine similarity between changes in the Qwen3-8B student’s predictions and the base model’s thinking direction under a non-thinking teacher.

Method	NuminaMath
	MATH-500	AIME24	AIME25	HMMT25	Avg.
Base model	84.0	24.7	21.1	10.8	35.2
OPSD, non-thinking teacher, correct ref.	82.5	31.4	21.7	13.6	37.3
OPSD, non-thinking teacher, mismatched ref.	82.6	40.3	32.5	22.8	44.5
OPSD, thinking teacher, correct ref.	90.9	56.7	48.3	34.2	57.5
OPSD, thinking teacher, no ref.	90.0	41.9	37.5	24.4	48.5

Appendix

Table 16: Qwen3-4B benchmark results under non-thinking evaluation. Evaluation reads follow Table 1 .

Method	NuminaMath
	MATH-500	AIME24	AIME25	HMMT25	Avg.
Base model	72.0	14.4	10.3	4.4	25.3
OPSD, non-thinking teacher, correct ref.	67.2	13.1	11.9	5.3	24.4
OPSD, non-thinking teacher, mismatched ref.	77.5	27.5	18.3	13.9	34.3
OPSD, thinking teacher, correct ref.	76.5	32.2	20.8	14.7	36.1
OPSD, thinking teacher, no ref.	79.1	22.2	16.7	10.0	32.0

Appendix

Table 17: Qwen3-1.7B benchmark results under non-thinking evaluation. Evaluation reads follow Table 1 .

On-Policy Self-Distillation

On-policy self-distillation trains a model on its own generated trajectories using a teacher view to provide dense supervision.

See all 157 papers

What Does Privileged Information Add to On-Policy Self-Distillation?
Sep 17, 2026XiuYu Zhang, Wei Chow, Junfeng Fang +3On-Policy Self-Distillation Learning Using Privileged Information
National University of Singapore
Privileged Solutions or Context-Induced Teacher Behavior? Dissecting On-Policy Self-Distillation
Aug 10, 2026Yuki Ichihara, Naoto Iwase, Mohammad Atif Quamar +1On-Policy Self-Distillation Language Model Distillation
Mohamed bin Zayed University of Artificial Intelligence · Nagoya University · RIKEN AIP
Is More Privileged Information Better? From Solution Traces to Problem-Solving Structure in Self-Distilled Reasoning
Aug 3, 2026Xuyang Zhao, Liting Zhang, Zichen Xu +4On-Policy Self-Distillation Knowledge Representation
1TMCC, College of Computer Science, Nankai University, Tianjin, China · 2Huawei Technologies Ltd., Beijing, China

Sep 17, 2026J/K move · Enter open · S save

What Does Privileged Information Add to On-Policy Self-Distillation?

XiuYu Zhang, Wei Chow, Junfeng Fang, Xingyu Zhu+2

National University of Singapore

On-Policy Self-Distillation Learning Using Privileged Information LLM Reasoning

Open paper AskPDF arXiv

On-policy self-distillation (OPSD) lets a language model learn from a frozen copy of itself that sees an answer or a worked solution. Giving the teacher this extra information seems to offer the student more to learn, but how much does it add beyond distillation itself? To isolate that contribution, we construct AMPLE-Math, a reusable suite of 5,319 mathematical problems with six reasoning views that share the same answer, and compare each view with matched reference-free distillation. With a thinking-enabled teacher supervising direct-response rollouts, reference-free distillation accounts for much of Qwen3-1.7B's improvement under thinking-enabled evaluation, both in domain and on external benchmarks. Evidence for an additional reference benefit is modest in Qwen, strongest for a polished solution, whereas complete traces add two percentage points in SmolLM3-3B at step 50. These benefits depend on the student being trained. At the same checkpoint, replacing short direct-response rollouts with long thinking-enabled rollouts turns gains into losses in both families while the problems, references, and evaluation stay fixed. Teacher profiles and matched loss interventions in Qwen further show that changing token-level supervision can leave student behavior largely unchanged. Together, these findings suggest that OPSD can improve access to existing reasoning capabilities through parameters shared by direct-response and thinking-enabled inference. The value of a privileged reference is what it adds to this cross-mode transfer, not how much of the solution it reveals.

Abstract

On-Policy Self-Distillation

Explore similar work

What Does Privileged Information Add to On-Policy Self-Distillation?

Privileged Solutions or Context-Induced Teacher Behavior? Dissecting On-Policy Self-Distillation

Is More Privileged Information Better? From Solution Traces to Problem-Solving Structure in Self-Distilled Reasoning