On-policy self-distillation (OPSD) trains a student on its own responses using token-level supervision from the same model conditioned on privileged reference information. We investigate whether performance gains from OPSD show that the student learned the information in the reference or instead reflect recovery of reasoning behavior already present in the base model. We perform OPSD experiments on science and mathematics datasets using Qwen3 models ranging from 1.7B to 8B. Our analysis framework separates the supervision induced by the reference from the supervision provided by the teacher without the reference and measures how each aligns with changes in the student's predictions. The correct reference does not provide a consistent performance benefit across teacher generation modes, model sizes, and training datasets. Students can improve without the correct reference, and a solution from another problem can outperform the correct solution on several mathematical reasoning benchmarks. The student's predictions align more strongly with the base model's thinking behavior than with the supervision induced by the reference, but controls constructed from other problems reproduce much of both alignments. Moreover, stronger alignment attributable to the correct reference does not reliably coincide with a greater performance benefit from the reference. Performance gains and distributional alignment alone therefore cannot determine how privileged reference information contributes to student learning in OPSD.
Figures & tables
Figure 1: OPSD setup and experimental conditions. The student remains in non-thinking mode while we vary the teacher’s generation mode and reference context.
Method
NuminaMath
MATH-500
AIME24
AIME25
HMMT25
Avg.
Base model
84.7
25.0
19.7
13.1
35.6
OPSD, non-thinking teacher, correct ref.
91.5
66.7
49.2
30.6
59.5
OPSD, non-thinking teacher, mismatched ref.
94.4
71.9
61.7
42.2
67.6
OPSD, thinking teacher, correct ref.
88.0
65.3
52.8
31.7
59.4
OPSD, thinking teacher, no ref.
89.5
63.6
51.1
36.9
60.3
Table 1: Main benchmark results for Qwen3-8B. The base model and all OPSD students are evaluated in non-thinking mode. Ref. denotes the reference solution. Avg. is the unweighted mean across benchmarks. AIME, HMMT, and GPQA-Diamond use Avg@12, MATH-500 uses Avg@4, and MMLU-Pro and UGPhysics use Avg@1.
Figure 2: Cosine similarity between the reference and thinking directions on NuminaMath under a non-thinking teacher, across five reference types. Error bars show 1.96 bootstrap standard errors across problems.
Figure 3: Alignment between changes in the Qwen3-8B student’s predictions and the teacher directions during training.
Figure 4: Changes in the Qwen3-8B student’s predictions measured against the teacher directions and their controls. (a) Projection coefficients onto Dref and Drec under a thinking teacher. (b) Cosine similarity with the same directions.
Teacher
Dataset
Evaluated problem
Control
Thinking
NuminaMath
0.94
0.96
Thinking
MegaScience
0.97
0.98
Non-thinking
NuminaMath
0.88
0.94
Non-thinking
MegaScience
0.94
0.96
Table 2: Fraction Rc2 of the squared change in the Qwen3-8B student’s log probabilities that lies in the span of the teacher directions at checkpoint 300.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Source or subject
Problems
NuminaMath
Olympiads
6,500
AoPS forum
2,000
Synthetic AMC
1,000
MATH
500
MegaScience
Physics
4,000
Mathematics
2,500
Appendix
Table 3: Composition of the training subsets.
Reference condition
Problems
Original dataset solution
4,953
Qwen3-8B reasoning trace
4,953
Qwen3-32B reasoning trace
4,953
Answer-only reference
4,953
Abstract hint
4,898
Appendix
Table 4: Training-set sizes for the reference-type comparison.
Condition
Information provided to the teacher
Original dataset solution
The complete solution supplied by the training dataset.
Answer-only
Only the final answer extracted from the last \ boxed{} expression in the original solution.
Abstract hint
Method-level guidance generated from the problem and original solution without a worked derivation or final answer.
Qwen3-8B trace
A verified full reasoning trace generated by Qwen3-8B in thinking mode.
Qwen3-32B trace
A verified full reasoning trace generated by Qwen3-32B in thinking mode.
Mismatched reference
The original dataset solution from another problem.
Appendix
Table 5: Reference conditions used during training.
Setting
Value
Base model
Qwen3-8B
Student mode
Non-thinking
LoRA rank / alpha
64 / 128
LoRA target modules
q,k,v,o,gate,up,down_proj
Learning rate
5×10−6
Optimizer
Fused AdamW
Appendix
Table 6: Shared OPSD training configuration.
Dataset
Teacher mode
Completion limit
KL clip per coordinate
NuminaMath
Non-thinking
4,096
10−7
NuminaMath
Thinking
1,024
0.06
MegaScience
Non-thinking
2,048
10−7
MegaScience
Thinking
1,024
0.06
Appendix
Table 7: Settings that vary by dataset and teacher mode.
Benchmark
Problems
Reported read
Grading
MATH-500
500
Avg@4
Final boxed answer with symbolic equivalence.
AIME 2024
30
Avg@12
Final boxed answer with symbolic equivalence.
AIME 2025
30
Avg@12
Final boxed answer with symbolic equivalence.
HMMT 2025
30
Avg@12
Final boxed answer with symbolic equivalence.
GPQA-Diamond
198
Avg@12
Parsed answer choice with exact letter matching.
MMLU-Pro
12,032
Avg@1
Parsed answer choice with exact letter matching.
Appendix
Table 8: Evaluation benchmarks and grading procedures.
Reference provided to the teacher
Avg@12
Difference from canonical
Paired 95% interval
Canonical solution
61.1
–
–
Answer only
66.7
+5.6
[−1.4,+12.8]
Abstract hint
31.1
−30.0
[−39.4,−20.6]
Qwen3-8B reasoning trace
69.4
+8.3
[+3.3,+13.9]
Qwen3-32B reasoning trace
63.3
+2.2
[−2.5,+6.7]
Appendix
Table 9: Reference-type comparison on AIME 2024 under a non-thinking teacher (Avg@12).
Benchmark
Correct solution
Solution from another problem
Difference
Paired 95% interval
AIME 2024
66.7
71.9
+5.3
[−0.6,+11.1]
AIME 2025
49.2
61.7
+12.5
[+6.4,+18.9]
HMMT 2025
30.6
42.2
+11.7
[+6.1,+17.5]
Appendix
Table 10: Correct and mismatched references under a non-thinking teacher (Avg@12).
Training data
Condition
Checkpoint
NuminaMath
Non-thinking teacher, correct reference
50
NuminaMath
Non-thinking teacher, mismatched reference
50
NuminaMath
Thinking teacher, correct reference
200
NuminaMath
Thinking teacher, no reference
200
MegaScience
Non-thinking teacher, correct reference
150
MegaScience
Non-thinking teacher, mismatched reference
150
Appendix
Table 11: Checkpoints used for the main benchmark comparison.
Benchmark
Completion limit (tokens)
MATH-500
38,912
AIME 2024, AIME 2025, HMMT 2025
32,768
GPQA-Diamond
32,768
MMLU-Pro
32,768
UGPhysics
32,768
Appendix
Table 12: Generation limits for the main reported evaluations.
Method
AIME24
AIME25
HMMT25
Avg.
Base model
74.72
67.78
45.00
62.50
OPSD, thinking teacher, correct ref.
75.28
70.56
47.50
64.45
OPSD, thinking teacher, no ref.
76.67
71.39
45.00
64.35
Appendix
Table 13: Thinking-mode evaluation of Qwen3-8B on mathematical reasoning benchmarks.
Package
Version
Package
Version
Python
3.10.12
PyTorch
2.8.0
Transformers
4.57.1
TRL
0.26.0
PEFT
0.17.1
DeepSpeed
0.18.2
Datasets
3.6.0
Accelerate
1.11.0
vLLM
0.11.0
FlashAttention
2.8.3
math-verify
0.8.0
Appendix
Table 14: Principal software versions.
Analysis
Comparison
NuminaMath
MegaScience
Figure 2
Canonical solution
[−0.146,−0.133]
–
Figure 2
Qwen3-8B trace
[−0.069,−0.062]
–
Figure 2
Qwen3-32B trace
[−0.081,−0.068]
–
Figure 2
Abstract hint
[−0.144,−0.136]
–
Figure 2
Answer only
[−0.150,−0.145]
–
Figure 3
Reference direction at step 150
[0.054,0.055]
[0.066,0.079]
Appendix
Table 15: Cosine-similarity differences across three control assignments. Bracketed entries show the minimum and maximum values, not confidence intervals.
Figure 5: Magnitude of the Qwen3-8B student’s log-probability change and its KL divergence from the reference-free teacher during training under a thinking teacher.
Figure 6: Policy entropy of the Qwen3-8B student during training on NuminaMath under thinking and non-thinking teachers.
Figure 7: Cosine similarity between changes in the Qwen3-8B student’s predictions and the base model’s thinking direction under a non-thinking teacher.
On-policy self-distillation (OPSD) lets a language model learn from a frozen copy of itself that sees an answer or a worked solution. Giving the teacher this extra information seems to offer the student more to learn, but how much does it add beyond distillation itself? To isolate that contribution, we construct AMPLE-Math, a reusable suite of 5,319 mathematical problems with six reasoning views that share the same answer, and compare each view with matched reference-free distillation. With a thinking-enabled teacher supervising direct-response rollouts, reference-free distillation accounts for much of Qwen3-1.7B's improvement under thinking-enabled evaluation, both in domain and on external benchmarks. Evidence for an additional reference benefit is modest in Qwen, strongest for a polished solution, whereas complete traces add two percentage points in SmolLM3-3B at step 50. These benefits depend on the student being trained. At the same checkpoint, replacing short direct-response rollouts with long thinking-enabled rollouts turns gains into losses in both families while the problems, references, and evaluation stay fixed. Teacher profiles and matched loss interventions in Qwen further show that changing token-level supervision can leave student behavior largely unchanged. Together, these findings suggest that OPSD can improve access to existing reasoning capabilities through parameters shared by direct-response and thinking-enabled inference. The value of a privileged reference is what it adds to this cross-mode transfer, not how much of the solution it reveals.
On-Policy Self-Distillation (OPSD) is commonly interpreted as the transfer of privileged information: a teacher observes the verified solution to the target problem and supervises the student's trajectory. However, this interpretation conflates two effects. The reference solution not only reveals the answer to the current instance but also changes the context under which the teacher provides token-level supervision. We investigate the role of target-specific privilege with OP2SD (On-Policy Self-Distillation from Other Problems), which replaces the paired reference with a problem and solution from a different example, while preserving the student rollout, teacher, and distillation objective. Across three models and three mathematics benchmarks, OP2SD improves over the base model, remains competitive with OPSD. The success of OP2SD implies that OPSD gains do not necessarily come from access to the reference solution, and that the teacher's context-induced behavior is an important factor.
Yuki Ichihara, Naoto Iwase, Mohammad Atif Quamar +1
Mohamed bin Zayed University of Artificial Intelligence · Nagoya University · RIKEN AIP
On-policy self-distillation (OPSD) improves reasoning by using a privileged view of a model conditioned on reference solutions to supervise a student view that observes only the question. However, the teacher-provided token-level targets may depend on reference-specific information unavailable at inference time. We propose Problem-Space-Guided OPSD (PS-OPSD), which replaces the complete solution with trajectory-grounded guidance describing the initial state, goal conditions, constraints, and a selected state-transition path. The student rollout and OPSD objective remain unchanged. Across three mathematical reasoning benchmarks and model scales ranging from 1.7B to 8B, PS-OPSD achieves the highest aggregate question-only accuracy among the compared methods. Controlled experiments further indicate that guidance relevance and path coherence contribute to these gains, highlighting the representation of privileged information as an important design choice in OPSD.
Xuyang Zhao, Liting Zhang, Zichen Xu +4
1TMCC, College of Computer Science, Nankai University, Tianjin, China · 2Huawei Technologies Ltd., Beijing, China