On-policy self-distillation (OPSD) improves reasoning by training a model on its own trajectories with dense token-level supervision from a privileged copy conditioned on a reference solution. Existing methods typically use a single teacher distribution conditioned on the complete reference. We identify a limitation of this formulation: a reference is only one realization of the reasoning process. Although it provides evidence about the answer, it also encodes an ordering, phrasing, and solution path. Full-reference conditioning can therefore produce correct targets that depend excessively on information unavailable to the student at inference. We show that replacing one valid reference with another, while holding the problem and student state fixed, can substantially alter the teacher distribution, a phenomenon we call reference-induced target instability. To address this instability, we introduce Monte Carlo Privileged Distillation (MCPD), which models privileged context as a distribution over partial reference views and distills the marginal teacher distribution. MCPD evaluates the teacher under multiple views and averages the resulting next-token distributions. We show that this probability-space average is the exact target induced by minimizing the expected teacher-to-student KL divergence over privileged views. Across mathematical and scientific reasoning benchmarks, we show that MCPD produces more reference-consistent supervision and consistently improves over full-reference OPSD.
Figures & tables
Figure 1: Reference-induced target instability. Left: Distribution of the per-problem sensitivity reduction ΔS=SOPSD−SMCPD (defined in Method) over 500 problems sampled from DeepMath-103K ( He et al. 2026 ) using Qwen3-8B . For each problem, we use two valid reference solutions, generate one unprivileged student rollout, and evaluate both methods at matched prefix positions along the same trajectory. MCPD averages K=4 partial views of each reference; positive values indicate lower reference sensitivity than OPSD. Right: Qualitative example with the problem and student prefix fixed, showing that full-reference OPSD produces more reference-dependent next-token targets, whereas MCPD yields more consistent marginal targets.
Method
AIME24
AIME25
HMMT25
AVG
Qwen3-14B
Base (Instruct)
80.8
68.9
50.0
66.6
+ SFT
79.7
73.9
49.7
67.7
+ GRPO
81.9
73.6
53.0
69.5
+ OPSD
81.1
73.9
52.2
69.1
+ AVSD
80.6
73.6
50.6
68.3
Table 1: Avg@12 accuracy on math benchmarks across four backbones. Best results are in bold. AVG refers to average.
Method
Chem
Physics
Biology
Materials
AVG
Qwen3-8B
41.2
59.2
30.8
58.9
47.5
+ OPSD
44.1
60.4
35.6
58.6
49.7
+ AVSD
44.3
59.2
37.2
60.6
50.3
+ MCPD
44.3
62.2
38.0
59.0
50.9
Table 2: Avg@16 across scientific reasoning tasks using Qwen3-8B. Best results are in bold. AVG refers to average.
Method
Chem
Physics
Biology
Materials
AVG
OPSD
41.2
59.7
31.1
58.7
47.7
AVSD
41.7
59.2
32.9
59.4
48.3
MCPD
42.4
60.5
32.9
60.3
49.0
Table 3: Avg@16 comparison of math-trained Qwen3-8B models with OPSD, AVSD, and our MCPD, on SciKnowEval scientific-reasoning domains. AVG refers to average.
Figure 2: Effect of reference-solution masking on Qwen3-14B. Left: Performance changes relative to OPSD across individual benchmarks. Right: Average performance for OPSD and each masking configuration across AIME24, AIME25, HMMT25, and AMO-Bench. Mix denotes mixed masking ratios of 0% , 5% , 10% , and 30% .
Figure 3: MCPD with different masking ratios on DeepSeek-R1-Distill-Qwen-7B, with the number of views fixed at K=4 . Full results are provided in the Appendix.
Figure 4: Effect of the number of views. Left : Performance changes relative to OPSD on three benchmarks. Right : Average performance across AIME24, AIME25, HMMT25, and AMO-Bench. The masking ratio is fixed at ρ=30% .
Figure 5: MCPD with different numbers of views on DeepSeek-R1-Distill-Qwen-7B with the masking ratio fixed at ρ=40% . Full results table in Appendix.
Method
AIME25
HMMT25
OPSD w/ Sampled-token distillation
82.1
57.3
AVSD w/ Sampled-token distillation
86.7
63.3
MCPD w/ Sampled-token distillation
86.7
66.7
Table 4: Sampled-token distillation results on Qwen3-4B with a maximum student completion length of 2,048 tokens. We report Pass@8 accuracy on AIME25 and HMMT25 as in Zhao et al. (2026) .
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Mean entropy
Paired rate
OPSD
0.34
–
MCPD
0.33
75.6%
Appendix
Table 5: Full-vocabulary target entropy at the matched student states used for reference-sensitivity evaluation. The paired rate is the percentage of problems on which MCPD produces lower problem-level entropy than OPSD.
Method
Training time
OPSD
1h 26m
MCPD (ours)
1h 36m
AVSD
1h 40m
GRPO
12h 30m
Appendix
Table 6: Wall-clock training time on DeepSeek-R1-Distill-Qwen-7B using four NVIDIA A100 80GB GPUs. Times correspond to the method-specific training configurations used in our experiments.
Method
AIME24
AIME25
HMMT25
Avg.
ρ=15%
57.8
40.0
25.6
41.1
ρ=40%
55.0
43.1
25.6
41.2
ρ=50%
55.6
41.9
25.3
40.9
ρ=60%
54.2
41.7
28.6
41.5
ρ=65%
55.8
41.1
26.1
41.0
ρ=70%
55.8
41.9
25.0
40.9
Appendix
Table 7: Complete numerical results underlying Figure 3 of the main paper. We vary the masking ratio ρ on DeepSeek-R1-Distill-Qwen-7B while fixing the number of views to K=4 . The average is computed across AIME24, AIME25, and HMMT25.
Method
AIME24
AIME25
HMMT25
Avg.
K=1
54.2
40.3
26.4
40.3
K=4
55.0
43.1
25.6
41.2
K=7
55.0
42.5
26.1
41.2
K=9
59.2
40.0
25.0
41.4
Appendix
Table 8: Complete numerical results underlying Figure 5 of the main paper. We vary the number of partial reference views K on DeepSeek-R1-Distill-Qwen-7B while fixing the masking ratio to ρ=40% . The average is computed across AIME24, AIME25, and HMMT25.
On-policy self-distillation (OPSD) trains a student on its own reasoning trajectories using token-level supervision from a reference-conditioned teacher initialized from the same model. However, access to the reference can induce teacher behaviors that undermine independent reasoning in thinking models, leading to reference hallucinations, where the student cites an unavailable reference to justify reasoning steps, and reconsideration suppression, where the teacher discourages verification and revision at uncertain steps. We introduce Steered-Teacher OPSD (ST-OPSD) to jointly address these failure modes through activation steering. ST-OPSD extracts two directions from model-generated reasoning trajectories and applies them to the teacher's hidden states during distillation target computation, suppressing reference-invoking behavior and encouraging reconsideration with marker-family calibration. The student learns from the resulting teacher distributions without being steered. Experiments on AIME and HMMT show that ST-OPSD improves accuracy over vanilla OPSD and other interventions while restoring reconsideration and reference-hallucination rates to base-model levels.
On-Policy Self-Distillation (OPSD) is commonly interpreted as the transfer of privileged information: a teacher observes the verified solution to the target problem and supervises the student's trajectory. However, this interpretation conflates two effects. The reference solution not only reveals the answer to the current instance but also changes the context under which the teacher provides token-level supervision. We investigate the role of target-specific privilege with OP2SD (On-Policy Self-Distillation from Other Problems), which replaces the paired reference with a problem and solution from a different example, while preserving the student rollout, teacher, and distillation objective. Across three models and three mathematics benchmarks, OP2SD improves over the base model, remains competitive with OPSD. The success of OP2SD implies that OPSD gains do not necessarily come from access to the reference solution, and that the teacher's context-induced behavior is an important factor.
Yuki Ichihara, Naoto Iwase, Mohammad Atif Quamar +1
Mohamed bin Zayed University of Artificial Intelligence · Nagoya University · RIKEN AIP
On-policy self-distillation (OPSD) trains a student on its own responses using token-level supervision from the same model conditioned on privileged reference information. We investigate whether performance gains from OPSD show that the student learned the information in the reference or instead reflect recovery of reasoning behavior already present in the base model. We perform OPSD experiments on science and mathematics datasets using Qwen3 models ranging from 1.7B to 8B. Our analysis framework separates the supervision induced by the reference from the supervision provided by the teacher without the reference and measures how each aligns with changes in the student's predictions. The correct reference does not provide a consistent performance benefit across teacher generation modes, model sizes, and training datasets. Students can improve without the correct reference, and a solution from another problem can outperform the correct solution on several mathematical reasoning benchmarks. The student's predictions align more strongly with the base model's thinking behavior than with the supervision induced by the reference, but controls constructed from other problems reproduce much of both alignments. Moreover, stronger alignment attributable to the correct reference does not reliably coincide with a greater performance benefit from the reference. Performance gains and distributional alignment alone therefore cannot determine how privileged reference information contributes to student learning in OPSD.