On-policy self-distillation enables the same model to act as student and teacher, with dense token-level supervision obtained by the teacher being given access to privileged information. However, conditioning the teacher on a single reference can concentrate its predictions on one reasoning trajectory and limit transfer to the student. We propose Multi-Mask Selective Self-Distillation (MASS), which approximates the unavailable marginal over diverse reference realizations using multiple independently masked views of one verified solution. MASS aggregates these complementary teacher predictions, selectively transfers reliable token-level corrections, and adaptively focuses masking on influential reference spans. Across mathematical reasoning benchmarks and model families, MASS consistently outperforms strong self-distillation baselines.
Figures & tables
Figure 1: PI Bias Scores for the in-context solution and a distinct correct solution under the full-reference teacher, a single masked view, and the arithmetic mean of multiple masked views.
Figure 2: Overview of MASS. Multiple masked reference provide complementary teacher predictions over the same student trajectory and are averaged into a marginal target. MASS selectively distills this target using reliability-aware weighting. In parallel, anchor-relative coalition utilities are decomposed into span-level attributions, which adapt future masking probabilities.
Model
Method
AIME24
AIME25
HMMT25
Avg.
Qwen3-1.7B
Base
51.39
36.67
23.06
37.04
OPSD
55.56
41.11
26.94
41.20
AVSD
59.17
45.28
29.72
44.72
RLSD
58.89
45.83
30.00
44.91
MASS
60.83
47.50
30.83
46.39
Qwen3-4B
Base
75.00
66.39
42.22
61.20
Table 1: Performance comparison on mathematical reasoning benchmarks. We report Avg@12 accuracy on AIME 2024, AIME 2025, and HMMT 2025. MASS consistently achieves the strongest average performance across all evaluated model settings.
Figure 3: Performance Dynamics over 250 Training Steps. We compare MASS with OPSD and RLSD on Qwen3-4B reporting Avg@12 accuracy on AIME25 within 250 training steps. OPSD peaks early, while RLSD and MASS sustain improvement throughout training.
Model
Target
Acc.
Qwen3-4B
Masked avg. (Base)
73.89
Masked avg. + anchor
72.50
OLMo-7B
Masked avg. (Base)
75.00
Masked avg. + anchor
73.89
Table 2: Ablation on the anchor teacher. AIME 2025 Avg@12 accuracy is reported.
Figure 4: Reasoning behavior during self-distillation. We measured average response length and epistemic token usage for (a) Qwen3-8B and (b) OLMo-7B on AIME25 evaluation.
Figure 5: Sensitivity to the number of teacher routes. Increasing the number of routes improves AIME 2025 Avg@12 accuracy on Qwen3-1.7B.
Figure 6: Sensitivity to masking strength. On Qwen3-4B, a moderate masking ratio gives the best AIME 2025 Avg@12 accuracy.
Variant
Weighting
Adaptive Masking
AIME25
Neither
✗
✗
70.83
Weighting only
✓
✗
72.22
Adaptive only
✗
✓
72.78
Full MASS
✓
✓
73.89
Table 3: Ablation of reliability weighting and adaptive masking. Both components improve AIME 2025 performance on Qwen3-4B, with the full configuration achieving the strongest result.
Model
Method
GPQA
BBH
Qwen3-8B
Base
57.7
84.4
OPSD
54.8
82.9
AVSD
56.6
83.7
MASS
58.3
84.6
OLMo3-7B-Think
Base
46.2
86.6
OPSD
43.9
85.0
Table 4: OOD evaluation. GPQA Avg@8 and BBH Avg@1 accuracy are reported.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Model
ϵ=0.01
ϵ=0.03
ϵ=0.05
ϵ=0.10
Qwen3-8B
79.08
79.44
79.17
78.61
OLMo-7B
74.62
75.00
74.72
73.89
Appendix
Table 5: Sensitivity analysis of the masking probability clipping We report Avg@12 performance on AIME 2025. All experiments use the same MASS training configuration except for the clipping threshold.
Distillation Objective
AIME24
AIME25
Backward KL
79.28
72.50
Forward KL (MASS)
81.11
73.89
Appendix
Table 6: Effect of KL divergence direction on Qwen3-4B. Results are reported as Avg@12 accuracy on AIME 2024 and AIME 2025.
Figure 7: Teacher prompts in OPSD and MASS. OPSD uses the complete reference solution, while MASS constructs multiple teacher views from independently masked versions of the same reference; one example is shown. Labels [S1] – [S5] and the surrounding braces are added only for visualization to clarify the span boundaries and are not included in the actual prompts. During training, selected spans are directly replaced by [MASK] .
Method
FLOPs / Step
Relative to OPSD
OPSD
1.31×1015
1.00×
MASS
3.41×1015
2.60×
RLSD
8.39×1015
6.40×
Appendix
Table 7: Estimated training computation per optimization step. FLOPs are normalized by OPSD.
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.
On-policy self-distillation trains reasoning models on their own trajectories using dense token distribution guidance from a privileged teacher with access to a verified solution. This supervision transfers what the teacher predicts without directly transferring where it attends within the preceding context. We introduce On-Policy Attention Self-Distillation (OPASD), which complements token-level supervision with solution-conditioned attention distillation. Because the privileged teacher can attend to verified solution tokens unavailable to the student, OPASD projects teacher attention onto student-visible positions and renormalizes the resulting distribution before alignment. Across three model sizes and four competition-level mathematics benchmarks, OPASD consistently outperforms token-only OPSD, improving average accuracy by 4.98 to 8.40 percentage points. OPASD also avoids the response-length inflation and performance degradation observed with token-only distillation, reducing generated rollout tokens by 73.9% and estimated model compute by 72.6% while training 1.53x faster. These results show that solution-conditioned attention provides a complementary supervision signal that makes on-policy self-distillation more accurate, stable, and compute-efficient.
Safaeid Hossain Arib, Rabeya Akter, Ismam Nur Swapnil +3
ACI PLC, Bangladesh · University of Dhaka, Bangladesh · BRAC University, Bangladesh +2
On-policy self-distillation, where a student is pulled toward a copy of itself conditioned on privileged context (e.g., a verified solution or feedback), offers a promising direction for advancing reasoning capability without a stronger external teacher. Yet in math reasoning the gains are inconsistent, even when the same approach succeeds elsewhere. A pointwise mutual information analysis traces the failure to the privileged context itself: it inflates the teacher's confidence on tokens already implied by the solution (structural connectives, verifiable claims) and deflates it on deliberation tokens ("Wait", "Let", "Maybe") that drive multi-step search. We propose Anti-Self-Distillation (AntiSD), which ascends a divergence between student and teacher rather than descending it: this reverses the per-token sign and yields a naturally bounded advantage in one step. An entropy-triggered gate disables the term once the teacher entropy collapses, completing a drop-in replacement for default self-distillation. Across five models from 4B to 30B parameters on math reasoning benchmarks, AntiSD reaches the GRPO baseline's accuracy in 2 to 10x fewer training steps and improves final accuracy by up to 11.5 points. AntiSD opens a path to scalable self-improvement, where a language model bootstraps its own reasoning through its training signal.
Guobin Shen, Xiang Cheng, Chenxiao Zhao +4
1Xiaohongshu Inc. · Institute of Automation, Chinese Academy of Sciences
On-policy self-distillation (OPSD) provides dense token-level supervision without a second model: one network acts as teacher with the reference solution and as student with only the problem. We identify a specific failure mode of this recipe. During training, student token entropy rises past the teacher's and remains elevated, a pattern we call entropy overshoot. We trace it to both sides of distillation. The reference-conditioned teacher is confident along its answer-directed reasoning path, but this confidence transfers poorly to student-generated prefixes, making its supervision overly tied to answer-specific cues rather than reusable reasoning patterns; meanwhile, the forward KL used by OPSD continually diffuses the student's predictive distribution without pulling it back. We introduce E2-OPSD to address both causes. Exemplar-guided teaching replaces the current answer with a retrieved solved neighboring problem, providing transferable reasoning guidance without revealing the destination and better matching student-reachable states. Entropy-aware distillation uses the student-teacher entropy gap to determine the direction and strength of each token's correction. E2-OPSD improves math reasoning by up to 4.3 points in mean@16 over OPSD, while out-of-domain evaluations show gains over the corresponding base models of up to 4.9 points in mean@16 and 5.5 points in pass@8. Despite these gains, E2-OPSD remains simple, requiring no additional forward passes or networks. Code repo: https://github.com/orionfei/E2-OPSD.
Yifei Liu, Minghao Fang, Xinyu Gu +6
The Chinese University of Hong Kong · Zhejiang University · Shanghai Artificial Intelligence Laboratory +5