Self-distillation has emerged as an effective post-training paradigm for LLMs, often improving performance while shortening reasoning traces. However, in mathematical reasoning, we find that it can reduce response length while degrading performance. We trace this degradation to the suppression of epistemic verbalization - the model's expression of uncertainty during reasoning. Through controlled experiments varying conditioning context richness and task coverage, we show that conditioning the teacher on rich information suppresses uncertainty expression, enabling rapid in-domain optimization with limited task coverage but harming OOD performance, where unseen problems benefit from expressing uncertainty and adjusting accordingly. Across Qwen3-1.7B/8B, DeepSeek-Distill-Qwen-7B, and Olmo3-7B-Instruct, we observe performance drops of up to 40%. Our findings highlight that exposing appropriate levels of uncertainty is crucial for robust reasoning and underscore the importance of optimizing reasoning behavior beyond merely reinforcing correct answer traces.
Figures & tables
Figure 1: (a) Training score and response length changes for GRPO and Self-Distillation (SDPO) ( Hübotter et al., 2026 ) in Chemistry, using results from SDPO Wandb logs ( Biewald, 2020 ) (link) . (b) Training score and response length changes on DAPO-Math-17k with GRPO and SDPO.
Figure 2
Prompt for unguided generation
{question} Please reason step by step, and put your final answer within \boxed{}.
Regeneration prompt (followed the prompt in Hübotter et al. (2026) )
{question} Please reason step by step, and put your final answer within \boxed{}. Correct solution: {previously correct solution} Correctly solve the original question.
Table 3
Avg. Score
Avg. Length
Epistemic Token Count
(1) Unguided
0.30
13,054
182.5
(2) Solution-Guided ( c=s )
0.98
1,873
8.8
(3) Solution-Guided ( c=s∖think )
0.78
12,036
159.8
(4) Regeneration-Conditioned
0.95
2,808
24.1
Table 1: Comparison of response characteristics under varying levels of rich information.
DeepSeek-R1-Distill-Qwen-7B
AIME24
AIME25
AMC23
MATH500
Base
54.79
37.92
89.06
92.19
SFT on Dug
51.04
40.00
87.66
90.93
SFT on Dsg
20.21
12.71
57.03
65.52
Table 2: Math benchmark performance of the base model DeepSeek-R1-Distill-Qwen-7B and its SFT checkpoints trained on unguided and solution-guided datasets.
Figure 3: On-policy self-distillation results for DeepSeek-R1-Distill-Qwen-7B. GRPO yields modest OOD gains with a slight increase in epistemic verbalization, whereas SDPO degrades both performance and epistemic token usage, particularly with c=s .
Figure 4: On-policy self-distillation results for Qwen3-8B (Thinking Mode: ON). Both GRPO and SDPO reduce response length and epistemic verbalization, but SDPO’s more aggressive suppression leads to greater degradation in OOD performance, particularly on AIME24.
Figure 5: On-policy self-distillation results for Qwen3-8B (Thinking Mode: OFF). GRPO rapidly increases response length via epistemic verbalization and achieves strong training gains, while SDPO reduces response length and struggles to improve, with slight OOD degradation on AIME24.
Figure 6: Effect of epistemic-token suppression during GRPO for DeepSeek-Distill-7B.
Figure 7: Fixed vs. moving target teacher for DeepSeek-R1-Distill-Qwen-7B. Even slow EMA updates (rate 0.05) amplify epistemic suppression via a feedback loop, causing greater performance degradation than a fixed teacher.
Train/Eval split: Total 131 questions for training and all 131 questions are used for both training and evaluation. Only 50% of the public test cases are used during training, while the full set including hidden test cases is used for evaluation.
DAPO-Math-17k
Train/Eval split: Total 14,000 questions for training and evaluation is conducted on standard benchmarks (AIME24/25, AMC23, MATH 500) using questions not part of the training data.
Table 3: Comparison of the total number of problems, problem composition, and train–evaluation splits in ScienceQ&A, LiveCodeBench v6, and our experiments.
Figure 8: Training score and response length comparison between GRPO and SDPO for ∣D∣∈{1,8,64,128,512} .
Figure 9: Evaluation performance on AIME24 and MATH500 and response length as ∣D∣ varies over {1, 8, 64, 128, 512}.
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
Rank
Qwen3 ( n=55,743 )
QwQ ( n=10,817 )
DeepSeek ( n=27,386 )
OLMo ( n=4,948 )
1
maybe (37.9%)
maybe (34.3%)
maybe (23.5%)
perhaps (31.4%)
2
let me check (12.3%)
hmm (16.0%)
hmm (19.5%)
maybe (19.9%)
3
wait (12.1%)
wait (12.3%)
I think (13.0%)
wait (18.4%)
4
hmm (8.6%)
let me check (8.6%)
wait (10.6%)
likely (5.1%)
5
I think (7.0%)
perhaps (7.2%)
perhaps (6.2%)
actually (3.1%)
6
let me think (4.4%)
I think (5.0%)
double- check (4.3%)
double- check (3.1%)
Appendix
Table 4: Top-15 uncertainty trigger tokens by frequency across model families. Percentages indicate the proportion of extracted trigger-token occurrences within each family.
Category
Parameter
Value
Data
Max. prompt length
2048
Max. response length
20480
Batching
Question batch size
256
Mini batch size
64 or 128
Number of rollouts
8
Rollout
Inference engine
vllm
Appendix
Table 5: Common hyperparameters shared by GRPO and SDPO.
Figure 10: GRPO and SDPO with the maximum response length restricted to 4,096 tokens.
Figure 11: Ablation study on top- k distillation and learning rate. Both modifications fail to prevent the convergence toward degraded reasoning behavior; a lower learning rate only delays the process.
Figure 12: Per-token breakdown of epistemic token usage across the four generation settings. Each bar represents the average number of occurrences per response for an individual epistemic token. All tokens follow the same ordering as the aggregate trend, with wait , maybe , and perhaps showing the largest variation across settings.
Figure 13: Comparison of epistemic token usage across DeepSeek-R1-Distill-Qwen-7B, Qwen3-8B (thinking enabled), and Qwen3-8B (thinking disabled).
Figure 14: Change in epistemic token counts relative to the base model on AIME24 for GRPO and SDPO with ∣D∣∈{1,64,512} .
Figure 15: Comparison of per-word frequency shifts ( Δm(w) ) for epistemic tokens ( T ) versus non-epistemic tokens ( V∖T ) in GRPO- and SDPO-trained models relative to the base model. Epistemic tokens exhibit disproportionately large frequency changes compared to the general vocabulary, indicating that the training procedures specifically amplify or suppress epistemic expressions rather than inducing uniform changes across the word distribution.
Uncertainty Expression
“maybe there’s a simpler way”
“Wait, hold on. Is that correct?”
“I think the key here is to find s and t first, right?”
“I’m not entirely sure”
“perhaps there’s a relationship here”
“Hmm, that seems contradictory”
“perhaps I made a miscalculation”
“it seems a bit tricky”
“Wait, but this might get complicated”
“it seems less likely”
“maybe we can assume some symmetry”
“Alternatively, perhaps it’s better
Appendix
Table 10: Representative examples of uncertainty expressions identified by GPT-5.4. Expressions are drawn from model-generated reasoning traces across all evaluated methods.
Table 27
Token
Variant
Mean KL
Count
Ratio
hmm
‘ Hmm’
0.516
542
6.8×
‘Hmm’
0.337
887
4.4×
maybe
‘Maybe’
0.432
52
5.7×
‘ Maybe’
0.192
937
2.5×
‘ maybe’
0.173
4,676
2.3×
actually
‘ actually’
0.188
1,134
2.5×
Appendix
Table 12: KL divergence between student and self-teacher for epistemic tokens T . The corpus-wide mean KL across all 1,642 frequent tokens is 0.076 .
Model
Unguided
Solution-guided
DeepSeek-R1-Distill-Qwen-7B
0.385
0.218
Appendix
Table 13: Token-level entropy under unguided and solution-guided generation.
Model
Initial
GRPO
SDPO
DeepSeek-R1-Distill-Qwen-7B
0.378
0.401
0.248
Qwen3-8B (Thinking Mode: OFF)
0.100
0.291
0.099
Appendix
Table 14: Token-level entropy before and after GRPO and SDPO training.
Figure 17: Average token-level entropy during GRPO and SDPO training.
Prompt for unguided generation
{question} Please reason step by step, and put your final answer within \boxed{}.
Regeneration prompt (followed the prompt in Hübotter et al. (2026) )
{question} Please reason step by step, and put your final answer within \boxed{}. Correct solution: {previously correct solution} Correctly solve the original question.
Regeneration prompt with encouraging uncertainty
{question} Please reason step by step, and put your final answer within \boxed{}. Correct solution: {previously correct solution} Solve the original question from scratch with a full, detailed reasoning trace, without skipping any steps, and include as much uncertainty or ambiguity in the explanation.
Appendix
Table 32
Avg. Length
Epistemic Token Count
Unguided
13,054
182.5
Solution-Guided (vanilla)
1,873
8.8
Solution-Guided (w/ encouraged uncertainty)
3,098
31.4
Appendix
Table 15: Response characteristics of solution-guided generation with and without the uncertainty-encouraging prompt.
Figure 18: Training dynamics and AIME24 evaluation of on-policy self-distillation using the uncertainty-encouraging regeneration prompt. (a) Training score and response length over training steps. (b) AIME24 Acc@16 and response length over training steps. While the uncertainty-encouraging prompt partially mitigates response shortening, the overall performance degradation persists.
Figure 19: Training dynamics of OPSD hybrid distillation on Qwen3-1.7B compared with our homogeneous chat-template setup. (a) In the hybrid setup, the thinking-enabled teacher initially improves student performance, but the gains diminish over time, whereas the homogeneous setup shows a consistent decline. (b) Response length and epistemic token usage under the homogeneous setup.
Figure 20: Extended results for Qwen3-8B (thinking mode off): training score-length trade-off, evaluation performance on AMC23 and AIME24, and change in epistemic token usage.
Figure 21: GRPO and SDPO results on Qwen3-1.7B (Thinking Mode: ON): training score and response length, and OOD evaluation on AIME24. Self-distillation consistently reduces response length but leads to a substantial drop in performance, with a more pronounced degradation compared to larger models.
Figure 22: SDPO results on OLMo-3-7B-Instruct: training score and response length, and OOD evaluation on AIME24.
Figure 23: Pass@16 on AMC23 and AIME24 for GRPO and SDPO across training steps.
On-policy self-distillation (OPSD) is an emerging LLM post-training paradigm in which the model serves as its own teacher: conditioned on privileged information such as a reference trace or hint, the same policy provides dense token-level supervision on its own rollouts. However, recent studies show that OPSD degrades complex reasoning by suppressing predictive uncertainty, which supports exploration and hypothesis revision. Our token-level analysis shows that this failure arises from applying a uniform direction of teacher supervision across tokens with different uncertainty levels: conformity to the privileged self-teacher suppresses exploration at high entropy, while deviation from the teacher degrades step accuracy at low entropy. Accordingly, we propose Direction-Adaptive Self-Distillation (DASD), which reframes privileged self-distillation from uniform teacher imitation into entropy-routed directional supervision: high-entropy tokens are pushed away from the privileged teacher to preserve exploration, while low-entropy tokens are pulled toward the teacher to stabilize step-level execution. Across six mathematical reasoning benchmarks, DASD achieves the best macro Avg@16 over strong RLVR and self-distillation baselines. Pass@k, reasoning-health, and generalization analyses show that these average gains come from preserving exploration without sacrificing step-level execution.
Hongbin Zhang, Chaozheng Wang, Kehai Chen +4
Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China · Keeta AI, Meituan, Beijing, China · Peng Cheng Laboratory, Shenzhen, China
On-Policy Self-Distillation (OPSD) has emerged as a popular paradigm for large language model (LLM) self-improvement, allowing models to act as their own teachers by leveraging privileged information such as ground-truth solutions. However, recent findings indicate that OPSD can severely degrade the performance of LLMs on complex reasoning tasks: By forcing the student to imitate an artificially confident reasoning trace conditioned on privileged information, OPSD inadvertently suppresses expressions of uncertainty and penalizes the exploratory, self-corrective behaviors required to solve challenging problems. To address this, we introduce Negative Self-Distillation (NSD), a new framework that optimizes LLMs by diverging from flawed reasoning rather than imitating privileged solutions. Instead of relying on ground-truth answers or external supervision, NSD uses the model itself to generate a question-specific negative condition (eg, acting as a ``careless reasoner'') and pushes the student's distribution away from this self-generated negative teacher. Naively applying unlearning objectives to achieve this divergence is problematic, as flawed reasoning tokens are confounded with basic linguistic tokens; indiscriminately penalizing both risks catastrophically degrading the model's foundational language capabilities. We resolve this by designing a dynamic gating mechanism that automatically identifies and isolates reasoning-critical tokens, ensuring gradient updates target only behavioral flaws while preserving the model's linguistic priors. Empirically, NSD consistently outperforms OPSD and other label-free, self-bootstrapping reinforcement learning (RL) baselines.
Rongcan Pei, Zhepei Wei, Shuyao Xu +3
Department of Computer Science, University of Virginia · Stanford University
Self-distillation is an emerging approach that trains a model to imitate its own predictions when conditioned on privileged information, such as correct answers or environmental feedback. Recent methods have reported better sample efficiency and generalization than reinforcement learning (RL), but it remains unclear whether these gains extend beyond in-distribution evaluations and whether they stem from privileged distillation. In this paper, we revisit two representative methods, Self-Distillation Policy Optimization (SDPO) and On-Policy Self-Distillation (OPSD), across code, science, tool use, and math reasoning tasks. We find that their reported gains do not consistently generalize to unseen problems and are largely explained by overfitting or transferring existing reasoning behavior, rather than effectively using privileged information. Moreover, self-distillation gradients show weak alignment with policy gradients that directly optimize expected reward, questioning whether they provide an effective direction for policy improvement. Our experiments across Qwen3 models (1.7B-14B) and OLMo-3-7B suggest that effectively leveraging privileged information to improve reasoning remains an open challenge. Our code is available at https://github.com/sleepypandu/does-sd-improve-reasoning
Jason Ken Adhinarta, Raina Wu, Ashutosh Sharma +5
Massachusetts Institute of Technology · MIT-IBM Computing Research Lab