Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?

Jeonghye Kim; Xufang Luo; Minbeom Kim; Sangmook Lee; Dohyung Kim; Jiwon Jeon; Dongsheng Li; Yuqing Yang

cs.CLMar 25, 2026

Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?

Authors: Jeonghye Kim, Xufang Luo, Minbeom Kim, Sangmook Lee, Dohyung Kim, Jiwon Jeon, Dongsheng Li, Yuqing Yang

Organizations: Microsoft Research · KAIST · SK hynix · Seoul National University

Abstract

Self-distillation has emerged as an effective post-training paradigm for LLMs, often improving performance while shortening reasoning traces. However, in mathematical reasoning, we find that it can reduce response length while degrading performance. We trace this degradation to the suppression of epistemic verbalization - the model's expression of uncertainty during reasoning. Through controlled experiments varying conditioning context richness and task coverage, we show that conditioning the teacher on rich information suppresses uncertainty expression, enabling rapid in-domain optimization with limited task coverage but harming OOD performance, where unseen problems benefit from expressing uncertainty and adjusting accordingly. Across Qwen3-1.7B/8B, DeepSeek-Distill-Qwen-7B, and Olmo3-7B-Instruct, we observe performance drops of up to 40%. Our findings highlight that exposing appropriate levels of uncertainty is crucial for robust reasoning and underscore the importance of optimizing reasoning behavior beyond merely reinforcing correct answer traces.

Figures & tables

Panel of Figure 1 — Figure 1: (a) Training score and response length changes for GRPO and Self-Distillation (SDPO) ( Hübotter et al., 2026 ) in Chemistry, using results from SDPO Wandb logs ( Biewald, 2020 ) (link) . (b) Training score and response length changes on DAPO-Math-17k with GRPO and SDPO.

Prompt for unguided generation	{question} Please reason step by step, and put your final answer within \boxed{}.
Regeneration prompt (followed the prompt in Hübotter et al. (2026) )	{question} Please reason step by step, and put your final answer within \boxed{}. Correct solution: {previously correct solution} Correctly solve the original question.

Table 3

	Avg. Score	Avg. Length	Epistemic Token Count
(1) Unguided	0.30	13,054	182.5
(2) Solution-Guided ( $c=s$ )	0.98	1,873	8.8
(3) Solution-Guided ( $c=s_{\setminus\text{think}}$ )	0.78	12,036	159.8
(4) Regeneration-Conditioned	0.95	2,808	24.1

Table 1: Comparison of response characteristics under varying levels of rich information.

DeepSeek-R1-Distill-Qwen-7B	AIME24	AIME25	AMC23	MATH500
Base	54.79	37.92	89.06	92.19
SFT on $\mathcal{D}_{\text{ug}}$	51.04	40.00	87.66	90.93
SFT on $\mathcal{D}_{\text{sg}}$	20.21	12.71	57.03	65.52

Table 2: Math benchmark performance of the base model DeepSeek-R1-Distill-Qwen-7B and its SFT checkpoints trained on unguided and solution-guided datasets.

Panel of Figure 3 — Figure 3: On-policy self-distillation results for DeepSeek-R1-Distill-Qwen-7B. GRPO yields modest OOD gains with a slight increase in epistemic verbalization, whereas SDPO degrades both performance and epistemic token usage, particularly with $c=s$ .

Panel of Figure 4 — Figure 4: On-policy self-distillation results for Qwen3-8B (Thinking Mode: ON). Both GRPO and SDPO reduce response length and epistemic verbalization, but SDPO’s more aggressive suppression leads to greater degradation in OOD performance, particularly on AIME24.

Panel of Figure 5 — Figure 5: On-policy self-distillation results for Qwen3-8B (Thinking Mode: OFF). GRPO rapidly increases response length via epistemic verbalization and achieves strong training gains, while SDPO reduces response length and struggles to improve, with slight OOD degradation on AIME24.

Panel of Figure 6 — Figure 6: Effect of epistemic-token suppression during GRPO for DeepSeek-Distill-7B.

Panel of Figure 7 — Figure 7: Fixed vs. moving target teacher for DeepSeek-R1-Distill-Qwen-7B. Even slow EMA updates (rate 0.05) amplify epistemic suppression via a feedback loop, causing greater performance degradation than a fixed teacher.

Domain	Dataset Analysis
ScienceQ&A (Chemistry)	Total 2,400 questions: Reaction Balancing (300), Molecular Descriptor Counting (300), Molecular Weight Calculation (600), Property Prediction (e.g., logS) (500), Precursor/Reactant Selection (Retrosynthesis) (300), Product Prediction (Organic Reactions) (400). Train/Eval split: 90% for training, 10% for evaluation.
LiveCodeBench v6	Train/Eval split: Total 131 questions for training and all 131 questions are used for both training and evaluation. Only 50% of the public test cases are used during training, while the full set including hidden test cases is used for evaluation.
DAPO-Math-17k	Train/Eval split: Total 14,000 questions for training and evaluation is conducted on standard benchmarks (AIME24/25, AMC23, MATH 500) using questions not part of the training data.

Table 3: Comparison of the total number of problems, problem composition, and train–evaluation splits in ScienceQ&A, LiveCodeBench v6, and our experiments.

Figure 8: Training score and response length comparison between GRPO and SDPO for

|D|\in\{1,8,64,128,512\}

Figure 9: Evaluation performance on AIME24 and MATH500 and response length as

|D|

varies over {1, 8, 64, 128, 512}.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Rank	Qwen3 ( $n=55{,}743$ )	QwQ ( $n=10{,}817$ )	DeepSeek ( $n=27{,}386$ )	OLMo ( $n=4{,}948$ )
1	maybe (37.9%)	maybe (34.3%)	maybe (23.5%)	perhaps (31.4%)
2	let me check (12.3%)	hmm (16.0%)	hmm (19.5%)	maybe (19.9%)
3	wait (12.1%)	wait (12.3%)	I think (13.0%)	wait (18.4%)
4	hmm (8.6%)	let me check (8.6%)	wait (10.6%)	likely (5.1%)
5	I think (7.0%)	perhaps (7.2%)	perhaps (6.2%)	actually (3.1%)
6	let me think (4.4%)	I think (5.0%)	double- check (4.3%)	double- check (3.1%)

Appendix

Table 4: Top-15 uncertainty trigger tokens by frequency across model families. Percentages indicate the proportion of extracted trigger-token occurrences within each family.

Category	Parameter	Value
Data	Max. prompt length	2048
	Max. response length	20480
Batching	Question batch size	256
	Mini batch size	64 or 128
	Number of rollouts	8
Rollout	Inference engine	vllm

Appendix

Table 5: Common hyperparameters shared by GRPO and SDPO.

Category	Parameter	Value
Loss	$\epsilon$ -high	0.28
	Rollout IS clip	2
	KL coefficient ( $\lambda$ )	0.0
Training	Learning rate	$1\times 10^{-6}$

Appendix

Table 6: GRPO-specific hyperparameters.

Category	Parameter	Value
Loss	Distillation divergence	Jensen–Shannon
	Top- $k$ distillation	100
	EMA update rate	0.0
Training	Learning rate	$1\times 10^{-5}$

Appendix

Table 7: SDPO-specific hyperparameters.

Model	Max Tokens	Temp.	Top- $p$	Top- $K$
DeepSeek-R1-Distill-7B	38912	0.6	0.95	20
Qwen3-8B (thinking)	38912	0.6	0.95	20
Qwen3-8B (non-thinking)	38912	0.7	0.8	20
OLMo-3-7B-Instruct	38912	0.6	0.95	20

Appendix

Table 8: Evaluation hyperparameters.

Model	Template
DeepSeek-R1-Distill-7B	<\|begin_of_sentence\|><\|User\|> {prompt} <\|Assistant\|><think>
Qwen3-8B (think ON)	<\|im_start\|>user\n {prompt} <\|im_end\|> <\|im_start\|>assistant
Qwen3-8B (think OFF)	<\|im_start\|>user\n {prompt} <\|im_end\|> <\|im_start\|>assistant\n<think>\n</think>
OLMo-3-7B-Instruct	<\|im_start\|>system\n {sys} <\|im_end\|> <\|im_start\|>user\n {prompt} <\|im_end\|>\n<\|im_start\|>assistant

Appendix

Table 9: Chat template formats by model series.

Figure 10: GRPO and SDPO with the maximum response length restricted to 4,096 tokens.

Panel of Figure 11 — Figure 11: Ablation study on top- $k$ distillation and learning rate. Both modifications fail to prevent the convergence toward degraded reasoning behavior; a lower learning rate only delays the process.

Figure 12: Per-token breakdown of epistemic token usage across the four generation settings. Each bar represents the average number of occurrences per response for an individual epistemic token. All tokens follow the same ordering as the aggregate trend, with wait , maybe , and perhaps showing the largest variation across settings.

Figure 13: Comparison of epistemic token usage across DeepSeek-R1-Distill-Qwen-7B, Qwen3-8B (thinking enabled), and Qwen3-8B (thinking disabled).

Figure 14: Change in epistemic token counts relative to the base model on AIME24 for GRPO and SDPO with

|D|\in\{1,64,512\}

Figure 15: Comparison of per-word frequency shifts (

\Delta_{m}(w)

) for epistemic tokens (

T

) versus non-epistemic tokens (

\mathcal{V}\setminus T

) in GRPO- and SDPO-trained models relative to the base model. Epistemic tokens exhibit disproportionately large frequency changes compared to the general vocabulary, indicating that the training procedures specifically amplify or suppress epistemic expressions rather than inducing uniform changes across the word distribution.

Uncertainty Expression
“maybe there’s a simpler way”	“Wait, hold on. Is that correct?”
“I think the key here is to find $s$ and $t$ first, right?”	“I’m not entirely sure”
“perhaps there’s a relationship here”	“Hmm, that seems contradictory”
“perhaps I made a miscalculation”	“it seems a bit tricky”
“Wait, but this might get complicated”	“it seems less likely”
“maybe we can assume some symmetry”	“Alternatively, perhaps it’s better

Appendix

Table 10: Representative examples of uncertainty expressions identified by GPT-5.4. Expressions are drawn from model-generated reasoning traces across all evaluated methods.

Token	Variant	Mean KL	Count	Ratio
hmm	‘ Hmm’	0.516	542	$6.8\times$
hmm	‘Hmm’	0.337	887	$4.4\times$
maybe	‘Maybe’	0.432	52	$5.7\times$
	‘ Maybe’	0.192	937	$2.5\times$
	‘ maybe’	0.173	4,676	$2.3\times$
actually	‘ actually’	0.188	1,134	$2.5\times$

Appendix

Table 12: KL divergence between student and self-teacher for epistemic tokens

\mathcal{T}

. The corpus-wide mean KL across all 1,642 frequent tokens is

0.076

Model	Unguided	Solution-guided
DeepSeek-R1-Distill-Qwen-7B	0.385	0.218

Appendix

Table 13: Token-level entropy under unguided and solution-guided generation.

Model	Initial	GRPO	SDPO
DeepSeek-R1-Distill-Qwen-7B	0.378	0.401	0.248
Qwen3-8B (Thinking Mode: OFF)	0.100	0.291	0.099

Appendix

Table 14: Token-level entropy before and after GRPO and SDPO training.

Figure 17: Average token-level entropy during GRPO and SDPO training.

Prompt for unguided generation	{question} Please reason step by step, and put your final answer within \boxed{}.
Regeneration prompt (followed the prompt in Hübotter et al. (2026) )	{question} Please reason step by step, and put your final answer within \boxed{}. Correct solution: {previously correct solution} Correctly solve the original question.
Regeneration prompt with encouraging uncertainty	{question} Please reason step by step, and put your final answer within \boxed{}. Correct solution: {previously correct solution} Solve the original question from scratch with a full, detailed reasoning trace, without skipping any steps, and include as much uncertainty or ambiguity in the explanation.

Appendix

Table 32

	Avg. Length	Epistemic Token Count
Unguided	13,054	182.5
Solution-Guided (vanilla)	1,873	8.8
Solution-Guided (w/ encouraged uncertainty)	3,098	31.4

Appendix

Table 15: Response characteristics of solution-guided generation with and without the uncertainty-encouraging prompt.

Panel of Figure 18 — Figure 18: Training dynamics and AIME24 evaluation of on-policy self-distillation using the uncertainty-encouraging regeneration prompt. (a) Training score and response length over training steps. (b) AIME24 Acc@16 and response length over training steps. While the uncertainty-encouraging prompt partially mitigates response shortening, the overall performance degradation persists.

Panel of Figure 19 — Figure 19: Training dynamics of OPSD hybrid distillation on Qwen3-1.7B compared with our homogeneous chat-template setup. (a) In the hybrid setup, the thinking-enabled teacher initially improves student performance, but the gains diminish over time, whereas the homogeneous setup shows a consistent decline. (b) Response length and epistemic token usage under the homogeneous setup.

Panel of Figure 20 — Figure 20: Extended results for Qwen3-8B (thinking mode off): training score-length trade-off, evaluation performance on AMC23 and AIME24, and change in epistemic token usage.

Panel of Figure 21 — Figure 21: GRPO and SDPO results on Qwen3-1.7B (Thinking Mode: ON): training score and response length, and OOD evaluation on AIME24. Self-distillation consistently reduces response length but leads to a substantial drop in performance, with a more pronounced degradation compared to larger models.

Panel of Figure 22 — Figure 22: SDPO results on OLMo-3-7B-Instruct: training score and response length, and OOD evaluation on AIME24.

Panel of Figure 23 — Figure 23: Pass@16 on AMC23 and AIME24 for GRPO and SDPO across training steps.

Self-Distillation

Self-distillation trains a model from its own predictions, representations, or experience to improve capability, efficiency, and robustness.

See all 137 papers

Tailoring Teaching to Aptitude: Direction-Adaptive Self-Distillation for LLM Reasoning
May 21, 2026Hongbin Zhang, Chaozheng Wang, Kehai Chen +4On-Policy Self-Distillation Mathematical Reasoning
Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China · Keeta AI, Meituan, Beijing, China · Peng Cheng Laboratory, Shenzhen, China
Negative Self-Distillation: Learning to Reason by Avoiding Flaws
Sep 10, 2026Rongcan Pei, Zhepei Wei, Shuyao Xu +3Language Model Distillation Self-Distillation
Department of Computer Science, University of Virginia · Stanford University
Does self-distillation improve reasoning?
Oct 13, 2026Jason Ken Adhinarta, Raina Wu, Ashutosh Sharma +5Self-Distillation On-Policy Self-Distillation
Massachusetts Institute of Technology · MIT-IBM Computing Research Lab

May 21, 2026J/K move · Enter open · S save

Tailoring Teaching to Aptitude: Direction-Adaptive Self-Distillation for LLM Reasoning

Hongbin Zhang, Chaozheng Wang, Kehai Chen, Youcheng Pan+3

Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China · Keeta AI, Meituan, Beijing, China · Peng Cheng Laboratory, Shenzhen, China

On-Policy Self-Distillation Mathematical Reasoning Self-Distillation Teacher-Student Learning LLM Reasoning

Open paper AskPDF arXiv

On-policy self-distillation (OPSD) is an emerging LLM post-training paradigm in which the model serves as its own teacher: conditioned on privileged information such as a reference trace or hint, the same policy provides dense token-level supervision on its own rollouts. However, recent studies show that OPSD degrades complex reasoning by suppressing predictive uncertainty, which supports exploration and hypothesis revision. Our token-level analysis shows that this failure arises from applying a uniform direction of teacher supervision across tokens with different uncertainty levels: conformity to the privileged self-teacher suppresses exploration at high entropy, while deviation from the teacher degrades step accuracy at low entropy. Accordingly, we propose Direction-Adaptive Self-Distillation (DASD), which reframes privileged self-distillation from uniform teacher imitation into entropy-routed directional supervision: high-entropy tokens are pushed away from the privileged teacher to preserve exploration, while low-entropy tokens are pulled toward the teacher to stabilize step-level execution. Across six mathematical reasoning benchmarks, DASD achieves the best macro Avg@16 over strong RLVR and self-distillation baselines. Pass@

k

, reasoning-health, and generalization analyses show that these average gains come from preserving exploration without sacrificing step-level execution.

Abstract

Self-Distillation

Explore similar work

Tailoring Teaching to Aptitude: Direction-Adaptive Self-Distillation for LLM Reasoning

Negative Self-Distillation: Learning to Reason by Avoiding Flaws

Does self-distillation improve reasoning?