Does self-distillation improve reasoning?

Jason Ken Adhinarta; Raina Wu; Ashutosh Sharma; Avinash Balakrishnan; Rameswar Panda; Polina Golland; Yoon Kim; Zhang-Wei Hong

cs.LGOct 13, 2026

Does self-distillation improve reasoning?

Authors: Jason Ken Adhinarta, Raina Wu, Ashutosh Sharma, Avinash Balakrishnan, Rameswar Panda, Polina Golland, Yoon Kim, Zhang-Wei Hong

Organizations: Massachusetts Institute of Technology · MIT-IBM Computing Research Lab

Abstract

Self-distillation is an emerging approach that trains a model to imitate its own predictions when conditioned on privileged information, such as correct answers or environmental feedback. Recent methods have reported better sample efficiency and generalization than reinforcement learning (RL), but it remains unclear whether these gains extend beyond in-distribution evaluations and whether they stem from privileged distillation. In this paper, we revisit two representative methods, Self-Distillation Policy Optimization (SDPO) and On-Policy Self-Distillation (OPSD), across code, science, tool use, and math reasoning tasks. We find that their reported gains do not consistently generalize to unseen problems and are largely explained by overfitting or transferring existing reasoning behavior, rather than effectively using privileged information. Moreover, self-distillation gradients show weak alignment with policy gradients that directly optimize expected reward, questioning whether they provide an effective direction for policy improvement. Our experiments across Qwen3 models (1.7B-14B) and OLMo-3-7B suggest that effectively leveraging privileged information to improve reasoning remains an open challenge. Our code is available at https://github.com/sleepypandu/does-sd-improve-reasoning

Figures & tables

Figure 1: SDPO reduces solution diversity on LiveCodeBench v6. AST-unique solutions per training problem (from $8$ samples) for Qwen3-14B before (Base) and after $200$ steps of GRPO or SDPO (mean $\pm$ std, four seeds). Diversity collapse suggests memorization.

Figure 2: The reference solution barely changes the self-distillation teacher. Teacher log-probabilities (thinking on) along a Qwen3-8B thinking-off student rollout on an OpenThoughts problem, with the problem’s reference solution in context (ref $z$ , top) and with no solution (no $z$ , bottom). In each panel, the first row is the sampled token and the three rows below are the student’s next three most likely tokens. The two teachers assign very similar probabilities, suggesting that OPSD distillation effects do not significantly depend on $z$ ( Section 4.3 ).

Figure 3: Qwen3-8B SDPO vs. GRPO on LiveCodeBench v6 across four seeds, scored as the fraction of problems with no failing test cases (mean $\pm$ std). train_seen and train_unseen evaluate training problems on seen and on held-out test cases respectively; test_unseen evaluates held-out problems. The $20000$ training generations reported by Hübotter et al. (2026) is marked in red.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Qwen3-14B SDPO vs. GRPO on LiveCodeBench v6 across four seeds, scored as the fraction of problems with no failing test cases (mean $\pm$ std). train_seen and train_unseen evaluate problems seen during training, on seen and on held-out test cases respectively; test_unseen evaluates held-out problems.

Figure 5: Olmo-3-7B-Instruct SDPO vs. GRPO on LiveCodeBench v6 across four seeds, scored as the fraction of problems with no failing test cases (mean $\pm$ std). train_seen and train_unseen evaluate problems seen during training, on seen and on held-out test cases respectively; test_unseen evaluates held-out problems.

Model	Method	Step 0	Step 80	Step 200
Qwen3-8B	SDPO	32.3 $\pm$ 1.6	30.1 $\pm$ 0.7	29.8 $\pm$ 1.1
	GRPO	30.6 $\pm$ 1.0	33.3 $\pm$ 1.9	34.3 $\pm$ 0.5
Qwen3-14B	SDPO	32.8 $\pm$ 0.7	33.9 $\pm$ 2.8	32.3 $\pm$ 1.3
	GRPO	33.6 $\pm$ 0.4	36.4 $\pm$ 3.7	38.5 $\pm$ 2.2
Olmo-3-7B-Instruct	SDPO	30.9 $\pm$ 1.0	25.2 $\pm$ 1.0	21.0 $\pm$ 1.4
	GRPO	31.5 $\pm$ 1.8	32.7 $\pm$ 1.2	34.1 $\pm$ 0.8

Appendix

Table 5: LiveCodeBench v6 score (fraction of problems with no failing test cases, mean

\pm

std over four seeds) on test_unseen for SDPO and GRPO at initialization, 80 steps, and 200 steps. Bold marks the better method at each step.

Model	Method	Step 0	Step 80	Step 200
Qwen3-8B	SDPO	25.8 $\pm$ 1.8	48.6 $\pm$ 0.7	48.9 $\pm$ 1.6
	GRPO	25.1 $\pm$ 0.7	43.8 $\pm$ 2.4	50.9 $\pm$ 1.1
Qwen3-14B	SDPO	29.4 $\pm$ 0.4	49.2 $\pm$ 1.3	51.9 $\pm$ 1.5
	GRPO	29.0 $\pm$ 2.0	45.4 $\pm$ 1.8	54.1 $\pm$ 2.5
Olmo-3-7B-Instruct	SDPO	28.9 $\pm$ 1.7	52.9 $\pm$ 2.3	47.2 $\pm$ 4.7
	GRPO	28.0 $\pm$ 1.9	39.7 $\pm$ 1.3	43.7 $\pm$ 2.4

Appendix

Table 6: LiveCodeBench v6 score (fraction of problems with no failing test cases, mean

\pm

std over four seeds) on train_seen for SDPO and GRPO at initialization, 80 steps, and 200 steps. Bold marks the better method at each step.

Model	Method	Step 0	Step 80	Step 200
Qwen3-8B	SDPO	25.6 $\pm$ 0.5	44.1 $\pm$ 1.0	44.0 $\pm$ 1.7
	GRPO	25.4 $\pm$ 1.0	41.2 $\pm$ 2.4	46.2 $\pm$ 2.0
Qwen3-14B	SDPO	28.2 $\pm$ 1.4	43.0 $\pm$ 0.2	45.4 $\pm$ 2.1
	GRPO	26.0 $\pm$ 1.0	41.8 $\pm$ 1.6	49.2 $\pm$ 2.0
Olmo-3-7B-Instruct	SDPO	25.9 $\pm$ 1.6	48.5 $\pm$ 1.4	42.6 $\pm$ 6.2
	GRPO	25.8 $\pm$ 0.4	36.2 $\pm$ 2.3	41.9 $\pm$ 0.8

Appendix

Table 7: LiveCodeBench v6 score (fraction of problems with no failing test cases, mean

\pm

std over four seeds) on train_unseen for SDPO and GRPO at initialization, 80 steps, and 200 steps. Bold marks the better method at each step.

Figure 6: Cross-domain validation accuracy of Qwen3-8B trained with GRPO and SDPO on SciKnowEval and ToolAlpaca, averaged over four seeds. Panels vary over training domains (across rows) and evaluation domains (across columns).

Baselines
Method	AIME24	AIME25	HMMT25	Avg.
Base	50.3	37.2	23.6	37.0
GRPO	51.9 $\pm$ 1.3	39.2 $\pm$ 1.0	25.6 $\pm$ 2.3	38.9
SFT	53.1 $\pm$ 0.5	40.4 $\pm$ 0.9	24.7 $\pm$ 1.2	39.4
OPSD (self-distill)
Ref $z$	55.4 $\pm$ 1.0	41.7 $\pm$ 0.4	26.4 $\pm$ 0.8	41.2

Appendix

Table 11: Olmo-3-7B-Instruct training methods evaluated under avg@12 . OPSD self-distill uses the frozen Olmo-3-7B-Instruct as the teacher, while OPSD cross-model uses Olmo-3-7B-Think as the teacher. Denote

z

as the reference solution shown to the teacher. We report the checkpoint with the best average performance (averaged over four seeds), reporting mean

\pm

std accuracy.

Baselines
Method	Cosine
GRPO	$+0.026\pm 0.021$
SFT	$-0.060\pm 0.021$
OPSD (self-distill)
Ref $z$	$+0.057\pm 0.052$
No $z$	$+0.051\pm 0.047$

Appendix

Table 19: Cosine similarity between the gradient of each method trained on OpenThoughts and the GRPO gradient on AIME24, AIME25, and HMMT25 for Olmo-3-7B-Instruct, measured at the base model (pooled over four seeds,

\pm

paired jackknife error). OPSD self-distill uses the frozen Olmo-3-7B-Instruct as the teacher, while OPSD cross-model uses Olmo-3-7B-Think as the teacher.

Baselines
Gradient	Qwen3-1.7B	Qwen3-4B	Qwen3-8B	Qwen3-14B	Olmo-3-7B-Instruct
Reference	$3.583$	$3.436$	$3.666$	$3.736$	$4.033$
GRPO ${}_{\text{think}}$	$3.979$	$4.108$	$3.889$	$3.786$	–
GRPO ${}_{\text{nothink}}$	$3.653$	$3.483$	$3.322$	$2.717$	$3.951$
SFT	$0.000$	$0.000$	$0.001$	$0.001$	$0.002$
OPSD (teacher thinking-on / cross-model)

Appendix

Table 20: Gradient noise scale across four seeds for the math probe, where each seed’s gradient is the mean over 64 iterations;

0

is a noiseless estimate. The reference is GRPO

{}_{\text{nothink}}

on AIME24, AIME25, and HMMT25. For Olmo-3-7B-Instruct, the teacher thinking-on and thinking-off rows are replaced by OPSD cross-model (Olmo-3-7B-Think teacher) and self-distill, respectively.

Gradient	Qwen3-8B	Qwen3-14B	Olmo-3-7B-Instruct
Reference	$1.968$	$3.152$	$1.999$
GRPO	$2.855$	$3.125$	$2.076$
SDPO	$0.043$	$0.018$	$0.020$

Appendix

Table 21: Gradient noise scale across four seeds for the LiveCodeBench v6 probe, where each seed’s gradient is the mean over 8 iterations;

0

is a noiseless estimate. The reference is GRPO on test_unseen ; GRPO and SDPO are trained on train_seen .

General
Parameter	GRPO	SDPO
Model	Qwen3-8B	Qwen3-8B
Thinking	False	False
Total steps	200	200
Data
Max. prompt length	2048	2048

Appendix

Table 23: GRPO and SDPO training hyperparameters on SciKnowEval and ToolAlpaca

Self-Distillation

Self-distillation trains a model from its own predictions, representations, or experience to improve capability, efficiency, and robustness.

See all 131 papers

Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information
May 12, 2026Guobin Shen, Xiang Cheng, Chenxiao Zhao +4On-Policy Self-Distillation RL for Language Model Reasoning
1Xiaohongshu Inc. · Institute of Automation, Chinese Academy of Sciences
Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularization
May 6, 2026Xin Yu, Liuchen Liao, Yiwen Zhang +3On-Policy Self-Distillation Language Model Distillation
1The Pennsylvania State University · 2TikTok
Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoning
May 13, 2026Junlong Ke, Zichen Wen, Weijia Li +2On-Policy Self-Distillation RL for Language Model Reasoning
2Tsinghua University · 1Shanghai Jiao Tong University · 3Shanghai AI Laboratory

May 12, 2026J/K move · Enter open · S save

Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information

Guobin Shen, Xiang Cheng, Chenxiao Zhao, Lei Huang+3

1Xiaohongshu Inc. · Institute of Automation, Chinese Academy of Sciences

On-Policy Self-Distillation RL for Language Model Reasoning Self-Distillation Teacher-Student Learning LLM Mathematical Reasoning

Open paper AskPDF arXiv

On-policy self-distillation, where a student is pulled toward a copy of itself conditioned on privileged context (e.g., a verified solution or feedback), offers a promising direction for advancing reasoning capability without a stronger external teacher. Yet in math reasoning the gains are inconsistent, even when the same approach succeeds elsewhere. A pointwise mutual information analysis traces the failure to the privileged context itself: it inflates the teacher's confidence on tokens already implied by the solution (structural connectives, verifiable claims) and deflates it on deliberation tokens ("Wait", "Let", "Maybe") that drive multi-step search. We propose Anti-Self-Distillation (AntiSD), which ascends a divergence between student and teacher rather than descending it: this reverses the per-token sign and yields a naturally bounded advantage in one step. An entropy-triggered gate disables the term once the teacher entropy collapses, completing a drop-in replacement for default self-distillation. Across five models from 4B to 30B parameters on math reasoning benchmarks, AntiSD reaches the GRPO baseline's accuracy in 2 to 10x fewer training steps and improves final accuracy by up to 11.5 points. AntiSD opens a path to scalable self-improvement, where a language model bootstraps its own reasoning through its training signal.

Abstract

Self-Distillation

Explore similar work

Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information

Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularization

Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoning