On-policy self-distillation provides dense, token-level supervision by conditioning a model on privileged information and distilling the resulting teacher distribution back into the model. However, privileged information can change not only what the teacher knows, but also how it behaves, entangling correctness-relevant learning signals with unintended behavioral shifts. We study this effect in reasoning tasks by contrasting attractive self-distillation, which moves the model toward a privileged teacher, with repulsive self-distillation, which moves it away from a privileged teacher. We find that both objectives can induce strong and opposing behavioral shifts: attraction suppresses exploratory reasoning and promotes shorter, more confident responses, whereas repulsion increases response length, can trigger unintended switches into a model's latent thinking mode, and ultimately becomes unstable. Motivated by these observations, we study contrastive self-distillation, which combines attraction toward a correct-solution-conditioned teacher with repulsion from an incorrect-solution-conditioned teacher. In contrast to prior work that combines such distillation signals with a GRPO objective, we isolate the self-distillation objective and study its behavior on its own. We find that the shared behavioral shifts of the two teachers largely cancel, leaving a token-level signal that more directly reflects correctness. Across non-thinking, instruct-only, and already-thinking models, this contrastive objective improves reasoning performance while maintaining stable response lengths.
Figures & tables
Figure 1: Contrastive self-distillation cancels opposing behavioral shifts. A teacher conditioned on a solution favors concise and confident responses, so distilling it induces a behavioral shift in the student ( ). Repulsion from a solution-conditioned teacher induces the opposite behavioral shift ( ) and leads to increased verbalized uncertainty. These behavioral shifts are largely independent of correctness. A contrastive objective ( ) balances the two opposing behavioral shifts, leaving the learning signal largely dominated by correctness. The fan of uncertainty around repulsion from a negative teacher illustrates that there are many more ways to be wrong than right.
Figure 2: Repulsive self-distillation activates dormant thinking. Repulsion from a teacher conditioned on a correct solution, starting from Qwen3-4B in non-thinking mode. The model switches into its pre-existing thinking mode without outperforming the thinking-mode baseline, before response growth leads to truncation and collapse. We observe three phases, marked by the colour of the trace: 1) training is initially stable, 2) the model enters the thinking regime, its response length increases and it starts emitting think tags, 3) continued growth of response lengths leads to truncation and collapse.
Figure 3: Attraction and repulsion. A student-generated excerpt, each token shaded by its advantage AtSDPO (Equation ( 3 )), the teacher-to-student log-probability ratio. Attraction suppresses exploratory reasoning tokens (a); reversing the sign reinforces them (b).
Figure 4: Repulsive briefly recovers thinking-mode behavior, before collapsing. Training on DeepMath from a non-thinking initialization and mean evaluation across six math benchmarks. Repulsive switches the model into thinking mode and rises sharply, but stays below the thinking-mode baseline and collapses under truncation of responses. Attractive improves training accuracy, but evaluation stays flat while responses shorten. Contrastive improves both steadily, with response length growing gradually and staying below the cap. The dashed grey lines mark the thinking-mode baseline and the 28k length cap.
Figure 5: Contrastive improves an instruct-only model. Training on DeepMath from an Instruct-only initialization and mean evaluation across six math benchmarks. Repulsive briefly reaches high training and evaluation accuracy before rapid response growth drives it into truncation collapse. Attractive improves training only, with responses staying short. Contrastive improves both steadily without approaching the cap. The dashed grey line marks the 28k length cap.
Figure 6: Contrastive self-distillation improves an already-thinking model. Training on Group Anagrams from a thinking-mode initialization and mean evaluation across four difficulty ranges. Repulsive improves through step 12, then collapses under truncation as responses grow. Attractive improves training and evaluation, while responses become shorter. Contrastive improves both training and evaluation steadily, with response length well below the cap. The dashed grey lines mark the thinking-mode baseline and the length cap.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Repulsion from a correct-solution-conditioned teacher does not improve an instruct-only model. The instruct-only counterpart of Figure 2 . Think-tag closure stays at zero throughout and is omitted. We observe two phases: 1) training and evaluation accuracy stagnate as response length increases, 2) truncation causes collapse.
Figure 8: Contrastive credit assignment: pooling and permutation. Group Anagrams training dynamics for Contrastive as token advantages are pooled over increasingly large windows or randomly permuted within each response.
Figure 9: Group Anagrams: the thinking context decides how much the model reasons. Thinking length of the training rollouts for Contrastive on Group Anagrams under three privileged contexts, with the fraction of generations that never close their thinking block as a truncation control. Thinking length is the number of tokens before the first closing think tag; responses that never close their thinking block are counted at their full response length.
Anagram groups
Words per group
Train
Evaluation
5–26
2–5
800
100
7–34
2–5
750
100
8–42
2–5
650
100
10–50
2–5
0
100
Total
2,200
400
Appendix
Table 1: Group Anagrams data. Counts are numbers of problems.
Parameter
Non-thinking math
Instruct-only math
Thinking Anagrams
Section 5.1
Section 5.2
Section 5.3
Model and data
Model
Qwen3-4B
Qwen3-4B- Instruct-2507
Qwen3-4B
Thinking enabled
False
False
True
Training problems
10,000
10,000
2,200
Token budgets
Appendix
Table 2: Hyperparameters for the three self-distillation experiment families.
Objective
Positive weight
Negative weight
Learning rate
λ
1−λ
Attractive
1
0
5×10−7
Repulsive
0
1
5×10−7
Contrastive
0.5
0.5
5×10−7
Appendix
Table 3: Objective weights in Equation 6 and learning rates. None of the three objectives includes an auxiliary GRPO loss.
Benchmark
Dataset
Problems
Responses
MATH-500
math-ai/math500
(subset) 128
512
AIME 2024
math-ai/aime24
30
120
AIME 2025
math-ai/aime25
30
120
AIME 2026
math-ai/aime26
30
120
AMC 2023
math-ai/amc23
40
160
HMMT February 2025
MathArena/hmmt_feb_2025
30
120
Appendix
Table 4: Math evaluation problems and number of sampled responses.
Self-distillation is an emerging approach that trains a model to imitate its own predictions when conditioned on privileged information, such as correct answers or environmental feedback. Recent methods have reported better sample efficiency and generalization than reinforcement learning (RL), but it remains unclear whether these gains extend beyond in-distribution evaluations and whether they stem from privileged distillation. In this paper, we revisit two representative methods, Self-Distillation Policy Optimization (SDPO) and On-Policy Self-Distillation (OPSD), across code, science, tool use, and math reasoning tasks. We find that their reported gains do not consistently generalize to unseen problems and are largely explained by overfitting or transferring existing reasoning behavior, rather than effectively using privileged information. Moreover, self-distillation gradients show weak alignment with policy gradients that directly optimize expected reward, questioning whether they provide an effective direction for policy improvement. Our experiments across Qwen3 models (1.7B-14B) and OLMo-3-7B suggest that effectively leveraging privileged information to improve reasoning remains an open challenge. Our code is available at https://github.com/sleepypandu/does-sd-improve-reasoning
Jason Ken Adhinarta, Raina Wu, Ashutosh Sharma +5
Massachusetts Institute of Technology · MIT-IBM Computing Research Lab
On-policy distillation is an efficient alternative to reinforcement learning, offering dense token-level training signals. However, its reliance on a stronger external teacher has driven recent work on on-policy self-distillation, where the same model serves as both teacher and student under different prompt contexts. Yet, existing self-distillation methods largely reduce learning to KL matching toward the context-augmented teacher model. This approach often suffers from training instability and can degrade reasoning performance over time. Moreover, self-distillation from the same model with prompt augmentation lacks the exploratory diversity provided by a genuine external teacher. To address these limitations, we move beyond fixed-teacher KL matching and propose Preference-Based Self-Distillation (PBSD), which revisits on-policy self-distillation through a reward-regularized perspective. Instead of directly matching the teacher distribution, we derive a reward-regularized objective whose analytic optimum is a reward-reweighted teacher distribution, yielding a target policy provably superior to the original teacher under this objective. Practically, PBSD optimizes preference gaps between teacher and student samples while maintaining on-policy student sampling. We support this framework with a statistical analysis of the induced preference-learning problem, formally establishing when on policy self-distillation is preferable to learning from an external teacher in our setting. Experiments on mathematical reasoning and tool-use benchmarks across multiple model scales demonstrate that PBSD consistently achieves the strongest average performance among comparable baselines, showing improved training stability over prior self-distillation baselines while preserving token efficiency.
On-policy self-distillation, where a student is pulled toward a copy of itself conditioned on privileged context (e.g., a verified solution or feedback), offers a promising direction for advancing reasoning capability without a stronger external teacher. Yet in math reasoning the gains are inconsistent, even when the same approach succeeds elsewhere. A pointwise mutual information analysis traces the failure to the privileged context itself: it inflates the teacher's confidence on tokens already implied by the solution (structural connectives, verifiable claims) and deflates it on deliberation tokens ("Wait", "Let", "Maybe") that drive multi-step search. We propose Anti-Self-Distillation (AntiSD), which ascends a divergence between student and teacher rather than descending it: this reverses the per-token sign and yields a naturally bounded advantage in one step. An entropy-triggered gate disables the term once the teacher entropy collapses, completing a drop-in replacement for default self-distillation. Across five models from 4B to 30B parameters on math reasoning benchmarks, AntiSD reaches the GRPO baseline's accuracy in 2 to 10x fewer training steps and improves final accuracy by up to 11.5 points. AntiSD opens a path to scalable self-improvement, where a language model bootstraps its own reasoning through its training signal.
Guobin Shen, Xiang Cheng, Chenxiao Zhao +4
1Xiaohongshu Inc. · Institute of Automation, Chinese Academy of Sciences