On Repulsive and Attractive Teachers: Separating Correctness from Behavior in Self-Distillation

Anton Baumann; Akmal Ashirmatov; Leo Schmidt-Traub; Frederike Lübeck; Jonas Hübotter; Thomas Kleine Buening; Andreas Krause

cs.LGSep 18, 2026

On Repulsive and Attractive Teachers: Separating Correctness from Behavior in Self-Distillation

Authors: Anton Baumann, Akmal Ashirmatov, Leo Schmidt-Traub, Frederike Lübeck, Jonas Hübotter, Thomas Kleine Buening, Andreas Krause

Organizations: ETH Zurich · Max Planck Institute for Intelligent Systems

Abstract

On-policy self-distillation provides dense, token-level supervision by conditioning a model on privileged information and distilling the resulting teacher distribution back into the model. However, privileged information can change not only what the teacher knows, but also how it behaves, entangling correctness-relevant learning signals with unintended behavioral shifts. We study this effect in reasoning tasks by contrasting attractive self-distillation, which moves the model toward a privileged teacher, with repulsive self-distillation, which moves it away from a privileged teacher. We find that both objectives can induce strong and opposing behavioral shifts: attraction suppresses exploratory reasoning and promotes shorter, more confident responses, whereas repulsion increases response length, can trigger unintended switches into a model's latent thinking mode, and ultimately becomes unstable. Motivated by these observations, we study contrastive self-distillation, which combines attraction toward a correct-solution-conditioned teacher with repulsion from an incorrect-solution-conditioned teacher. In contrast to prior work that combines such distillation signals with a GRPO objective, we isolate the self-distillation objective and study its behavior on its own. We find that the shared behavioral shifts of the two teachers largely cancel, leaving a token-level signal that more directly reflects correctness. Across non-thinking, instruct-only, and already-thinking models, this contrastive objective improves reasoning performance while maintaining stable response lengths.

Figures & tables

Panel of Figure 1 — Figure 1: Contrastive self-distillation cancels opposing behavioral shifts. A teacher conditioned on a solution favors concise and confident responses, so distilling it induces a behavioral shift in the student ( ). Repulsion from a solution-conditioned teacher induces the opposite behavioral shift ( ) and leads to increased verbalized uncertainty. These behavioral shifts are largely independent of correctness. A contrastive objective ( ) balances the two opposing behavioral shifts, leaving the learning signal largely dominated by correctness. The fan of uncertainty around repulsion from a negative teacher illustrates that there are many more ways to be wrong than right.

Panel of Figure 2 — Figure 2: Repulsive self-distillation activates dormant thinking. Repulsion from a teacher conditioned on a correct solution, starting from Qwen3-4B in non-thinking mode. The model switches into its pre-existing thinking mode without outperforming the thinking-mode baseline, before response growth leads to truncation and collapse. We observe three phases, marked by the colour of the trace: 1) training is initially stable, 2) the model enters the thinking regime, its response length increases and it starts emitting think tags, 3) continued growth of response lengths leads to truncation and collapse.

Panel of Figure 3 — Figure 3: Attraction and repulsion. A student-generated excerpt, each token shaded by its advantage $A_{t}^{\mathrm{SDPO}}$ (Equation ( 3 )), the teacher-to-student log-probability ratio. Attraction suppresses exploratory reasoning tokens (a); reversing the sign reinforces them (b).

Panel of Figure 4 — Figure 4: Repulsive briefly recovers thinking-mode behavior, before collapsing. Training on DeepMath from a non-thinking initialization and mean evaluation across six math benchmarks. Repulsive switches the model into thinking mode and rises sharply, but stays below the thinking-mode baseline and collapses under truncation of responses. Attractive improves training accuracy, but evaluation stays flat while responses shorten. Contrastive improves both steadily, with response length growing gradually and staying below the cap. The dashed grey lines mark the thinking-mode baseline and the 28k length cap.

Panel of Figure 5 — Figure 5: Contrastive improves an instruct-only model. Training on DeepMath from an Instruct-only initialization and mean evaluation across six math benchmarks. Repulsive briefly reaches high training and evaluation accuracy before rapid response growth drives it into truncation collapse. Attractive improves training only, with responses staying short. Contrastive improves both steadily without approaching the cap. The dashed grey line marks the 28k length cap.

Panel of Figure 6 — Figure 6: Contrastive self-distillation improves an already-thinking model. Training on Group Anagrams from a thinking-mode initialization and mean evaluation across four difficulty ranges. Repulsive improves through step 12, then collapses under truncation as responses grow. Attractive improves training and evaluation, while responses become shorter. Contrastive improves both training and evaluation steadily, with response length well below the cap. The dashed grey lines mark the thinking-mode baseline and the length cap.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Repulsion from a correct-solution-conditioned teacher does not improve an instruct-only model. The instruct-only counterpart of Figure 2 . Think-tag closure stays at zero throughout and is omitted. We observe two phases: 1) training and evaluation accuracy stagnate as response length increases, 2) truncation causes collapse.

Panel of Figure 8 — Figure 8: Contrastive credit assignment: pooling and permutation. Group Anagrams training dynamics for Contrastive as token advantages are pooled over increasingly large windows or randomly permuted within each response.

Panel of Figure 9 — Figure 9: Group Anagrams: the thinking context decides how much the model reasons. Thinking length of the training rollouts for Contrastive on Group Anagrams under three privileged contexts, with the fraction of generations that never close their thinking block as a truncation control. Thinking length is the number of tokens before the first closing think tag; responses that never close their thinking block are counted at their full response length.

Anagram groups	Words per group	Train	Evaluation
5–26	2–5	800	100
7–34	2–5	750	100
8–42	2–5	650	100
10–50	2–5	0	100
Total		2,200	400

Appendix

Table 1: Group Anagrams data. Counts are numbers of problems.

Parameter	Non-thinking math	Instruct-only math	Thinking Anagrams
	Section 5.1	Section 5.2	Section 5.3
Model and data
Model	Qwen3-4B	Qwen3-4B- Instruct-2507	Qwen3-4B
Thinking enabled	False	False	True
Training problems	10,000	10,000	2,200
Token budgets

Appendix

Table 2: Hyperparameters for the three self-distillation experiment families.

Objective	Positive weight	Negative weight	Learning rate
	$\lambda$	$1-\lambda$
Attractive	1	0	$5\times 10^{-7}$
Repulsive	0	1	$5\times 10^{-7}$
Contrastive	0.5	0.5	$5\times 10^{-7}$

Appendix

Table 3: Objective weights in Equation 6 and learning rates. None of the three objectives includes an auxiliary GRPO loss.

Benchmark	Dataset	Problems	Responses
MATH-500	math-ai/math500	(subset) 128	512
AIME 2024	math-ai/aime24	30	120
AIME 2025	math-ai/aime25	30	120
AIME 2026	math-ai/aime26	30	120
AMC 2023	math-ai/amc23	40	160
HMMT February 2025	MathArena/hmmt_feb_2025	30	120

Appendix

Table 4: Math evaluation problems and number of sampled responses.

Reasoning Distillation

Reasoning distillation transfers step-by-step problem-solving skills from powerful teachers to smaller, more efficient models.

See all 22 papers

Does self-distillation improve reasoning?
Oct 13, 2026Jason Ken Adhinarta, Raina Wu, Ashutosh Sharma +5Self-Distillation On-Policy Self-Distillation
Massachusetts Institute of Technology · MIT-IBM Computing Research Lab
Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularization
May 6, 2026Xin Yu, Liuchen Liao, Yiwen Zhang +3On-Policy Self-Distillation Language Model Distillation
1The Pennsylvania State University · 2TikTok
Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information
May 12, 2026Guobin Shen, Xiang Cheng, Chenxiao Zhao +4On-Policy Self-Distillation RL for Language Model Reasoning
1Xiaohongshu Inc. · Institute of Automation, Chinese Academy of Sciences

Oct 13, 2026J/K move · Enter open · S save

Does self-distillation improve reasoning?

Jason Ken Adhinarta, Raina Wu, Ashutosh Sharma, Avinash Balakrishnan+4

Massachusetts Institute of Technology · MIT-IBM Computing Research Lab

Self-Distillation On-Policy Self-Distillation Knowledge Distillation RL for Language Model Reasoning

Open paper AskPDF arXiv

Self-distillation is an emerging approach that trains a model to imitate its own predictions when conditioned on privileged information, such as correct answers or environmental feedback. Recent methods have reported better sample efficiency and generalization than reinforcement learning (RL), but it remains unclear whether these gains extend beyond in-distribution evaluations and whether they stem from privileged distillation. In this paper, we revisit two representative methods, Self-Distillation Policy Optimization (SDPO) and On-Policy Self-Distillation (OPSD), across code, science, tool use, and math reasoning tasks. We find that their reported gains do not consistently generalize to unseen problems and are largely explained by overfitting or transferring existing reasoning behavior, rather than effectively using privileged information. Moreover, self-distillation gradients show weak alignment with policy gradients that directly optimize expected reward, questioning whether they provide an effective direction for policy improvement. Our experiments across Qwen3 models (1.7B-14B) and OLMo-3-7B suggest that effectively leveraging privileged information to improve reasoning remains an open challenge. Our code is available at https://github.com/sleepypandu/does-sd-improve-reasoning

Abstract

Reasoning Distillation

Explore similar work

Does self-distillation improve reasoning?

Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularization

Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information