Self-distillation methods for large language models with an exponential moving average (EMA) teacher already retain a history of student updates. Muon nevertheless stores a separate momentum buffer, increasing persistent training memory. We propose Muon with Reused Teacher (Muon-RT), which reuses the teacher-student displacement to eliminate momentum buffers for hidden weight matrices. We show that this displacement encodes an exponentially weighted history of applied updates and smooths temporally uncorrelated noise under a local model. Muon-RT removes its radial component, normalizes it spectrally, and mixes it with the current gradient after Newton-Schulz (NS) orthogonalization. Under explicit regularity and alignment assumptions, we establish linear contraction of an expected objective-gap bound toward a residual error level. With Self-Distillation Fine-Tuning (SDFT) on Qwen2.5-7B-Instruct, Muon-RT improves average accuracy over AdamW by 2.6 points with 42% less persistent memory, and over momentum Muon by 5.4 points with 27% less persistent memory. With Self-Distillation Policy Optimization (SDPO) on Olmo-3-7B-Instruct, it improves average accuracy over AdamW by 2.0 points.
Figures & tables
Figure 1: Muon-RT with SDFT on Qwen2.5-7B-Instruct. (a) Average accuracy over 2 datasets against memory per GPU. (b) Gain over AdamW in points.
Figure 2: Overview of Muon-RT. (a) Each hidden matrix maintains student weights WS and EMA-teacher weights WT , eliminating Muon’s momentum buffer M . (b) Newton–Schulz approximately orthogonalizes the gradient G into O . The radial component of the teacher–student displacement WT−WS is removed to obtain H , which is spectrally normalized into P . The update combines both directions as U=(1−α)O+αP . (c) The EMA-teacher tracks the student’s parameter trajectory with a temporal lag.
Task
Method
Optimizer
W/O CL
CL Tasks
Accuracy
HS
HE
IFE
MMLU
TQA
Wino
Avg.
Science
Base model †
–
32.1
62.0
65.8
74.3
71.7
47.9
71.1
65.5
SFT †
AdamW
66.2
55.0
54.8
35.3
64.6
36.8
73.7
53.4
SFT + re-invoke †
AdamW
66.0
61.6
63.4
52.9
68.7
45.2
70.0
60.3
DFT †
AdamW
54.8
57.6
67.0
60.4
69.4
38.8
68.2
60.2
SDFT †
AdamW
70.2
60.9
68.9
66.8
70.7
46.5
73.1
64.5
Table 1: The accuracy without continual learning (W/O CL) and with continual learning (CL) task for SDFT (Qwen2.5-7B). HS, HE, IFE, TQA, and Wino denote HellaSwag, HumanEval, IFEval, TruthfulQA, and WinoGrande; Avg. is the mean across previous tasks. † denotes results from SDFT, with averages recomputed. Muon-RT shows significant improvement without CL setting, however for the CL setting its highly competitive.
Method
Optimizer
1 h
5 h
Biology
Base model †
–
16.2
GRPO †
AdamW
35.6
55.8
GRPO (on-policy) †
AdamW
49.8
49.8
SDPO †
AdamW
48.0
52.8
SDPO
Muon ( μ=0 )
48.7
53.1
SDPO
MuonClip ( μ=0 )
48.5
54.4
Table 2: On-policy post-training with SDPO (Olmo-3-7B). Best avg@16 accuracy (%) within 1 and 5 hours. † from SDPO. Best in bold, second-best underlined.
Figure 3: Training dynamics with SDFT. (a, b) Best accuracy reached up to each step. (c) Self-distillation KL at learning rate 5×10−5 . (d) Persistent memory per GPU between steps, with a bfloat16 mixed-precision student (32-bit master weights) and a 32-bit teacher for every optimizer.
Figure 4: Analysis of the teacher branch. (a) Measured and predicted teacher–student displacement during training. (b) Cosine with the mean gradient direction under gradient noise on a Qwen2.5-7B-Instruct weight matrix. (c) Change of the update caused by the teacher direction. (d) Gradient norm during training.
Figure 5: Projected persistent memory of self-distillation versus model size, with all states in 32 bits and gradients included.
Variant
Tool use
Science
(a) Optimizer state
Muon ( μ=0.95 )
66.9
68.3
MuonClip ( μ=0.95 )
67.0
68.8
Muon ( μ=0 )
69.1
69.0
MuonClip ( μ=0 )
73.2
71.0
Muon-RT (ours)
74.2
71.8
Table 3: Ablations with SDFT (accuracy, %).
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Value
Model
Qwen2.5-7B-Instruct
Hardware
2 × H100 80 GB, FSDP2
Prompts per step
32 (1 rollout each)
Epochs (steps)
2 (252 tool use, 166 science)
Learning rate
{3,5}×10−5 , cosine
Warmup
10% of steps
Appendix
Table 4: SDFT training settings for all optimizers.
Setting
Value
Model
Olmo-3-7B-Instruct
Data
SciKnowEval (2 subsets)
Max. prompt / response
2048 / 8192
Questions / mini-batch
32 / 32
Rollouts / temperature
8 / 1.0
Validation / temperature / top- p
avg@16 / 0.6 / 0.95
Appendix
Table 5: SDPO training settings. All entries above the line are those of Hübotter et al. (2026) .
Figure 6: (a) Distance from the initial weights for the student and for teachers in 2 precisions. (b) Size of the teacher–student displacement. (c) Alignment of the 2 branches. (d) Best tool-use accuracy versus learning rate.
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.
Self-distillation with privileged context adapts a language model from demonstrations by letting the model, once conditioned on a reference response, teach its context-free copy token by token. Our taxonomy reveals existing methods differ along three entangled axes: (i) the rollout source (student or teacher), (ii) the teacher coupling (frozen, or an exponential moving average of the student at some coupling rate) and (iii) the KL direction (reverse or forward), yet these axes are usually studied in fixed combinations and have led to conflicting conclusions. We formalize a unifying framework to encompass all self-distillation methods vs classic supervised fine-tuning: we train every combination of the three axes, on Qwen2.5-7B and Ministral-3-3B across ordinary and contradictory tasks, totaling 1,200 adaptation runs, to systematically investigate the impact of the above axes. We propose a controlled model of the same objective to explain the resulting acquisition-retention trade-offs. We find that (i) the rollout source matters mostly where the task contradicts the pretrained behavior: there teacher rollouts raise acquisition well above what student rollouts achieve, with almost no change in retention; (ii) the teacher coupling changes acquisition most, on every task: acquisition rises with the coupling rate, then falls past a task-specific rate; (iii) switching the KL direction costs retention in one model but not the other so which axis to tune first depends on the model. The controlled model reproduces the three trends.
On-policy Self-distillation (OPSD) has proven effective for post-training large language models by constructing a teacher conditioned on privileged references. However, existing approaches keep the teacher fixed either throughout training by consistently exposing it to the full reference, or within each response by adaptively varying partial reference exposure only across responses. In this paper, we argue that neither of them is sufficient. Instead, every token needs its own teacher, as the teacher preference can vary token by token within the same response. Through a teacher-intervention pilot study, we validate that the teacher preference differs across tokens, and reveal that a preferred teacher can be identified for each token through majority voting and KL-based selection. Motivated by these findings, we propose Multi-teacher Adaptive On-policy Self-Distillation (MTA-OPSD), the first token-level adaptive OPSD method. Our method constructs a multi-teacher pool consisting of candidate teachers, and enables every token to adaptively select its preferred teacher during training. Experiments with Qwen3 models on mathematical reasoning tasks show that MTA-OPSD consistently outperforms existing OPSD approaches with an average gain of up to +5.9 over the base model, highlighting the importance of adapting teacher supervision at the token level. The code is available at https://github.com/xingzhejun/MTA-OPSD.
Yifu Luo, Chuang Ma, Zeyu Chen +3
TU München · Munich Center for Machine Learning (MCML) · relAI – Konrad Zuse School of Excellence in Reliable AI +4
On-policy self-distillation (OPSD) improves large language models by letting a self-teacher with privileged information provide dense token-level supervision on the model's own trajectories. Yet existing methods typically construct the self-teacher from the current, initial, or slowly averaged policy state, leaving the quality of supervision constrained by the teacher's ability to exploit privileged information. We ask whether the model's own optimization progress can instead be recycled into a stronger self-teacher. In this paper, we introduce Bootstrapped On-Policy Self-Distillation (B-OPSD), which temporarily trains the policy ahead to obtain a future teacher, restores the student to the original policy state, and then uses the future teacher to supervise the restarted student. The future teacher improves supervision in two complementary ways, it can generate more reliable privileged trajectories and, conditioned on them, provide more informative token-level targets along the restarted student's on-policy trajectories. Experiments on mathematical reasoning with Qwen3-4B and Qwen3-8B show consistent improvements over standard OPSD in both settings, including gains from 27.50 to 41.30 and from 48.80 to 64.44 in the rollout-privileged setting. Our findings point to a broader principle for self-improving models that future learning progress can be distilled backward, preserving acquired knowledge while bootstrapping beyond the optimization state that produced it.
Zheng Zhang, Xinyue Tan, Lufei Li +3
School of Information Science and Technology, ShanghaiTech University · State Key Laboratory of General Artificial Intelligence, BIGAI · Hefei University of Technology