Muon has recently emerged as a promising optimizer for pretraining large language models, offering a speedup over AdamW in minimizing pretraining loss. However, it remains unclear whether Muon's advantage persists after fine-tuning. In this paper, we study the downstream behavior of Muon and AdamW and uncover a surprising trend: beyond a sufficiently large pretraining data budget, AdamW outperforms Muon after fine-tuning. Moreover, as model size increases, AdamW overtakes Muon at progressively smaller pretraining token-parameter ratios. To explain this reversal, we view fine-tuning as a perturbation of the outputs and introduce the margin-to-spread ratio, which measures how far the correct logit lies above the wrong ones relative to how widely the wrong logits are spread. In language models, Muon has the larger margin-to-spread ratio at small data budgets and AdamW at large ones, mirroring the fine-tuning trend. In a linear associative memory, we prove that after one update the margin-to-spread ratio of gradient descent gains on that of Muon as the data budget grows, and simulations show that this trend persists after extended training.
Figures & tables
Figure 1: Beyond a sufficiently large data budget, AdamW outperforms Muon. We pretrain OLMo-60M models with Muon and AdamW and fine-tune on StackMathQA. For each pretraining optimizer, we take the pretrained model with the lowest pretraining loss, sweep across all fine-tuning configurations, and plot the resulting learning–forgetting frontier, from 2.4 B to 38.4 B pretraining tokens. At 2.4 B tokens, Muon’s frontier dominates AdamW’s; at larger budgets the trend reverses and AdamW surpasses Muon. This reversal happens despite Muon achieving lower pretraining loss across all data budgets.
Figure 2: Crossover TPR decreases with model size. In the lowest-pretraining-loss setting, where each pretraining optimizer’s lowest-loss pretrained model is swept across all fine-tuning hyperparameters, Muon’s advantage reverses at substantially smaller pretraining data budgets as model size increases. The exact crossover TPR varies depending on the fine-tuning dataset, but the high-level trend is consistent.
Figure 3: The Muon–AdamW fine-tuning gap widens with increasing pretraining data budget, and Muon’s advantage shrinks with model size. (a) The fine-tuning gap after fine-tuning OLMo-60M models on StackMathQA, in the lowest-pretraining-loss setting (each pretraining optimizer’s lowest-loss pretrained model swept across all fine-tuning configurations). As the pretraining data budget increases, the fine-tuning gap transitions from negative to positive, with AdamW outperforming Muon beyond the crossover TPR. (b) Pretraining loss at 40 TPR. Muon’s advantage over AdamW shrinks as model size grows. (c) Degradation at 40 TPR, when we accept a StackMathQA loss 10% above the best in exchange for less forgetting. Muon degrades increasingly more than AdamW as model size grows.
Figure 4: The margin-to-spread ratio of GD gains on Muon’s as the data budget grows. (a, b) After t=104 steps with model dimension d=256 , GD’s MSR ρ overtakes Muon’s as γ grows, and the ordering of the perturbed robustness σ50 turns over with it. (c, d) After one step with d=512 , GD’s normalized MSR ρ/d/γ overtakes Muon’s (c), so the MSR ratio R=ρGD/ρMuon rises with γ (d), following the theory of Section 4.2 .
Figure 5: In real language models, the margin-to-spread ratio of AdamW gains on Muon’s as the token budget grows. (a) MSR ρ of AdamW and Muon against pretraining TPR for 30M, 60M, and 100M models: Muon has the larger MSR at small budgets and AdamW at large ones. (b) The MSR ratio ρAdamW/ρMuon crosses one as the budget grows, at every model size.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: The perturbation degradation gap grows with pretraining data budget. Under a perturbation of magnitude δ=0.02 , Muon degrades increasingly more than AdamW as the pretraining data budget grows. Points show the mean over three noise draws.
Figure 7: The margin-to-spread ratio tracks robustness to weight perturbations. (a) For the linear models trained under GD and Muon, we plot the perturbation σ50 required to lose half of the recovered facts against its MSR, hˉ/κˉ ( Section 4.1 ). A higher MSR requires more noise to cause the same loss. (b) We measure the MSR of 72 pretrained language models (30M–600M) as the median of h/κ over correctly predicted tokens, and plot the perturbation δ∗ at which held-out loss rises by ΔL . The same relationship holds.
Size
Optimizer
TPR
5
10
20
40
80
160
320
640
1,280
2,560
30M
AdamW
0.08
0.04
0.04
0.014
0.02
0.014
0.02
0.014
0.01
–
Muon
0.01
0.01
0.01
0.014
0.02
0.01
0.014
0.01
0.014
–
60M
AdamW
0.056
0.04
0.02
0.02
0.01
0.014
0.01
0.01
0.007
0.01
Muon
0.014
0.01
0.014
0.01
0.014
0.01
0.014
0.007
0.005
0.005
100M
AdamW
0.04
0.02
0.01
0.01
0.01
0.007
0.007
0.007
–
–
Appendix
Table 1: Tuned peak pretraining learning rates per model size and budget. For Muon, the Muon learning rate; the AdamW-parameter learning rate is fixed to the tuned AdamW value.
Figure 8: Muon is insensitive to the learning rate of its AdamW-updated parameters. Best pretraining loss over Muon learning rates for each AdamW learning rate, relative to the tuned AdamW value (dashed), for 60M models.
Figure 9: Muon reaches lower pretraining loss than AdamW. DCLM held-out loss of the loss-optimal base models across pretraining data budgets and model sizes.
Figure 10: Fine-tuning gap on every fine-tuning dataset, loss-optimal base models (60M). Negative values mean the Muon frontier dominates.
Figure 11: Learning–forgetting frontiers on every fine-tuning dataset, loss-optimal base models (60M). Each row shows the budgets around that dataset’s crossover TPR.
Figure 12: Fine-tuning gap on every fine-tuning dataset, best-frontier setting (60M). Negative values mean the Muon frontier dominates.
Figure 13: Learning–forgetting frontiers on every fine-tuning dataset, best-frontier setting (60M). Each row shows the budgets around that dataset’s crossover TPR.
Figure 14: Crossover TPR decreases with model size, in the best-frontier setting. As model size increases, Muon’s advantage reverses with substantially smaller pretraining data budgets.
Figure 15: Muon’s greater forgetting comes from the pretrained model, not the fine-tuning optimizer. Learning–forgetting frontiers of the best 60M base models on GSM8K, split by fine-tuning optimizer (solid: AdamW; dashed: Muon). AdamW fine-tuning is always better for AdamW-pretrained models, while the two fine-tuning optimizers give nearly the same frontier for Muon-pretrained models. At large budgets, the AdamW-pretrained model dominates under either choice.
Figure 16: Learning–forgetting frontiers with replay of 10 – 30% DCLM data, loss-optimal 60M base models. Each row shows the budgets around that dataset’s crossover under replay.
Figure 17: Learning–forgetting frontiers with EWC , λ∈{103,4⋅103,104,4⋅104} , loss-optimal 60M base models. Each row shows the budgets around that dataset’s crossover under EWC.
Figure 18: Fine-tuning gap under fine-tuning regularization , with replay (solid) and EWC (dashed), loss-optimal 60M base models. Negative values mean the Muon frontier dominates.
Figure 19: One step at d=512 , GD against Muon (top) and the AdamW surrogate against Muon (bottom); solid is experiment, dashed the closed form of Section 4.2 .
Figure 20: Top: saturated ρ against the data budget , the two curves behind Figure 21 . Bottom: perturbed robustness σ50 against the data budget , below capacity only. One column per d .
Figure 21: Saturated R against the data budget , one panel per d .
Perturbation, δ
Fine-tuning
Size
0.05
0.1
0.13
0.2
Alpaca
GSM8K
MusicPile
StackMathQA
30M
160
40
40
20
160
80
640
320
60M
80
40
40
20
80
80
80
80
100M
160
40
20
10
80
40
80
40
300M
80†
40
20
10
40
10
40
20
600M
>20
20
10
≤5
20
5
10
10
Appendix
Table 2: Crossover budgets, in TPR (tokens per parameter). Left: the smallest budget at which AdamW’s loss under a perturbation of magnitude δ is below Muon’s (loss-optimal checkpoints, mean over three perturbation draws); >20 : Muon’s is lower up to the largest 600M budget, 20 TPR; ≤5 : AdamW’s is lower already at the smallest budget; † : Muon’s is lower again at 160 TPR, by 0.003 nats. Right: the fine-tuning crossover of Figure 2 .
Figure 22: The perturbation degradation gap is positive at every perturbation magnitude. Muon minus AdamW increase in held-out DCLM loss. Left three panels: one δ each; right: 100M across δ .
Table 25
Figure 23: Top: σ50 against ρ in the linear model , one panel per model dimension, one point per γ cell, below capacity only. Bottom: real language models, δ∗ against ρ at every threshold ; marker shape is the model size.
Muon has emerged as an efficient alternative to Adam for pretraining, yet remains underused for fine-tuning. A key obstacle is that most open models are pretrained with Adam, and naively switching to Muon for fine-tuning leads to degraded performance due to an optimizer mismatch. We investigate this mismatch through controlled experiments and relate it to the distinct implicit biases of Adam and Muon. We provide evidence that the mismatch disrupts pretrained knowledge, and that this disruption scales with update strength. This leads us to hypothesize that constraining updates should mitigate the mismatch. We validate this with LoRA: across language and vision tasks, LoRA reduces the performance gap between Adam and Muon observed under full fine-tuning. Studies on LoRA rank, catastrophic forgetting, and LoRA variants further confirm that mismatch severity correlates with update strength. These results shed light on how optimizer mismatch affects fine-tuning and how it can be mitigated. Our code is available at https://github.com/XingyuQu/muon-finetune.
Orthogonal optimisers such as Muon can substantially accelerate large language model pretraining relative to Adam, yet the mechanism remains incompletely understood. We investigate this through an out-of-sample spectral probing analysis of Transformer loss landscapes. At checkpoints along real training trajectories, we decompose each momentum buffer into its singular directions and estimate the loss-optimal step size along each direction on held-out data. The resulting spectral profile is anisotropic yet stable across batches and training stages, and consistent across the optimisers and model scales: a volatile head operating at the Edge-of-Stability supports a much smaller step size than the tolerant bulk, which permits substantially larger steps. This profile provides a unified spectral allocation account of why Muon outperforms Adam, which outperforms SGD. It also exposes a limitation of Muon's uniform scaling: it still underutilises the bulk. Guided by this finding, we introduce Spectral-Aware Muon (SAMuon), which holds the head at the Muon scale and amplifies the bulk using a static spectral prior. We provide two variants: the complete SAMuon follows the measured profile using a low-rank randomised SVD and the simplified SAMuon-lite uses a two-level approximation via rank-one power iteration. Neither method adds persistent optimiser state or notable extra FLOPs beyond Muon at scale, and the idealised exact-whitening versions of both retain Muon's asymptotic convergence rate under standard assumptions. Across "modded-nanogpt" models from 124M to 1B parameters, both variants outperform tuned AdamW and Muon (Scion implementation) baselines in all evaluated model-scale and batch-size configurations. SAMuon requires 13.3% to 24.0% fewer training tokens to reach the same validation loss as Muon, while SAMuon-lite retains most of this gain with near-zero wall-clock overhead.
Xiaodong Wu, Wenyi Yu, Chao Zhang +1
Department of Engineering University of Cambridge · Department of Electronic Engineering Tsinghua University
Muon has emerged as a strong competitor to AdamW for language model pre-training, yet its behavior at scale is sensitive to weight decay. Recent work has observed that, for Muon without decoupled weight decay, the spectral norm of weight matrices drifts upward over training. Through a decomposition of the spectral norm into a row-magnitude factor and a row-coherence factor, we identify the former as the empirical driver of this drift under Muon, while the latter remains well-behaved along the trajectory. Motivated by this diagnosis, we introduce Muown, a drop-in replacement for Muon that treats the row-magnitude vector as an explicit optimizer variable, updating it under the ℓ∞ geometry induced by the decomposition, while applying Muon unchanged to the remaining direction component. We prove that Muown attains the optimal non-convex rates in both deterministic and stochastic regimes under a dual norm aligned with the underlying geometries and with a stochastic noise coefficient that empirically remains below that of Muon throughout training. Across GPT-style pre-training on FineWeb-Edu with model sizes from 124M up to 2.7B parameters, Muown improves perplexity over Muon, SOAP, AdamW, and Lion. It also widens the plateau of near-optimal learning rates across model scales, reduces sensitivity to weight decay, and avoids the spectral norm drift at negligible step-time overhead when appropriately sharded.
Kai Lion, Florian Hübler, Bingcong Li +2
Department of Computer Science, ETH Zurich, Switzerland · Department of Mathematics, School of Computation, Information and Technology; Technical University of Munich, Germany · ELLIS Institute Tübingen, MPI-IS, Tübingen AI Center, Germany