Pretraining Data Scale Reverses the Muon-AdamW Fine-Tuning Gap

Catherine Li; Jingchu Gai; Andrej Risteski; Jacob Mitchell Springer; Aditi Raghunathan

cs.LGNov 2, 2026

Pretraining Data Scale Reverses the Muon-AdamW Fine-Tuning Gap

Authors: Catherine Li, Jingchu Gai, Andrej Risteski, Jacob Mitchell Springer, Aditi Raghunathan

Organizations: Carnegie Mellon University

Abstract

Muon has recently emerged as a promising optimizer for pretraining large language models, offering a speedup over AdamW in minimizing pretraining loss. However, it remains unclear whether Muon's advantage persists after fine-tuning. In this paper, we study the downstream behavior of Muon and AdamW and uncover a surprising trend: beyond a sufficiently large pretraining data budget, AdamW outperforms Muon after fine-tuning. Moreover, as model size increases, AdamW overtakes Muon at progressively smaller pretraining token-parameter ratios. To explain this reversal, we view fine-tuning as a perturbation of the outputs and introduce the margin-to-spread ratio, which measures how far the correct logit lies above the wrong ones relative to how widely the wrong logits are spread. In language models, Muon has the larger margin-to-spread ratio at small data budgets and AdamW at large ones, mirroring the fine-tuning trend. In a linear associative memory, we prove that after one update the margin-to-spread ratio of gradient descent gains on that of Muon as the data budget grows, and simulations show that this trend persists after extended training.

Figures & tables

Figure 1: Beyond a sufficiently large data budget, AdamW outperforms Muon. We pretrain OLMo-60M models with Muon and AdamW and fine-tune on StackMathQA. For each pretraining optimizer, we take the pretrained model with the lowest pretraining loss, sweep across all fine-tuning configurations, and plot the resulting learning–forgetting frontier, from

2.4

B to

38.4

B pretraining tokens. At

2.4

B tokens, Muon’s frontier dominates AdamW’s; at larger budgets the trend reverses and AdamW surpasses Muon. This reversal happens despite Muon achieving lower pretraining loss across all data budgets.

Figure 2: Crossover TPR decreases with model size. In the lowest-pretraining-loss setting, where each pretraining optimizer’s lowest-loss pretrained model is swept across all fine-tuning hyperparameters, Muon’s advantage reverses at substantially smaller pretraining data budgets as model size increases. The exact crossover TPR varies depending on the fine-tuning dataset, but the high-level trend is consistent.

Figure 3: The Muon–AdamW fine-tuning gap widens with increasing pretraining data budget, and Muon’s advantage shrinks with model size. (a) The fine-tuning gap after fine-tuning OLMo-60M models on StackMathQA, in the lowest-pretraining-loss setting (each pretraining optimizer’s lowest-loss pretrained model swept across all fine-tuning configurations). As the pretraining data budget increases, the fine-tuning gap transitions from negative to positive, with AdamW outperforming Muon beyond the crossover TPR. (b) Pretraining loss at 40 TPR. Muon’s advantage over AdamW shrinks as model size grows. (c) Degradation at 40 TPR, when we accept a StackMathQA loss

10\%

above the best in exchange for less forgetting. Muon degrades increasingly more than AdamW as model size grows.

Figure 4: The margin-to-spread ratio of GD gains on Muon’s as the data budget grows. (a, b) After

t=10^{4}

steps with model dimension

d=256

, GD’s MSR

\rho

overtakes Muon’s as

\gamma

grows, and the ordering of the perturbed robustness

\sigma_{50}

turns over with it. (c, d) After one step with

d=512

, GD’s normalized MSR

\rho/\sqrt{d/\gamma}

overtakes Muon’s (c), so the MSR ratio

\mathcal{R}=\rho^{\mathrm{GD}}/\rho^{\mathrm{Muon}}

rises with

\gamma

(d), following the theory of Section 4.2 .

Figure 5: In real language models, the margin-to-spread ratio of AdamW gains on Muon’s as the token budget grows. (a) MSR

\rho

of AdamW and Muon against pretraining TPR for 30M, 60M, and 100M models: Muon has the larger MSR at small budgets and AdamW at large ones. (b) The MSR ratio

\rho^{\mathrm{AdamW}}/\rho^{\mathrm{Muon}}

crosses one as the budget grows, at every model size.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: The perturbation degradation gap grows with pretraining data budget. Under a perturbation of magnitude

\delta=0.02

, Muon degrades increasingly more than AdamW as the pretraining data budget grows. Points show the mean over three noise draws.

Figure 7: The margin-to-spread ratio tracks robustness to weight perturbations. (a) For the linear models trained under GD and Muon, we plot the perturbation

\sigma_{50}

required to lose half of the recovered facts against its MSR,

\bar{h}/\bar{\kappa}

( Section 4.1 ). A higher MSR requires more noise to cause the same loss. (b) We measure the MSR of

72

pretrained language models (30M–600M) as the median of

h/\kappa

over correctly predicted tokens, and plot the perturbation

\delta^{\ast}

at which held-out loss rises by

\Delta L

. The same relationship holds.

Size	Optimizer	TPR
		$\mathbf{5}$	$\mathbf{10}$	$\mathbf{20}$	$\mathbf{40}$	$\mathbf{80}$	$\mathbf{160}$	$\mathbf{320}$	$\mathbf{640}$	$\mathbf{1{,}280}$	$\mathbf{2{,}560}$
30M	AdamW	$0.08$	$0.04$	$0.04$	$0.014$	$0.02$	$0.014$	$0.02$	$0.014$	$0.01$	–
30M	Muon	$0.01$	$0.01$	$0.01$	$0.014$	$0.02$	$0.01$	$0.014$	$0.01$	$0.014$	–
60M	AdamW	$0.056$	$0.04$	$0.02$	$0.02$	$0.01$	$0.014$	$0.01$	$0.01$	$0.007$	$0.01$
60M	Muon	$0.014$	$0.01$	$0.014$	$0.01$	$0.014$	$0.01$	$0.014$	$0.007$	$0.005$	$0.005$
100M	AdamW	$0.04$	$0.02$	$0.01$	$0.01$	$0.01$	$0.007$	$0.007$	$0.007$	–	–

Appendix

Table 1: Tuned peak pretraining learning rates per model size and budget. For Muon, the Muon learning rate; the AdamW-parameter learning rate is fixed to the tuned AdamW value.

Figure 8: Muon is insensitive to the learning rate of its AdamW-updated parameters. Best pretraining loss over Muon learning rates for each AdamW learning rate, relative to the tuned AdamW value (dashed), for 60M models.

Figure 9: Muon reaches lower pretraining loss than AdamW. DCLM held-out loss of the loss-optimal base models across pretraining data budgets and model sizes.

Figure 10: Fine-tuning gap on every fine-tuning dataset, loss-optimal base models (60M). Negative values mean the Muon frontier dominates.

Figure 11: Learning–forgetting frontiers on every fine-tuning dataset, loss-optimal base models (60M). Each row shows the budgets around that dataset’s crossover TPR.

Figure 12: Fine-tuning gap on every fine-tuning dataset, best-frontier setting (60M). Negative values mean the Muon frontier dominates.

Figure 13: Learning–forgetting frontiers on every fine-tuning dataset, best-frontier setting (60M). Each row shows the budgets around that dataset’s crossover TPR.

Figure 14: Crossover TPR decreases with model size, in the best-frontier setting. As model size increases, Muon’s advantage reverses with substantially smaller pretraining data budgets.

Figure 15: Muon’s greater forgetting comes from the pretrained model, not the fine-tuning optimizer. Learning–forgetting frontiers of the best 60M base models on GSM8K, split by fine-tuning optimizer (solid: AdamW; dashed: Muon). AdamW fine-tuning is always better for AdamW-pretrained models, while the two fine-tuning optimizers give nearly the same frontier for Muon-pretrained models. At large budgets, the AdamW-pretrained model dominates under either choice.

Figure 16: Learning–forgetting frontiers with replay of

10

–

30\%

DCLM data, loss-optimal 60M base models. Each row shows the budgets around that dataset’s crossover under replay.

Figure 17: Learning–forgetting frontiers with EWC ,

\lambda\in\{10^{3},4\cdot 10^{3},10^{4},4\cdot 10^{4}\}

, loss-optimal 60M base models. Each row shows the budgets around that dataset’s crossover under EWC.

Figure 18: Fine-tuning gap under fine-tuning regularization , with replay (solid) and EWC (dashed), loss-optimal 60M base models. Negative values mean the Muon frontier dominates.

Figure 19: One step at

d=512

, GD against Muon (top) and the AdamW surrogate against Muon (bottom); solid is experiment, dashed the closed form of Section 4.2 .

Figure 20: Top: saturated

\rho

against the data budget , the two curves behind Figure 21 . Bottom: perturbed robustness

\sigma_{50}

against the data budget , below capacity only. One column per

d

Figure 21: Saturated

\mathcal{R}

against the data budget , one panel per

d

	Perturbation, $\bm{\delta}$				Fine-tuning
Size	$\mathbf{0.05}$	$\mathbf{0.1}$	$\mathbf{0.13}$	$\mathbf{0.2}$	Alpaca	GSM8K	MusicPile	StackMathQA
30M	$160$	$40$	$40$	$20$	$160$	$80$	$640$	$320$
60M	$80$	$40$	$40$	$20$	$80$	$80$	$80$	$80$
100M	$160$	$40$	$20$	$10$	$80$	$40$	$80$	$40$
300M	$80^{\dagger}$	$40$	$20$	$10$	$40$	$10$	$40$	$20$
600M	$>20$	$20$	$10$	$\leq 5$	$20$	$5$	$10$	$10$

Appendix

Table 2: Crossover budgets, in TPR (tokens per parameter). Left: the smallest budget at which AdamW’s loss under a perturbation of magnitude

\delta

is below Muon’s (loss-optimal checkpoints, mean over three perturbation draws);

>20

: Muon’s is lower up to the largest 600M budget,

20

TPR;

\leq 5

: AdamW’s is lower already at the smallest budget; † : Muon’s is lower again at

160

TPR, by

0.003

nats. Right: the fine-tuning crossover of Figure 2 .

Figure 22: The perturbation degradation gap is positive at every perturbation magnitude. Muon minus AdamW increase in held-out DCLM loss. Left three panels: one

\delta

each; right: 100M across

\delta

Figure 23: Top:

\sigma_{50}

against

\rho

in the linear model , one panel per model dimension, one point per

\gamma

cell, below capacity only. Bottom: real language models,

\delta^{\ast}

against

\rho

at every threshold ; marker shape is the model size.

Muon Optimizer

Muon optimizer uses momentum and matrix orthogonalization to shape neural-network updates, especially in large language-model training.

See all 153 papers

Can Muon Fine-tune Adam-Pretrained Models?
May 11, 2026Xingyu Qu, Peigeng Huang, Samuel HorvathFine-Tuning Muon Optimizer
MBZUAI · Nanjing University
Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muon
Aug 26, 2026Xiaodong Wu, Wenyi Yu, Chao Zhang +1Deep Learning Optimization Muon Optimizer
Department of Engineering University of Cambridge · Department of Electronic Engineering Tsinghua University
Muown: Row-Norm Control for Muon Optimization
May 11, 2026Kai Lion, Florian Hübler, Bingcong Li +2Language Model Pretraining Muon Optimizer
Department of Computer Science, ETH Zurich, Switzerland · Department of Mathematics, School of Computation, Information and Technology; Technical University of Munich, Germany · ELLIS Institute Tübingen, MPI-IS, Tübingen AI Center, Germany

May 11, 2026J/K move · Enter open · S save

Can Muon Fine-tune Adam-Pretrained Models?

Xingyu Qu, Peigeng Huang, Samuel Horvath

MBZUAI · Nanjing University

Fine-Tuning Muon Optimizer

Open paper AskPDF arXiv

Muon has emerged as an efficient alternative to Adam for pretraining, yet remains underused for fine-tuning. A key obstacle is that most open models are pretrained with Adam, and naively switching to Muon for fine-tuning leads to degraded performance due to an optimizer mismatch. We investigate this mismatch through controlled experiments and relate it to the distinct implicit biases of Adam and Muon. We provide evidence that the mismatch disrupts pretrained knowledge, and that this disruption scales with update strength. This leads us to hypothesize that constraining updates should mitigate the mismatch. We validate this with LoRA: across language and vision tasks, LoRA reduces the performance gap between Adam and Muon observed under full fine-tuning. Studies on LoRA rank, catastrophic forgetting, and LoRA variants further confirm that mismatch severity correlates with update strength. These results shed light on how optimizer mismatch affects fine-tuning and how it can be mitigated. Our code is available at https://github.com/XingyuQu/muon-finetune.

Abstract

Muon Optimizer

Explore similar work

Can Muon Fine-tune Adam-Pretrained Models?

Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muon

Muown: Row-Norm Control for Muon Optimization