Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muon

Xiaodong Wu; Wenyi Yu; Chao Zhang; Philip Woodland

cs.LGAug 26, 2026

Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muon

Authors: Xiaodong Wu, Wenyi Yu, Chao Zhang, Philip Woodland

Organizations: Department of Engineering University of Cambridge · Department of Electronic Engineering Tsinghua University

Abstract

Orthogonal optimisers such as Muon can substantially accelerate large language model pretraining relative to Adam, yet the mechanism remains incompletely understood. We investigate this through an out-of-sample spectral probing analysis of Transformer loss landscapes. At checkpoints along real training trajectories, we decompose each momentum buffer into its singular directions and estimate the loss-optimal step size along each direction on held-out data. The resulting spectral profile is anisotropic yet stable across batches and training stages, and consistent across the optimisers and model scales: a volatile head operating at the Edge-of-Stability supports a much smaller step size than the tolerant bulk, which permits substantially larger steps. This profile provides a unified spectral allocation account of why Muon outperforms Adam, which outperforms SGD. It also exposes a limitation of Muon's uniform scaling: it still underutilises the bulk. Guided by this finding, we introduce Spectral-Aware Muon (SAMuon), which holds the head at the Muon scale and amplifies the bulk using a static spectral prior. We provide two variants: the complete SAMuon follows the measured profile using a low-rank randomised SVD and the simplified SAMuon-lite uses a two-level approximation via rank-one power iteration. Neither method adds persistent optimiser state or notable extra FLOPs beyond Muon at scale, and the idealised exact-whitening versions of both retain Muon's asymptotic convergence rate under standard assumptions. Across "modded-nanogpt" models from 124M to 1B parameters, both variants outperform tuned AdamW and Muon (Scion implementation) baselines in all evaluated model-scale and batch-size configurations. SAMuon requires 13.3% to 24.0% fewer training tokens to reach the same validation loss as Muon, while SAMuon-lite retains most of this gain with near-zero wall-clock overhead.

Figures & tables

Figure 1: Out-of-sample loss-optimal step profile across training. Spectral probing ( Section 4 ) of the updated momentum buffer , along a Muon (Scion) trajectory of a 64M-parameter “modded-nanogpt” model (exact run in Pethick et al. (2025) ), with checkpoints from iterations 200–3500 in the constant-learning-rate stage. Each point shows the mean over 20 held-out batches, and error bars indicate the standard deviation across batches. The horizontal axis runs from the deep tail (rank 512) to the head (rank 1) in log scale. Left: cumulative spectral energy, concentrated in the leading ranks (the head alone carries the majority). Middle: per-rank optimal spectral allocation

\eta^{*}_{d}

in log scale (the LQA-estimated loss-optimal step size along each probe): this spectral profile is mostly flat across the bulk and drops sharply near the volatile head; the head’s optimal step coincides with the actual (uniform) allocation of Muon (red dashed line,

0.01221

), pinned at the Edge-of-Stability ( Cohen et al., 2021 ) . Right: cumulative loss reduction (LQA estimate), accumulated for each probe from the deep tail to the head and normalised by the per-iteration optimal total, under two allocation policies: each rank at its own optimal allocation (solid, reaching

100\%

by construction) versus every rank at the actual uniform allocation of Muon (dashed). The head-pinned allocation of Muon realises only a small fraction of this idealised per-iteration optimum.

Figure 2: The near-head decline of the loss-optimal step profile is close to linear in log rank, across model scales. We replot the probed spectral profile with a linear

y

-axis against log rank. Left: the width-512 (64M) model (the middle panel of Figure 1 ); Right: the width-768 (124M) model (see appendix C ). At both scales, the loss-optimal step decreases approximately linearly with log rank from the bulk plateau to the head. The visually estimated plateau onset shifts from rank

\sim 32

at width 512 to rank

\sim 40

at width 768, consistent with the square-root width heuristic used for SAMuon in Section 5 .

Figure 3: Optimisers differ in how they scale the momentum-buffer spectrum, and SAMuon / SAMuon-lite match the measured optimum most closely. The updates generated by different optimisers implicitly assign a per-rank scaling to the singular components of the momentum buffer , and we overlay these allocations against the optimal ∗ profile (the iteration-1000 out-of-sample loss-optimal step size profile of Muon from Figure 1 ). All profiles are head-normalised to

1

and evaluated at the iteration-1000 checkpoint, with the Adam allocation drawn from the AdamW trajectory (see appendix E ). Left: scaling per spectral rank (log axes). Right: the same allocations as a function of normalised singular value

\sigma

(linear axes). SAMuon-lite (

\gamma=5

) raises the bulk to a single uniform level well above the Muon scale while holding the head at the same scale as Muon, and SAMuon (

\gamma=7.07

) additionally follows the measured log-rank-linear decline over the leading ranks, the closest shape match to the measured optimum profile. From SGD

\to

Adam

\to

Muon

\to

SAMuon-lite

\to

SAMuon, each successive optimiser aligns its allocation more closely with what the spectral profile suggests.

Model	Batch	Tokens	AdamW	Muon	SAMuon		SAMuon-lite
Model	Batch	Tokens	AdamW	Muon	loss	improv.	loss	improv.
124M	1024	10B	3.2105	3.1622	3.1435	20.3%	3.1438	20.3%
	2048	10B	3.2410	3.1726	3.1484	23.8%	3.1511	21.5%
	4096	10B	3.3050	3.1953	3.1658	24.0%	3.1678	22.1%
300M	1024	10B	3.0297	2.9836	2.9640	17.7%	2.9663	15.9%
	2048	10B	3.0658	2.9941	2.9713	18.8%	2.9738	17.1%

Table 1: Final validation loss and token-efficiency improvement across the model-scale

\times

batch-size grid. The hyperparameters for each optimiser are tuned for every batch size setup on the 124M model. The larger model scales (300M and 1B) inherit the tuned hyperparameters from the corresponding batch size of the 124M model, following the scaling convention of Pethick et al. (2025) (see Section 6.1 ). The final loss is reported at the end of the training budget (10B tokens for 124M and 300M, 20B tokens for 1B). The token-efficiency improvement (improv.) is the token budget saving

1-T^{\star}/N

of a SAMuon variant over the Muon baseline, where

N

is the schedule length of the tuned Muon baseline and

T^{\star}

the shortest schedule with which the variant still reaches the same final loss ( appendix H ). The best (unrounded) loss and token-efficiency improvement in each row are highlighted in bold.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Validation of the local-quadratic estimate against the exact line search. Left: model-wide loss-optimal update norm

s_{d}^{*}

, estimated by the LQA (x-axis) and by the golden-section line-search reference (y-axis), for the 773 probe directions sampled from 24 checkpoint–batch analyses spanning all training stages and both Muon and AdamW trajectories; the dashed line marks perfect agreement (

y=x

). Right: relative difference between the two as a function of the LQA estimate, with the windowed median and interquartile range overlaid. The median deviation stays near

5\%

over the operationally relevant range, rising only for extremely flat directions where the loss minimum is shallow.

Figure 5: Out-of-sample loss-optimal step profile along an AdamW trajectory (same 64M-parameter “modded-nanogpt” model as Figure 1 , checkpoints from iterations 200–3500). Out-of-sample spectral probing ( Section 4 ) of the updated first-moment (momentum) buffer, with each per-rank probe’s optimal step size measured on a disjoint out-of-sample batch; the horizontal axis runs from the deep tail (rank 512) to the head (rank 1, log scale). Left: cumulative spectral energy, again concentrated in the leading ranks. Middle: per-rank optimal spectral allocation

\eta^{*}_{d}

(the loss-optimal step along each probe; LQA estimate; log axis). As with Muon, the bulk is mostly flat and the optimal allocation drops sharply near the head, which tolerates a far smaller step. Because the AdamW update is unnormalised, the realised update norm varies, and the head’s loss-optimal step therefore varies more freely across checkpoints; the slower convergence of AdamW also prolongs the burn-in, giving a more unstable profile in early checkpoints (iterations 200–500) that settles to the same volatile-head shape later (iterations 1000–3500). Right: cumulative loss reduction (LQA estimate): as for Muon, the leading ranks account for most of the achievable reduction and the deep tail contributes a modest residual share. The close structural agreement with Figure 1 shows that the volatile-head and tolerant-bulk structure persists across the evaluated Muon and AdamW trajectories.

Figure 6: Out-of-sample loss-optimal step profile at width 768 (124M model, Muon trajectory). Panels, axes, and protocol as in Figure 1 . The qualitative structure of the width-512 analysis is reproduced.

Figure 7: In-sample loss-optimal step profile along a Muon trajectory , in the same panel layout as the out-of-sample Figure 1 (same 64M-parameter “modded-nanogpt” model and checkpoints), but with each probe’s optimal step size measured on the same batch whose gradient formed the buffer (

\mathcal{B}_{\mathrm{o}}=\mathcal{B}_{\mathrm{g}}

). Two differences from the out-of-sample profile stand out. First, the entire bulk is raised well above its out-of-sample level (middle panel), widening the anisotropy. Second, the deep tail contributes far more to the idealised cumulative loss reduction (right panel). Both indicate that, in-sample, the bulk mainly yields refitting of the batch that formed the update, a gain that does not transfer to unseen data, which is exactly the confounder the main-text out-of-sample protocol removes.

Figure 8: In-sample versus out-of-sample spectral profiles along a Muon trajectory (64M-parameter “modded-nanogpt”, checkpoints from iterations 200–3500). Solid curves are the out-of-sample (main-text) measurement; dashed curves are the in-sample variant. Left: per-rank optimal spectral allocation. The in-sample allocation is systematically higher across the entire bulk, several times the out-of-sample level. Both collapse together at the head, pinned at the actual Muon scale (red dashed,

0.01221

). Right: per-probe optimal loss reduction (log axis). In-sample, every probe across the spectrum, particularly the deep tail, yields a large reduction; out-of-sample, the per-probe reduction in the tail is far smaller and only climbs towards the head. The gap between the two reflects that the tail spectrum of the gradient batch mainly contributes to overfitting that batch, with limited transferable capacity to out-of-sample batches.

Figure 9: In-sample loss-optimal step profile along an AdamW trajectory , the AdamW analogue of the Muon Figure 7 and the in-sample counterpart of the out-of-sample Figure 5 (same 64M-parameter modded-nanogpt model and checkpoints; three-panel layout as Figure 1 ), with each probe’s optimal step size measured on the same batch whose gradient formed the buffer (

\mathcal{B}_{\mathrm{o}}=\mathcal{B}_{\mathrm{g}}

). As for Muon ( Figure 7 ), the bulk is lifted above its out-of-sample level (middle panel) and the deep tail contributes far more of the idealised cumulative loss reduction (right panel), the same batch-refitting signatures that the main-text out-of-sample protocol removes.

Figure 10: In-sample versus out-of-sample spectral profiles along an AdamW trajectory (64M-parameter “modded-nanogpt”, checkpoints from iterations 200–3500), the AdamW analogue of the Muon Figure 8 ; panels and axes as there, with solid curves out-of-sample and dashed in-sample. As for Muon, the in-sample allocation is systematically more permissive across the bulk (widest in the deep tail) and every in-sample probe yields a large loss reduction, whereas out-of-sample the tail’s contribution collapses; the gap is the batch-refitting capacity that does not transfer to unseen data.

Specification	64M	124M	300M	1B
Layers ( $L$ )	12	12	12	12
Model Dimension ( $d_{\text{model}}$ )	512	768	1280	2560
Attention Heads ( $H$ )	4	6	10	20
Head Dimension ( $d_{h}$ )	128	128	128	128
Vocabulary Size	50,304	50,304	50,304	50,304
Context Length	1024	1024	1024	1024

Appendix

Table 2: Model architectures. The 64M, 124M, 300M and 1B models used for probing and/or training experiments. All utilise a head dimension of 128 and a depth of 12 layers.

Figure 11: The spectral anisotropy emerges over early training, motivating the slow warmup. Out-of-sample probing along the Muon trajectory (64M-parameter “modded-nanogpt”), against training iteration (log axis). Left: the head gap (rank-2 over rank-1 optimal spectral allocation ratio, the largest single gap in the measured profile) rises from near-isotropic (

\approx 2

) at iteration 50 towards its final level of

\approx 5

(red dotted), on which the

\gamma

search grid of our experiments is centred, within a few hundred iterations, and stays there; the isotropic-curvature assumption (

=1

) is violated throughout and increasingly so. Middle: the head’s concentration of spectral energy,

\sigma_{1}^{2}/\sum_{i}\sigma_{i}^{2}

; Right: the head’s concentration of directional curvature,

\big(\Delta\bm{\theta}^{(1)\top}\mathbf{H}\,\Delta\bm{\theta}^{(1)}\big)\big/\sum_{d}\Delta\bm{\theta}^{(d)\top}\mathbf{H}\,\Delta\bm{\theta}^{(d)}

, using the rank-

d

directional curvature of Equation 7 ; both concentrate over the same window, from

{\sim}70\%

{\sim}85

–

95\%

. Early in training the spectrum is closer to isotropic, so SAMuon and SAMuon-lite delay the full anisotropic shaping and warm the profile from the uniform scaling of Muon to its target over the first

30\%

of training (much slower than standard warmup schedules).

Global Settings
Parameter	AdamW	Muon (Scion)	SAMuon-lite	SAMuon
Weight Decay	0.0	0.0	0.0	0.0
Gradient Clip	1.0	None	None	None
LR Schedule	Trapezoidal	Trapezoidal	Trapezoidal	Trapezoidal
LR Warmup	5%	0%	0%	0%
Hyperparameter sweeps at 124M (optima at bs 1024/2048/4096)

Appendix

Table 3: Optimiser hyperparameters. All values are tuned on the 124M model per batch size, with optima listed as bs

1024/2048/4096

and the full sweeps in Table 5 . Both SAMuon instantiations run at the fixed spectral radius

50

. The profile rank

k

of SAMuon follows the fixed width rule described below and is not tuned. All values transfer across model scales, except the AdamW learning rate, which is halved at each scale step.

Iteration $\ell$	$a_{\ell}$	$b_{\ell}$	$c_{\ell}$
1	5.30697775	$-9.73226547$	4.52926445
2	3.99123669	$-4.20899105$	1.18235242
3	2.66316843	$-2.16650701$	0.56325209
4	1.93040931	$-1.31219244$	0.38289258

Appendix

Table 4: Coefficients of the custom

4\times 3

Newton–Schulz scheme.

Figure 12: Response of the custom

4\times 3

Newton–Schulz scheme after four iterations. The dashed line denotes exact orthogonalisation, which maps every non-zero singular value to one. For normalised input singular values

\sigma\geq 0.02

, the maximum response error is

0.0069

AdamW — learning rate ( $\log_{2}$ )
Hyperparameter	bs 1024	bs 2048	bs 4096
$-10.5$	3.2363	–	–
$-10$	–	3.2620	–
$-9.5$	3.2113	–	3.3266
$-9$	–	3.2410	–
$-8.5$	3.2105	–	3.3050

Appendix

Table 5: Hyperparameter searches at 124M (width 768, FineWeb-100B,

{\sim}

10B tokens), for each batch size. Entries are final validation loss, and the per-batch optimum of each sweep is in bold. AdamW is swept over learning rate (

2^{x}

), Muon over its spectral radius, and both SAMuon instantiations over the tail boost

\gamma

at the re-anchored radius

50

, the Scion default. The Muon optimum shifts with batch size (

70.71/50/70.71

), whereas the SAMuon optimum is

\gamma=7.07

at all batch sizes. Searches at the superseded radius

70.71

are omitted for the SAMuon instantiations.

Component	ms/iter	% of Muon iter
Forward + backward ( $\times 64$ microbatches, compiled, bf16)	8133.8	91.2%
Muon: Newton-Schulz ( $4\times 3$ )	782.1	8.8%
SAMuon: $+$ torch.svd_lowrank	657.8	7.4%
SAMuon-lite: $+$ power iteration (bf16)	41.2	0.5%

Appendix

Table 6: Per-GPU wall-clock cost of one training iteration (1B model, batch-size-1024 configuration), measured on a single NVIDIA RTX 6000 Ada GPU. The full Muon iteration is the forward+backward computation plus the Newton-Schulz whitening (

8916

ms in total), and percentages are relative to this total. The SAMuon and SAMuon-lite rows give the additional cost of each variant’s head estimate on top of the Muon iteration.

				Token-effic. improv. $1-T^{\star}/N$
Model	Batch	Tokens	$N$	SAMuon	SAMuon-lite
124M (w768)	1024	10B	9600	20.3%	20.3%
	2048	10B	4800	23.8%	21.5%
	4096	10B	2400	24.0%	22.1%
300M (w1280)	1024	10B	9600	17.7%	15.9%
	2048	10B	4800	18.8%	17.1%

Appendix

Table 7: Token-efficiency improvement of SAMuon over tuned Muon at matched final validation loss.

N

is the tuned-Muon schedule length, and

T^{\star}

is the estimated SAMuon or SAMuon-lite schedule length that matches Muon’s final loss, found using Algorithm 2 and checked with a confirmation run. SAMuon uses spectral radius

50

; Muon uses its tuned radius for each batch size. The shaping warmup remains fixed at

0.30N

absolute steps, making the reported improvements conservative. The 1B runs use 20B tokens; the others use 10B.

Figure 13: Per-weight-matrix sign-alignment probability. For the checkpoint at iteration 1000, the probability that a weight matrix’s rank- $d$ buffer direction $\bm{u}_{j,d}\bm{v}_{j,d}^{\top}$ is aligned (has positive projection) with the out-of-sample gradient, estimated over 20 held-out batches. Rows group the six weight types (attention query, key, value, and output projections; multilayer perceptron (MLP) up- and down-projections), each shown for all twelve layers L0–L11; columns run from the deep tail (rank 512) to the head (rank 1, log axis). Blue marks near-certain alignment, red near-certain anti-alignment, and white a coin-flip. Two patterns stand out. First, the top few ranks are almost always aligned (the deep-blue right edge across all weight matrices), so the head’s orientation is effectively deterministic. Second, in the tail the four attention projections drift towards $0.5$ and below, i.e. their low-energy directions flip sign freely from batch to batch, whereas the MLP projections stay strongly aligned across almost the entire spectrum. The query and key projections show the worst tail alignment, so their tail directions are the least trustworthy; compare this observation with the QK-clip practice of Kimi Team [2025] . This weight-matrix-dependent sign-flip structure is the inter-block variation that our per-rank clustering averages over ( Section 7 ); the cross-batch cancellation it produces in the tail is one source of the mostly flat bulk.

Deep Learning Optimization

Deep learning optimization develops algorithms and training strategies that help neural networks converge, generalize, and use compute efficiently.

See all 198 papers

Oct 7, 2026

Spectrally Targeted Muon

The Muon optimizer orthogonalizes each update matrix, setting all of its singular values to one, and has proven highly effective for training large language models. It remains unclear, however, whether this success comes from amplifying small singular directions that gradient descent neglects or from suppressing large, degenerate directions that disrupt training. We introduce Spectrally Targeted Muon, which orthogonalizes only the singular values above or below a threshold

τ

, so that varying

τ

interpolates between normalized SGD and Muon. It isolates the relevant singular subspaces with projections computed by Newton-Schulz iteration on a shifted Gram matrix, so no SVD is needed. We evaluate these variants on the CIFAR-10 and NanoGPT speedruns, tracking the effective rank of gradient, update, and weight matrices and a new metric, the alignment of updates with the tangent space of the weight matrix's isospectral manifold. We find three things. First, the small singular values of the momentum are not noise. Orthogonalizing everything except the few largest singular values of each matrix nearly matches Muon while touching only a small fraction of the momentum, whereas orthogonalizing only the top falls well short even though it holds almost all of it. On language models every momentum singular value is far below one, so targeted orthogonalization can only amplify, and Muon wins by making directions that are too small to train on at their raw scale trainable. Second, shrinking the largest singular values is what keeps the parameter spectrum flat; this is cheap, and it is not what drives the loss. Third, AdamW differs from Muon mainly in how slowly it builds structure, which explains its slower start and why warmup helps AdamW but only hurts Muon.

Vishrut Goyal, Rohan Ramkumar

Department of Computer Science, Stanford University · Department of Electrical Engineering, Stanford University.

Spectral Optimization Deep Learning Optimization Muon Optimizer LLM Training Spectral Methods

Ask PDF

Sep 28, 2026

LionMuon: Alternating Spectral and Sign Descent for Efficient Training

Pretraining a language model takes enormous compute, and the right optimizer can save a good part of it. Muon's spectral step gives a stronger direction than a sign step, but it is expensive. Every step runs Newton-Schulz iterations on the full matrix and, in distributed training, an extra all-reduce. Sign steps, as in Lion and Signum, are cheap and stay local to each device. We propose LionMuon, which takes one Muon step every

P

iterations and Lion steps in between, with a single dual-EMA momentum buffer shared by both. Muon's compute and communication are paid once per

P

steps, and the optimizer state is half of AdamW's. A single-EMA variant, SignMuon, already improves on Muon. We prove complexity bounds under heavy-tailed noise in which the period sets an interpolation between Muon's and Lion's smoothness and noise constants, and which say when LionMuon is faster than both. On 124M and 355M models trained on FineWeb, LionMuon with

P=2

and

P=5

reaches a lower loss than Muon, AdamW, Lion and Signum at the same number of tokens. Under 4-GPU data-parallel training it reaches Muon's final loss with a third less wall-clock on PCIe, and it beats the communication-efficient Muon variants Dion and MuonBP on loss at no more exposed communication, while keeping the exact gradient. Code: https://github.com/brain-lab-research/lion-muon

Arman Bolatov, Artem Riabinin, Nikita Kornilov +4

Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · Applied Artificial Intelligence Institute +1

Stochastic Optimization Language Model Pretraining Sign-Based Optimization Communication-Efficient Distributed Training Gradient Descent Efficient Language Model Training

Ask PDF

May 18, 2026

AMO: Operator-level Adaptive Muon Orthogonalization

Muon has recently emerged as a competitive alternative to AdamW for large-scale pre-training, with orthogonalization via Newton-Schulz (NS) iteration as its core operation. Standard Muon applies a uniform NS schedule to all parameter matrices, overlooking possible differences in orthogonalization difficulty and its impact on performance. Through a systematic empirical study, we show that this per-matrix heterogeneity is pervasive and strongly associated with matrix geometry, which evolves dynamically across operator types, training stages, and network depths. Therefore, uniform NS schedules can lead to uneven orthogonalization quality across the model. Motivated by these findings, we propose Operator-level Adaptive Muon Orthogonalization (AMO), an observe-then-commit method that measures weight geometry by operator type early in training and then uses these signals to allocate the NS budget for the remainder of training. AMO delivers consistent improvements over uniform-schedule Muon across standard, prolonged, and continual pre-training, surpassing the strongest baseline by +0.76 on Llama3.1-1.4B and +0.51 on Qwen3-1.7B in average downstream performance of 12 evaluation tasks, with gains persisting at Llama3.1-4B scale.

Xinlin Zhuang, Panyi Ouyang, Yichen Li +7

The Chinese University of Hong Kong · Shopee · MBZUAI +3

Language Model Pretraining Newton-Schulz Iteration Muon Optimizer Second-Order Optimization Adaptive Gradient Methods

Ask PDF

Abstract

Deep Learning Optimization

Explore similar work

Spectrally Targeted Muon

LionMuon: Alternating Spectral and Sign Descent for Efficient Training

AMO: Operator-level Adaptive Muon Orthogonalization