A Quadratic Lens on Muon's Polar Update: Benefits, Pitfalls, and the Role of Newton-Schulz

Egor Shulgin; Sam Laing; Antonio Orvieto; Peter Richtárik

cs.LGOct 21, 2026

A Quadratic Lens on Muon's Polar Update: Benefits, Pitfalls, and the Role of Newton-Schulz

Authors: Egor Shulgin, Sam Laing, Antonio Orvieto, Peter Richtárik

Organizations: King Abdullah University of Science and Technology (KAUST), Saudi Arabia · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), United Arab Emirates · ELLIS Institute Tübingen and MPI for Intelligent Systems, Tübingen AI Center, Germany · OpenEuroLLM

Abstract

Muon updates matrix-valued parameters along approximately orthogonalized directions and often trains large language models faster than AdamW, yet it remains unclear when and why orthogonalization helps. We study PolarGD, the orthogonalized update at the core of Muon, on matrix quadratics, where optimizers can be analyzed in closed form, and compare update directions under line search to remove the effect of learning-rate tuning. Orthogonalization pays off when the gradient is far from low rank and its largest singular directions see high curvature. PolarGD reaches the minimizer in one step whenever the rows of the error are orthogonal with equal norms, for any Hessian, whereas gradient descent needs all Hessian eigenvalues to be equal; and unlike sign descent, PolarGD is unaffected by rotations of the problem. Over many steps, PolarGD acts as gradient descent preconditioned by the current iterate, so its speed is governed by the conditioning of the iterate rather than of the Hessian. This explains fast regimes as well as a slow one, caused by tiny singular values of the iterate, in which PolarGD is much slower than gradient descent. Finally, the few Newton-Schulz iterations used in Muon treat these tiny singular values as gradient descent does, so fewer iterations can converge faster than exact orthogonalization; in a transformer trained with Muon, about a quarter of the singular values of the attention matrices fall in this regime.

Figures & tables

Figure 1: Convergence across initialization–Hessian regimes. Performance of dual-adapted GD , SignGD and PolarGD (the core Muon update) on matrix quadratic objectives

f(X)=\frac{1}{2}\mathrm{tr}(X^{\top}HX)

with tuned fixed step sizes, not line search (see App. A.1 ). To isolate structural dependencies, we test three initializations: Gaussian (

X_{0}\sim\mathcal{N}(0,1)

), Orthogonal and Vertex (

X_{0}\sim\{-1,1\}

). We evaluate these against three Hessian profiles: Isotropic (

H=I_{d}

), Dense Marchenko–Pastur (

\kappa=386

) and Diagonal (matching the spectrum in the dense case). Consistent with our analysis, each optimizer favors a distinct structural regime: GD is robust to initialization but degrades under ill-conditioning, SignGD benefits from coordinate-aligned curvature, and PolarGD is rotation-equivariant and benefits from row-isotropic initialization. No single optimizer dominates across all regimes.

Figure 2: Zipf quadratic with

\lambda_{k}(H)\propto k^{-\zeta}

\zeta\in\{0.25,0.5,1\}

, for diagonal and randomly rotated

H

with the target

X^{\star}

kept fixed; best-tuned dual-adapted fixed steps, median and range over 5 seeds. PolarGD is unaffected by the rotation, while SignGD degrades significantly. Details in App. A.3 .

Figure 3: Alternating local winner. A

2\times 2

diagonal quadratic with

\kappa(H)=10

(details in App. C ). Along the GD trajectory, the one-step advantage

\Delta_{\mathrm{GD}}-\Delta_{\mathrm{Pol}}

alternates sign every iteration. Loss curves are shown in Figure 8 .

Figure 4: Spectral recovery near rank deficiency. (Left) Loss

f(X_{k})

, (Right) exact progress coefficient

p(X_{k})

. We initialize at

X_{0}=O^{\mathsf{T}}\mathrm{diag}(1,\delta,\ldots,\delta)O

for

\delta=10^{-4}

O

random orthogonal,

H=I_{d}

and

d=10

; with

H=I_{d}

, GD is one-step exact, so this setting isolates the effect of the iterate’s spectrum on PolarGD . All methods use exact line search. Exact PolarGD contracts at the fixed rate

(1-2/d)^{2}

per step after the first (Corollary 5 ), while Newton--Schulz damps the tiny singular values and recovers quickly, the more so with fewer steps. GD is shown for reference. Further details in Appendix A .

Figure 5: Muon departs from exact polar on a sizable part of the attention spectrum. Momentum matrices $M$ that Muon orthogonalizes in a 90M-parameter transformer (final checkpoint), with singular values normalized by $\left\lVert M\right\rVert_{\mathrm{F}}$ as in training. (a) Top: output vs. input singular value for exact polar, normalized GD , Muon ’s five Newton--Schulz (NS) steps, and damped polar with $\varepsilon=\ell_{5}^{-2}$ ; below the damping scale $1/\ell_{5}$ (shaded, GD -like), Muon ’s output is proportional to its input, as for GD , and above it Muon behaves like exact polar. Bottom: distribution of the singular values. (b) Percentage of singular values below $1/\ell_{5}$ , by layer and matrix; the embedding and the LM head have none.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: The Zipf quadratic of Figure 2 with diagonal

H

, with

H

rotated and

X^{\star}

fixed (as in Figure 2 ), and with

H

and

X^{\star}

rotated together (

X^{\star}\mapsto R^{\mathsf{T}}X^{\star}

). Same tuning; median and range over 5 seeds. In the third panel, the PolarGD curves coincide with those of the first by Theorem 2 .

Figure 7: Tracking the rank for PolarGD . For the experiments of App. A.2 (Figure 1 ), we track the rank of the gradient

G=HX_{k}

for the best performing PolarGD learning rate

\gamma_{\mathrm{best}}

along with

0.1\gamma_{\mathrm{best}}

and

10\gamma_{\mathrm{best}}

. The rank remains near full across the nine regimes, validating the full-rank assumption used in Theorem 3 .

Direction $D$	gain $\langle G,D\rangle$	curvature $\mathrm{tr}(D^{\mathsf{T}}HD)$	one-step exactness for $H=\lambda I$
GD ( $D=G$ )	$\left\lVert G\right\rVert_{\mathrm{F}}^{2}$	$\lambda\left\lVert G\right\rVert_{\mathrm{F}}^{2}$	always (any $X$ )
PolarGD ( $D=Q$ )	$\left\lVert G\right\rVert_{*}$	$\lambda\left\lVert Q\right\rVert_{\mathrm{F}}^{2}=\lambda\mathrm{rank}(G)$	iff nonzero singular values of $X$ are equal
SignGD ( $D=S$ )	$\left\lVert G\right\rVert_{1}$	$\lambda\left\lVert S\right\rVert_{\mathrm{F}}^{2}$	iff nonzero entries of $X$ have equal magnitude

Appendix

Table 1: Line search reveals distinct “isotropy targets” for different methods on isotropic quadratics.

Figure 8: Local advantage oscillates along the GD path.

2\times 2

diagonal quadratic with

H=\mathrm{diag}(\kappa,1)

\kappa=10

, initialized at

X_{0}=\mathrm{diag}(1,\sqrt{\kappa})

. (Left) Loss trajectories for GD (exact line search) and a Greedy comparison. At each iterate

X_{k}

, Greedy selects the direction

D_{k}\in\{G_{k},Q_{k}\}

with the larger one-step decrease. Despite always choosing the locally better step, Greedy underperforms GD globally. (Right) The one-step advantage

\Delta_{\mathrm{GD}}-\Delta_{\mathrm{Pol}}

along the GD trajectory.

Figure 9: Diminishing returns of line search. Comparing three fixed LMO-normalized learning rates,

X_{k+1}=X_{k}-\gamma\,\mathrm{LMO}(G_{k})

(best from the grid of App. A.1 ), to line search for GD , PolarGD , and SignGD on a quadratic with dense MP Hessian (

d=20

, aspect ratio

d/N=0.8

), and Gaussian initialization (

X_{0}\sim\mathcal{N}(0,1)

). We see that while line search typically outperforms a tuned learning rate for GD , this is not true for PolarGD and SignGD , where a fixed learning rate can significantly outperform line search. The rightmost panel displays the step sizes selected by each line search method. For PolarGD and SignGD this step size, measured in units of the normalized direction, is positively homogeneous of degree one in

X_{k}

(Proposition 4 ). This reflects only the unit scale of

\mathrm{polar}(G)

and

\mathrm{sign}(G)

: in dual-adapted units the PolarGD line-search step is constant when

G_{k}

has full row rank (Theorem 3 ), and the gap to the tuned rate comes from the trajectory.

Figure 10: Line search vs. tuned dual-adapted steps. Dense Marchenko–Pastur Hessian (aspect ratio

d/N=0.8

d=m=100

\kappa(H)\approx 192

; a separate draw from Figure 1 ), Gaussian initialization, 150 iterations; details in App. A.5 . (a) Relative loss. (b) Progress coefficient

p(X_{k})

along the two PolarGD runs. Exact line search is the dual-adapted rate

\alpha=1

in ( 17 ); the tuned rate

\alpha\approx 1.7

keeps

p(X_{k})

near

0.6

, whereas line search drives the iterate toward shapes with small

p(X_{k})

. For GD , line search and the tuned fixed step perform similarly. (c) Iterations to reach

f(X_{k})/f(X_{0})\leq 10^{-6}

versus

\kappa(H)

for a dense Hessian with log-spaced spectrum (

d=m=50

, Gaussian initialization, median over 3 seeds). GD ’s iteration count grows roughly linearly with

\kappa(H)

and tuned PolarGD ’s only slowly, so the gap exceeds two orders of magnitude at

\kappa(H)=10^{4}

; PolarGD with line search is slower than its tuned fixed step throughout.

Figure 11: Line search on an MLP. Training loss on CIFAR-5K under fixed dual-adapted learning rates and line search (dashed). The three fixed rates shown are the best-performing from a grid search. For GD , line search matches or slightly beats the best fixed rate; for PolarGD , a fixed learning rate outperforms it. This is consistent with the local-global gap observed in the quadratic case (Section 4 ). The rightmost panel plots the line-search step size: for GD it oscillates, whereas for PolarGD it decays by about an order of magnitude over 100 iterations.

Data	Nemotron-CC ( Su et al., 2025 ) , 1.8B tokens; GPT-2 BPE tokenizer, vocabulary $50{,}304$ ; sequence length $2048$
Model	decoder-only, $89.8$ M parameters ( $31.9$ M non-embedding); $8$ layers, width $576$ , $9$ heads of dimension $64$ ; RoPE ( $\theta=5\cdot 10^{5}$ ), pre-RMSNorm, fused SwiGLU MLP with hidden width $1536$ ; untied embedding and LM head
Training	$3433$ steps, batch $524{,}288$ tokens ( $32$ sequences $\times$ $8$ accumulation micro-steps); one NVIDIA A100; bfloat16 mixed precision (fp32 master weights and gradients); gradient clipping at $1.0$
Schedule ( Muon )	linear warmup over $10\%$ of the steps to $8\cdot 10^{-3}$ , then cosine decay to $10^{-6}$
Muon	50 matrices: query, key, value, and output projections and both MLP matrices of each layer, embedding, LM head; momentum $0.9$ without Nesterov; weight decay $0$ ; per-matrix scaling $0.2\sqrt{\max(d_{\mathrm{out}},d_{\mathrm{in}})}$
Newton--Schulz	$5$ steps, coefficients $(a,b,c)=(3.4445,-4.7750,2.0315)$ , applied to $M/(\left\lVert M\right\rVert_{\mathrm{F}}+10^{-7})$
Other parameters	normalization gains, Adam with $\beta=(0.95,0.95)$

Appendix

Table 2: Configuration of the transformer run.

Figure 12: (Left, middle) Distribution of the normalized singular values of the momentum matrices at four checkpoints, for the attention matrices (left) and for the MLP, embedding, and LM-head matrices (middle); the shaded band is the GD -like range. (Right) Percentage of singular values in the GD -like range over training for the momentum

M

and the raw gradient

G

, for the attention matrices and for all 50 matrices.

Figure 13: Singular value trajectories in

\mathbb{R}^{2}

. Rows correspond to different initializations; the right column shows the loss curves. For each row, optimizer trajectories in singular value space, all start from the same initialization (

\star

), with the von Neumann bound ( 59 ) overlaid. Terminal iterates are marked with a square (

\square

). Here

H=O\mathrm{diag}(1,10^{4})O^{\mathsf{T}}

with

O

a random orthogonal matrix, and the rows use orthogonal, Gaussian, and low-rank initializations. PolarGD ’s singular value trajectory maintains a

45^{\circ}

direction in singular value space. By contrast GD is seen to optimize the larger eigen-direction first. SignGD does not obviously exploit singular geometry here.

Muon Optimizer

Muon optimizer uses momentum and matrix orthogonalization to shape neural-network updates, especially in large language-model training.

See all 146 papers

May 29, 2026

How Much Orthogonalization Does Muon Need?

Muon optimizers improve neural-network training by replacing ill-conditioned momentum updates with approximately semi-orthogonal updates. This motivates a practical question: how much orthogonalization does Muon actually require? We study this question using a relaxed cubic Newton--Schulz schedule derived directly for Muon's low precision singular value band. The resulting five-step cubic construction uses ten dominant matrix multiplications, compared with fifteen for five quintic Newton--Schulz iterations. The cubic schedule is not intended as a more accurate polar solver; instead, it is a principled low-cost variant that lets us probe the relation between polar accuracy, spectral shaping, and training quality. Across synthetic diagnostics, NanoGPT ablations, and training experiments on hybrid MoE/Mamba models, we find that training quality is not governed monotonically by polar-decomposition accuracy: truncated Polar Express, Muon-Jordan, cubic Newton--Schulz, and an explicit FP32 SVD polar factor can reach nearly indistinguishable final loss on GPT-2 Small, and cubic5 matches the Muon-Jordan quintic update within about

10^{-3}

validation loss on hybrid MoE/Mamba models with one billion to four billion parameters. These results support cubic5 as a practical low-cost Muon orthogonalization variant, with empirical evidence of training-quality parity in the settings tested.

Hua Huang

NVIDIA

Muon Optimizer Matrix Factorization Neural Network Optimization Newton-Schulz Iteration Second-Order Optimization Neural Network Training Dynamics+1

Ask PDF

Oct 19, 2026

Polar Switch: Fast Muon Orthogonalization with Polynomial Circuits

Muon, a 2024-born optimizer for language models, differs from the 2015-born spectral gradient descent mainly in its fast, GPU-friendly Newton--Schulz polynomial for orthogonalizing the gradient matrix. Accordingly, a line of research proposes faster polynomial iterations for gradient orthogonalization, such as Polar Express and Gram Newton--Schulz, both built on quintic polynomials. In this paper, we introduce Polar Switch, a new approach to polynomial gradient orthogonalization. Using simple polynomial circuits, Polar Switch reaches a given high polynomial degree at a lower FLOP cost than quintic iterations. We define a circuit as a short sequence of symmetric matrix products on a few square buffers, and circuits with

k

products form a restricted family of odd polynomials of degree up to

2^{k+1}+1

. We choose the sequence of products in each circuit, their coefficients, and how the circuits are composed by numerically minimizing a worst-case bound on the polar approximation error. This minimization is subject to a FLOP budget that accounts for symmetric matrix multiplication kernels and a rounding constraint for numerical stability in half precision. Our two recommended Polar Switch methods have 15.5--18.7% lower runtime than Polar Express on square matrices and 8.6--16.2% lower runtime than Gram Newton--Schulz on rectangular matrices with aspect ratios 3.25, 3.5, and 8, with comparable approximation accuracy. In pretraining Modded-NanoGPT #91, Qwen3-0.6B, and Gemma 3-1B, Polar Switch reaches final validation losses similar to those of established orthogonalization methods. Across the more than 14 million matrices that each of the two recommended methods orthogonalized in these runs, the estimated output spectral norms never exceed the designed bounds by more than 3%, demonstrating their numerical stability.

Zhehang Du, Weijie Su

University of Pennsylvania

Orthogonal Optimization Muon Optimizer

Ask PDF

May 13, 2026

Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence

Muon orthogonalizes the momentum buffer before each update, replacing its singular values with ones via Newton-Schulz iterations. This simple change lets Muon tolerate far larger learning rates and converge faster than other optimizers, but why? We show that the mechanism is spectral flattening, and develop two results around it. First, we prove that Muon's maximal stable step size scales with the average singular value of the gradient rather than the largest, which bottlenecks standard gradient descent. Second, we recast Muon as a preconditioned gradient method and show, under a Kronecker-factored curvature model, that it improves the effective convergence factor, with the improvement controlled by the spectrum of the gradient covariance. Extensive experiments validate both results: Muon remains stable at learning rates that cause SGD to diverge within the first few iterations, and reaches accuracy milestones several epochs earlier even at identical step sizes. Taken together, our results offer a principled, geometric explanation for Muon's empirical success.

Tien-Phat Nguyen, Truong Nguyen, Minh-Phuc Truong +3

Hanoi University of Science and Technology Hanoi, Vietnam · Indiana University 107 S. Indiana Ave, Bloomington, IN 47405, USA · Monash University Clayton, VIC 3800, Australia

Momentum Methods Gradient Descent Muon Optimizer Spectral Methods Stochastic Gradient Descent

Ask PDF

Abstract

Muon Optimizer

Explore similar work

How Much Orthogonalization Does Muon Need?

Polar Switch: Fast Muon Orthogonalization with Polynomial Circuits

Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence