When Is Muon Optimal? Radial Fragility and Minimax Geometry

Ao Xu

stat.MLOct 15, 2026

When Is Muon Optimal? Radial Fragility and Minimax Geometry

Authors: Ao Xu

Organizations: Jilin University, Changchun 130012, China · Zhongguancun Academy, Beijing 100094, China

Abstract

Muon replaces a matrix gradient by an approximate polar factor and has shown strong empirical performance in neural-network training. The Isotropic Curvature Model (ICM) provides a curvature-based explanation for this update by asking when gradient orthogonalization minimizes a one-step local-loss surrogate. We investigate the stability of this explanation when the fixed input radius of the ICM is replaced by a general radial law. We introduce the Random-Radius Isotropic Curvature Model (RR-ICM) and the Polar-Factor Admissibility Set, which encodes exact scaled-polar optimality as a geometric condition on the positive polar factor of the gradient. We derive an exact characterization of this set for arbitrary radial laws and show that all nonscalar admissible responses arise from radial atoms aligned with curvature kinks. Consequently, every atomless radial law exhibits zero-capacity rigidity: exact scaled-polar optimality is possible only when the gradient is already a scaled column isometry. To connect the curvature-based framework with the established spectral geometry of Muon, we introduce the Minimax RR-ICM by replacing directional averaging with worst-case optimization over probability laws on the sphere. We prove that every nonzero global minimizer lies on the polar ray of the gradient, while its scale is determined by a one-dimensional convex problem. The corresponding admissibility geometry is a positive-semidefinite trace slab, in sharp contrast to the spherical-zonoid geometry of the RR-ICM. Together, these results expose the radial fragility of average-curvature explanations and provide a new minimax variational characterization of Muon's polar direction.

Figures & tables

Figure 1: Radial averaging is a small perturbation at the level of function values but a singular perturbation at a kink. For

H(t)=(t-1)_{+}

and

R_{\varepsilon}\sim\operatorname{Unif}[1-\varepsilon,1+\varepsilon]

, the averaged profile

H_{\varepsilon}(t)=\mathbb{E}(R_{\varepsilon}t-1)_{+}

differs from

H

only within an

O(\varepsilon)

transition layer. Nevertheless, the interval

\partial H(1)=[0,1]

is replaced by the unique derivative

H_{\varepsilon}^{\prime}(1)=1/2+\varepsilon/4

Figure 2: Two polar-factor admissibility geometries in dimension

n=2

. Panel (a) shows the RR-ICM admissibility set

b_{c}I_{2}+w_{c}\mathcal{K}_{2}

in spectral coordinates. Its nonscalar extent is controlled by the atom–kink response capacity

w_{c}

and collapses to the singleton

\{b_{c}I_{2}\}

when

w_{c}=0

. Panel (b) shows the Minimax RR-ICM admissibility set, where

\alpha_{-}:=\min\partial\overline{H}_{R}(c)

and

\alpha_{+}:=\max\partial\overline{H}_{R}(c)

. Positive semidefiniteness confines the spectrum to the first quadrant, while trace matching restricts it to the slab

\alpha_{-}\leq\lambda_{1}+\lambda_{2}\leq\alpha_{+}

. Thus, the RR-ICM restricts spectral shape through spherical-zonoid geometry, whereas the Minimax RR-ICM restricts only the total spectral mass.

Muon Optimizer

Muon optimizer uses momentum and matrix orthogonalization to shape neural-network updates, especially in large language-model training.

See all 146 papers

Jun 7, 2026

The Spectral Dynamics and Noise Geometry of Muon

Muon replaces a matrix gradient

G=UΣV^\top

by its polar factor

UV^\top

. This keeps the singular directions selected by the gradient, but makes the update spectrum flat. We study the optimization bias created by this operation. Under explicit alignment assumptions, we prove that the polar update is the one-step entropy-maximizing choice among bounded updates that use the gradient singular directions and do not adapt to the current weight spectrum. In an underdetermined regression model, we derive exact singular-value dynamics for continuous-time Muon and identify a measurement-dependent condition under which the normalized spectrum moves toward equal nonzero singular values. This geometry also rules out a common low-rank interpretation: at fixed Frobenius norm, Muon's distinguished state has a flat spectrum, whereas nuclear-norm minimization favors spectral concentration. Controlled matrix-sensing experiments separate the effect from simple gradient rescaling, show that norm-matched gradient descent does not reproduce Muon, and recover the predicted flattening trend across broad ablations. In small NanoGPT pretraining, Muon preserves stable rank, has a broad learning-rate plateau, and improves validation loss relative to AdamW; in a matched small-ViT control, the ranking reverses. The resulting picture is regime-dependent: Muon is not universally superior, but its flat-spectrum bias can help when many spectral directions need to remain active.

Pierfrancesco Beneventano, Mahmoud Abdelmoneum, Tomaso Poggio

Massachusetts Institute of Technology

Muon Optimizer Matrix Optimization Spectral Methods Implicit Regularization Gradient Descent Dynamics Spectral Optimization

Ask PDF

Jun 3, 2026

Why Muon Outperforms Adam: A Curvature Perspective

Muon improves training efficiency over Adam in large language-model training by about two times, but the local geometric source of this advantage remains unclear. Our work takes a first step toward demystifying Muon's superiority over Adam from a curvature perspective. First, we apply a second-order Taylor approximation to the training landscape and show that Muon achieves a larger one-step loss decrease than Adam at matched validation loss. The two optimizers have comparable first-order gains, but Muon consistently incurs a smaller second-order curvature penalty. Second, we decompose this curvature penalty into the squared update norm and Normalized Directional Sharpness (NDS). We find that Muon and Adam have comparable update norms, so Muon's smaller curvature penalty is driven by lower NDS, not update scale. Third, we study how training data and model structure shape Muon's NDS advantage. Using Zipf-Probabilistic Context-Free Grammar (PCFG) data with controlled imbalance, we show that data imbalance amplifies Muon's NDS advantage over Adam. A within-/cross-layer decomposition further shows that, in the middle and late stages of training, Muon's lower NDS is mainly sustained by smaller within-layer curvature. Beyond empirical evidence, we analyze stylized quadratic problems with heterogeneous curvature and gradient alignment toward high-curvature modes. We prove that Muon attains a smaller average NDS than GD by balancing update energy across curvature groups; when curvature heterogeneity is sufficiently strong, this also yields lower local quadratic loss after the same number of steps.

Shuche Wang, Fengzhuo Zhang, Jiaxiang Li +2

National University of Singapore · Yale University · University of Minnesota

Loss Landscape Sharpness Neural Network Optimization Muon Optimizer Adaptive Gradient Methods Loss Landscape Geometry Curvature-Aware Optimization

Ask PDF

May 13, 2026

Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence

Muon orthogonalizes the momentum buffer before each update, replacing its singular values with ones via Newton-Schulz iterations. This simple change lets Muon tolerate far larger learning rates and converge faster than other optimizers, but why? We show that the mechanism is spectral flattening, and develop two results around it. First, we prove that Muon's maximal stable step size scales with the average singular value of the gradient rather than the largest, which bottlenecks standard gradient descent. Second, we recast Muon as a preconditioned gradient method and show, under a Kronecker-factored curvature model, that it improves the effective convergence factor, with the improvement controlled by the spectrum of the gradient covariance. Extensive experiments validate both results: Muon remains stable at learning rates that cause SGD to diverge within the first few iterations, and reaches accuracy milestones several epochs earlier even at identical step sizes. Taken together, our results offer a principled, geometric explanation for Muon's empirical success.

Tien-Phat Nguyen, Truong Nguyen, Minh-Phuc Truong +3

Hanoi University of Science and Technology Hanoi, Vietnam · Indiana University 107 S. Indiana Ave, Bloomington, IN 47405, USA · Monash University Clayton, VIC 3800, Australia

Momentum Methods Gradient Descent Muon Optimizer Spectral Methods Stochastic Gradient Descent

Ask PDF

Abstract

Muon Optimizer

Explore similar work

The Spectral Dynamics and Noise Geometry of Muon

Why Muon Outperforms Adam: A Curvature Perspective

Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence