cs.LGJun 7, 2026

The Spectral Dynamics and Noise Geometry of Muon

Authors: Pierfrancesco Beneventano, Mahmoud Abdelmoneum, Tomaso Poggio

Organizations: Massachusetts Institute of Technology

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Muon replaces a matrix gradient G=UΣV⊤G=UΣV^\top by its polar factor UV⊤UV^\top. This keeps the singular directions selected by the gradient, but makes the update spectrum flat. We study the optimization bias created by this operation. Under explicit alignment assumptions, we prove that the polar update is the one-step entropy-maximizing choice among bounded updates that use the gradient singular directions and do not adapt to the current weight spectrum. In an underdetermined regression model, we derive exact singular-value dynamics for continuous-time Muon and identify a measurement-dependent condition under which the normalized spectrum moves toward equal nonzero singular values. This geometry also rules out a common low-rank interpretation: at fixed Frobenius norm, Muon's distinguished state has a flat spectrum, whereas nuclear-norm minimization favors spectral concentration. Controlled matrix-sensing experiments separate the effect from simple gradient rescaling, show that norm-matched gradient descent does not reproduce Muon, and recover the predicted flattening trend across broad ablations. In small NanoGPT pretraining, Muon preserves stable rank, has a broad learning-rate plateau, and improves validation loss relative to AdamW; in a matched small-ViT control, the ranking reverses. The resulting picture is regime-dependent: Muon is not universally superior, but its flat-spectrum bias can help when many spectral directions need to remain active.

Explore similar work

CardsList
May 13, 2026cs.LG

Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence

Muon orthogonalizes the momentum buffer before each update, replacing its singular values with ones via Newton-Schulz iterations. This simple change lets Muon tolerate far larger learning rates and converge faster than other optimizers, but why? We show that the mechanism is spectral flattening, and develop two results around it. First, we prove that Muon's maximal stable step size scales with the average singular value of the gradient rather than the largest, which bottlenecks standard gradient descent. Second, we recast Muon as a preconditioned gradient method and show, under a Kronecker-factored curvature model, that it improves the effective convergence factor, with the improvement controlled by the spectrum of the gradient covariance. Extensive experiments validate both results: Muon remains stable at learning rates that cause SGD to diverge within the first few iterations, and reaches accuracy milestones several epochs earlier even at identical step sizes. Taken together, our results offer a principled, geometric explanation for Muon's empirical success.
Tien-Phat Nguyen, Truong Nguyen, Minh-Phuc Truong +3
Muon OptimizerMuonConvergence AnalysisNewtonConvergence
AskPDF
Jul 14, 2026cs.LG

Reassessing Muon for Matrix Factorization

Muon has recently emerged as a strong optimizer for large-scale deep learning, where it reshapes gradient updates through approximate orthogonalization and has been reported to outperform Adam and AdamW in large language model training. Its empirical success has motivated a growing body of theoretical work that interprets Muon as steepest descent under the spectral norm. Yet it remains unclear which of Muon's advantages stem from its update rule itself and which are artifacts of the scale, architecture, and data of modern deep networks. In this work, we isolate the optimizer from these confounding factors by studying Muon on a simple, well-understood, and spectrally structured problem: low-rank matrix factorization. Through a controlled comparison against carefully tuned adaptive baselines, we find that Muon does not consistently outperform AdamW in this setting and that several previously reported advantages are sensitive to hyperparameter choices. Our results provide a more nuanced picture of when spectrum-aware orthogonalization is beneficial and argue for evaluating modern optimizers on controlled problems in addition to end-to-end benchmarks.
Ali Parviz, Gal Mishne, Alex Cloninger
Muon OptimizerMuonOptimizer DesignAdamModern Deep Networks
AskPDF
Jun 11, 2026cs.LG

Muonp^p: Muon with Fractional Spectral Powers

Muon is an increasingly widely used optimizer that replaces a gradient G=USV⊤G=USV^\top with its polar factor UV⊤UV^\top, thereby flattening the singular spectrum. However, full flattening discards singular-value information that may matter for adaptation. We introduce Muonp^p, a Muon-style optimizer that instead uses fractional spectral-power updates USpV⊤US^pV^\top for rational p∈(0,1)p\in(0,1), interpolating between Muon and gradient descent. To make it practical, we prove that fractional spectral powers cannot be computed by any fixed univariate polynomial iteration, and furthermore derive low-degree odd bivariate recurrences that approximate USpV⊤US^pV^\top using only matrix multiplications, preserving Muon's matrix-multiplication-only structure and compute complexity. We show that Muonp^p maximizes the linear improvement in loss under the Schatten qq-norm for q=1+1pq=1+\frac{1}{p}. Empirically, Muonp^p is especially effective for finetuning: on billion-scale models, Muonp^p improves validation perplexity and downstream task performance. We further analyze when Muonp^p is less suitable, through the lens of spectral geometry. Our results reveal important insights on when preserving the singular spectrum can bring significant gains, and introduce a principled way to achieve them.
Yihe Dong, Will Sawin
Muon OptimizerMuonGradient DescentMatrix Multiplication
AskPDF