Constant Step Sizes for Muon-Type Spectral Optimizers on Strongly Convex Quadratics

Pratik Jawanpuria; Bhabatosh Garai; Bamdev Mishra

cs.LGOct 20, 2026

Constant Step Sizes for Muon-Type Spectral Optimizers on Strongly Convex Quadratics

Authors: Pratik Jawanpuria, Bhabatosh Garai, Bamdev Mishra

Organizations: Centre for Machine Intelligence and Data Science, IIT Bombay, India · Microsoft, India

Abstract

Muon and related optimizers update a weight matrix with a spectrally transformed gradient. In this work, we study such spectral optimizers with a constant step size on strongly convex matrix quadratics, which isolate the effect of the update from the confounds of nonconvexity, noise, and step size schedules. We analyze an update using two quantities: the gain, which is the factor by which it amplifies the gradient along each singular direction, and the length, which is the norm of the update. These two quantities place the popular spectral updates in three regimes with different behavior: (i) with a bounded gain, the iterates behave like preconditioned gradient descent and converge for step sizes set by the largest curvature; (ii) with an unbounded gain but a vanishing length, they still converge, with the admissible step sizes set by the total curvature instead; and (iii) with a non-vanishing length, they do not converge for almost every initialization. Interestingly, the practical variants of Muon fall in all three regimes, depending only on how the update is normalized and scaled and not on the polynomial that approximates the polar factor. Experiments validate these findings.

Figures & tables

Update	$g_{i}$	Gain $g_{i}/\sigma_{i}$	Regime
GD	$\sigma_{i}$	$1$	bounded gain
Fixed shaping	$h(\sigma_{i})$	$h(\sigma_{i})/\sigma_{i}$	bounded gain †
NS, $\varepsilon>0$	$p\big(\tfrac{\sigma_{i}}{\lVert G\rVert_{\mathrm{F}}+\varepsilon}\big)$	$\leq C_{p}/\varepsilon$	bounded gain
NS, $\varepsilon=0$	$p\big(\tfrac{\sigma_{i}}{\lVert G\rVert_{\mathrm{F}}}\big)$	unbounded	non-vanish. length
Polar map (Muon)	$1$	$1/\sigma_{i}$	non-vanish. length
PolarGrad	$\lVert G\rVert_{*}$	$\lVert G\rVert_{*}/\sigma_{i}$	vanishing length

Table 1: Instances of the spectral update and their regime. GD has

h(\sigma)=\sigma

and the polar map has

h\equiv 1

. The polynomial

p

is the composition of the Newton–Schulz (NS) stages.

Figure 1: Left, diagonal problem: (a) P1,

\tanh\sigma

below, at, and above the threshold. (b) P2, measured error factors of four bounded-gain updates on the curve

\rho(\alpha)

. Right, dense

16\times 8

problem, P3: (c) relative loss and (d) update length of the normalized NS iteration with

\varepsilon=0

and

\varepsilon=3.08

, the polar map, and GD at the same step size, and PolarGrad at

\alpha\tr(H)=1.9

Muon Optimizer

Muon optimizer uses momentum and matrix orthogonalization to shape neural-network updates, especially in large language-model training.

See all 149 papers

Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence
May 13, 2026Tien-Phat Nguyen, Truong Nguyen, Minh-Phuc Truong +3Momentum Methods Gradient Descent
Hanoi University of Science and Technology Hanoi, Vietnam · Indiana University 107 S. Indiana Ave, Bloomington, IN 47405, USA · Monash University Clayton, VIC 3800, Australia
Beyond the Matrix Sign: Quadratic Spectral Descent
Sep 7, 2026Qiaozhe Zhang, Jun Sun, Yingzhuang LiuSign-Based Optimization Muon Optimizer
School of Electronic Information and Communications Huazhong University of Science and Technology
The Spectral Dynamics and Noise Geometry of Muon
Jun 7, 2026Pierfrancesco Beneventano, Mahmoud Abdelmoneum, Tomaso PoggioMuon Optimizer Matrix Optimization
Massachusetts Institute of Technology

May 13, 2026J/K move · Enter open · S save

Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence

Tien-Phat Nguyen, Truong Nguyen, Minh-Phuc Truong, Tuc Nguyen+2

Hanoi University of Science and Technology Hanoi, Vietnam · Indiana University 107 S. Indiana Ave, Bloomington, IN 47405, USA · Monash University Clayton, VIC 3800, Australia

Momentum Methods Gradient Descent Muon Optimizer Spectral Methods Stochastic Gradient Descent

Open paper AskPDF arXiv

Muon orthogonalizes the momentum buffer before each update, replacing its singular values with ones via Newton-Schulz iterations. This simple change lets Muon tolerate far larger learning rates and converge faster than other optimizers, but why? We show that the mechanism is spectral flattening, and develop two results around it. First, we prove that Muon's maximal stable step size scales with the average singular value of the gradient rather than the largest, which bottlenecks standard gradient descent. Second, we recast Muon as a preconditioned gradient method and show, under a Kronecker-factored curvature model, that it improves the effective convergence factor, with the improvement controlled by the spectrum of the gradient covariance. Extensive experiments validate both results: Muon remains stable at learning rates that cause SGD to diverge within the first few iterations, and reaches accuracy milestones several epochs earlier even at identical step sizes. Taken together, our results offer a principled, geometric explanation for Muon's empirical success.

Abstract

Muon Optimizer

Explore similar work

Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence

Beyond the Matrix Sign: Quadratic Spectral Descent

The Spectral Dynamics and Noise Geometry of Muon