From Spectral to Frobenius: When Muon Loses Its Spectral Implicit Bias

Yongtao Wu; Mahdi Mansouri; Zhenyu Zhu

cs.LGOct 14, 2026

From Spectral to Frobenius: When Muon Loses Its Spectral Implicit Bias

Authors: Yongtao Wu, Mahdi Mansouri, Zhenyu Zhu

Organizations: The University of Hong Kong

Abstract

Prior work on implicit bias shows that gradient descent selects the Frobenius max-margin direction on linearly separable problems, whereas Fan et al. (2025) established spectral max-margin behavior for spectral-norm steepest descent methods, including idealized Muon with exact orthogonalization. In practice, however, Muon approximates orthogonalization using finite Newton-Schulz iterations and a positive stabilization constant. We prove that these implementation choices cause Muon to select the Frobenius rather than the spectral max-margin direction in linear classification with separable data. We also establish this behavior for both exponential moving average and Nesterov-style momentum. The key technical ingredient is a summable perturbation argument showing that the resulting parameter trajectories remain at bounded distance from those of gradient descent. We then disentangle the roles of Newton-Schulz iteration count, the stabilization constant, and the Newton-Schulz polynomial, revealing distinct asymptotic behaviors, including directions that achieve neither Frobenius nor spectral max-margin. Finally, we show that Muon can recover its spectral max-margin bias with Newton-Schulz iterations growing logarithmically over training.

Figures & tables

Table 3: Idealized and practical Muon for matrix parameters. The idealized update uses an SVD polar factor. The practical update normalizes Nesterov-style momentum and applies

K

Newton–Schulz iterations.

Setting	Polynomial	Frobenius	Spectral	Other
Previous results
Gradient descent	–	${\color[rgb]{0.1328,0.5469,0.1328}\checkmark}$
Idealized Muon with exact-polar	–		${\color[rgb]{0.1328,0.5469,0.1328}\checkmark}$
Our results on practical Muon
Frobenius Max-Margin Bias
Fixed $K\geq 1$ , $\varepsilon>0$ ( Theorems 2 and 3 )	General( Assumption 2 )	${\color[rgb]{0.1328,0.5469,0.1328}\checkmark}$

Table 4: Comparison of implicit-bias behaviors. Frobenius checkmarks indicate max-margin direction convergence. Spectral checkmarks indicate spectral max-margin convergence, while Other indicates counterexamples where the direction differs from both or the relevant directional limit fails to exist.

Figure 1: Muon training using cubic (top) and quintic (bottom) Newton–Schulz iterations following the setting of Section 5.1 . We track the predictor ratio

W_{t,11}/W_{t,22}

, with reference values

(\widehat{W}_{\mathrm{F}})_{11}/(\widehat{W}_{\mathrm{F}})_{22}=2

and

(\widehat{W}_{\mathrm{op}})_{11}/(\widehat{W}_{\mathrm{op}})_{22}=1

for the Frobenius and spectral max-margin directions, respectively.

Figure 2: Frobenius max-margin selection and spectral recovery under the experimental settings described in Section 5.2 . Results are averaged over five random seeds.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Type	Description
$A_{i}$ , $W_{t}$	Matrices	Signed matrix feature and parameter at training iteration $t$ , both in $\mathbb{R}^{d_{1}\times d_{2}}$ .
$\mathcal{L}$ , $G_{t}$	Loss, gradient	Empirical loss and $G_{t}=\nabla\mathcal{L}(W_{t})$ .
$\langle A,B\rangle$	Inner product	Frobenius inner product $\operatorname{tr}(A^{\top}B)$ .
$\lVert\cdot\rVert_{\mathrm{F}}$ , $\lVert\cdot\rVert_{\mathrm{op}}$ , $\lVert\cdot\rVert_{*}$	Matrix norms	Frobenius, spectral, and nuclear norms.
$R$ , $R_{*}$ , $\gamma$	$>0$	Feature bounds $R=\max_{i}\lVert A_{i}\rVert_{\mathrm{F}}$ and $R_{}=\max_{i}\lVert A_{i}\rVert_{}$ , and a separating margin $0<\gamma\leq\gamma_{\mathrm{F}}$ from Assumption 1 .
$H$	$>0$	Local Hessian bound in the auxiliary lemmas and proofs, specified where it is used.

Appendix

Table 5: Core symbols and constants used in this paper.

Figure 3: Direction distances during Muon training with cubic, quintic, and quintic with Nesterov-style momentum, under the experimental settings described in Section E.2 . Results are averaged over five random seeds.

Muon Optimizer

Muon optimizer uses momentum and matrix orthogonalization to shape neural-network updates, especially in large language-model training.

See all 143 papers

Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence
May 13, 2026Tien-Phat Nguyen, Truong Nguyen, Minh-Phuc Truong +3Momentum Methods Gradient Descent
Hanoi University of Science and Technology Hanoi, Vietnam · Indiana University 107 S. Indiana Ave, Bloomington, IN 47405, USA · Monash University Clayton, VIC 3800, Australia
The Spectral Dynamics and Noise Geometry of Muon
Jun 7, 2026Pierfrancesco Beneventano, Mahmoud Abdelmoneum, Tomaso PoggioMuon Optimizer Matrix Optimization
Massachusetts Institute of Technology
Training-Loss Guarantees for Muon with Finite-Step Newton--Schulz Orthogonalization
Oct 2, 2026Amartya Roy, Souvik ChakrabortyMuon Optimizer Neural Network Training Dynamics
The School of Interdisciplinary Research Indian Institute of Technology Delhi, India · Robert Bosch GmbH, India · Department of Applied Mechanics Indian Institute of Technology Delhi, India +1

May 13, 2026J/K move · Enter open · S save

Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence

Tien-Phat Nguyen, Truong Nguyen, Minh-Phuc Truong, Tuc Nguyen+2

Hanoi University of Science and Technology Hanoi, Vietnam · Indiana University 107 S. Indiana Ave, Bloomington, IN 47405, USA · Monash University Clayton, VIC 3800, Australia

Momentum Methods Gradient Descent Muon Optimizer Spectral Methods Stochastic Gradient Descent

Open paper AskPDF arXiv

Muon orthogonalizes the momentum buffer before each update, replacing its singular values with ones via Newton-Schulz iterations. This simple change lets Muon tolerate far larger learning rates and converge faster than other optimizers, but why? We show that the mechanism is spectral flattening, and develop two results around it. First, we prove that Muon's maximal stable step size scales with the average singular value of the gradient rather than the largest, which bottlenecks standard gradient descent. Second, we recast Muon as a preconditioned gradient method and show, under a Kronecker-factored curvature model, that it improves the effective convergence factor, with the improvement controlled by the spectrum of the gradient covariance. Extensive experiments validate both results: Muon remains stable at learning rates that cause SGD to diverge within the first few iterations, and reaches accuracy milestones several epochs earlier even at identical step sizes. Taken together, our results offer a principled, geometric explanation for Muon's empirical success.

Abstract

Muon Optimizer

Explore similar work

Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence

The Spectral Dynamics and Noise Geometry of Muon

Training-Loss Guarantees for Muon with Finite-Step Newton--Schulz Orthogonalization