math.OCNov 13, 2026

Muon Attains O(ε−3)\mathcal{O}(\varepsilon^{-3}) Iteration Complexity under Shuffling

Authors: Kun Huang, Shi Pu, Karl Henrik Johansson

Organizations: KTH Royal Institute of Technology, Department of Decision and Control Systems (DCS), Digital Futures, Stockholm, Sweden · The Chinese University of Hong Kong, Shenzhen, School of Data Science (SDS), Shenzhen, Guangdong, China

Abstract

Muon is a stochastic first-order optimizer for matrix-valued parameters. It maintains an exponential moving average of stochastic gradients in a momentum buffer with coefficient ββ, and orthogonalizes the buffer to obtain the update direction. Muon has been reported to accelerate and stabilize the training of large language models. In practice, it is often used with shuffling (sampling without replacement), where each epoch visits the components of a finite-sum objective in a permuted order. However, its iteration complexity in this setting remains open. We analyze Muon under shuffling, with the momentum buffer orthogonalized by the polar map. For smooth nonconvex finite-sum problems, we establish an O(ε−3)\mathcal{O}(\varepsilon^{-3}) iteration complexity for finding an ε\varepsilon-stationary point. This guarantee holds for every sequence of permutations, covering random reshuffling, shuffle once, and the incremental gradient method. This complexity matches that of stochastic gradient descent under shuffling in its dependence on ε\varepsilon, and improves on Muon's O(ε−4)\mathcal{O}(\varepsilon^{-4}) complexity under sampling with replacement. Our guarantee requires a problem-dependent ββ that approaches one as the number of components grows. In contrast, no fixed ββ guarantees convergence to stationarity for every number of components. We show this by constructing a family of smooth convex problems on which, for any fixed β∈[0,1)β\in[0,1), Muon remains bounded away from stationarity once the number of components is sufficiently large. This failure occurs for every sequence of permutations and persists even with small stepsizes. These results suggest two practical tuning rules: if the training loss stops decreasing even after reducing the stepsize, increase ββ; when increasing ββ, reduce the stepsize accordingly.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Oct 25, 2026

Last-Iterate Stationarity of Muon with Finitely Many Newton-Schulz Iterations

Muon has demonstrated strong empirical performance in large-scale language-model training, motivating a deeper theoretical understanding of its practical implementation. Unlike the idealized polar-factor update, practical Muon employs a fixed number of quintic Newton-Schulz iterations. Existing analyses primarily establish average- or best-iterate stationarity guarantees, while last-iterate convergence remains less understood. In this work, we investigate last-iterate gradient convergence of practical finite-step Muon on smooth, lower-bounded nonconvex objectives. In the deterministic setting, we show that Muon without input stabilization can exhibit cyclic behavior even on a smooth convex quadratic. In contrast, with positive stabilization, we establish last-iterate gradient convergence and square-summability of gradients for any fixed momentum coefficient below one and a sufficiently small constant learning rate. In the stochastic setting, we first demonstrate that persistent bounded, unbiased noise can prevent convergence under fixed learning rates and momentum, even with positive stabilization. We then establish almost-sure and mean-square last-iterate gradient convergence under bounded-variance stochastic gradients by combining diminishing learning rates with increasing momentum. Finally, we construct a smooth counterexample demonstrating that neither convergence theorem implies a pointwise polynomial convergence rate. Nevertheless, we establish density-one convergence rates, guaranteeing polynomial gradient decay for all but an asymptotically vanishing fraction of iterations.
May 13, 2026

Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence

Muon orthogonalizes the momentum buffer before each update, replacing its singular values with ones via Newton-Schulz iterations. This simple change lets Muon tolerate far larger learning rates and converge faster than other optimizers, but why? We show that the mechanism is spectral flattening, and develop two results around it. First, we prove that Muon's maximal stable step size scales with the average singular value of the gradient rather than the largest, which bottlenecks standard gradient descent. Second, we recast Muon as a preconditioned gradient method and show, under a Kronecker-factored curvature model, that it improves the effective convergence factor, with the improvement controlled by the spectrum of the gradient covariance. Extensive experiments validate both results: Muon remains stable at learning rates that cause SGD to diverge within the first few iterations, and reaches accuracy milestones several epochs earlier even at identical step sizes. Taken together, our results offer a principled, geometric explanation for Muon's empirical success.
Aug 26, 2026

Muon with Finite Newton-Schulz: The Smoothing Benefit in Nonsmooth Nonconvex Optimization

Muon has emerged as a strong optimizer for the matrix-valued parameters in large language model pretraining, approximately orthogonalizing its momentum with a few Newton-Schulz iterations. Existing theory either replaces this iteration with the exact polar factor it approximates, or treats its finite depth as an approximation error, and thus the iteration Muon actually runs can only hurt the guarantees. We show that finite Newton-Schulz can instead be beneficial for nonsmooth nonconvex optimization. To this end, we analyze Muon through the online-to-nonconvex conversion, which views the update rule as an online learner and converts its regret bound into a stationarity guarantee. The finite Newton-Schulz iteration smooths the discontinuous polar map into a Lipschitz map of the singular values, and Muon with finite Newton-Schulz can be regarded as an online learner with a smoothed spectral potential. This smoothing is exactly what the conversion needs: we prove that a Newton-Schulz depth growing only logarithmically in the target accuracy suffices for convergence to stationary points in nonsmooth nonconvex optimization, whereas Muon with the exact-polar update may fail to converge. The resulting sample complexity bounds match the best-known guarantees for nonsmooth nonconvex optimization and are optimal for smooth nonconvex optimization up to problem-dependent factors. The argument extends beyond Newton-Schulz to general spectral maps with the same smoothing property.