cs.LGOct 25, 2026

Last-Iterate Stationarity of Muon with Finitely Many Newton-Schulz Iterations

Authors: Ping Li, Vincent Y. F. Tan

Organizations: Department of Mathematics, National University of Singapore (NUS) · Department of Electrical and Computer Engineering, NUS

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Muon has achieved strong empirical performance in large-scale language-model training, motivating a better theoretical understanding of its practical implementation. Unlike the idealized polar-factor update, practical Muon uses a fixed number of quintic Newton--Schulz iterations. Existing analyses mainly establish average- or best-iterate stationarity guarantees, leaving last-iterate convergence less understood. We study last-iterate gradient convergence of practical finite-step Muon on smooth, lower-bounded nonconvex objectives. For deterministic gradients, we show that without input stabilization, Muon can cycle even on a smooth convex quadratic. With positive stabilization, we establish last-iterate gradient convergence for any fixed momentum coefficient below one and a sufficiently small constant learning rate, together with square-summability of gradients. For stochastic gradients, we first show that persistent bounded, unbiased noise can prevent convergence with fixed learning rates and momentum, even with positive stabilization. We then establish almost-sure and mean-square last-iterate gradient convergence by combining decreasing learning rates with increasing momentum under bounded-variance stochastic gradients. Finally, we construct a smooth counterexample showing that neither convergence theorem guarantees a pointwise polynomial rate. We further establish density-one convergence rates, guaranteeing polynomial gradient decay for all but a vanishing fraction of iterations.

Figures & tables

Figure 1: Schematic illustration of an objective with sparse gradient peaks; horizontal distances are not to scale.
More in this topic

Muon Optimizer

Muon optimizer uses momentum and matrix orthogonalization to shape neural-network updates, especially in large language-model training.

See all 150 papers

Explore similar work

CardsList
Sep 30, 2026

Convergence of Practical Muon with Finite Newton-Schulz Iterations and Nesterov Momentum

Practical Muon maintains momentum and performs a small, fixed number of Newton--Schulz iterations separately for each parameter matrix, often with a Nesterov correction. We analyze these layer-wise finite-step updates jointly on a coupled nonconvex objective, rather than replacing them by exact polar factors or one global orthogonalization. Under gradient-dependent (L0,L1,q)(\mathcal L_0,\mathcal L_1,q)-smoothness and conditionally unbiased stochastic gradients with bounded layer-wise variance, we establish an O(T−1/4)\mathcal O(T^{-1/4}) bound on the expected average Frobenius gradient norm. The analysis retains the Nesterov recursion and requires neither bounded stochastic gradients, symmetric noise, nor a uniform positive lower bound on the nonzero output singular values. Its constants contain no explicit matrix-dimension or rank factors when the number of blocks and problem constants are fixed. The proof follows a descent inequality and a decomposition of the momentum tracking error into initialization, noise, and drift. For the original five-step quintic, we verify the required scalar-map bounds analytically; the result also allows step-dependent coefficients satisfying the same bounds. A complementary nuclear-norm result quantifies rank dependence under a stronger spectral condition. The vanishing rate uses coupled learning-rate and momentum schedules, including the standard single-coefficient Nesterov rule.

Hanyng Peng, Hui Wang, Yue Yu

Pengcheng Laboratory, Shenzhen, China

Newton-Schulz IterationGradient DescentNonconvex Stochastic OptimizationMuon OptimizerStochastic Optimization Convergence
AskPDF
Sep 24, 2026

Convergence guarantees for Muon: New parameter regimes and generalizations

In this paper, we establish the first asymptotic convergence guarantees for the Muon algorithm through a more accurate proxy for the Newton-Schultz iteration than the typical matrix sign function. We prove that, for appropriate choices of hyperparameters, the iterates satisfy lim⁡k→∞∥∇f(xk)∥=0\lim_{k\to\infty}\|\nabla f(x_k)\|=0, and, under a global Polyak-Łojasiewicz condition, that the sequence of function values converges linearly. The key insight is that the regularization, implicit in Muon's Newton-Schulz implementation, induces a bounded preconditioner, exposing Muon as a preconditioned Polyak heavy-ball method and enabling a classical Lyapunov analysis. This observation naturally motivates applying the same preconditioning structure to the Nesterov gradient evaluation. We formalize this idea by introducing Muesterov, a Nesterov-based variant of Muon, and prove that it enjoys the same convergence guarantees, extending the theoretical framework beyond the heavy-ball setting. Numerical experiments on a scalar cross-entropy problem corroborate the theory and illuminate the joint role of the learning rate and the Newton-Schulz regularizer in controlling convergence. Preliminary numerical simulations training the nanoGPT dataset provide intuition regarding the relevance of the observations in this paper to practical applications.

Arthur C. B. de Oliveira, Dhruv D. Jatkar, Guilherme S. Vicinansa +1

Department of Electrical and Computer Engineering Northeastern University Boston, MA 02115 · Department of Telecommunications and Control Escola Politecnica, University of São Paulo São Paulo, SP 05508-010

Muon OptimizerOptimization Convergence AnalysisAccelerated Gradient Methods
AskPDF
May 13, 2026

Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence

Muon orthogonalizes the momentum buffer before each update, replacing its singular values with ones via Newton-Schulz iterations. This simple change lets Muon tolerate far larger learning rates and converge faster than other optimizers, but why? We show that the mechanism is spectral flattening, and develop two results around it. First, we prove that Muon's maximal stable step size scales with the average singular value of the gradient rather than the largest, which bottlenecks standard gradient descent. Second, we recast Muon as a preconditioned gradient method and show, under a Kronecker-factored curvature model, that it improves the effective convergence factor, with the improvement controlled by the spectrum of the gradient covariance. Extensive experiments validate both results: Muon remains stable at learning rates that cause SGD to diverge within the first few iterations, and reaches accuracy milestones several epochs earlier even at identical step sizes. Taken together, our results offer a principled, geometric explanation for Muon's empirical success.

Tien-Phat Nguyen, Truong Nguyen, Minh-Phuc Truong +3

Hanoi University of Science and Technology Hanoi, Vietnam · Indiana University 107 S. Indiana Ave, Bloomington, IN 47405, USA · Monash University Clayton, VIC 3800, Australia

Momentum MethodsGradient DescentMuon OptimizerSpectral MethodsStochastic Gradient Descent
AskPDF