Convergence guarantees for Muon: New parameter regimes and generalizations
Authors: Arthur C. B. de Oliveira, Dhruv D. Jatkar, Guilherme S. Vicinansa, Eduardo D. Sontag
Organizations: Department of Electrical and Computer Engineering Northeastern University Boston, MA 02115 · Department of Telecommunications and Control Escola Politecnica, University of São Paulo São Paulo, SP 05508-010
In this paper, we establish the first asymptotic convergence guarantees for the Muon algorithm through a more accurate proxy for the Newton-Schultz iteration than the typical matrix sign function. We prove that, for appropriate choices of hyperparameters, the iterates satisfy limk→∞∥∇f(xk)∥=0, and, under a global Polyak-Łojasiewicz condition, that the sequence of function values converges linearly. The key insight is that the regularization, implicit in Muon's Newton-Schulz implementation, induces a bounded preconditioner, exposing Muon as a preconditioned Polyak heavy-ball method and enabling a classical Lyapunov analysis. This observation naturally motivates applying the same preconditioning structure to the Nesterov gradient evaluation. We formalize this idea by introducing Muesterov, a Nesterov-based variant of Muon, and prove that it enjoys the same convergence guarantees, extending the theoretical framework beyond the heavy-ball setting. Numerical experiments on a scalar cross-entropy problem corroborate the theory and illuminate the joint role of the learning rate and the Newton-Schulz regularizer in controlling convergence. Preliminary numerical simulations training the nanoGPT dataset provide intuition regarding the relevance of the observations in this paper to practical applications.
Figures & tables
Figure 1: Simulations for f(x)=x2 with α=0.1 and β=0.5 . Columns (a)-(c) show the phase portrait ( xk×Hk ) for Canonical Muon ( 4 ), Ideal Muon ( 5 ), and Soft Muon ( 8 ) respectively; (d) shows the loss across iterations. The first row uses ε=10−7 and δ=kδε10−7=484.87610−7 , where δ denotes the regularizer inside the Newton-Schulz algorithm. To generate the second row, the values of δ and ε were increased until Canonical Muon and Soft Muon respectively started presenting a converging trajectory. Ideal Muon has no such regularizer and instead always settles into a periodic orbit around x∗=0 .
Figure 2: Synthetic simulations for Canonical Muon ( 4 ) (first row, gray), Soft Muon ( 8 ) (second row, blue), and Muesterov ( 16 ) (third row, red). Column (a) depicts four training cost trajectories for α=10−3 , β=0.5 , and increasing values of ε . Column (b) depicts the training cost for β=0.5 , ε=10−7 , and increasing values of α . Columns (c) and (d) repeat the same simulations as column (b) but for much larger values of ε for the larger α .
Figure 3: Fixed-batch nanoGPT overfitting at 100× the baseline script-level learning rate ( α=0.36 instead of 0.0036 ). The Muon optimizer trains the transformer blocks at 0.1α while AdamW remains at the baseline rate ( 0.0036 ) on the output head. After the first minibatch is drawn it is reused at every step, so the plotted train loss is a fixed-batch diagnostic rather than a sampled training curve. Faint traces show the per-step values; bold traces show a 50 -step moving average. Top Panel. With Newton-Schulz regularizer δ=0.1 , the trajectory exhibits sustained peak–trough oscillations superimposed on an overall descent. Bottom Panel. With δ=1 , the same learning rate produces a single early overshoot followed by smooth, near-monotone descent toward a fixed point. The contrast is consistent with the soft-sign / Newton-Schulz analysis: larger δ reduces the effective gain of the normalized update and damps the oscillatory regime.
The School of Interdisciplinary Research Indian Institute of Technology Delhi, India · Robert Bosch GmbH, India · Department of Applied Mechanics Indian Institute of Technology Delhi, India +1