cs.LGFeb 18, 2026

The Implicit Bias of Adam and Muon on Smooth Homogeneous Neural Networks

Authors: Eitan Gronich, Gal Vardi

Organizations: Department of Computer Science and Applied Mathematics, Weizmann Institute of Science, Rehovot, Israel

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We study the implicit bias of momentum-based optimizers on smooth homogeneous models. We show that momentum steepest descent algorithms like Muon (spectral norm), MomentumGD (ℓ2\ell_2 norm), and Signum (ℓ∞\ell_\infty norm) are approximate steepest descent trajectories under a decaying learning rate schedule, proving that these algorithms have a bias towards KKT points of the corresponding margin maximization problem. We extend the analysis to Adam (without the stability constant), which maximizes the ℓ∞\ell_\infty margin, and to Muon-Signum and Muon-Adam, which maximize a hybrid norm. Our experiments corroborate the theory and show that the identity of the margin maximized depends on the choice of optimizer. Overall, our results extend earlier lines of work on steepest descent in homogeneous models and momentum-based optimizers in linear models.

Figures & tables

Figure 1: (a) Margin values vs. loss for different optimizers. A lighter/darker color signifies the squared-ReLU / ReLU activations respectively. Dotted lines represent optimizers with momentum disabled. Lines are mean values over 10 random seeds, while filled areas represent one standard deviation. (b) Cosine similarity to last iterate ⟨θt∥θt∥2,θlast∥θlast∥2⟩\left\langle\frac{\bm{\theta}_{t}}{\left\lVert\bm{\theta}_{t}\right\rVert_{2}},\frac{\bm{\theta}_{\text{last}}}{\left\lVert\bm{\theta}_{\text{last}}\right\rVert_{2}}\right\rangle , plotted on a normalized linear time scale.
Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

ActivationNGD w.o. momentumNGDSignumAdamMuonMuon-Adam
ReLU2.02.08×10−18\times 10^{-1}5×10−35\times 10^{-3}1×10−21\times 10^{-2}2×10−12\times 10^{-1}5×10−25\times 10^{-2}
Squared ReLU1.51.53×10−13\times 10^{-1}3×10−33\times 10^{-3}5×10−35\times 10^{-3}8×10−28\times 10^{-2}5×10−25\times 10^{-2}
Appendix
Table 1: Learning Rates for MNIST Task
Figure 2: Margin values vs. loss for different optimizers. A lighter/darker color signifies the squared-ReLU / ReLU activations respectively. Lines are mean values over 10 random seeds, while filled areas represent one standard deviation.
Figure 3: (a) Margin values vs. loss for different optimizers. Lines are mean values over 10 random seeds, while filled areas represent one standard deviation. (b) Cosine similarity to last iterate ⟨θt∥θt∥2,θlast∥θlast∥2⟩\left\langle\frac{\bm{\theta}_{t}}{\left\lVert\bm{\theta}_{t}\right\rVert_{2}},\frac{\bm{\theta}_{\text{last}}}{\left\lVert\bm{\theta}_{\text{last}}\right\rVert_{2}}\right\rangle , plotted on a normalized linear time scale.
ActivationNGDSignumAdamMuon-Adam
ReLU5×10−15\times 10^{-1}3×10−33\times 10^{-3}1×10−21\times 10^{-2}3×10−23\times 10^{-2}
Appendix
Table 2: Learning Rates for CIFAR10 Task
More in this topic

Gradient Descent

Gradient descent iteratively updates model parameters to reduce a loss using its slope, with variants for speed, scale, and constraints.

See all 116 papers

Explore similar work

CardsList

  1. From Spectral to Frobenius: When Muon Loses Its Spectral Implicit Bias

    Oct 14, 2026Yongtao Wu, Mahdi Mansouri, Zhenyu ZhuMuon OptimizerImplicit Regularization

    The University of Hong Kong

  2. Convergence of Steepest Descent and Adam under Non-Uniform Smoothness

    May 28, 2026Sharan Vaswani, Yifan Sun, Reza BabanezhadOptimization Convergence AnalysisGradient Descent

    1Simon Fraser University · 2Stony Brook University · 3Samsung AI, Montreal.

  3. To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters

    Feb 28, 2026Sara Dragutinović, Yedi Zhang, Rajesh RanganathNeural Network GeneralizationMuon Optimizer

    Courant Institute School of Mathematics, Computing, and Data Science, New York University · Gatsby Computational Neuroscience Unit, University College London