Muon as a Residual Connection

Hao Huang

cs.LGJul 1, 2026

Muon as a Residual Connection

Authors: Hao Huang

Organizations: College of Computer Science and Technology Zhejiang University Hangzhou, Zhejiang, China

Abstract

Muon has recently emerged as one of the most effective optimizers for training large neural networks, yet its empirical success has been explained from several different perspectives. In this paper, we propose a simple mechanistic interpretation: Muon can be understood as an implicit residual connection during training. Specifically, orthogonalizing the update can sacrifice some immediate gradient fidelity while improving representation preservation for downstream layers. We study this trade-off in controlled linear optimization settings, where Muon can learn representations that are slower to fit a local target but easier for downstream layers to exploit. Our results suggest a conceptual explanation for Muon and a design perspective for optimizers that balance local descent with downstream usability.

Explore similar work

To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters
Feb 28, 2026Sara Dragutinović, Yedi Zhang, Rajesh RanganathMuon Optimizer Muon
Reassessing Muon for Matrix Factorization
Jul 14, 2026Ali Parviz, Gal Mishne, Alex CloningerMuon Optimizer Muon
Denoise First, Orthogonalize Later: Understanding Momentum in Muon via Spectral Filtering
Jun 2, 2026Xianliang Li, Zihan Zhang, Weiyang Liu +1Muon Optimizer Muon

Feb 28, 2026 · cs.LGJ/K move · Enter open · S save

To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters

Sara Dragutinović, Yedi Zhang, Rajesh Ranganath

Courant Institute School of Mathematics, Computing, and Data Science, New York University · Gatsby Computational Neuroscience Unit, University College London

While Adam has long been the ubiquitous default optimizer for deep neural networks, Muon has recently seen rapid adoption due to its superior training speed. Although much of the literature focuses on validating the benefits of Muon, our work investigates the potential downsides of the mechanism driving this speedup. On the theoretical front, we analyze the learning dynamics of simplified Muon on deep linear networks and linear attention. Our analysis reveals that Muon gains speed by avoiding saddle points, but does so at the expense of the simplicity bias characteristic of Gradient Descent (GD), where the complexity of the functional solution learned grows sequentially. Experiments demonstrate the consequences of losing the simplicity bias, showing that Muon struggles to uncover common underlying structure across tasks and may be prone to fitting spurious features. More broadly, this paper serves as a reminder that faster optimization is rarely a free lunch; improvements in optimization can come at the cost of changes in the inductive biases that shape generalization.

Muon Optimizer Muon Optimizer Design Adam Deep Network Simplification

Open paper Ask about it PDF

Abstract

Explore similar work

To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters

Reassessing Muon for Matrix Factorization

Denoise First, Orthogonalize Later: Understanding Momentum in Muon via Spectral Filtering