Muon Optimizer

Follow this topic Ask about this topic

Muon is a matrix-aware optimizer that accumulates momentum for weight matrices and approximately orthogonalizes each update, often with Newton–Schulz iterations, rather than scaling each parameter independently as Adam does. Research explores variants with curvature or variance preconditioning, different spectral weighting and normalization, and cheaper or distributed orthogonalization. Studies test Muon in language-model training, vision, recommendation, diffusion, and scientific machine learning, while theory examines convergence, noise, implicit bias, and generalization. A central open question is when its spectral updates improve learning enough to justify their cost: results depend on architecture, batch size, and training phase, and practical approximations can change both behavior and guarantees.

Latest papers

146 papers32 papers in the last four weeks, up 146% on the four weeks before. 0.2% of all new papers.

When and How Does Muon Precondition? The Role of Structured Gradient Noise
Oct 22, 2026Xiaoran Chen, Binghui Li, Liwei Wang +1Muon Optimizer
School of EECS, Peking University · Center for Machine Learning Research, Peking University · Center for Data Science, Peking University +2
A Quadratic Lens on Muon's Polar Update: Benefits, Pitfalls, and the Role of Newton-Schulz
Oct 21, 2026Egor Shulgin, Sam Laing, Antonio Orvieto +1Muon Optimizer Orthogonal Optimization
King Abdullah University of Science and Technology (KAUST), Saudi Arabia · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), United Arab Emirates · ELLIS Institute Tübingen and MPI for Intelligent Systems, Tübingen AI Center, Germany +1
Constant Step Sizes for Muon-Type Spectral Optimizers on Strongly Convex Quadratics
Oct 20, 2026Pratik Jawanpuria, Bhabatosh Garai, Bamdev MishraMuon Optimizer Spectral Optimization
Centre for Machine Intelligence and Data Science, IIT Bombay, India · Microsoft, India
Polar Switch: Fast Muon Orthogonalization with Polynomial Circuits
Oct 19, 2026Zhehang Du, Weijie SuOrthogonal Optimization Muon Optimizer
University of Pennsylvania
T-MUON: Tensorizing Muon via t-SVD for Large Language Model Pre-training
Oct 18, 2026Dongyang Li, Mingsong Yan, Zheng Zhang +1Muon Optimizer Language Model Pretraining
University of California, Santa Barbara Santa Barbara, CA 93106, USA
How Does Muon Work in Private Deep Learning? Scalable Differentially Private Training via Orthogonalization
Oct 17, 2026Alexander Gaponov, Grigory Malinovsky, Egor Shulgin +1Muon Optimizer Deep Learning Optimization
King Abdullah University of Science and Technology (KAUST) KAUST Center of Excellence for Generative AI Thuwal, Saudi Arabia
Muon Operates Beyond the Edge of Stability
Oct 16, 2026Niko Covic, Baraq Lipshitz, Alexander Padula +1Muon Optimizer Edge of Stability
Department of Informatics, ETH Zurich, Switzerland.
BMuon: Budgeting Muon's Spectral Geometry for Structured Compressibility
Oct 16, 2026Rida Assalouh, Tom Jacobs, Rebekka Burkholz +2Muon Optimizer Structured Sparsity
Inria, Paris, France · ´Ecole polytechnique, Institut Polytechnique de Paris, Palaiseau, France · CISPA Helmholtz Center for Information Security, Saarbrücken, Germany +3
The Ephemeral Cost of Switching Optimizers Across LLM Training Stages
Oct 15, 2026Xinyu Zhao, Davis Wertheimer, Ashutosh Sharma +2LLM Training LLM Post-Training
MIT · MIT-IBM Watson AI Lab
When Does Muon Actually Win? A Paired Data-, Time-, and Tuning-Budget Benchmark
Oct 15, 2026Mushir Akhtar, Hazem N. Nounou, Majdi Mansouri +1Muon Optimizer
When Is Muon Optimal? Radial Fragility and Minimax Geometry
Oct 15, 2026Ao XuMuon Optimizer Neural Network Optimization
Jilin University, Changchun 130012, China · Zhongguancun Academy, Beijing 100094, China
Long-Horizon RL Needs Muon
Oct 14, 2026Kai Ruan, Jinghao Lin, Tao Chen +3Long-Horizon RL Long-Horizon LLM Agents
Gaoling School of Artificial Intelligence, Renmin University of China · Duke University · Institute of Automation, Chinese Academy of Sciences +1
From Spectral to Frobenius: When Muon Loses Its Spectral Implicit Bias
Oct 14, 2026Yongtao Wu, Mahdi Mansouri, Zhenyu ZhuMuon Optimizer Implicit Regularization
The University of Hong Kong
Taming Softmax at the Limits of Muon: A Unified Approach through Hilbert Geometry
Oct 13, 2026Aditya Somasundaram, Charles Guille-Escuret, Alexander Moreno +2Muon Optimizer Softmax Attention
Institute of Foundation Models, Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)
Monarch-Muon: Structure-Aware Muon for Fast and Memory-Efficient Pretraining
Oct 12, 2026Alexander Miasnikov, Ruslan Kabirov, Gleb Molodtsov +7Muon Optimizer LLM Training
BRAIn Lab · HSE University · MBZUAI
From Parallel to Sequential: Small-Batch Noise Restores Simplicity Bias of Muon
Oct 12, 2026Tom Jacobs, Rebekka BurkholzMuon Optimizer Nonconvex Optimization
CISPA Helmholtz Center for Information Security, Germany
Spectrally Targeted Muon
Oct 7, 2026Vishrut Goyal, Rohan RamkumarSpectral Optimization Deep Learning Optimization
Department of Computer Science, Stanford University · Department of Electrical Engineering, Stanford University.
The Best Optimizer Depends on Batch Size
Oct 6, 2026Xingyu Dang, Kaiyue Wen, Sadhika MalladiDeep Learning Optimization Hyperparameter Transfer
Princeton University · Stanford University · University of California, San Diego
Does Muon Need Fine-Grained Spectral Shaping?
Oct 5, 2026Meher Chaitanya, Tianyi Zhou, Aristides GionisDeep Learning Optimization Muon Optimizer
KTH Royal Institute of Technology Stockholm, Sweden
DGA-Muon: Decoupled Geometry-Aligned Adaptive Scaling for Muon
Oct 5, 2026Wenpeng Zhang, Runsheng YuMuon Optimizer Adaptive Gradient Methods
Independent Reasearcher
Muon Is Theoretically Wrong For Convolutions, But Empirically Effective
Oct 5, 2026Thibaut Boissin, Thomas Massena, Mathieu Serrurier +1Newton-Schulz Iteration Neural Network Optimization
IRIT · IRIT, DTIPG - SNCF, UT3 · Institut de Recherche Technologique Saint-Exupery, France
Training-Loss Guarantees for Muon with Finite-Step Newton--Schulz Orthogonalization
Oct 2, 2026Amartya Roy, Souvik ChakrabortyMuon Optimizer Neural Network Training Dynamics
The School of Interdisciplinary Research Indian Institute of Technology Delhi, India · Robert Bosch GmbH, India · Department of Applied Mechanics Indian Institute of Technology Delhi, India +1
Muon Learns Facts Better: Understanding the Role of Spectral Orthogonalization
Oct 2, 2026Xuheng Li, Qiwei Di, Yuan Cao +1Neural Network Training Dynamics Muon Optimizer
Department of Computer Science, University of California, Los Angeles, CA 90095, USA · School of Computing and Data Science, the University of Hong Kong
Inner Momentum for Differentially Private Muon
Oct 2, 2026Bishnu Bhusal, Minh Vu, Ben Southworth +3Muon Optimizer Privacy-Preserving Language Models
Los Alamos National Laboratory · University of Missouri · Georgia Institute of Technology
MuonIO: Principled Norm-Aware Descent for Embedding Tables and Language Model Heads
Oct 2, 2026Linkai Ma, Xinyu Luo, Mengbo Wang +3Muon Optimizer Language Model Pretraining
Department of Computer Science, Purdue University
AF-Muon: An AdamW-Free Muon Optimizer for Tied-Embedding Models
Oct 1, 2026Arash Lagzian, Paniz Halvachi, Junming Zhang +2Memory-Efficient Optimization Muon Optimizer
National University of Singapore, Singapore · Independent Researcher · LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China
Convergence of Practical Muon with Finite Newton-Schulz Iterations and Nesterov Momentum
Sep 30, 2026Hanyng Peng, Hui Wang, Yue YuNewton-Schulz Iteration Gradient Descent
Pengcheng Laboratory, Shenzhen, China
The Row Normalization Puzzle in Muon
Sep 30, 2026Jiayu Zhang, Tianyi LinStochastic Optimization Optimization Convergence Analysis
Department of Industrial Engineering and Operations Research Columbia University
Physical Muon: Orthogonalization as an Equilibrium Computation
Sep 29, 2026Yuren HaoPhysical Neural Networks Neural Network Optimization
University of Illinois at Urbana-Champaign
Second-Moment Stochastic Approximation Methods
Sep 29, 2026Tao Jiang, Lin XiaoStochastic Approximation Muon Optimizer
Fundamental AI Research (FAIR), Meta Superintelligence Labs

Oct 22, 2026J/K move · Enter open · S save

When and How Does Muon Precondition? The Role of Structured Gradient Noise

Xiaoran Chen, Binghui Li, Liwei Wang, Pinyan Lu

School of EECS, Peking University · Center for Machine Learning Research, Peking University · Center for Data Science, Peking University+2

Muon Optimizer

Open paper AskPDF arXiv

Muon has become a popular optimizer for large language model training, yet its theoretical foundations remain poorly understood. In this work, we study how gradient noise geometry shapes Muon's preconditioning benefits. We first show, theoretically and empirically, that under the classical isotropic noise assumption, Muon offers no acceleration beyond a rescaling of the learning rate. This motivates us to investigate structured noise arising from mini-batch sampling in associative memory with heterogeneous task frequencies. We establish an exact balancing threshold for two tasks and extend the analysis to multiple tasks, revealing how task coverage and gradient reliability jointly shape learning. Our analysis shows that Muon can soften the dependence on task frequency from linear to square-root, helping balance the learning of frequent and rare tasks. We further identify rotation-induced negative transfer in SignSGD that Muon avoids, and explain how momentum reduces sampling noise while tracking changing gradients. Controlled numerical experiments validate these mechanisms, and a preliminary LLM pretraining study illustrates the practical promise of fast-slow momentum.