Muon Optimizer

Follow this topic Ask about this topic

Muon is a matrix-aware optimizer that accumulates momentum for weight matrices and approximately orthogonalizes each update, often with Newton–Schulz iterations, rather than scaling each parameter independently as Adam does. Research explores variants with curvature or variance preconditioning, different spectral weighting and normalization, and cheaper or distributed orthogonalization. Studies test Muon in language-model training, vision, recommendation, diffusion, and scientific machine learning, while theory examines convergence, noise, implicit bias, and generalization. A central open question is when its spectral updates improve learning enough to justify their cost: results depend on architecture, batch size, and training phase, and practical approximations can change both behavior and guarantees.

Latest papers

146 papers32 papers in the last four weeks, up 146% on the four weeks before. 0.2% of all new papers.

How the Optimizer Shapes Learned Solutions in Equivariant Neural Networks
May 26, 2026Teodor-Mihai Stupariu, Andrei ManolacheNeural Representation Geometry Neural Network Optimization
1Bitdefender, Romania · 2Tudor Vianu High School of Com · University of Stuttgart, Germany +1
Convergence of Spectral Descent for Non-smooth Optimization
May 26, 2026Yixuan Yang, Yuqing He, Song LiMuon Optimizer Nonsmooth Convex Optimization
School of Mathematical Sciences, Zhejiang University, Hangzhou, China
When Muon Optimizer Meets Adversarial Training: A Theoretical and Empirical Study
May 26, 2026Jun Yan, Weiquan Huang, Jiankai Zuo +4Adversarial Training Orthogonal Optimization
IT College, Shanghai Ocean University, Shanghai, 201306 · School of Computer Science and Technology, Tongji University, Shanghai, 201804 · SEIE, Suzhou University of Science and Technology, Suzhou, 215009 +2
MONA: Muon Optimizer with Nesterov Acceleration for Scalable Language Model Training
May 26, 2026Jiacheng Li, Jianchao Tan, Hongtao Xu +5Language Model Pretraining Gradient Descent
Meituan, Beijing, China
MuCon: Clipped Muon Updates for LLM Training
May 26, 2026Albert YiMuon Optimizer Spectral Methods
Muon in Vision Transformers: Optimizer-Recipe Interactions and Gradient Spectra
May 23, 2026Ben S. Southworth, Shuai Jiang, Daniel McBride +2Vision Transformer Muon Optimizer
Los Alamos National Laboratories Los Alamos, NM 87545 · Sandia National Laboratories Albuquerque, NM 87123 · Lehigh University Bethlehem, PA 18015
Move on Muon : A Hamiltonian probability gradient flow perspective of Muon optimizer
May 22, 2026Aratrika Mustafi, Soumya Mukherjee, Bharath K. SriperumbudurMomentum Methods Muon Optimizer
Department of Statistics, Pennsylvania State University
AMUSE: Anytime Muon with Stable Gradient Evaluation
May 21, 2026Jueun Kim, Baekrok Shin, Jihun Yun +3Deep Learning Optimization Muon Optimizer
KAIST · KRAFTON · Seoul National University, KRAFTON
LionMuon: Alternating Spectral and Sign Descent for Efficient Training
May 19, 2026Arman Bolatov, Artem Riabinin, Nikita Kornilov +4Stochastic Optimization Sign-Based Optimization
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · Applied Artificial Intelligence Institute +1
From SGD to Muon: Adaptive Optimization via Schatten-p Norms
May 19, 2026Thomas Massena, Corentin Friedrich, Mathieu SerrurierLMO Methods Neural Network Optimization
DTIPG - SNCF, IRIT · IRIT
MiMuon: Mixed Muon Optimizer with Improved Generalization for Large Models
May 19, 2026Feihu Huang, Yuning Luo, Songcan ChenNeural Network Generalization Deep Learning Optimization
College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, China · College of Design and Engineering, National University of Singapore, Singapore
Rethinking Muon Beyond Pretraining: Spectral Failures and High-Pass Remedies for VLA and RLVR
May 19, 2026Chongyu Fan, Gaowen Liu, Mingyi Hong +2Muon Optimizer Vision-Language-Action Models
Michigan State University · Cisco · University of Minnesota +1
Distance-Aware Muon: Adaptive Step Scaling for Normalized Optimization
May 18, 2026Yury Demidovich, Abhishek Chakraborty, Grigory Malinovsky +2Muon Optimizer Nonconvex Optimization
King Abdullah University of Science and Technology · Arizona State University
AMO: Operator-level Adaptive Muon Orthogonalization
May 18, 2026Xinlin Zhuang, Panyi Ouyang, Yichen Li +7Language Model Pretraining Newton-Schulz Iteration
The Chinese University of Hong Kong · Shopee · MBZUAI +3
DynMuon: A Dynamic Spectral Shaping View of Muon
May 16, 2026Fangzhou Wu, Rikhav Shah, Sandeep Silwal +1Neural Network Optimization Muon Optimizer
University of Wisconsin–Madison · MIT · Elorian AI
Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence
May 13, 2026Tien-Phat Nguyen, Truong Nguyen, Minh-Phuc Truong +3Momentum Methods Gradient Descent
Hanoi University of Science and Technology Hanoi, Vietnam · Indiana University 107 S. Indiana Ave, Bloomington, IN 47405, USA · Monash University Clayton, VIC 3800, Australia
DP-Muon: Differentially Private Optimization via Matrix-Orthogonalized Momentum
May 13, 2026Jihwan Kim, Chenglin FanDeep Learning Optimization Differentially Private Stochastic Gradient Descent
Seoul National University
Constrained Stochastic Spectral Preconditioning Converges for Nonconvex Objectives
May 12, 2026Konstantinos Oikonomidis, Jan Quan, Kimon Antonakopoulos +3Nonconvex Stochastic Optimization Muon Optimizer
KU Leuven, Department of Electrical Engineering, STADIUS Center for Dynamical Systems, Signal Processing, and Data Analytics · ´Ecole Polytechnique F´ed´erale de Lausanne (EPFL), Laboratory for Information and Inference Systems (LIONS) · CVN, CentraleSup´elec, Universit´e Paris-Saclay, Inria
MuonQ: Enhancing Low-Bit Muon Quantization via Directional Fidelity Optimization
May 12, 2026Yupeng Su, Ruijie Zhang, Ziyue Liu +24-Bit Quantization Muon Optimizer
University of California, Santa Barbara
Muon is Not That Special: Random or Inverted Spectra Work Just as Well
May 11, 2026Zakhar Shumaylov, Nathaël Da Costa, Peter Zaika +6Stochastic Optimization Deep Learning Optimization
University of Cambridge · University of Tübingen · University of Oxford
Muown: Row-Norm Control for Muon Optimization
May 11, 2026Kai Lion, Florian Hübler, Bingcong Li +2Language Model Pretraining Muon Optimizer
Department of Computer Science, ETH Zurich, Switzerland · Department of Mathematics, School of Computation, Information and Technology; Technical University of Munich, Germany · ELLIS Institute Tübingen, MPI-IS, Tübingen AI Center, Germany
Can Muon Fine-tune Adam-Pretrained Models?
May 11, 2026Xingyu Qu, Peigeng Huang, Samuel HorvathFine-Tuning Muon Optimizer
MBZUAI · Nanjing University
Phases of Muon: When Muon Eclipses SignSGD
May 10, 2026Elliot Paquette, Noah Marshall, Lucas Benigni +3Sign-Based Optimization Muon Optimizer
Mathematics and Statistics Department, McGill University · Mathematics and Statistics Department, Université de Montréal · Google DeepMind
Dimension-Free Saddle-Point Escape in Muon
May 10, 2026Yanlin Long, Yufei Gu, Zeke XieMuon Optimizer Nonconvex Optimization
xLeaF Lab, The Hong Kong University of Science and Technology (Guangzhou)
Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifolds
May 10, 2026Yibang Li, Bihari Lal Pandey, Ravi Sah +4Stiefel Manifold Optimization Riemannian Optimization
Nanyang Technological University, Singapore. · Indian Institute of Technology Bombay, India. · University of Sydney, Australia. +1
Accelerating Zeroth-Order Spectral Optimization with Partial Orthogonalization from Power Iteration
May 9, 2026Jiahe Chen, Ziye MaZeroth-Order Optimization LLM Fine-Tuning
Department of Computer Science City University of Hong Kong
Muon Does Not Converge on Convex Lipschitz Functions
May 9, 2026Tetiana Parshakova, Ahmed Khaled, Michael Crawshaw +2Deep Learning Optimization Error Feedback
Center for Computational Mathematics, Flatiron Institute, Simons Foundation · Princeton University, Google DeepMind · George Mason University +1
Muon-OGD: Muon-based Spectral Orthogonal Gradient Projection for LLM Continual Learning
May 9, 2026Binghang Lu, Zheyuan Deng, Runyu Zhang +6Continual Learning for LLMs Gradient Descent
Purdue University · Brown University · Massachusetts Institute of Technology +4
When and Why Grouping Attention Heads Accelerates Muon Optimization
May 9, 2026Hongtao Zhang, Wenjie Zhou, Wei Chen +1Attention Head Analysis Muon Optimizer
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · University of Chinese Academy of Sciences
OrScale: Orthogonalised Optimization with Layer-Wise Trust-Ratio Scaling
May 8, 2026Yuxuan Lou, Yang YouOrthogonal Optimization Neural Network Optimization
National University of Singapore

May 26, 2026J/K move · Enter open · S save

How the Optimizer Shapes Learned Solutions in Equivariant Neural Networks

Teodor-Mihai Stupariu, Andrei Manolache

1Bitdefender, Romania · 2Tudor Vianu High School of Com · University of Stuttgart, Germany+1

Neural Representation Geometry Neural Network Optimization Equivariant Neural Networks Muon Optimizer Equivariant Representation Learning Neural Network Training Dynamics

Open paper AskPDF arXiv

Equivariant neural networks encode geometric symmetries by construction, yet they are often difficult to optimize and can underperform less constrained architectures. A growing body of work addresses this through architectural modifications such as constraint relaxation or approximate equivariance, while the role of the optimizer remains comparatively underexplored. We study this direction by comparing Muon and Adam across several equivariant and geometric architectures under pointcloud and molecular learning settings. On ModelNet40, where the comparison is clearest, Muon consistently improves over Adam across all architectures considered. We then analyze the trained ModelNet40 checkpoints through Hessian estimates, loss surface visualizations, and spectral properties of learned weights and intermediate representations. The checkpoints reached by Muon have larger Hessian curvature summaries but more regular loss surfaces, and their learned weights and representations have higher stable and effective ranks. These observations suggest that the interaction between optimizer design and geometric inductive bias deserves further attention from the community.