Muon

Follow this topic Ask about this topic

Momentum

18 papers in the last four weeks, up 350% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 111

All topics

Does Muon Need Fine-Grained Spectral Shaping?
Oct 5, 2026Meher Chaitanya, Tianyi Zhou, Aristides GionisMuon Reweighting
KTH Royal Institute of Technology Stockholm, Sweden
DGA-Muon: Decoupled Geometry-Aligned Adaptive Scaling for Muon
Oct 5, 2026Wenpeng Zhang, Runsheng YuMuon Sequential Scaling
Independent Reasearcher
Muon Is Theoretically Wrong For Convolutions, But Empirically Effective
Oct 5, 2026Thibaut Boissin, Thomas Massena, Mathieu Serrurier +1Muon Neural Network Optimization
IRIT · IRIT, DTIPG - SNCF, UT3 · Institut de Recherche Technologique Saint-Exupery, France
TACO: Ternary Absolute-max Column-wise One-sparse Optimizer for LLM Fine-Tuning
Oct 1, 2026Jichao Jiang, Cristian McGee, El Houcine Bergou +2Model Fine-Tuning Large Language Model Fine-Tuning
University of Central Florida · Mohammed VI Polytechnic University
Muon meets Tamed Langevin: Momentum Preconditioning beyond Convex and gradient-Lipschitz Potentials
Oct 1, 2026Nikolaos Makras, Sotirios SabanisLangevin Dynamics Lipschitz Continuity
School of Mathematics, University of Edinburgh, UK · National Technical University of Athens, Greece · Athena/Archimedes Research Centre, Greece
AF-Muon: An AdamW-Free Muon Optimizer for Tied-Embedding Models
Oct 1, 2026Arash Lagzian, Paniz Halvachi, Junming Zhang +2Muon Adam
National University of Singapore, Singapore · Independent Researcher · LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China
Convergence of Practical Muon with Finite Newton-Schulz Iterations and Nesterov Momentum
Sep 30, 2026Hanyng Peng, Hui Wang, Yue YuMomentum Stochastic Gradient Descent Newton
Pengcheng Laboratory, Shenzhen, China
The Row Normalization Puzzle in Muon
Sep 30, 2026Jiayu Zhang, Tianyi LinMuon Normalization
Department of Industrial Engineering and Operations Research Columbia University
Normalize-Then-Precondition: A Hierarchical Approach to Marginal Scale and Interaction Geometry for LLM Training
Sep 29, 2026Zixuan Gong, Zeyu Gan, Jiaye Teng +1Spectral Preconditioning Large Language Model Pretraining
Gaoling School of Artificial Intelligence Renmin University of China · School of Statistics and Management Shanghai University of Finance and Economics
Second-Moment Stochastic Approximation Methods
Sep 29, 2026Tao Jiang, Lin XiaoStochastic Approximation Adaptive Optimizers
Fundamental AI Research (FAIR), Meta Superintelligence Labs
MeqMuon: Matrix-Equilibrating Muon for LLM Pretraining
Sep 28, 2026Chang-Wei Shi, Xu Wang, Wu-Jun LiLarge Language Model Pretraining Muon
National Key Laboratory for Novel Software Technology, School of Computer Science, Nanjing University, P. R. China
LionMuon: Alternating Spectral and Sign Descent for Efficient Training
Sep 28, 2026Arman Bolatov, Artem Riabinin, Nikita Kornilov +4Muon Descent
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · Applied Artificial Intelligence Institute +1
Muon Sublates the Edge of Stability in LLM Pretraining
Sep 28, 2026Yanzhe Chen, Qifang Zhao, Xiaoxiao Xu +1Muon Large Language Model Alignment
School of Mathematical Sciences, Shanghai Jiao Tong University, China · Alibaba Inc., Hangzhou, China · School of Mathematical Sciences, Institute of Natural Sciences and MOE-LSC, Shanghai Jiao Tong University, China
NS-ATTENTION: Newton-Schulz Transformations of Attention Outputs in Vision Transformers
Sep 23, 2026Xiaohe Jiang, Guoqiang Zhang, Tianjin Huang +1Transformer Attention Linear Attention
University of Exeter · University of Exeter, Exeter, United Kingdom
Muon Can Outperform Dedicated Continual Learning Methods
Sep 21, 2026Sebastian George Sincari, Bogdan Alexandru Gheorghe, Antonio BarbalauReplay-Based Continual Learning Continual Learning
Faculty of Mathematics and Computer Science, University of Bucharest, Bucharest, Romania · Faculty of Mathematics and Computer Science University of Bucharest Bucharest, Romania · Bitdefender, Bucharest, Romania +1
Design of the IBM Granite 5.0 TurboCTC ASR Model
Sep 17, 2026Brian Kingsbury, George Saon, Masayuki Suzuki +5End-To-End Automatic Speech Recognition Models Muon
IBM Research
Derivative-Free Structured Updates for Muon
Sep 15, 2026Pengcheng XieMuon Differentiable Optimization
Lawrence Berkeley National Laboratory & University of California Berkeley, CA 94720, USA
Fast BIB simulation at a future Muon Collider with generative machine learning
Sep 14, 2026Radha Mastandrea, Shiyu Peng, Benjamin Rosser +1Muon Generative Models
Enrico Fermi Institute, The University of Chicago, Chicago, IL 60637, USA · Data Science Institute, The University of Chicago, Chicago, IL 60637, USA · Department of Physics, Brown University, Providence, RI 02912, USA +1
Muon-C: Operator-Aligned Muon for Convolutional Kernels
Sep 9, 2026Jiaxin Qing, Lexin LiMuon Deep Unfolding Networks
University of California, Berkeley Berkeley, CA 94720-1776, USA
Equivariance Breaks the Learning Rate
Sep 8, 2026Andrei Manolache, Mathias NiepertAdam Equivariant Neural Networks
University of Stuttgart, Germany · Bitdefender, Romania
Beyond the Matrix Sign: Quadratic Spectral Descent
Sep 7, 2026Qiaozhe Zhang, Jun Sun, Yingzhuang LiuMuon Local Curvature
School of Electronic Information and Communications Huazhong University of Science and Technology
Toward a First-Principles Update Geometry for the Language-Model Head
Aug 23, 2026Aditya Somasundaram, Charles Guille-Escuret, Alexander Moreno +2Spectral Norm Multi-Head Attention
MBZUAI
Federated Compositional Muon Optimizer for Matrix-Wise Models
Aug 13, 2026Wang Yan, Feihu HuangMuon Stochastic Optimization
College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, China
Dion3: Full-Stack Orthogonal Updates
Aug 12, 2026Noah Amsel, Jack Zhang, Kwangjun Ahn +5Muon Newton
New York University · Microsoft Research · Princeton University +3
Second-Order Muon Done Right: A Principled Marriage of Spectral Geometry and Curvature
Aug 10, 2026Tong CheMuon Newton
NVIDIA Research
Post-Grokking Collapse at the Representation-Readout Interface in Muon-Trained Transformers
Aug 7, 2026Ali Janati, Kaoutar El Maghraoui, Anass BelfatmiMuon Grokking
Data Science Institute, Columbia University, New York, NY, USA · Department of Computer Science, Columbia University, New York, NY, USA · CentraleSupélec, Gif-sur-Yvette, France
Muon on the Stiefel Manifold Admits an Exact Closed-Form Update
Aug 6, 2026Mikhail Solonko, Molozhavenko Alexander, Maxim RakhubaMuon Manifolds
HSE University, Moscow, Russia.
MALT: Lightweight Curvature-Aware Muon via Diagonal Preconditioning
Aug 5, 2026Tongle Wu, Huanyu Dong, Ying Sun +1Spectral Preconditioning Muon
School of Electrical Engineering and Computer Science, The Pennsylvania State University, University Park, PA, USA · Department of Computer Science, City University of Hong Kong, Hong Kong SAR, China
Muon Meets Mamba: Spectral Optimization for State Space Models
Aug 4, 2026Arslan Battalov, Karim Kramin, Alexander Markotenko +1Muon State Space Models
HSE University · Moscow, Russia
Joint Affine Spectral Shaping: Coupling Weight and Bias Updates Beyond Weight-Only Muon
Aug 4, 2026Gongyue Zhang, Honghai LiuAffine Muon
State Key Laboratory of Robotics and Systems, Harbin Institute of Technology, Shenzhen, Shenzhen, 518055, China.
CMuon: Accelerating and Stabilizing Diffusion Transformer Training via Chunked Momentum Orthogonalization
Aug 3, 2026Chuyan Chen, Peng Sun, Kun YuanDiffusion Transformers Muon
1Peking University · 2Westlake University · 3Zhejiang University
Sign compression for Muon: SignMuon, MuonSign, and the Limits of Error Feedback
Jul 31, 2026Maria Smirnova, Alexey KravatskiyMuon Momentum Stochastic Gradient Descent
MIRIAI Moscow, Russia
Sharpness-Aware Minimization and Muon: Robustness under the Spectral Norm
Jul 28, 2026Wenzhi Zhong, Edward Milsom, Michael MurraySharpness-Aware Minimization Regularization
Department of Mathematical Sciences University of Bath · Department of Computer Science University of Bath
Scale Weight Decay and Train Better
Jul 26, 2026Anuj ApteWeight Decay Batch
Global Technology Applied Research, JPMorganChase, New York, NY 10001
An Isotropy-Preserving Spectral Cap for Muon: Theory and Three Case Studies
Jul 22, 2026Jiachun LiMuon Spectral
PoLoRA: A Preconditioned Orthogonalized LoRA Optimizer
Jul 20, 2026Nikhil Ghosh, Tetiana Parshakova, Robert M. GowerMultimodal Continual Instruction Tuning Muon
Center for Computational Mathematics, Flatiron Institute, New York, NY
When Does Muon Help Agentic Reinforcement Learning?
Jul 17, 2026Kai Ruan, Jinghao Lin, Zihe Huang +4Muon Adam
Gaoling School of Artificial Intelligence, Renmin University of China · Independent Researcher · Institute of Computing Technology, Chinese Academy of Sciences +3
Muse: Representation Geometry of Muon Beyond Normalized Momentum
Jul 16, 2026Da Chang, Qiankun Shi, Lvgang Zhang +4Muon Representation Space
1Pengcheng Laboratory · 2Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences +3
Mono-Z Dark Matter Search with Neural Spline Flows Using CMS Run 2015D Open Data
Jul 15, 2026Hitesh Rasineni, Bhavishya ChebroluGeneral Relativity Muon
VIT-AP University, Amaravati, India · 1VIT-AP University, Amaravati, 522241, India · Mohan Babu University, Tirupati, India +1
Reassessing Muon for Matrix Factorization
Jul 14, 2026Ali Parviz, Gal Mishne, Alex CloningerMuon Adam
†. Halicioğlu Data Science Institute, UC San Diego. · ∗. Department of Mathematics, UC San Diego.
Efficient Long-Horizon Learning for Learned Optimization
Jul 7, 2026Xiaolong Huang, Benjamin Thérien, James Harrison +1Meta-Learning Adaptive Optimizers
Mila - Quebec AI Institute · Concordia University · Université de Montréal +1
Beyond Adam: SOAP and Muon for Faster, Label-Efficient Training of Machine Learning Interatomic Potentials
Jul 2, 2026Gil Harari, Yoel Zimmermann, Ola Tangen Kulseng +4Interatomic Potentials Adam
John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA · Robert Bosch LLC Research and Technology Center, Watertown, MA, USA
Muon as a Residual Connection
Jul 1, 2026Hao HuangMuon Residual Connections
College of Computer Science and Technology Zhejiang University Hangzhou, Zhejiang, China
MNAR- $k$ -means: A $k$ -means Clustering for Data Missing Not at Random with Magnitude-Decaying Probability
Jun 30, 2026Xin GuanK-Means Imputation
Graduate School of Information Sciences Tohoku University Sendai, Miyagi 980-8579, Japan
Muon learns balanced solutions in matrix factorization without slow saddle-to-saddle dynamics
Jun 29, 2026Mark Rhee, Jamie Simon, Dhruva KarkadaMuon Restricted Bernoulli Matrix Factorization
UC Berkeley · Imbue
Aurora: A Leverage-Aware Spectral Optimizer
Jun 26, 2026Alec Dewulf, Dhruv Pai, Li Yang +2Muon Batch Normalization
Tilde Research
Hierarchical Muon: Tiled Newton-Schulz Updates for Efficient Muon Optimization
Jun 25, 2026Ziyuan Tang, Tianshi Xu, Yousef Saad +1Muon Newton
Department of Computer Science and Engineering, University of Minnesota, Minneapolis · Department of Mathematics, Emory University, Atlanta, GA 30322
DMuon: Efficient Distributed Muon Training with Near-Adam Overhead
Jun 25, 2026Vincent Chen, Starrick Liu, Regis Cheng +8Muon Inference Workloads
X SQUARE ROBOT TEAM
Tensorion: A Tensor-Aware Generalization of the Muon Optimizer
Jun 24, 2026Vladimir Bogachev, Vladimir Aletov, Alexander Molozhavenko +2Adam Muon
HSE University · Basic Research of Artificial Intelligence Laboratory (BRAIn Lab)
Muown Implicitly Performs Angular Step-size Decay
Jun 22, 2026Florian Hübler, Kai Lion, Antonio Orvieto +1Muon Adaptive Optimizers
Department of Computer Science · ETH Zurich, Switzerland · ELLIS Institute Tübingen, MPI-IS +1
Towards Understanding the Power and Limits of the Muon Optimizer: A River-Valley Perspective
Jun 19, 2026Tianqi Shen, Jinji Yang, Runze Shi +3Muon Adam
Department of Computer Science City University of Hong Kong (CityUHK) · Department of Statistics and Data Science University of Pennsylvania · School of Statistics and Data Science Shanghai University of Finance and Economics
Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization
Jun 15, 2026Kaiyue Wen, Xingyu Dang, Kaifeng Lyu +2Adaptive Optimizers Weight Decay
†Stanford University. · ‡Princeton University. · §Tsinghua University.
CacheMuon: Using Temporal Preconditioning To Approximate Polar Factor
Jun 15, 2026Bishnu Dev, Sushil Bohara, Martin Takáč +1Muon Spectral Preconditioning
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · Mohamed bin Zayed University of Artificial Intelligence
Schattor: Schatten-family methods for deep learning optimization
Jun 14, 2026Bohao Ma, Junyu Zhang, Chuan HeNeural Network Optimization Stochastic Optimization
School of Data Science, The Chinese University of Hong Kong (Shenzhen), Shenzhen, Guangdong, China · Department of Industrial Systems Engineering and Management, National University of Singapore, Singapore · Department of Mathematics, Linköping University, Sweden
When to use what Schatten- $p$ norm in deep learning?
Jun 13, 2026Thomas PethickSpectral Norm Muon
Free Heavy-Tailed Lunch for Muon: A Theoretical Justification of Empirical Success
Jun 12, 2026Florian Hübler, Thomas Pethick, Suvrit SraSpectral Norm Muon
Department of Computer Science, ETH Zurich, Switzerland · Department of Mathematics, Technical University of Munich, Germany · Munich Center for Machine Learning (MCML)
Muon $^p$ : Muon with Fractional Spectral Powers
Jun 11, 2026Yihe Dong, Will SawinMuon Spectral
Princeton University
LoRA-Muon: Spectral Steepest Descent on the Low-Rank Manifold
Jun 11, 2026Franz Louis Cesista, Katherine Crowson, Cédric Simal +1Low-Rank Structure Muon
Ateneo de Manila University · EleutherAI · NaXys +1
Agentic Hybrid RAG for Evidence-Grounded Muon Collider Analysis
Jun 9, 2026Ruobing Jiang, Dawei Fu, Cheng Jiang +6Agentic Retrieval-Augmented Generation Systems Muon
State Key Laboratory of Nuclear Physics and Technology, Peking University, China
Muon Learns More Robust and Transferable Features than Adam
Jun 8, 2026Tianyu Ruan, Fengzhuo Zhang, Shuche Wang +1Adam Feature Learning
Yale University · University of Chinese Academy of Sciences · Academy of Mathematics and Systems Science, CAS +1

Oct 5, 2026, cs.AIJ/K move · Enter open · S save

Does Muon Need Fine-Grained Spectral Shaping?

Meher Chaitanya, Tianyi Zhou, Aristides Gionis

KTH Royal Institute of Technology Stockholm, Sweden

Muon Reweighting

Open paper AskPDF arXiv

Muon combines current and past gradients into matrix momentum. For

M=UΣV^\top

, the idealized polar update

Q=UV^\top

gives every singular direction the same weight. We refer to this as the flat profile. Several recent optimizers replace this flat profile with fine-grained spectral maps that give each direction its own gain. We ask how much of this spectral detail a Muon update needs. Our spectral diagnostics show that approximately

94

97\%

of measured singular modes lie below an estimated noise edge, yet collectively align positively with a reference gradient. We introduce BulkBoost, a two-band spectral reweighting framework with fixed-rank and noise-calibrated variants. The latter uses split-minibatch gradient differences to calibrate a Marchenko--Pastur reference edge for Muon's Nesterov input, separating the bulk below the edge from the spikes above it. Both variants increase the bulk's relative weight through one shared gain while preserving the Frobenius norm of each matrix's unreweighted direction. For a fixed partition, our theory gives the first-order condition under which moving weight toward the bulk lowers the loss. It also quantifies the fraction of the maximal first-order improvement rate, over all per-mode reallocations, that two bands can capture. Across 30 continued-pretraining settings spanning Pythia-14M to 410M and six corpora, two-band reweighting is competitive with the fine-grained power-law profile of Freon and outperforms Spectra. Measured against Muon's flat profile, Freon reduces final loss by

0.022\%

of the pre-adaptation loss on average, whereas the two-band variants achieve reductions of

0.073

0.147\%

. These observations suggest that useful departures from the flat profile are surprisingly low-dimensional: a single bulk-to-spike gain captures at least as much benefit as the fine-grained spectral profiles.

Momentum

Related topics

Latest papers 111