Overview

Muon is a matrix-aware optimizer that accumulates momentum for weight matrices and approximately orthogonalizes each update, often with Newton–Schulz iterations, rather than scaling each parameter independently as Adam does. Research explores variants with curvature or variance preconditioning, different spectral weighting and normalization, and cheaper or distributed orthogonalization. Studies test Muon in language-model training, vision, recommendation, diffusion, and scientific machine learning, while theory examines convergence, noise, implicit bias, and generalization. A central open question is when its spectral updates improve learning enough to justify their cost: results depend on architecture, batch size, and training phase, and practical approximations can change both behavior and guarantees.

Latest papers

CardsList

159 papers26 papers in the last four weeks, up 13% on the four weeks before. 0.2% of all new papers.

  1. Muon Attains O(ε−3)\mathcal{O}(\varepsilon^{-3}) Iteration Complexity under Shuffling

    Nov 13, 2026Kun Huang, Shi Pu, Karl Henrik JohanssonMuon OptimizerStochastic Optimization

  2. Byzantine-Robust Muon: Resilient Momentum Orthogonalization for Distributed Learning

    Nov 9, 2026Grigory Malinovsky, Yury Demidovich, Peter RichtárikMuon OptimizerFault-Tolerant Distributed Training

  3. BiMuon: A Geometry-Aware Stochastic Bilevel Optimization Approach

    Nov 6, 2026Zhen Qin, Zhiyao Zhang, Yingbin Liang +1Stochastic Bilevel OptimizationMatrix Optimization

  4. Theoretical Analysis of Muon on Transformers for Next-Token Prediction

    Nov 3, 2026Shengwei Wu, Yixuan Yang, Gao Huang +1Transformer Training DynamicsMuon Optimizer

  5. Pretraining Data Scale Reverses the Muon-AdamW Fine-Tuning Gap

    Nov 2, 2026Catherine Li, Jingchu Gai, Andrej Risteski +2Muon OptimizerLLM Fine-Tuning

  6. Muon Makes the First Mixture-of-Experts Layer Harder to Balance

    Oct 27, 2026Marcos HernanzMixture-of-Experts Language ModelsExpert Load Balancing

  7. MuonGS: Pretraining LLMs with Relaxed Weight-Norm Control

    Oct 27, 2026Athanasios Glentis, Dawei Li, Mingyi HongGradient OrthogonalizationMuon Optimizer

  8. Sharpness-Aware Muon: Matrix-Aware Optimization with Momentum-Guided Perturbations

    Oct 27, 2026Zhiwei Zhai, Wenjing Yan, Ying-Jun Angela ZhangMuon OptimizerSharpness-Aware Minimization

  9. Last-Iterate Stationarity of Muon with Finitely Many Newton-Schulz Iterations

    Oct 25, 2026Ping Li, Vincent Y. F. TanMuon OptimizerLast-Iterate Convergence

  10. When and How Does Muon Precondition? The Role of Structured Gradient Noise

    Oct 22, 2026Xiaoran Chen, Binghui Li, Liwei Wang +1Muon Optimizer

  11. A Quadratic Lens on Muon's Polar Update: Benefits, Pitfalls, and the Role of Newton-Schulz

    Oct 21, 2026Egor Shulgin, Sam Laing, Antonio Orvieto +1Muon OptimizerOrthogonal Optimization

  12. Constant Step Sizes for Muon-Type Spectral Optimizers on Strongly Convex Quadratics

    Oct 20, 2026Pratik Jawanpuria, Bhabatosh Garai, Bamdev MishraMuon OptimizerSpectral Optimization

  13. T-MUON: Tensorizing Muon via t-SVD for Large Language Model Pre-training

    Oct 18, 2026Dongyang Li, Mingsong Yan, Zheng Zhang +1Muon OptimizerLanguage Model Pretraining

  14. How Does Muon Work in Private Deep Learning? Scalable Differentially Private Training via Orthogonalization

    Oct 17, 2026Alexander Gaponov, Grigory Malinovsky, Egor Shulgin +1Muon OptimizerDeep Learning Optimization

  15. Muon Operates Beyond the Edge of Stability

    Oct 16, 2026Niko Covic, Baraq Lipshitz, Alexander Padula +1Muon OptimizerEdge of Stability

  16. BMuon: Budgeting Muon's Spectral Geometry for Structured Compressibility

    Oct 16, 2026Rida Assalouh, Tom Jacobs, Rebekka Burkholz +2Muon OptimizerStructured Sparsity

  17. The Ephemeral Cost of Switching Optimizers Across LLM Training Stages

    Oct 15, 2026Xinyu Zhao, Davis Wertheimer, Ashutosh Sharma +2LLM TrainingLLM Post-Training

  18. When Does Muon Actually Win? A Paired Data-, Time-, and Tuning-Budget Benchmark

    Oct 15, 2026Mushir Akhtar, Hazem N. Nounou, Majdi Mansouri +1Muon Optimizer

  19. When Is Muon Optimal? Radial Fragility and Minimax Geometry

    Oct 15, 2026Ao XuMuon OptimizerNeural Network Optimization

  20. Long-Horizon RL Needs Muon

    Oct 14, 2026Kai Ruan, Jinghao Lin, Tao Chen +3Long-Horizon RLLong-Horizon LLM Agents

  21. From Spectral to Frobenius: When Muon Loses Its Spectral Implicit Bias

    Oct 14, 2026Yongtao Wu, Mahdi Mansouri, Zhenyu ZhuMuon OptimizerImplicit Regularization

  22. Taming Softmax at the Limits of Muon: A Unified Approach through Hilbert Geometry

    Oct 13, 2026Aditya Somasundaram, Charles Guille-Escuret, Alexander Moreno +2Muon OptimizerSoftmax Attention

  23. Monarch-Muon: Structure-Aware Muon for Fast and Memory-Efficient Pretraining

    Oct 12, 2026Alexander Miasnikov, Ruslan Kabirov, Gleb Molodtsov +7Muon OptimizerLLM Training

  24. From Parallel to Sequential: Small-Batch Noise Restores Simplicity Bias of Muon

    Oct 12, 2026Tom Jacobs, Rebekka BurkholzMuon OptimizerNonconvex Optimization

  25. Spectrally Targeted Muon

    Oct 7, 2026Vishrut Goyal, Rohan RamkumarSpectral OptimizationDeep Learning Optimization

  26. The Best Optimizer Depends on Batch Size

    Oct 6, 2026Xingyu Dang, Kaiyue Wen, Sadhika MalladiDeep Learning OptimizationHyperparameter Transfer