Newton-Schulz Iteration

Follow this topic Ask about this topic

Momentum

2 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 12

All topics

Muon Is Theoretically Wrong For Convolutions, But Empirically Effective
Oct 5, 2026Thibaut Boissin, Thomas Massena, Mathieu Serrurier +1Newton-Schulz Iteration Neural Network Optimization
IRIT · IRIT, DTIPG - SNCF, UT3 · Institut de Recherche Technologique Saint-Exupery, France
Convergence of Practical Muon with Finite Newton-Schulz Iterations and Nesterov Momentum
Sep 30, 2026Hanyng Peng, Hui Wang, Yue YuNewton-Schulz Iteration Gradient Descent
Pengcheng Laboratory, Shenzhen, China
NS-ATTENTION: Newton-Schulz Transformations of Attention Outputs in Vision Transformers
Sep 23, 2026Xiaohe Jiang, Guoqiang Zhang, Tianjin Huang +1Newton-Schulz Iteration Vision Transformer
University of Exeter · University of Exeter, Exeter, United Kingdom
Dion3: Full-Stack Orthogonal Updates
Aug 12, 2026Noah Amsel, Jack Zhang, Kwangjun Ahn +5Newton-Schulz Iteration GPU Kernel Optimization
New York University · Microsoft Research · Princeton University +3
Newton-Schulz Retraction-Based Inference Enables Hidden Quantum Markov Models to Outperform Classical HMMs
Aug 6, 2026Ning NingNewton-Schulz Iteration Markov Models
Department of Statistics, Texas A&M University College Station, Texas, USA
On MUON optimization: From non-convergence to an error analysis with Polar Express and the Newton-Schulz polynomial from implementations
Aug 5, 2026Thang Do, Steffen Dereich, Arnulf JentzenStochastic Optimization Newton-Schulz Iteration
School of Data Science, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China · Department of Probability and Statistics, Institute of Mathematics, Vietnam Academy of Science and Technology, Vietnam · Institute for Mathematical Stochastics, Faculty of Mathematics and Computer Science, University of Münster, Germany +2
Hierarchical Muon: Tiled Newton-Schulz Updates for Efficient Muon Optimization
Jun 25, 2026Ziyuan Tang, Tianshi Xu, Yousef Saad +1Newton-Schulz Iteration Neural Network Optimization
Department of Computer Science and Engineering, University of Minnesota, Minneapolis · Department of Mathematics, Emory University, Atlanta, GA 30322
Spectral Scaling Laws of Muon
Jun 2, 2026Gagik Magakyan, Pablo Parrilo, Asuman OzdaglarNewton-Schulz Iteration Momentum Methods
MIT
How Much Orthogonalization Does Muon Need?
May 29, 2026Hua HuangNewton-Schulz Iteration Neural Network Optimization
NVIDIA
AMO: Operator-level Adaptive Muon Orthogonalization
May 18, 2026Xinlin Zhuang, Panyi Ouyang, Yichen Li +7Language Model Pretraining Newton-Schulz Iteration
The Chinese University of Hong Kong · Shopee · MBZUAI +3
A second-order method landing on the Stiefel manifold via Newton $\unicode{x2013}$ Schulz iteration
May 4, 2026Xinhui Xiong, Bin Gao, P. -A. AbsilNewton-Schulz Iteration Stiefel Manifold Optimization
State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, and University of Chinese Academy of Sciences, China · State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, China · ICTEAM Institute, UCLouvain, Belgium
Turbo-Muon: Almost-Orthogonal Pre-Conditioning for Fast Muon Updates
Dec 4, 2025Thibaut Boissin, Thomas Massena, Franck Mamalet +1Newton-Schulz Iteration Neural Network Optimization
IRIT-MISFIT · Institut de Recherche Technologique Saint-Exupery, France · DTIPG - SNCF, IRIT-MISFIT +2

Oct 5, 2026, cs.LGJ/K move · Enter open · S save

Muon Is Theoretically Wrong For Convolutions, But Empirically Effective

Thibaut Boissin, Thomas Massena, Mathieu Serrurier, Franck Mamalet

IRIT · IRIT, DTIPG - SNCF, UT3 · Institut de Recherche Technologique Saint-Exupery, France

Newton-Schulz Iteration Neural Network Optimization Muon Optimizer Convolutional Neural Networks

Open paper AskPDF arXiv

Muon, an optimizer known for its efficiency, has a clear interpretation for matrix-valued updates, but convolutional kernels are stored as four-dimensional tensors. Standard implementations reshape these tensors into matrices, a shortcut which breaks the theoretical understanding behind Muon. To investigate this, we formalize the corresponding optimization objective directly in convolutional operator geometry and introduce Convolutional Newton-Schulz (Conv-NS), which approximates the polar factor in this geometry while preserving kernel support. When applied in fast training experiments, Conv-NS and reshape-based Muon are both computationally efficient and achieve comparable accuracy on CIFAR-10 and ImageNet classification tasks. However, as one could expect a theoretically aligned Conv-NS to outperform reshape-based Muon, we investigate this mismatch between practice and theoretical understanding, with the hypothesis that exact convolutional orthogonalization may overconstrain updates. These findings highlight Muon's strong practical performance while opening directions for its further development on convolutions. Our code is publicly available at github conv-muon.

Momentum

Related topics

Latest papers 12