Spectral Optimization

Follow this topic Ask about this topic

Latest papers

23 papers6 papers in the last four weeks, up 20% on the four weeks before. 0.0% of all new papers.

Constant Step Sizes for Muon-Type Spectral Optimizers on Strongly Convex Quadratics
Oct 20, 2026Pratik Jawanpuria, Bhabatosh Garai, Bamdev MishraMuon Optimizer Spectral Optimization
Centre for Machine Intelligence and Data Science, IIT Bombay, India · Microsoft, India
Corrected Block Sparse Spectral Optimization: Exact Models, Finite Identification, and Local Dynamics
Oct 19, 2026Omid Shams SolariSpectral Optimization Structured Sparsity
sAIm Labs
Bridging Compressed Communication and Local Steps via Spectral Method in Federated Learning
Oct 16, 2026Dmitriy Bystrov, George Chirkov, Daniil Medyakov +2Federated Learning Communication-Efficient Distributed Training
Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · Federated Learning Problems Laboratory · Innopolis University
Which Geometry on Which Layer? Measured Sign-Spectral Assignment in Mixed-Optimizer Training
Oct 12, 2026Hiroki Naganuma, Laura Gomezjurado, Mahdi Ghaznavi +5Spectral Optimization Deep Learning Optimization
Mila · Université de Montréal · Stanford University +4
Spectrally Targeted Muon
Oct 7, 2026Vishrut Goyal, Rohan RamkumarSpectral Optimization Deep Learning Optimization
Department of Computer Science, Stanford University · Department of Electrical Engineering, Stanford University.
ORCA: The Annealed Spectral Conditioning Optimizer for Faster, Better LLM Training
Oct 5, 2026Yuanshi Liu, Boyuan Jiang, Liang Hou +4Spectral Regularization Deep Learning Optimization
School of Intelligence Science and Technology, Peking University · Kling Team
Neural networks for spectral optimization
Sep 28, 2026Alexis de Villeroché, Beniamin Bogosel, Stéphane Breuils +2Neural Surrogate Modeling Spectral Optimization
Orthogonal Witness Control for Muon Optimization via Sigmoid Spectral Reshaping
Sep 27, 2026Dat Phi Van, Ngo Vu Minh, Tuc Nguyen +3Neural Network Optimization Spectral Optimization
Center for AI Research, VinUniversity, Hanoi, Vietnam · Indiana University, Bloomington, IN, USA · Monash University, Clayton, VIC, Australia
Online Supervised Dimension Reduction with Random Features: Diagnostics and Computational Trade-offs
Sep 17, 2026Zhenlin Yao, Wei XiongPrincipal Component Analysis Random Feature Methods
School of Statistics University of International Business and Economics Beijing, China
The Drift Contract: Spectral Updates for Depth-Robust Local Learning
Sep 16, 2026Fabien PollySpectral Optimization Local Learning
Independent researcher
Beyond the Matrix Sign: Quadratic Spectral Descent
Sep 7, 2026Qiaozhe Zhang, Jun Sun, Yingzhuang LiuSign-Based Optimization Muon Optimizer
School of Electronic Information and Communications Huazhong University of Science and Technology
Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muon
Aug 26, 2026Xiaodong Wu, Wenyi Yu, Chao Zhang +1Deep Learning Optimization Muon Optimizer
Department of Engineering University of Cambridge · Department of Electronic Engineering Tsinghua University
Joint Affine Spectral Shaping: Coupling Weight and Bias Updates Beyond Weight-Only Muon
Aug 4, 2026Gongyue Zhang, Honghai LiuNeural Network Optimization Muon Optimizer
State Key Laboratory of Robotics and Systems, Harbin Institute of Technology, Shenzhen, Shenzhen, 518055, China.
Aurora: A Leverage-Aware Spectral Optimizer
Jun 26, 2026Alec Dewulf, Dhruv Pai, Li Yang +2Neural Network Optimization Muon Optimizer
Tilde Research
Muon $^p$ : Muon with Fractional Spectral Powers
Jun 11, 2026Yihe Dong, Will SawinNeural Network Optimization Muon Optimizer
Princeton University
The Spectral Dynamics and Noise Geometry of Muon
Jun 7, 2026Pierfrancesco Beneventano, Mahmoud Abdelmoneum, Tomaso PoggioMuon Optimizer Matrix Optimization
Massachusetts Institute of Technology
Anytime Training with Schedule-Free Spectral Optimization
May 21, 2026Anuj Apte, Pranav Deshpande, Niraj Kumar +2Deep Learning Optimization Learning Rate Scheduling
Global Technology Applied Research, JPMorganChase New York, NY 10001, USA
Constrained Stochastic Spectral Preconditioning Converges for Nonconvex Objectives
May 12, 2026Konstantinos Oikonomidis, Jan Quan, Kimon Antonakopoulos +3Nonconvex Stochastic Optimization Muon Optimizer
KU Leuven, Department of Electrical Engineering, STADIUS Center for Dynamical Systems, Signal Processing, and Data Analytics · ´Ecole Polytechnique F´ed´erale de Lausanne (EPFL), Laboratory for Information and Inference Systems (LIONS) · CVN, CentraleSup´elec, Universit´e Paris-Saclay, Inria
Phases of Muon: When Muon Eclipses SignSGD
May 10, 2026Elliot Paquette, Noah Marshall, Lucas Benigni +3Sign-Based Optimization Muon Optimizer
Mathematics and Statistics Department, McGill University · Mathematics and Statistics Department, Université de Montréal · Google DeepMind
Accelerating Zeroth-Order Spectral Optimization with Partial Orthogonalization from Power Iteration
May 9, 2026Jiahe Chen, Ziye MaZeroth-Order Optimization LLM Fine-Tuning
Department of Computer Science City University of Hong Kong
Orth-Dion: Eliminating Geometric Mismatch in Distributed Low-Rank Spectral Optimization
May 7, 2026Tatsuhiro Nakamori, Laura Gomezjurado Gonzalez, Ganesh Talluri +5Communication-Efficient Distributed Training Distributed Optimization
1Keio University · 2Stanford University · 3Midwestern University +3
Subspace Control: Turning Constrained Model Steering into Controllable Spectral Optimization
Apr 5, 2026Yancheng Huang, Changsheng Wang, Chongyu Fan +7Gradient Orthogonalization Spectral Optimization
OPTML, Michigan State University · MIT-IBM Watson AI Lab, IBM Research · University of Minnesota +2
Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memory
Mar 27, 2026Juno Kim, Eshaan Nichani, Denny Wu +2Spectral Optimization Neural Network Optimization
UC Berkeley · Princeton University · New York University +1

Oct 20, 2026J/K move · Enter open · S save

Constant Step Sizes for Muon-Type Spectral Optimizers on Strongly Convex Quadratics

Pratik Jawanpuria, Bhabatosh Garai, Bamdev Mishra

Centre for Machine Intelligence and Data Science, IIT Bombay, India · Microsoft, India

Muon Optimizer Spectral Optimization

Open paper AskPDF arXiv

Muon and related optimizers update a weight matrix with a spectrally transformed gradient. In this work, we study such spectral optimizers with a constant step size on strongly convex matrix quadratics, which isolate the effect of the update from the confounds of nonconvexity, noise, and step size schedules. We analyze an update using two quantities: the gain, which is the factor by which it amplifies the gradient along each singular direction, and the length, which is the norm of the update. These two quantities place the popular spectral updates in three regimes with different behavior: (i) with a bounded gain, the iterates behave like preconditioned gradient descent and converge for step sizes set by the largest curvature; (ii) with an unbounded gain but a vanishing length, they still converge, with the admissible step sizes set by the total curvature instead; and (iii) with a non-vanishing length, they do not converge for almost every initialization. Interestingly, the practical variants of Muon fall in all three regimes, depending only on how the update is normalized and scaled and not on the polynomial that approximates the polar factor. Experiments validate these findings.