Adaptive Gradient Methods

Follow this topic Ask about this topic

Momentum

13 papers in the last four weeks, up 160% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 113

All topics

Why Muon Outperforms Adam: A Curvature Perspective
Jun 3, 2026Shuche Wang, Fengzhuo Zhang, Jiaxiang Li +2Neural Network Optimization Muon Optimizer
National University of Singapore · Yale University · University of Minnesota
Beyond Gradient Descent: Adam for Analog Ising Machines
Jun 2, 2026Stijn Van Vooren, Guy Van der Sande, Guy VerschaffeltCombinatorial Optimization Adaptive Gradient Methods
Applied Physics research group, Vrije Universiteit Brussel, Pleinlaan 2, 1050 Brussels, Belgium∗
MAdam: Metric-Aware Multi-Objective Adam
Jun 2, 2026Fengbei Liu, Rachit Saluja, Sunwoo Kwak +5Adaptive Gradient Methods Multi-Objective Optimization
Cornell Tech · Delft University of Technology · Weill Cornell Medicine
Stochastic convergence of parallel asynchronous adaptive first-order methods
Jun 1, 2026Serge Gratton, Philippe L. TointStochastic Optimization Nonconvex Stochastic Optimization
Universit´e de Toulouse, INP, IRIT, Toulouse, France · Universit´e de Toulouse, INP, IRIT, Toulouse, France, and NAXYS, University of Namur, Namur, Belgium
Rethinking Bregman Divergences in Kronecker-Factored Optimizers
May 30, 2026Bing Liu, Wenjie Zhou, Chengcheng ZhaoSecond-Order Optimization Adaptive Gradient Methods
College of Control Science and Engineering, Zhejiang University · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences
Convergence of Steepest Descent and Adam under Non-Uniform Smoothness
May 28, 2026Sharan Vaswani, Yifan Sun, Reza BabanezhadOptimization Convergence Analysis Gradient Descent
1Simon Fraser University · 2Stony Brook University · 3Samsung AI, Montreal.
Singularity-aware Optimization via Randomized Geometric Probing: Towards Stable Non-smooth Optimization
May 28, 2026Ruoran Xu, Borong She, Xiaobo Jin +1Nonconvex Stochastic Optimization Stochastic Optimization Convergence
Xi’an Jiaotong-Liverpool University
A Theoretical and Experimental Study of a Novel Adaptive Learning Algorithm
May 28, 2026Sakshi Kumari, Shyam Kumar M, Sushmitha POptimization Convergence Analysis Stochastic Optimization Convergence
#Department of Mathematics Indian Institute of Technology Patna Bihta 801 106 India · *Department of Mechanical Engineering Indian Institute of Technology Kharagpur2026 Kharagpur IndiaMay 28
Can Entry-Wise Clipping Give Spectral Control of Stochastic Gradients?
May 26, 2026Zitao Song, Cedar Site Bai, Zhe Zhang +2Gradient Clipping Heavy-Tailed Noise
Department of Computer Science, Purdue University, West Lafayette, IN, USA. · School of Industrial Engineering, Purdue University, West Lafayette, IN, USA.
PILOT: Policy-Informed Learned Optimization for Adaptive Deep Network Training
May 23, 2026Sattam Altuuaim, Lama Ayash, Muhammad Mubashar +1Deep Learning Optimization Learning to Optimize
King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia · University of Strathclyde, Glasgow, Scotland
Non-normal spectral signatures of instability in neural network training dynamics
May 22, 2026Souvik GhoshDynamical Systems Neural Network Optimization
Department of Physics, National Sun Yat-sen University, Kaohsiung 80424, Taiwan
An Improved Adaptive PID Optimizer with Enhanced Convergence and Stability for Deep Learning
May 21, 2026Saurabh Saini, Kapil Ahuja, Thomas Wick +1Deep Learning Optimization Stochastic Optimization Convergence
Department of Computer Science & Engineering, Indian Institute of Technology Indore, India. · Leibniz Universität Hannover, Institut für Angewandte Mathematik, Germany. · National Remote Sensing Centre, Indian Space Research Organisation, India.
Correcting Stochastic Update Bias in Preconditioned Language Model Optimizers
May 20, 2026Nikhil Nayak, Julia White, Urchade Zaratiana +7Efficient Language Model Training Adaptive Gradient Methods
Fastino Labs
Ada2MS: A Hybrid Optimization Algorithm Based on Exponential Mixing of Elementwise and Global Second-Moment Estimates
May 19, 2026Meng Zhu, Quan Xiao, Weidong MinDeep Learning Optimization Momentum Methods
School of Information Management and Mathematics, Jiangxi University of Finance and Economics, Nanchang 330032, China · School of Mathematics and Computer Science, Nanchang University, Nanchang 330031, China · Institute of Metaverse, Nanchang University, Nanchang 330031, China +1
From SGD to Muon: Adaptive Optimization via Schatten-p Norms
May 19, 2026Thomas Massena, Corentin Friedrich, Mathieu SerrurierNeural Network Optimization Muon Optimizer
IRIT, DTIPG - SNCF, UT3 · IRIT · IRT Saint Exupery
Understanding Dynamics of Adam in Zero-Sum Games: An ODE Approach
May 19, 2026Yi Feng, Weiming Ou, Xiao WangZero-Sum Games Min-Max Optimization
Aarhus University, Aarhus, Denmark · Shanghai University of Finance and Economics, Shanghai, China · MoE Key Laboratory of Interdisciplinary Research of Computation and Economics, Shanghai University of Finance and Economics, Shanghai, China
Distance-Aware Muon: Adaptive Step Scaling for Normalized Optimization
May 18, 2026Yury Demidovich, Abhishek Chakraborty, Grigory Malinovsky +2Muon Optimizer Nonconvex Optimization
King Abdullah University of Science and Technology · Arizona State University
Can Adaptive Gradient Methods Converge under Heavy-Tailed Noise? A Case Study of AdaGrad
May 18, 2026Zijian LiuStochastic Optimization Nonconvex Stochastic Optimization
Stern School of Business, New York University
AMO: Operator-level Adaptive Muon Orthogonalization
May 18, 2026Xinlin Zhuang, Panyi Ouyang, Yichen Li +7Language Model Pretraining Newton-Schulz Iteration
The Chinese University of Hong Kong · Shopee · MBZUAI +3
Revisiting the Adam-SGD Gap in LLM Pre-Training: The Role of Large Effective Learning Rates
May 18, 2026Athanasios Glentis, Dawei Li, Chung-Yiu Yau +1Language Model Pretraining Gradient Clipping
University of Minnesota
Stochastic Non-Smooth Convex Optimization with Unbounded Gradients
May 15, 2026Dmitry KovalevStochastic Optimization Gradient Clipping
Yandex Research
Novel Dynamic Batch-Sensitive Adam Optimiser for Vehicular Accident Injury Severity Prediction
May 14, 2026Daniel Asare Kyei, Alimatu Saadia-Yussiff, Maame G. Asante-Mensah +3Class-Imbalanced Learning Deep Learning Optimization
Department of Computer Science and Information Technology, University of Cape Coast, Cape Coast, Ghana
Adam-SHANG: A Convergent Adam-Type Method for Stochastic Smooth Convex Optimization
May 13, 2026Yaxin Yu, Long Chen, Minfu FengStochastic Optimization Momentum Methods
School of Mathematics Sichuan University Chengdu, China · Department of Mathematics University of California, Irvine Irvine, CA, USA
PowerStep: Memory-Efficient Adaptive Optimization via $\ell_p$ -Norm Steepest Descent
May 11, 2026Yao Lu, Dengdong Fan, Shixun Zhang +1Gradient Descent Memory-Efficient Optimization
Pengcheng Laboratory · Peking University
Refresh-Scaling the Memory of Balanced Adam
May 11, 2026Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre +2Deep Learning Optimization Momentum Methods
Universitat Polit`ecnica de Val`encia, Val`encia, Spain · Universitat Jaume I, Castell´o de la Plana, Spain
Cosine-Gated Adam-Decay: Drop-In Staleness-Aware Outer Optimization for Decoupled DiLoCo
May 9, 2026Vatsal Shah, Jiahao SunCommunication-Efficient Distributed Training Distributed Optimization
SURGE: Surrogate Gradient Adaptation in Binary Neural Networks
May 9, 2026Haoyu Huang, Boyu Liu, Linlin Yang +6Binary Neural Networks Adaptive Gradient Methods
National College for Excellent Engineers, Beihang University, Beijing, China · School of Artificial Intelligence, Beihang University, Beijing, China · State Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing, China +3
AdaPreLoRA: Adafactor Preconditioned Low-Rank Adaptation
May 9, 2026Ziyun Liu, Fengmiao Bian, Jian-Feng CaiLow-Rank Adaptation Adaptive Gradient Methods
Department of Mathematics The Hong Kong University of Science and Technology
AdamFLIP: Adaptive Momentum Feedback Linearization Optimization for Hard Constrained PINN Training
May 8, 2026Binghang Lu, Runyu Zhang, Changhong Mou +2Neural PDE Solvers Constrained Optimization
Purdue University · Massachusetts Institute of Technology · Utah State University +1
PolarAdamW: Disentangling Spectral Control and Schur Gauge-Equivariance in Matrix Optimisation
May 8, 2026Haozhou ZhangMuon Optimizer Matrix Optimization
Department of Mathematics and Statistics University of Idaho

Jun 3, 2026, cs.LGJ/K move · Enter open · S save

Why Muon Outperforms Adam: A Curvature Perspective

Shuche Wang, Fengzhuo Zhang, Jiaxiang Li, Dirk Bergemann+1

National University of Singapore · Yale University · University of Minnesota

Neural Network Optimization Muon Optimizer Adaptive Gradient Methods Loss Landscape Geometry

Open paper AskPDF arXiv

Muon improves training efficiency over Adam in large language-model training by about two times, but the local geometric source of this advantage remains unclear. Our work takes a first step toward demystifying Muon's superiority over Adam from a curvature perspective. First, we apply a second-order Taylor approximation to the training landscape and show that Muon achieves a larger one-step loss decrease than Adam at matched validation loss. The two optimizers have comparable first-order gains, but Muon consistently incurs a smaller second-order curvature penalty. Second, we decompose this curvature penalty into the squared update norm and Normalized Directional Sharpness (NDS). We find that Muon and Adam have comparable update norms, so Muon's smaller curvature penalty is driven by lower NDS, not update scale. Third, we study how training data and model structure shape Muon's NDS advantage. Using Zipf-Probabilistic Context-Free Grammar (PCFG) data with controlled imbalance, we show that data imbalance amplifies Muon's NDS advantage over Adam. A within-/cross-layer decomposition further shows that, in the middle and late stages of training, Muon's lower NDS is mainly sustained by smaller within-layer curvature. Beyond empirical evidence, we analyze stylized quadratic problems with heterogeneous curvature and gradient alignment toward high-curvature modes. We prove that Muon attains a smaller average NDS than GD by balancing update energy across curvature groups; when curvature heterogeneity is sufficiently strong, this also yields lower local quadratic loss after the same number of steps.

Momentum

Related topics

Latest papers 113