Nonconvex Stochastic Optimization

Follow this topic Ask about this topic

Momentum

10 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 67

All topics

Non-Euclidean SGD for Structured Optimization: Unified Analysis and Improved Rates
Nov 14, 2025Dmitry Kovalev, Ekaterina BorodichNonconvex Stochastic Optimization Stochastic Optimization Convergence
Yandex Research
Convergence Analysis of the ProbAbilistic Gradient Estimator Algorithm for Weakly Convex Finite-Sum Optimization
Aug 31, 2025Laurent Condat, Peter RichtárikStochastic Optimization Nonconvex Stochastic Optimization
King Abdullah University of Science and Technology (KAUST) Thuwal, Kingdom of Saudi Arabia
Better Convergence Guarantees for Sign-Based Momentum Methods
Jul 16, 2025Wei Jiang, Dingzhi Yu, Sifan Yang +3Sign-Based Optimization Momentum Methods
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · School of Artificial Intelligence, Nanjing University, Nanjing, China
A stochastic smoothing framework for nonconvex-nonconcave minEmax problems with applications to Wasserstein distributionally robust optimization
Feb 24, 2025Wei Liu, Muhammad Khan, Gabriel Mancino-Ball +1Randomized Smoothing Nonconvex Stochastic Optimization
Department of Applied Mathematics, Hong Kong Polytechnic University, Hong Kong, China · Institute for Math & AI, Wuhan (IMAI), Wuhan university, China · Department of Mathematical Sciences Rensselaer Polytechnic Institute, Troy, NY, USA
Preconditioned Inexact Stochastic ADMM for Deep Model
Feb 15, 2025Shenglong Zhou, Ouya Wang, Ziyan Luo +2Stochastic Optimization Deep Learning Optimization
School of Mathematics and Statistics, Beijing Jiaotong University, China · ITP Lab, Department of EEE, Imperial College London, UK · National Mobile Communications Research Laboratory, Southeast University, China
Convergence Rate Analysis of LION
Nov 12, 2024Yiming Dong, Huan Li, Zhouchen LinSign-Based Optimization Deep Learning Optimization
State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University, Beijing, 100871, China · Institute of Robotics and Automatic Information Systems, College of Artificial Intelligence, Nankai University, Tianjin, 300071, China · Pazhou Laboratory (Huangpu), Guangzhou, 510555, China
Accelerated Stochastic Min-Max Optimization Based on Bias-corrected Momentum
Jun 18, 2024Haoyuan Cai, Sulaiman A. Alghunaim, Ali H. SayedMomentum Methods Nonconvex Stochastic Optimization
Ecole Polytechnique Fédérale de Lausanne, Switzerland · Kuwait University, Kuwait

Nov 14, 2025, math.OCJ/K move · Enter open · S save

Non-Euclidean SGD for Structured Optimization: Unified Analysis and Improved Rates

Dmitry Kovalev, Ekaterina Borodich

Yandex Research

Nonconvex Stochastic Optimization Stochastic Optimization Convergence Stochastic Gradient Descent

Open paper AskPDF arXiv

Recently, several instances of non-Euclidean SGD, including SignSGD, Lion, and Muon, have attracted significant interest from the optimization community due to their practical success in training deep neural networks. Consequently, a number of works have attempted to explain this success by developing theoretical convergence analyses. Unfortunately, these results cannot properly justify the superior performance of these methods, as they could not beat the convergence rate of vanilla Euclidean SGD. We resolve this important open problem by developing a new unified convergence analysis under the structured smoothness and gradient noise assumption. In particular, our results indicate that non-Euclidean SGD (i) can exploit the sparsity or low-rank structure of the upper bounds on the Hessian and gradient noise, (ii) can provably benefit from popular algorithmic tools such as extrapolation or momentum variance reduction, and (iii) can match the state-of-the-art convergence rates of adaptive and more complex optimization algorithms such as AdaGrad and Shampoo.

Momentum

Related topics

Latest papers 67