Nonconvex Stochastic Optimization

Follow this topic Ask about this topic

Momentum

10 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 67

All topics

Demystifying Pipeline Parallelism: First Theory for PipeDream
Jun 2, 2026Ivan Ilin, Peter RichtárikNonconvex Stochastic Optimization Distributed Training
KAUST
Beyond $\ell_2$ -norm and $\ell_\infty$ -norm: A Curvature-Inspired $\ell_p$ -Norm Scheme for Deep Neural Networks
Jun 1, 2026Jianhao Xu, Zhuang YangNeural Network Optimization Nonconvex Stochastic Optimization
School of Computer Science and Technology, Soochow University, Suzhou, 215006, China
Stochastic convergence of parallel asynchronous adaptive first-order methods
Jun 1, 2026Serge Gratton, Philippe L. TointStochastic Optimization Nonconvex Stochastic Optimization
Universit´e de Toulouse, INP, IRIT, Toulouse, France · Universit´e de Toulouse, INP, IRIT, Toulouse, France, and NAXYS, University of Namur, Namur, Belgium
In-Expectation Convergence of Stochastic Gradient Methods under Heavy-Tailed Noise
May 30, 2026Zijian LiuStochastic Optimization Nonconvex Stochastic Optimization
Stern School of Business, New York University
Softsign: Smooth Sign in Your Optimizer For Better Parameter Heterogeneity Handling
May 29, 2026Dmitrii Feoktistov, Timofey Belinsky, Andrey Veprikov +2Stochastic Optimization Sign-Based Optimization
HSE University, Moscow, Russia · BRAIn Lab, Moscow, Russia · Yandex Research, Moscow, Russia +3
MoSSP: A Momentum-Based Single-Loop Stochastic Penalty Method for Nonconvex Constrained DC-Regularized Optimization
May 28, 2026Luxuan Li, Chunfeng Cui, Xiao WangStochastic Optimization Nonconvex Stochastic Optimization
School of Mathematical Sciences, Beihang University, Beijing 100191, China · School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou 510006, China.
Singularity-aware Optimization via Randomized Geometric Probing: Towards Stable Non-smooth Optimization
May 28, 2026Ruoran Xu, Borong She, Xiaobo Jin +1Nonconvex Stochastic Optimization Stochastic Optimization Convergence
Xi’an Jiaotong-Liverpool University
Can Adaptive Gradient Methods Converge under Heavy-Tailed Noise? A Case Study of AdaGrad
May 18, 2026Zijian LiuStochastic Optimization Nonconvex Stochastic Optimization
Stern School of Business, New York University
Scale-Invariant Neural Network Optimization: Norm Geometry and Heavy-Tailed Noise
May 18, 2026Jiayu Zhang, Tianyi LinStochastic Optimization Neural Network Optimization
Department of Industrial Engineering and Operations Research Columbia University
Ringmaster LMO: Asynchronous Linear Minimization Oracle Momentum Method
May 18, 2026Abdurakhmon Sadiev, Artavazd Maranjyan, Ivan Ilin +1Momentum Methods Distributed Optimization
KAUST
Accelerated Gradient Descent for Faster Convergence with Minimal Overhead
May 15, 2026Manuel Graca, L. Miguel Silveira, Arlindo Oliveira +1Deep Learning Optimization Gradient Descent
IST, ULisboa Lisbon, Portugal · Old Dominion University Norfolk, VA USA
Stochastic Compositional Optimization via Hybrid Momentum Frank--Wolfe
May 14, 2026El Mahdi ChaytiStochastic Optimization Momentum Methods
Machine Learning and Optimization Laboratory (MLO) EPFL, Switzerland
Beyond Bounded Variance: Variance-Reduced Normalized Methods for Nonconvex Optimization under Blum-Gladyshev Noise
May 14, 2026Antesh Upadhyay, Arda Fazla, Abolfazl HashemiNonconvex Stochastic Optimization Variance Reduction
School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN 47907, USA.
Rescaled Asynchronous SGD: Optimal Distributed Optimization under Data and System Heterogeneity
May 13, 2026Ammar Mahran, Artavazd Maranjyan, Peter RichtárikDistributed Optimization Nonconvex Stochastic Optimization
KAUST
Population Risk Bounds for Kolmogorov-Arnold Networks Trained by DP-SGD with Correlated Noise
May 12, 2026Puyu Wang, Jan Schuchardt, Nikita Kalinin +4Differentially Private Stochastic Gradient Descent Nonconvex Stochastic Optimization
RPTU Kaiserslautern-Landau, Kaiserslautern, Germany · Machine Learning Research, Morgan Stanley · Institute of Science and Technology, Klosterneuburg, Austria +1
Constrained Stochastic Spectral Preconditioning Converges for Nonconvex Objectives
May 12, 2026Konstantinos Oikonomidis, Jan Quan, Kimon Antonakopoulos +3Nonconvex Stochastic Optimization Muon Optimizer
KU Leuven, Department of Electrical Engineering, STADIUS Center for Dynamical Systems, Signal Processing, and Data Analytics · ´Ecole Polytechnique F´ed´erale de Lausanne (EPFL), Laboratory for Information and Inference Systems (LIONS) · CVN, CentraleSup´elec, Universit´e Paris-Saclay, Inria
Gradient Clipping Beyond Vector Norms: A Spectral Approach for Matrix-Valued Parameters
May 12, 2026Alexander Yukhimchuk, Mladen Kolar, Martin Takáč +1Stochastic Optimization Gradient Clipping
MBZUAI · University of Southern California
Rennala MVR: Improved Time Complexity for Parallel Stochastic Optimization via Momentum-Based Variance Reduction
May 9, 2026Zhirayr Tovmasyan, Artavazd Maranjyan, Peter RichtárikStochastic Optimization Distributed Optimization
KAUST
Robust stochastic first order methods in heavy-tailed noise via medoid mini-batch gradient sampling
May 8, 2026Manojlo Vukovic, Dusan JakoveticStochastic Optimization Nonconvex Stochastic Optimization
Faculty of Technical Sciences, University of Novi Sad, Novi Sad, Serbia · Faculty of Sciences, University of Novi Sad, Novi Sad, Serbia
Sample Complexity of Stochastic Optimization with Integer Variables
May 8, 2026Hongyu Cheng, Yinghao Zheng, Marco Molinaro +1Stochastic Optimization Sample Complexity
Dept. of Applied Mathematics & Statistics Johns Hopkins University Baltimore, MD 21218 · Microsoft Research (Redmond) · Dept. of Computer Science, PUC-Rio
Muon with Nesterov Momentum: Heavy-Tailed Noise and (Randomized) Inexact Polar Decomposition
May 7, 2026Sayantan Choudhury, Xiaoran Cheng, Martin Takáč +2Stochastic Optimization Momentum Methods
MBZUAI · Penn State University · Georgia Institute of Technology +1
From Dual Tracking to Clipping: Provably Faster Distributionally Robust Multi-Objective Optimization
May 7, 2026Yufeng Yang, Fangning Zhuo, Ziyi Chen +2Gradient Clipping Gradient Descent
Texas A&M University · A*STAR · University of Maryland
Robust and Fast Training via Per-Sample Clipping
May 4, 2026Davide Nobile, Philipp GrohsGradient Clipping Nonconvex Stochastic Optimization
Department of Mathematics University of Vienna · RICAM Austrian Academy of Sciences
Anon: Extrapolating Adaptivity Beyond SGD and Adam
May 4, 2026Yiheng Zhang, Kaiyan Zhao, Shaowu Wu +5Nonconvex Stochastic Optimization Adaptive Gradient Methods
Wuhan University · University of Macau · University of Calgary
Bringing Order to Asynchronous SGD: Towards Optimality under Data-Dependent Delays with Momentum
May 3, 2026Tehila Dahan, Roie Reshef, Sharon Goldstein +1Momentum Methods Distributed Optimization
Technion – Israel Institute of Technology, Haifa, Israel
A unified convergence theory for adaptive first-order methods in the nonconvex case, including AdaNorm, full and diagonal AdaGrad, Shampoo and Muo
Apr 19, 2026S. Gratton, Ph. L. TointMomentum Methods Nonconvex Stochastic Optimization
Université de Toulouse, INP, IRIT, Toulouse, France. · Université de Toulouse, INP, IRIT, Toulouse, France, and NAXYS, University of Namur, Namur, Belgium.
Towards Fully Parameter-Free Stochastic Optimization: Grid Search with Self-Bounding Analysis
Apr 18, 2026Yuheng Zhao, Yu-Hu Yan, Amit Attia +3Stochastic Optimization Nonconvex Stochastic Optimization
National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · Tel Aviv University
Lower Bounds and Proximally Anchored SGD for Non-Convex Minimization Under Unbounded Variance
Apr 17, 2026Arda Fazla, Ege C. Kaya, Antesh Upadhyay +1Stochastic Optimization Nonconvex Stochastic Optimization
School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN 47907, USA.
StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Models
Apr 16, 2026Dingzhi Yu, Rui Pan, Yuxing Liu +1Stochastic Optimization Sign-Based Optimization
University of Illinois Urbana-Champaign
CLion: Efficient Cautious Lion Optimizer with Enhanced Generalization
Apr 16, 2026Feihu Huang, Guanyi Zhang, Songcan ChenStochastic Optimization Sign-Based Optimization
College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, China

Jun 2, 2026, cs.LGJ/K move · Enter open · S save

Demystifying Pipeline Parallelism: First Theory for PipeDream

Ivan Ilin, Peter Richtárik

KAUST

Nonconvex Stochastic Optimization Distributed Training Pipeline Parallelism

Open paper AskPDF arXiv

Training modern machine learning models increasingly requires computation to be distributed across many accelerators. Data parallelism remains the default choice and is often paired with tensor-parallel sharding, but model parallelism becomes unavoidable once parameters, activations, or optimizer states no longer fit on a single device. This paper studies pipeline model parallelism through the lens of PipeDream (PD) (Harlap et al., 2018). Our first contribution is theoretical: we introduce Randomized PipeDream (RPD), a stale block-SGD abstraction that yields, to our knowledge, the first clean nonconvex convergence guarantee for a PD-style method. Our second contribution is a scaling diagnosis: we prove that the delay induced by steady-state PD grows as

S^2 - S/2 + O(1)

for

S

stages, so the stale-read contribution in the convergence theorem scales as

Θ(γ^2 S^4)

, equivalently as

Θ(S^4/K)

in the tuned-rate form. Our third contribution is a comparison with LocalSGD, whose periodic model averaging trades weight staleness for synchronization bubbles. In our reported simulated-time experiments, the better-performing method depends on the objective: PD performs better on the quadratic objective and on a small language-modeling training-loss task, while for logistic regression LocalSGD becomes superior as the number of stages increases.

Momentum

Related topics

Latest papers 67