Stochastic Optimization Convergence

Follow this topic Ask about this topic

Momentum

21 papers in the last four weeks, against 2 the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 141

All topics

Factor Augmented High-Dimensional SGD
May 19, 2026Shubo Li, Yuefeng Han, Xiufan YuStochastic Optimization Stochastic Optimization Convergence
Department of Statistics The Pennsylvania State University University Park, PA 16802, USA · Department of Applied and Computational Mathematics and Statistics University of Notre Dame Notre Dame, IN 46556, USA
Can Adaptive Gradient Methods Converge under Heavy-Tailed Noise? A Case Study of AdaGrad
May 18, 2026Zijian LiuStochastic Optimization Nonconvex Stochastic Optimization
Stern School of Business, New York University
Perfect Parallelization in Mini-Batch SGD with Classical Momentum Acceleration
May 18, 2026Sachin Garg, Michał DerezińskiMomentum Methods Stochastic Optimization Convergence
University of Michigan
Scale-Invariant Neural Network Optimization: Norm Geometry and Heavy-Tailed Noise
May 18, 2026Jiayu Zhang, Tianyi LinStochastic Optimization Neural Network Optimization
Department of Industrial Engineering and Operations Research Columbia University
Ringmaster LMO: Asynchronous Linear Minimization Oracle Momentum Method
May 18, 2026Abdurakhmon Sadiev, Artavazd Maranjyan, Ivan Ilin +1Momentum Methods Distributed Optimization
KAUST
High-dimensional Limit of SGD for Diagonal Linear Networks
May 16, 2026Begoña García Malaxechebarría, Courtney Paquette, Maryam Fazel +1Stochastic Optimization Convergence Neural Network Training Dynamics
University of Washington · McGill University · Google DeepMind +2
Global Convergence of Sampling-Based Nonconvex Optimization through Diffusion-Style Smoothing
May 15, 2026Zeji Yi, Chaoyi Pan, Guanya Shi +1Optimization Convergence Analysis Black-Box Optimization
Carnegie Mellon University
Stochastic Non-Smooth Convex Optimization with Unbounded Gradients
May 15, 2026Dmitry KovalevStochastic Optimization Gradient Clipping
Yandex Research
Stochastic global optimization of continuous functions via random walks on Grassmannians
May 13, 2026Kartik Gupta, Stephen D. Miller, Pradeep Ravikumar +1Stochastic Optimization Black-Box Optimization
Adam-SHANG: A Convergent Adam-Type Method for Stochastic Smooth Convex Optimization
May 13, 2026Yaxin Yu, Long Chen, Minfu FengStochastic Optimization Momentum Methods
School of Mathematics Sichuan University Chengdu, China · Department of Mathematics University of California, Irvine Irvine, CA, USA
Constrained Stochastic Spectral Preconditioning Converges for Nonconvex Objectives
May 12, 2026Konstantinos Oikonomidis, Jan Quan, Kimon Antonakopoulos +3Nonconvex Stochastic Optimization Muon Optimizer
KU Leuven, Department of Electrical Engineering, STADIUS Center for Dynamical Systems, Signal Processing, and Data Analytics · ´Ecole Polytechnique F´ed´erale de Lausanne (EPFL), Laboratory for Information and Inference Systems (LIONS) · CVN, CentraleSup´elec, Universit´e Paris-Saclay, Inria
Phases of Muon: When Muon Eclipses SignSGD
May 10, 2026Elliot Paquette, Noah Marshall, Lucas Benigni +3Sign-Based Optimization Muon Optimizer
Mathematics and Statistics Department, McGill University · Mathematics and Statistics Department, Université de Montréal · Google DeepMind
Adversary-Robust Learning from Fully Asynchronous Directional Derivative Estimates
May 10, 2026Anik Kumar Paul, Nibedita Roy, Nagesh Talagani +3Stochastic Optimization Sign-Based Optimization
Computer Science and Automation, Indian Institute of Science, Bengaluru · Edwardson School of Industrial Engineering, Purdue University · Department of Computer Science, IMT Atlantique
\mathsf{VISTA}: Decentralized Machine Learning in Adversary Dominated Environments
May 8, 2026Hanzaleh Akbari Nodehi, Parsa Moradi, Soheil Mohajer +1Mechanism Design Byzantine-Robust Federated Learning
University of Minnesota
Robust stochastic first order methods in heavy-tailed noise via medoid mini-batch gradient sampling
May 8, 2026Manojlo Vukovic, Dusan JakoveticStochastic Optimization Nonconvex Stochastic Optimization
Faculty of Technical Sciences, University of Novi Sad, Novi Sad, Serbia · Faculty of Sciences, University of Novi Sad, Novi Sad, Serbia
SGD for Variational Inference: Tackling Unbounded Variance via Preconditioning and Dynamic Batching
May 8, 2026Hippolyte Labarrière, Cesare Molinari, Silvia Villa +1Stochastic Optimization Convergence Variational Inference
MaLGa, DIBRIS Università degli Studi di Genova Genoa, Italy · MaLGa, DIMA Università degli Studi di Genova Genoa, Italy · MaLGa, DIBRIS, IIT Università degli Studi di Genova Genoa, Italy
Locally Near Optimal Piecewise Linear Regression in High Dimensions via Difference of Max-Affine Functions
May 7, 2026Haitham Kanj, Kiryung LeeGradient Descent Sample Complexity
Department of Electrical and Computer Engineering The Ohio State University, Columbus, OH, USA
When and Why SignSGD Outperforms SGD: A Theoretical Study Based on $\ell_1$ -norm Lower Bounds
May 7, 2026Hongyi Tao, Dingzhi Yu, Lijun ZhangSign-Based Optimization Optimization Convergence Analysis
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China · School of Artificial Intelligence, Nanjing University, Nanjing 210023, China
Directional Consistency as a Complementary Optimization Signal: The GONO Framework
May 7, 2026Victor Daniel GeraDeep Learning Optimization Stochastic Optimization Convergence
Department of Artificial Intelligence Anurag University, Hyderabad, India
Accelerating LMO-Based Optimization via Implicit Gradient Transport
May 7, 2026Won-Jun Jang, Si-Hyeon LeeMuon Optimizer Stochastic Optimization Convergence
School of Electrical Engineering · Korea Advanced Institute of Science and Technology (KAIST) · Daejeon, South Korea
On Adaptivity in Zeroth-Order Optimization
May 5, 2026Hassan Dbouk, Nidham Gazagnadou, Matthias Reisser +1Zeroth-Order Optimization LLM Fine-Tuning
Qualcomm AI Research
Robust and Fast Training via Per-Sample Clipping
May 4, 2026Davide Nobile, Philipp GrohsGradient Clipping Nonconvex Stochastic Optimization
Department of Mathematics University of Vienna · RICAM Austrian Academy of Sciences
Bringing Order to Asynchronous SGD: Towards Optimality under Data-Dependent Delays with Momentum
May 3, 2026Tehila Dahan, Roie Reshef, Sharon Goldstein +1Momentum Methods Distributed Optimization
Technion – Israel Institute of Technology, Haifa, Israel
High-Probability Convergence in Decentralized Stochastic Optimization with Gradient Tracking
Apr 30, 2026Aleksandar Armacki, Haoyuan Cai, Ali H. SayedConvergence Analysis Stochastic Optimization Convergence
Ecole Polytechnique Fédérale de Lausanne, Lausanne, Switzerland
Enhancing SignSGD: Small-Batch Convergence Analysis and a Hybrid Switching Strategy
Apr 28, 2026Haoran Chen, Wentao WangSign-Based Optimization Stochastic Optimization Convergence
´Ecole polytechnique Palaiseau, France
From Cursed to Competitive: Closing the ZO-FO Gap via Input-to-State Stability
Apr 28, 2026Amir Ali Farzin, Philipp Braun, Iman ShamesZeroth-Order Optimization Stochastic Optimization Convergence
School of Engineering · Australian National University · Canberra, ACT 2601 +3
Complex Stochastic Gradient Descent and Directional Bias in Reproducing Kernel Hilbert Spaces
Apr 24, 2026Natanael Alpay, Emeric BattagliaKernel Regression Reproducing Kernel Hilbert Spaces
A unified convergence theory for adaptive first-order methods in the nonconvex case, including AdaNorm, full and diagonal AdaGrad, Shampoo and Muo
Apr 19, 2026S. Gratton, Ph. L. TointMomentum Methods Nonconvex Stochastic Optimization
Université de Toulouse, INP, IRIT, Toulouse, France. · Université de Toulouse, INP, IRIT, Toulouse, France, and NAXYS, University of Namur, Namur, Belgium.
Lower Bounds and Proximally Anchored SGD for Non-Convex Minimization Under Unbounded Variance
Apr 17, 2026Arda Fazla, Ege C. Kaya, Antesh Upadhyay +1Stochastic Optimization Nonconvex Stochastic Optimization
School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN 47907, USA.
StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Models
Apr 16, 2026Dingzhi Yu, Rui Pan, Yuxing Liu +1Stochastic Optimization Sign-Based Optimization
University of Illinois Urbana-Champaign

May 19, 2026, stat.MLJ/K move · Enter open · S save

Factor Augmented High-Dimensional SGD

Shubo Li, Yuefeng Han, Xiufan Yu

Department of Statistics The Pennsylvania State University University Park, PA 16802, USA · Department of Applied and Computational Mathematics and Statistics University of Notre Dame Notre Dame, IN 46556, USA

Stochastic Optimization Stochastic Optimization Convergence Online Learning Stochastic Gradient Descent

Open paper AskPDF arXiv

Stochastic gradient descent (SGD) is a fundamental optimization algorithm widely used in modern machine learning. In this paper, we propose Factor-Augmented SGD (FSGD), a new optimization method that leverages latent factor representations in high-dimensional learning tasks. Unlike standard two-stage dimension reduction approaches that rely on offline representation learning and full data storage, a key novelty of FSGD is that it operates purely on streaming data, making it scalable to large-scale and high-dimensional problems. Furthermore, we establish the first theoretical framework that explicitly incorporates latent factor estimation error into the analysis of SGD, and provide moment convergence in

\ell^s

norm under decaying step sizes and mini-batch updates. Our results provide a new foundation for employing SGD reliably and scalably in high-dimensional machine learning systems.

Momentum

Related topics

Latest papers 141