Stochastic Gradient Descent

Also known as SGD

Ask about this topic

Overview

Stochastic Gradient Descent (SGD) updates model parameters using gradients from individual examples or mini-batches, making optimization practical when full-dataset gradients are too costly. Research studies convergence and generalization, and develops variants such as momentum, adaptive learning rates, clipping, variance reduction, and decentralized or federated updates. Analyses span convex and nonconvex objectives, neural-network dynamics, heavy-tailed or dependent noise, and statistical inference from training trajectories. A central tension is that gradient noise can improve exploration and favor flatter solutions, yet can also destabilize training or slow convergence, so its effects depend on batch size, learning-rate schedule, model geometry, and data conditions.

Latest papers

CardsList

225 papers42 papers in the last four weeks, up 100% on the four weeks before. 0.2% of all new papers.

  1. Implicit Bias of (Stochastic) Gradient Descent on Deep Linear Diagonal Networks

    Oct 29, 2026Wiebke Bartolomaeus, Holger RauhutImplicit BiasImplicit Regularization

    Ludwig-Maximilians-University Munich, Department of Mathematics, Germany · Munich Center for Machine Learning (MCML)

  2. Rennala NSGD: Asynchronous Stochastic Optimization Beyond Euclidean Geometry

    Oct 28, 2026Igor Sokolov, Alexander Tyurin, Peter RichtárikStochastic OptimizationStochastic Gradient Descent

    Center of Excellence for Generative AI, King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia. · AXXX, Moscow, Russia. · Applied AI Institute, Moscow, Russia.

  3. B2P-Corr: Batch-to-Population Gradient Estimators for Non-Decomposable Correlation Losses

    Oct 28, 2026Jingquan Yan, Yuwei Miao, Peiran Yu +1Stochastic Gradient Descent

    Department of Computer Science and Engineering University of Texas at Arlington Arlington, TX, USA

  4. Finite-step statistical mechanics of scale-invariant learning: spectral equations of state and curvature transport

    Oct 27, 2026Alexander ItinStatistical Physics of LearningStochastic Gradient Descent

    Radboud University, Nijmegen, The Netherlands · Independent researcher, Hamburg, Germany

  5. The Cost of an Unknown Horizon for the Last Iterate of Clipped SGD under Heavy-Tailed Noise

    Oct 24, 2026Yuxing Peng, Zhiqing Tang, Weijia JiaLast-Iterate ConvergenceConvergence Analysis

    Faculty of Arts and Sciences, Beijing Normal University, Zhuhai 519087, China · Institute of Artificial Intelligence and Future Networks, Faculty of Arts and Sciences, Beijing Normal University, Zhuhai 519087, China · Guangdong Key Lab of AI and Multi-Modal Data Processing, Beijing Normal-Hong Kong Baptist University, Zhuhai 519087, China

  6. Sharpness-aware Orthogonalized Momentum Algorithms

    Oct 23, 2026Geetika, Shivani Singh Chauhan, Gauranshi Gupta +3Deep Learning OptimizationSharpness-Aware Minimization

    Department of Computer Science and Engineering Indraprastha Institute of Information Technology, Delhi, India. · Department of Mathematics Indraprastha Institute of Information Technology, Delhi, India.

  7. Convergence of stochastic gradient descent for learning linear neural networks

    Oct 20, 2026Gabin Maxime Nguegnang, Bubacarr Bah, Holger Rauhut +1Stochastic Optimization ConvergenceStochastic Gradient Descent

  8. Weak-LαL^α Noise, Strong Guarantees: Gradient Clipping under Affine Heavy Tails

    Oct 20, 2026Alyona Nekrasova, Makar Sidorov, Savelii Chezhegov +4Gradient ClippingHeavy-Tailed Stochastic Optimization

    Independent researcher · Yaroslavl State University · Moscow Independent Research Institute of Artificial Intelligence +2

  9. When Momentum Helps Polyak--Ruppert Averaging

    Oct 20, 2026Alexandra Suvorikova, Darina Dvinskikh, Yuriy Dorn +1Stochastic OptimizationStochastic Optimization Convergence

    Independent researcher · HSE University · MSU AI Institute +1

  10. Training with ML-labeled Data via Stochastic First- and Zeroth-Order Optimization

    Oct 19, 2026Maria-Eleni Sfyraki, Jun-Kun WangLearning with Noisy LabelsZeroth-Order Optimization

    University of California San Diego

  11. Near-Optimal Last-Iterate Convergence of SGD in the Smooth Interpolation Regime

    Oct 18, 2026Matan Schliserman, Tomer KorenLast-Iterate ConvergenceStochastic Optimization Convergence

    Blavatnik School of Computer Science and AI, Tel Aviv University · Blavatnik School of Computer Science and AI, Tel Aviv University, and Google Research Tel Aviv

  12. Sampling and Curvature in First-Exit Comparisons for Momentum SGD

    Oct 18, 2026Haiyi LiStochastic Gradient DescentNeural Network Training Dynamics

  13. Understanding Stability of Outer Nesterov Momentum on Stochastic Pseudo-Gradients

    Oct 15, 2026Nishant Sharma, Laura Gomezjurado Gonzalez, Ganesh Talluri +2Neural Network StabilityNeural Network Training Dynamics

    Chandar Research Lab · Mila – Quebec AI Institute · Université de Montréal +4

  14. Pairwise AUC Optimization Needs Corrective Power: A Unified View

    Oct 15, 2026Jia Chen, Zhiyong Yang, Shilong Bao +2U-StatisticsLearning to Rank

    School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing, China · Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China

  15. Personalized Projected Ascent for Federated Stackelberg Optimization

    Oct 15, 2026M Yashwanth, Sharannya Ghosh, Saikiran Bulusu +1Federated LearningStochastic Gradient Descent

    Indian Institute of Science, Bangalore · CISPA, Germany · IIIT-Hyderabad

  16. On the Last-Iterate Convergence of Clipped Gradient Methods

    Oct 14, 2026Zijian LiuLast-Iterate ConvergenceGradient Clipping

    New York University

  17. Stochastic Gradient Descent: A Nonlinear Time Series Perspective

    Oct 14, 2026Jiaqi Li, Zhipeng Lou, Stefan Richter +1Richardson ExtrapolationStochastic Gradient Descent

    Department of Statistics, Rice University, USA · Department of Mathematics, University of California, San Diego, USA · Institute of Applied Mathematics, Heidelberg University, Germany +1

  18. Memory-Efficient Subspace Optimization with Residual Correction: From Momentum SGD to Muon

    Oct 14, 2026Sifan Yang, Hongyi Tao, Dingzhi Yu +5Subspace OptimizationMemory-Efficient Training

    State Key Laboratory of Novel Software Technology, Nanjing University · School of Artificial Intelligence, Nanjing University · Alibaba Token Hub, Alibaba Group +1

  19. Large-Step Last-Iterate Convergence of Random-Reshuffling Stochastic Gradient Descent

    Oct 14, 2026Zijun Chen, Zihan ZhangStochastic OptimizationLast-Iterate Convergence

    Department of Computer Science and Engineering, Hong Kong University of Science and Technology

  20. Generalization Analysis of Biased Stochastic Gradient Methods for Minimax Problems

    Oct 14, 2026Shuang Zeng, Yunwen Lei, Yiming YingMin-Max OptimizationGeneralization Bounds

    Department of Mathematics, The University of Hong Kong, Hong Kong, China · School of Mathematics and Statistics, The University of Sydney, Sydney, Australia

  21. How Inner Optimizers Shape Replica Scaling

    Oct 13, 2026Eugene Belilovsky, Stefan Horoi, Edouard Oyallon +1LLM TrainingDistributed Training

    Mila · Concordia University · Université de Montréal +1

  22. A Rate Transfer Method: Convergence Rates in Stochastic Gradient Descent via Stochastic Calculus

    Oct 13, 2026Neil WaltonControl Lyapunov FunctionsStochastic Optimization Convergence

    Durham University

  23. Last-Iterate Convergence of Clipped SGD with Linear Decay

    Oct 13, 2026Wei Jiang, Haomin Bai, Zechao LiLast-Iterate ConvergenceStochastic Gradient Descent

    School of Computer Science and Engineering, Nanjing University of Science and Technology · National Key Laboratory for Novel Software Technology, Nanjing University

  24. NBCLT: Parameter-Agnostic Norm-Ball Clipped Local Training

    Oct 13, 2026Linxuan Pan, Shucheng Yang, Junchi YangCommunication-Efficient Distributed TrainingDecentralized Learning

  25. From Parallel to Sequential: Small-Batch Noise Restores Simplicity Bias of Muon

    Oct 12, 2026Tom Jacobs, Rebekka BurkholzMuon OptimizerNonconvex Optimization

    CISPA Helmholtz Center for Information Security, Germany

  26. Feature Normalization under Finite Training Budgets

    Oct 12, 2026Johan LarssonStatistical Learning TheoryStochastic Gradient Descent

    Department of Mathematical Sciences, University of Copenhagen, Universitetsparken 5, 2100 Copenhagen, Denmark

  27. Asymptotic Risk of Learning-Rate Schedules for Least-Squares Regression with Discretely Distributed Features

    Oct 12, 2026Noé Colomban, Francis Bach, Fabian SchaippLearning Rate SchedulingStochastic Gradient Descent

    Inria - École Normale Supérieure PSL Research University

  28. A Nonlinear Time-Series Theory for Constant-step SGD with Momentum

    Oct 12, 2026Younghun Lee, Jake Zummo, Wei Biao WuStochastic Gradient DescentStochastic Optimization Convergence

    Department of Statistics, University of Chicago

  29. Probabilistic Multi-Source Data Selection via Learnable Categorical Priors: Variance Reduction and Guaranteed Derandomization

    Oct 10, 2026Shizhan Cai, Qixin Zhang, Siyuan Ma +3Data SelectionBilevel Optimization

    College of Computing and Data Science Nanyang Technological University

  30. Formalizing Forgetting: Theory of Forgetting for Online Continual Learning

    Oct 10, 2026Renjie Cao, Yifei Cheng, Li ShenOnline Continual LearningCatastrophic Forgetting

    Boston College · Sun Yat-sen University