Stochastic Gradient Descent

Also known as SGD

Overview

Stochastic Gradient Descent (SGD) updates model parameters using gradients from individual examples or mini-batches, making optimization practical when full-dataset gradients are too costly. Research studies convergence and generalization, and develops variants such as momentum, adaptive learning rates, clipping, variance reduction, and decentralized or federated updates. Analyses span convex and nonconvex objectives, neural-network dynamics, heavy-tailed or dependent noise, and statistical inference from training trajectories. A central tension is that gradient noise can improve exploration and favor flatter solutions, yet can also destabilize training or slow convergence, so its effects depend on batch size, learning-rate schedule, model geometry, and data conditions.

Latest papers

CardsList

252 papers28 papers in the last four weeks, down 33% on the four weeks before. 0.2% of all new papers.

  1. Decomposing SGD Dynamics in Neural Networks: Teacher-Induced Spikes and Variance Inflation

    Nov 18, 2026Samet Demir, Zafer DoganNeural Network GeneralizationNeural Network Training Dynamics

  2. Taming the Tails of Stochastic Gradient Descent in the Interpolation Regime

    Nov 18, 2026Leonardo Galli, Daniela Angela Parletta, Andrea PaudiceStochastic Optimization ConvergenceStochastic Gradient Descent

  3. Nearly Tight Almost Sure Convergence Rates for the Last iterate of SGD

    Nov 18, 2026Weiming Ou, Andrea Paudice, Xiao WangLast-Iterate ConvergenceConvergence Analysis

  4. An inexact stochastic variance-reduced gradient method for constrained convex finite-sum optimization

    Nov 17, 2026Zhaosong Lu, Yifeng XiaoConstrained OptimizationStochastic Optimization

  5. LACCO: Local Accumulation with Computation--Communication Overlap

    Nov 13, 2026Daniele Kopyshevskiy, Benjamin Th{é}rien, Nicolas Pinto +1Communication-Efficient Distributed TrainingDistributed Training

  6. When and How SignSGD Outperforms SGD: A Functional Scaling Law Perspective

    Nov 12, 2026Zilin Wang, Binghui Li, Jia-Nan Wang +3Sign-Based OptimizationStochastic Gradient Descent

  7. Improved last-iterate bounds for SGD with and without replacement

    Nov 10, 2026Yimin Zhong, Jiaxin ZhouLast-Iterate ConvergenceStochastic Optimization Convergence

  8. Settling the Burn-in Gap: Sharp Risk Bounds for Tail-Averaged SGD in Linear Regression

    Nov 10, 2026Qingyue Zhao, Xuheng Li, Quanquan GuStochastic Gradient DescentStatistical Learning Theory

  9. Learning Rate Decay Can Exponentially Accelerate SGD for Global Optimization of Nonconvex Functions

    Nov 9, 2026Guneykan Ozgul, Dylan Herman, Junhyung Lyle Kim +2Nonconvex Stochastic OptimizationGlobal Optimization

  10. Orthogonal Updates for the Win: Towards Accelerated Adaptive Minimax Optimization

    Nov 8, 2026Zhiwei Zhai, Xinyu Wang, Wenjing Yan +2Min-Max OptimizationAdaptive Gradient Methods

  11. Full Fine-Tuning is not the Problem: Why Adam Fails and SGD Succeeds in Few-Shot CLIP Adaptation

    Nov 8, 2026Nairouz Mrabah, Ismail Ben Ayed, Eric GrangerFew-Shot VLM AdaptationAdaptive Gradient Methods

  12. On the Few-Shot Adaptation of Vision-Language Models: A Few Gradient Signs Are All You Need

    Nov 5, 2026Nairouz Mrabah, Ismail Ben Ayed, Mahdi S. Hosseini +1Few-Shot VLM AdaptationVLM Adaptation

  13. Compressed Per-Player Local Methods for Multiplayer Federated Learning: Exact Convergence via Increment Sharing

    Nov 5, 2026Bhavika Balani, TaeHo Yoon, Nicolas LoizouFederated LearningCommunication-Efficient Distributed Training

  14. Last-Iterate Convergence of Stochastic Gradient Methods for Unbounded Problems

    Nov 5, 2026Kevin Kurian Thomas Vaidyan, Ahmet Alacaoglu, Michael P. FriedlanderLast-Iterate ConvergenceStochastic Optimization Convergence

  15. Stepsize modulation for cross entropy from local quadratic optimization

    Nov 3, 2026Bruno Bauwens, Viacheslav Krivodub, Bingyu ZhangLearning Rate SchedulingAdaptive Learning Rate

  16. ZEMA: confidence-calibrated signed descent with only one state

    Nov 3, 2026Ali Dadsetan, Domenic Rosati, Frank RudziczAdaptive Gradient MethodsSign-Based Optimization

  17. Lower Bounds on Inference Complexity for Polyak-Ruppert Averaged Stochastic Gradient Descent

    Oct 31, 2026Denis Belomestny, Alexey Naumov, Sergey SamsonovStochastic Gradient DescentPolyak-Ruppert Averaging

  18. Stochastic learning flow: moment equations for stochastic gradient descent at finite step size

    Oct 30, 2026Christian Schmid, James M MurrayNeural Network Training DynamicsStochastic Gradient Descent

  19. Exact Theory and Optimal Stepsizes for Stochastic Gradient Descent in Smooth Convex Minimization

    Oct 30, 2026Benjamin Grimmer, Qining ShenSmooth Convex OptimizationStochastic Gradient Descent

  20. Variance Reduction for Decentralized Stochastic Optimization with Multiple Local Updates

    Oct 30, 2026Jie Liu, Nicola Bastianello, Wenying Xu +2Decentralized OptimizationVariance Reduction

  21. Where, When, and How Many Bits? Scaling Laws for SGD with Stochastic Rounding

    Oct 29, 2026Zhichao Wang, Michael W. MahoneyLow-Precision TrainingStochastic Rounding

  22. Implicit Bias of (Stochastic) Gradient Descent on Deep Linear Diagonal Networks

    Oct 29, 2026Wiebke Bartolomaeus, Holger RauhutImplicit BiasImplicit Regularization

  23. Rennala NSGD: Asynchronous Stochastic Optimization Beyond Euclidean Geometry

    Oct 28, 2026Igor Sokolov, Alexander Tyurin, Peter RichtárikStochastic OptimizationStochastic Gradient Descent

  24. B2P-Corr: Batch-to-Population Gradient Estimators for Non-Decomposable Correlation Losses

    Oct 28, 2026Jingquan Yan, Yuwei Miao, Peiran Yu +1Stochastic Gradient Descent

  25. Finite-step statistical mechanics of scale-invariant learning: spectral equations of state and curvature transport

    Oct 27, 2026Alexander ItinStatistical Physics of LearningStochastic Gradient Descent

  26. The Cost of an Unknown Horizon for the Last Iterate of Clipped SGD under Heavy-Tailed Noise

    Oct 24, 2026Yuxing Peng, Zhiqing Tang, Weijia JiaLast-Iterate ConvergenceConvergence Analysis

  27. Sharpness-aware Orthogonalized Momentum Algorithms

    Oct 23, 2026Geetika, Shivani Singh Chauhan, Gauranshi Gupta +3Deep Learning OptimizationSharpness-Aware Minimization

  28. Weak-LαL^α Noise, Strong Guarantees: Gradient Clipping under Affine Heavy Tails

    Oct 20, 2026Alyona Nekrasova, Makar Sidorov, Savelii Chezhegov +4Smooth Convex OptimizationGradient Clipping