Stochastic Gradient Descent

Also known as SGD

Overview

Stochastic Gradient Descent (SGD) updates model parameters using gradients from individual examples or mini-batches, making optimization practical when full-dataset gradients are too costly. Research studies convergence and generalization, and develops variants such as momentum, adaptive learning rates, clipping, variance reduction, and decentralized or federated updates. Analyses span convex and nonconvex objectives, neural-network dynamics, heavy-tailed or dependent noise, and statistical inference from training trajectories. A central tension is that gradient noise can improve exploration and favor flatter solutions, yet can also destabilize training or slow convergence, so its effects depend on batch size, learning-rate schedule, model geometry, and data conditions.

Latest papers

CardsList

252 papers28 papers in the last four weeks, down 33% on the four weeks before. 0.2% of all new papers.

  1. High-Probability PL-SGD with Markovian Noise: Optimal Mixing and Tail Dependence

    Jun 24, 2026Dhruv Sarkar, Aprameyo Chakrabartty, Vaneet AggarwalStochastic OptimizationOptimization Convergence Analysis

  2. GRAIN: Group Aggregation via Min-Norm Objective

    Jun 22, 2026Nghia Bui, Jiarui Yao, Lijing WangAlgorithmic StabilityNeural Network Training Dynamics

  3. Fisher-Geometric Sharpness and the Implicit Bias of SGD toward Flat Minima

    Jun 18, 2026Md Sakir Ahmed, Kumaresh Sarmah, Hemen DuttaInformation GeometryImplicit Bias

  4. Compute Efficiency and Serial Runtime Tradeoffs for Stochastic Momentum Methods

    Jun 17, 2026Depen Morwani, Alexandru Meterez, Pranav Nair +1Stochastic OptimizationMomentum Methods

  5. Mixed-Precision Communication-Avoiding SGD for Generalized Linear Models on GPUs

    Jun 16, 2026Aditya Devarakonda, Irene Simó Muñoz, Giulia GuidiGPU AccelerationCommunication-Efficient Distributed Training

  6. Beyond a Single Explanation of the Adam--SGD Gap

    Jun 12, 2026Chenxiang Zhang, Rustem Islamov, Enea Monzio Compagnoni +3Deep Learning OptimizationAdaptive Gradient Methods

  7. Clipping Makes Distributed and Federated Asynchronous SGD Robust to Stragglers

    Jun 11, 2026Samuel Erickson, Mikael JohanssonGradient ClippingStochastic Optimization Convergence

  8. Fourier fractal dimension to predict the generalization of deep neural networks

    Jun 6, 2026Joao B. Florindo, Davi Wanderley MisturiniNeural Network GeneralizationNeural Network Training Dynamics

  9. Uniform Stability and Generalization Error of GD and SGD on Fixed-Point Parameters

    Jun 5, 2026Jonghyun Shin, Sejun ParkAlgorithmic StabilityGradient Descent

  10. Generalization in Deep Neural Networks: Minimax Rates for Gradient Methods

    Jun 4, 2026Junyu Zhou, Puyu Wang, Yunwen Lei +2Neural Network GeneralizationGradient Descent Dynamics

  11. Optimal Rates for Generalization of Gradient Descent Methods with Deep Neural Networks

    Jun 4, 2026Junyu Zhou, Puyu Wang, Yunwen Lei +2Neural Network GeneralizationReLU Neural Networks

  12. Local MixVR: Breaking the Communication-Sample Dependence in Distributed Learning

    May 31, 2026Tehila Dahan, Bassel Hamoud, Roie Reshef +2Communication-Efficient Distributed TrainingDistributed Optimization

  13. Accurate Large-sample Uncertainty Quantification using Stochastic Gradient Markov Chain Monte Carlo

    May 29, 2026Yu Wang, Jie Ding, Jonathan H. HugginsMarkov Chain Monte CarloStochastic Gradient Langevin Dynamics

  14. Do Deep Networks Forget Initialization? A Forgetting-Time View of Practical Inductive Bias

    May 27, 2026Mohua Das, Pierfrancesco Beneventano, Shibshankar Dey +2Neural Network GeneralizationNeural Network Initialization

  15. Dynamics of Stochastic Momentum with Sparse Updates in High Dimensions

    May 27, 2026Katie Everett, Elliot PaquetteHigh-Dimensional AsymptoticsMomentum Methods

  16. Stochastic Gradient Descent with Momentum is Algorithmically Stable

    May 27, 2026Yunwen Lei, Zimeng Wang, Xiaoming YuanMomentum MethodsAlgorithmic Stability

  17. Learning Theory of the SVRG: Generalization and Convergence Analysis

    May 27, 2026Yunwen Lei, Zimeng Wang, Xiaoming YuanStochastic Optimization ConvergenceVariance Reduction

  18. Worker Disagreement Reveals Sharp Directions in Local SGD

    May 26, 2026Tolga Dimlioglu, Kristi Topollai, Anna ChoromanskaNeural Network Training DynamicsLoss Landscape Geometry

  19. Can Entry-Wise Clipping Give Spectral Control of Stochastic Gradients?

    May 26, 2026Zitao Song, Cedar Site Bai, Zhe Zhang +2Gradient ClippingHeavy-Tailed Noise

  20. Statistical Inference for Stochastic Gradient Descent Beyond Finite Variance

    May 25, 2026Jose Blanchet, Peter Glynn, Wenhao YangStochastic OptimizationConfidence Region Estimation

  21. From One-Pass SGD to Data Reuse: Mini-Batch Scaling Laws in Sketched Linear Regression

    May 23, 2026Ziyan Chen, Zhongzhu Zhou, Ding-Xuan ZhouLinear RegressionScaling Laws

  22. Why SGD is not Brownian Motion: A New Perspective on Stochastic Dynamics

    May 21, 2026Igor Ignashin, Anna Radovskaya, Andrew Semenov +7Fokker–Planck EquationNeural Network Training Dynamics

  23. Implicit Regularization of Mini-Batch Training in Graph Neural Networks

    May 21, 2026Clement Wang, Antoine Vialle, Robin Vaysse +1Graph Neural NetworksImplicit Regularization

  24. Training Neural Networks with Optimal Double-Bayesian Learning

    May 19, 2026Vy Bui, Hang Yu, Karthik Kantipudi +2Neural Network OptimizationBayesian Decision Theory