Stochastic Gradient Descent

Also known as SGD

Overview

Stochastic Gradient Descent (SGD) updates model parameters using gradients from individual examples or mini-batches, making optimization practical when full-dataset gradients are too costly. Research studies convergence and generalization, and develops variants such as momentum, adaptive learning rates, clipping, variance reduction, and decentralized or federated updates. Analyses span convex and nonconvex objectives, neural-network dynamics, heavy-tailed or dependent noise, and statistical inference from training trajectories. A central tension is that gradient noise can improve exploration and favor flatter solutions, yet can also destabilize training or slow convergence, so its effects depend on batch size, learning-rate schedule, model geometry, and data conditions.

Latest papers

CardsList

252 papers28 papers in the last four weeks, down 33% on the four weeks before. 0.2% of all new papers.

  1. Adaptive Bregman Proximal Stochastic Gradient with a Stabilized Barzilai--Borwein Step Size

    Aug 12, 2026Chenhan Jin, Shengze Xu, Binghui Xie +4Nonconvex Stochastic OptimizationAdaptive Gradient Methods

  2. Distribution of hitting times for dissipative random dynamical systems on Rd\mathbb{R}^d, with application to stochastic gradient descent

    Aug 11, 2026Stéphane Galatolo, Stéphane ChrétienStochastic Optimization ConvergenceStochastic Gradient Descent

  3. Generalized Convexity and Smoothness via Conjugate Duality, with Applications to Deep Network Training

    Aug 10, 2026Binchuan QiGradient DescentNeural Network Optimization

  4. Stochastic gradient descent with discontinuity across a manifold

    Aug 7, 2026Vivek S. BorkarStochastic ApproximationGradient Descent Dynamics

  5. Stochastic Saddle Avoidance Beyond Unit Excitation and Smoothness: A Pathwise Lyapunov-Perron Framework

    Aug 4, 2026Junwen Qiu, Bohao Ma, Andre Milzarek +1Stochastic OptimizationNonconvex Stochastic Optimization

  6. Scale Weight Decay and Train Better

    Jul 26, 2026Anuj ApteWeight DecayNeural Network Optimization

  7. Mini-batch Noise Lowers Sharpness via Dominant-Subspace Fluctuations

    Jul 25, 2026Junho So, Dongwook ShinNeural Network Training DynamicsGradient Descent Dynamics

  8. RELTA-SGLD: Relative-Growth Localized Taming for Nonconvex Stochastic-Gradient Langevin Learning

    Jul 21, 2026Yiwei Zhou, Ziheng ChenStochastic Gradient Langevin DynamicsLangevin Dynamics

  9. Scaling Limits of Constant-Stepsize SGD at Flat Minima

    Jul 17, 2026Jingyi Zhang, Cheng Mao, Debankur MukherjeeStochastic Optimization ConvergenceFlat Minima

  10. What's in a Smoothness Constant? Tighter Rates for Local SGD with Bounded Second-order Heterogeneity

    Jul 16, 2026Kumar Kshitij Patel, Rustem Islamov, Sebastian U Stich +3Optimization Convergence AnalysisStochastic Optimization Convergence

  11. Same Loss, Same Noise, Opposite Schedules: Noise Structure and Optimizer Normalization Jointly Determine Whether Learning-Rate Cooldown Helps

    Jul 14, 2026Subham Singh, Ashutosh Mishra, Subha RautLearning Rate SchedulingGradient Descent

  12. Understanding Schedule-Free Methods in Nonconvex Optimization: Rate Guarantees and Escaping Saddles

    Jul 10, 2026Jiseok Chae, Donghwan KimLearning Rate SchedulingOptimization Convergence Analysis

  13. Implicit Bias of SGD in Multivariate ReLU Networks: Effective Width Collapse

    Jul 3, 2026Shuang Liang, Tom Jacobs, Guido MontúfarShallow Neural NetworksReLU Neural Networks

  14. The Multiscale Single-Index Model: A Stylized Model for Hierarchical Feature Learning

    Jul 3, 2026Joan BrunaSingle-Index ModelsHierarchical Representation Learning

  15. Revisiting the Volume Hypothesis

    Jun 30, 2026Ari Pakman, Lior Kreimer, Yakir BerchenkoNeural Network GeneralizationStochastic Gradient Descent

  16. Convergence of Large-Stepsize SGD for the Cross-entropy Loss via Stochastic Lyapunov Stability

    Jun 29, 2026Konstantinos Emmanouilidis, Lachlan MacDonald, Salma Tarmoun +1Edge of StabilityStochastic Optimization Convergence

  17. Curvature-Weighted Gradient Diversity: A Noise Measure for Geometry-Adaptive SGD Schedules

    Jun 29, 2026Muhammad Hamza, Ayush GoelLearning Rate SchedulingStochastic Optimization Convergence

  18. SGD Provably Prioritizes a Shortcut Spurious Feature in the XOR Model

    Jun 29, 2026Tyler LaBonte, Vidya MuthukumarRepresentation LearningReLU Neural Networks

  19. Characterizing Optimizer-Dependent Training Dynamics Through Hessian Eigenvector Displacement and Localization

    Jun 29, 2026Marcelina Marjankowska, Valerio Modugno, Paolo BaruccaSpectral MethodsAdaptive Gradient Methods

  20. How AI settled the complexity of the oldest SGD algorithm

    Jun 28, 2026Michał Dereziński, Xiaoyu DongOptimization Convergence AnalysisStochastic Gradient Descent

  21. Spectral phase transitions and trainability in neural network learning dynamics

    Jun 26, 2026Chanju Park, Dario Bocchi, Francesco D'Amico +2Random Matrix TheoryNeural Network Training Dynamics

  22. Stochastic Gradient Optimization with Model-Assisted Sampling

    Jun 25, 2026Jonne Pohjankukka, Jukka HeikkonenVariance ReductionStochastic Gradient Descent