Stochastic Gradient Descent

Also known as SGD

Overview

Stochastic Gradient Descent (SGD) updates model parameters using gradients from individual examples or mini-batches, making optimization practical when full-dataset gradients are too costly. Research studies convergence and generalization, and develops variants such as momentum, adaptive learning rates, clipping, variance reduction, and decentralized or federated updates. Analyses span convex and nonconvex objectives, neural-network dynamics, heavy-tailed or dependent noise, and statistical inference from training trajectories. A central tension is that gradient noise can improve exploration and favor flatter solutions, yet can also destabilize training or slow convergence, so its effects depend on batch size, learning-rate schedule, model geometry, and data conditions.

Latest papers

CardsList

252 papers28 papers in the last four weeks, down 33% on the four weeks before. 0.2% of all new papers.

  1. Uniform-in-time concentration in two-layer neural networks via transportation inequalities

    Mar 2, 2026Arnaud Guillin, Boris Nectoux, Paul StosNeural Network Training DynamicsStochastic Gradient Descent

  2. Neural network optimization strategies and the topography of the loss landscape

    Feb 24, 2026Jianneng Yu, Alexandre V. MorozovNeural Network OptimizationStochastic Gradient Descent

  3. Stability and Generalization of Push-Sum Based Decentralized Optimization over Directed Graphs

    Feb 24, 2026Yifei Liang, Yan Sun, Xiaochun Cao +1Decentralized OptimizationDecentralized Learning

  4. Exponential Convergence of (Stochastic) Gradient Descent for Separable Logistic Regression

    Feb 21, 2026Sacchit Kale, Piyushi Manupriya, Pierre Marion +2Gradient DescentStochastic Optimization Convergence

  5. Interactive Learning of Single-Index Models via Stochastic Gradient Descent

    Feb 19, 2026Nived Rajaraman, Yanjun HanSingle-Index ModelsGeneralized Linear Bandits

  6. Optimizer choice matters for the emergence of Neural Collapse

    Feb 18, 2026Jim Zhao, Tin Sum Cheng, Wojciech Masarczyk +1Neural CollapseNeural Network Training Dynamics

  7. Conjugate Learning Theory: Uncovering the Mechanisms of Trainability and Generalization in Deep Neural Networks

    Feb 18, 2026Binchuan QiNeural Network GeneralizationNeural Network Optimization

  8. Robust Stochastic Gradient Posterior Sampling with Lattice Based Discretisation

    Feb 17, 2026Zier Mensch, Lars Holdijk, Samuel Duffield +4Posterior SamplingMarkov Chain Monte Carlo

  9. Fractional-Order Federated Learning

    Feb 17, 2026Mohammad Partohaghighi, Roummel Marcia, YangQuan ChenFederated LearningFederated Learning Aggregation

  10. On the Stability of Nonlinear Dynamics in GD and SGD: Beyond Quadratic Potentials

    Feb 16, 2026Rotem Mulayoff, Sebastian U. StichGradient Descent DynamicsStochastic Gradient Descent

  11. Constant-Stepsize Stochastic Approximation: Finite-Time Convergence, Gaussian Approximation, and Tail Bounds

    Feb 15, 2026Zedong Wang, Yuyang Wang, Ijay Narang +3Stochastic ApproximationStochastic Gradient Descent

  12. Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decay

    Feb 6, 2026Binghui Li, Zilin Wang, Fengling Chen +3Learning Rate SchedulingKernel Regression

  13. Limitations of SGD for Multi-Index Models Beyond Statistical Queries

    Feb 5, 2026Daniel Barzilai, Ohad ShamirSingle-Index ModelsStatistical Learning Theory

  14. Do We Need Asynchronous SGD? On the Near-Optimality of Synchronous Methods

    Feb 3, 2026Grigory Begunov, Alexander TyurinDistributed OptimizationStochastic Gradient Descent

  15. Adaptive Momentum and Nonlinear Damping for Neural Network Training

    Jan 30, 2026Aikaterini Karoni, Rajit Rajpal, Benedict Leimkuhler +1Momentum MethodsStochastic Optimization Convergence

  16. Understanding and inverse design of implicit bias in stochastic learning: a geometric perspective

    Jan 10, 2026Nicola Aladrah, Emanuele Ballarin, Matteo Biagetti +3Implicit BiasNeural Network Training Dynamics

  17. Can Stationary Distributions of Scale-Invariant Neural Networks Be Described by the Thermodynamics of an Ideal Gas?

    Nov 10, 2025Ildus Sadrtdinov, Ekaterina Lobacheva, Ivan Klimov +3Statistical Physics of LearningNeural Network Training Dynamics

  18. Bilevel Learning via Strategically Inexact Stochastic Gradient Descent

    Nov 10, 2025Mohammad Sadegh Salehi, Subhadip Mukherjee, Lindon Roberts +1Stochastic Bilevel OptimizationHypergradient Optimization

  19. Cross-Epoch Bias in SGD-based Leave-One-Out Influence: Analysis and Accumulative Correction

    Oct 30, 2025Yunxiao Shi, Shuo Yang, Yixin Su +2Influence FunctionsTraining Data Attribution