Stochastic Gradient Descent

Also known as SGD

Overview

Stochastic Gradient Descent (SGD) updates model parameters using gradients from individual examples or mini-batches, making optimization practical when full-dataset gradients are too costly. Research studies convergence and generalization, and develops variants such as momentum, adaptive learning rates, clipping, variance reduction, and decentralized or federated updates. Analyses span convex and nonconvex objectives, neural-network dynamics, heavy-tailed or dependent noise, and statistical inference from training trajectories. A central tension is that gradient noise can improve exploration and favor flatter solutions, yet can also destabilize training or slow convergence, so its effects depend on batch size, learning-rate schedule, model geometry, and data conditions.

Latest papers

CardsList

252 papers28 papers in the last four weeks, down 33% on the four weeks before. 0.2% of all new papers.

  1. Online Covariance Estimation in Averaged SGD: Improved Batch-Mean Rates and Minimax Optimality via Trajectory Regression

    Apr 12, 2026Yijin Ni, Xiaoming HuoPolyak-Ruppert AveragingCovariance Estimation

  2. Shuffling the Data, Stretching the Step-size: Sharper Bias in constant step-size SGD

    Apr 11, 2026Konstantinos Emmanouilidis, Emmanouil-Vasileios Vlatakis-Gkaragkounis, Rene VidalRandom ReshufflingRichardson Extrapolation

  3. Byzantine-Robust Distributed SGD: A Unified Analysis and Tight Error Bounds

    Apr 11, 2026Boyuan Ruan, Xiaoyu Wang, Ya-Feng LiuDistributed OptimizationFault-Tolerant Distributed Training

  4. New Hybrid Fine-Tuning Paradigm for LLMs: Algorithm Design and Convergence Analysis Framework

    Apr 10, 2026Shaocong Ma, Peiran Yu, Heng HuangFine-TuningLLM Fine-Tuning

  5. Last-Iterate Convergence of Randomized Kaczmarz and SGD with Greedy Step Size

    Apr 10, 2026Michał Dereziński, Xiaoyu DongLast-Iterate ConvergenceStochastic Optimization

  6. Stochastic Gradient Descent in the Saddle-to-Saddle Regime of Deep Linear Networks

    Apr 7, 2026Guillaume Corlouer, Avi Semler, Alexander Strang +1Gradient Descent DynamicsStochastic Gradient Descent

  7. Fine-grained Analysis of Stability and Generalization for Stochastic Bilevel Optimization

    Apr 5, 2026Xuelin Zhang, Hong Chen, Bin Gu +2Stochastic Bilevel OptimizationBilevel Optimization

  8. Isomorphic Functionalities between Ant Colony and Ensemble Learning: Part III -- Gradient Descent, Neural Plasticity, and the Emergence of Deep Intelligence

    Apr 3, 2026Ernest Fokoué, Gregory Babbitt, Yuval LeventalEnsemble LearningStochastic Gradient Descent

  9. Convergence of Byzantine-Resilient Gradient Tracking via Probabilistic Edge Dropout

    Apr 1, 2026Amirhossein Dezhboro, Fateme Maleki, Arman Adibi +2Decentralized OptimizationStochastic Gradient Descent

  10. Learning to Shuffle: Block Reshuffling and Reversal Schemes for Stochastic Optimization

    Mar 31, 2026Lam M. Nguyen, Dzung T. Phan, Jayant KalagnanamStochastic Optimization ConvergenceStochastic Gradient Descent

  11. Towards The Implicit Bias on Multiclass Separable Data Under Norm Constraints

    Mar 24, 2026Shengping Xie, Zekun Wu, Quan Chen +1Implicit BiasGradient Descent

  12. Optimal low-rank stochastic gradient estimation for LLM training

    Mar 21, 2026Zehao Li, Tao Ren, Zishi Zhang +2LLM TrainingMemory-Efficient Training

  13. Heavy-Tailed and Long-Range Dependent Noise in Stochastic Approximation: A Finite-Time Analysis

    Mar 20, 2026Siddharth Chandak, Anuj Yadav, Ayfer Ozgur +1Stochastic ApproximationStochastic Optimization Convergence

  14. Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization method

    Mar 19, 2026Steffen Dereich, Thang Do, Arnulf JentzenStochastic OptimizationStochastic Optimization Convergence

  15. High-Probability Bounds for SGD under the Polyak-Lojasiewicz Condition with Markovian Noise

    Mar 15, 2026Avik Kar, Siddharth Chandak, Rahul Singh +3Optimization Convergence AnalysisStochastic Optimization Convergence

  16. On Interpolation Formulas Describing Neural Network Generalization

    Mar 14, 2026Jin Guo, Roy Y. He, Jean-Michel MorelNeural Network GeneralizationNeural Tangent Kernel

  17. Byzantine-Robust Optimization under (L0,L1)(L_0, L_1)-Smoothness

    Mar 12, 2026Arman Bolatov, Samuel Horváth, Martin Takáč +1Distributed OptimizationByzantine-Robust Federated Learning

  18. On-Average Stability of Multipass Preconditioned SGD and Effective Dimension

    Mar 12, 2026Simon Vary, Tyler Farghly, Ilja Kuzborskij +1Stochastic Gradient DescentGeneralization Bounds

  19. On the Learning Dynamics of Two-layer Linear Networks with Label Noise SGD

    Mar 11, 2026Tongcheng Zhang, Zhanpeng Zhou, Mingze Wang +4Two-Layer Neural NetworksNeural Network Training Dynamics

  20. Large Spikes in Stochastic Gradient Descent: A Large-Deviations View

    Mar 10, 2026Benjamin Gess, Daniel HeydeckerLarge Deviation PrinciplesGradient Descent Dynamics

  21. SHANG++: Robust Stochastic Acceleration under Multiplicative Noise

    Mar 10, 2026Yaxin Yu, Long Chen, Minfu FengAccelerated Gradient MethodsStochastic Optimization Convergence

  22. Dynamic Momentum Recalibration in Online Gradient Learning

    Mar 6, 2026Zhipeng Yao, Rui Yu, Guisong Chang +3Deep Learning OptimizationStochastic Gradient Descent

  23. When to restart? Exploring escalating restarts on convergence

    Mar 4, 2026Ayush K. Varshney, Šarūnas Girdzijauskas, Konstantinos Vandikas +1Deep Learning OptimizationLearning Rate Scheduling

  24. Why Adam Can Beat SGD: Second-Moment Normalization Yields Sharper Tails

    Mar 3, 2026Ruinan Jin, Yingbin Liang, Shaofeng ZouStochastic Optimization ConvergenceAdaptive Gradient Methods

  25. Convex and Non-convex Federated Learning with Stale Stochastic Gradients: Diminishing Step Size is All You Need

    Mar 3, 2026Xinran Zheng, Tara Javidi, Behrouz TouriStochastic Optimization ConvergenceStochastic Gradient Descent

  26. Mini-Batch Covariance, Diffusion Limits, and Oracle Complexity in Stochastic Gradient Descent: A Sampling-Design Perspective

    Mar 2, 2026Daniel Zantedeschi, Kumar MuthuramanStochastic Gradient DescentStochastic Optimization Convergence

  27. Scaling Laws of SignSGD in Linear Regression: When Does It Outperform SGD?

    Mar 2, 2026Jihwan Kim, Dogyoon Song, Chulhee YunSign-Based OptimizationRandom Feature Methods