Stochastic Gradient Descent

Also known as SGD

Overview

Stochastic Gradient Descent (SGD) updates model parameters using gradients from individual examples or mini-batches, making optimization practical when full-dataset gradients are too costly. Research studies convergence and generalization, and develops variants such as momentum, adaptive learning rates, clipping, variance reduction, and decentralized or federated updates. Analyses span convex and nonconvex objectives, neural-network dynamics, heavy-tailed or dependent noise, and statistical inference from training trajectories. A central tension is that gradient noise can improve exploration and favor flatter solutions, yet can also destabilize training or slow convergence, so its effects depend on batch size, learning-rate schedule, model geometry, and data conditions.

Latest papers

CardsList

252 papers28 papers in the last four weeks, down 33% on the four weeks before. 0.2% of all new papers.

  1. Increasing Missingness to Reduce Bias: Richardson-SGD with Missing Data

    May 19, 2026Ferdinand Genans, Erwan ScornetLearning with Missing DataRichardson Extrapolation

  2. MiMuon: Mixed Muon Optimizer with Improved Generalization for Large Models

    May 19, 2026Feihu Huang, Yuning Luo, Songcan ChenNeural Network GeneralizationDeep Learning Optimization

  3. Adynamical systems view of training generativemodels and the memorization phenomenon

    May 19, 2026Siva Athreya, Chiranjib Bhattacharya, Vivek S. BorkarNeural Network MemorizationMemorization in Generative Models

  4. Factor Augmented High-Dimensional SGD

    May 19, 2026Shubo Li, Yuefeng Han, Xiufan YuStochastic OptimizationStochastic Optimization Convergence

  5. High-dimensional Limit of SGD for Diagonal Linear Networks

    May 16, 2026Begoña García Malaxechebarría, Courtney Paquette, Maryam Fazel +1Stochastic Optimization ConvergenceNeural Network Training Dynamics

  6. Beyond Bounded Variance: Variance-Reduced Normalized Methods for Nonconvex Optimization under Blum-Gladyshev Noise

    May 14, 2026Antesh Upadhyay, Arda Fazla, Abolfazl HashemiNonconvex Stochastic OptimizationVariance Reduction

  7. Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence

    May 13, 2026Tien-Phat Nguyen, Truong Nguyen, Minh-Phuc Truong +3Momentum MethodsGradient Descent

  8. Gradient Clipping Beyond Vector Norms: A Spectral Approach for Matrix-Valued Parameters

    May 12, 2026Alexander Yukhimchuk, Mladen Kolar, Martin Takáč +1Stochastic OptimizationGradient Clipping

  9. The Benefits of Temporal Correlations: SGD Learns k-Juntas from Random Walks Efficiently

    May 11, 2026Elisabetta Cornacchia, Dan Mikulincer, Elchanan MosselStatistical Learning TheoryStochastic Gradient Descent

  10. Robust stochastic first order methods in heavy-tailed noise via medoid mini-batch gradient sampling

    May 8, 2026Manojlo Vukovic, Dusan JakoveticStochastic OptimizationNonconvex Stochastic Optimization

  11. SGD for Variational Inference: Tackling Unbounded Variance via Preconditioning and Dynamic Batching

    May 8, 2026Hippolyte Labarrière, Cesare Molinari, Silvia Villa +1Stochastic Optimization ConvergenceVariational Inference

  12. When and Why SignSGD Outperforms SGD: A Theoretical Study Based on ℓ1\ell_1-norm Lower Bounds

    May 7, 2026Hongyi Tao, Dingzhi Yu, Lijun ZhangSign-Based OptimizationOptimization Convergence Analysis

  13. Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimization

    May 5, 2026Sharan Sahu, Abir Sarkar, Cameron J. Hogan +1Stochastic OptimizationStochastic Optimization Convergence

  14. Robust and Fast Training via Per-Sample Clipping

    May 4, 2026Davide Nobile, Philipp GrohsGradient ClippingNonconvex Stochastic Optimization

  15. Bringing Order to Asynchronous SGD: Towards Optimality under Data-Dependent Delays with Momentum

    May 3, 2026Tehila Dahan, Roie Reshef, Sharon Goldstein +1Momentum MethodsDistributed Optimization

  16. Stability and Generalization for Decentralized Markov SGD

    May 3, 2026Jiahuan Wang, Ziqing Wen, Ping Luo +2Algorithmic StabilityDecentralized Optimization

  17. Bridging Graph Drawing and Dimensionality Reduction with Stochastic Stress Optimization

    May 1, 2026Daniel Hangan, Stephen Kobourov, Jacob MillerScientific VisualizationData Visualization

  18. Information-Theoretic Generalization Bounds for Stochastic Gradient Descent with Predictable Virtual Noise

    Apr 30, 2026Mohammad PartohaghighiInformation-Theoretic Generalization BoundsStochastic Gradient Descent

  19. Refining Covariance Matrix Estimation in Stochastic Gradient Descent Through Bias Reduction

    Apr 23, 2026Ziyang Wei, Wanrong Zhu, Jingyang Lyu +1Covariance EstimationStochastic Gradient Descent

  20. SGD at the Edge of Stability: The Stochastic Sharpness Gap

    Apr 22, 2026Fangshuo Liao, Afroditi Kolomvaki, Anastasios KyrillidisEdge of StabilityNeural Network Optimization

  21. On the Stability and Generalization of First-order Bilevel Minimax Optimization

    Apr 22, 2026Xuelin Zhang, Peipei YuanMin-Max OptimizationBilevel Optimization

  22. Optimizing Stochastic Gradient Push under Broadcast Communications

    Apr 16, 2026Tuan Nguyen, Ting HeCommunication-Efficient Distributed TrainingFederated Learning

  23. Momentum Further Constrains Sharpness at the Edge of Stochastic Stability

    Apr 15, 2026Arseniy Andreyev, Advikar Ananthkumar, Marc Walden +2SGD with MomentumStochastic Gradient Descent

  24. Gradient Descent's Last Iterate is Often (slightly) Suboptimal

    Apr 15, 2026Guy Kornowski, Ohad ShamirLast-Iterate ConvergenceGradient Descent

  25. Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluations

    Apr 14, 2026Tong Zhang, Jiangning Zhang, Zhucun Xue +9Deep Learning OptimizationAdaptive Gradient Methods