Stochastic Gradient Descent

Follow this topic Ask about this topic

Recent momentum

×6

18 papers in the last 28 days · 0.3% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-21

6 new papers

A weekly snapshot of new work published in Stochastic Gradient Descent.

Period ending 2026-09-14

5 new papers

A weekly snapshot of new work published in Stochastic Gradient Descent.

Period ending 2026-09-07

3 new papers

A weekly snapshot of new work published in Stochastic Gradient Descent.

175 papers

Latest in Stochastic Gradient Descent

Browse all topics →

Improved Convergence Analysis of Topology Dependence in Decentralized SGD
Jun 8, 2026Yuki Takezawa, Anastasia Koloskova, Sebastian U. StichConvergence Analysis Stochastic Gradient Descent
Fourier fractal dimension to predict the generalization of deep neural networks
Jun 6, 2026Joao B. Florindo, Davi Wanderley MisturiniStochastic Gradient Descent Fourier
Accelerated Decentralized Stochastic Gradient Descent for Strongly Convex Optimization
Jun 5, 2026Ming Sun, Kun YuanStochastic Smooth Convex Optimization Stochastic Gradient Descent
Second-Order Path Kernel Interpolation Formulas in Machine Learning
Jun 5, 2026Jin Guo, Roy Y. He, Jean-Michel MorelStochastic Gradient Descent Interpolation
Beyond Linear and Overcomplete Regimes: A Mean-Field Analysis of Bottleneck Autoencoders
Jun 5, 2026Santanu Das, Ramyak Bilas, Pascal Esser +1Masked Autoencoders Two-Layer Neural Networks
Uniform Stability and Generalization Error of GD and SGD on Fixed-Point Parameters
Jun 5, 2026Jonghyun Shin, Sejun ParkStochastic Gradient Descent Generalization Bounds
Generalization in Deep Neural Networks: Minimax Rates for Gradient Methods
Jun 4, 2026Junyu Zhou, Puyu Wang, Yunwen Lei +2Over-Parameterized Neural Networks Neural Tangent Kernel
Optimal Rates for Generalization of Gradient Descent Methods with Deep Neural Networks
Jun 4, 2026Junyu Zhou, Puyu Wang, Yunwen Lei +2Gradient Descent Stochastic Gradient Descent
Quantifying Uncertainty In Wide Two-Layer Neural Networks: On The Law Of The Limiting Fluctuation Process
Jun 4, 2026Arnaud Descours, Arnaud Guillin, Geoffrey Lacour +3Two-Layer Neural Networks Central Limit Theorem
DP-MacAdam: Differentially Private Mechanism with Adaptive Clipping and Adaptive Momentum
Jun 3, 2026Naima Tasnim, Lalitha Sankar, Oliver KosutDifferential Privacy Gradient Clipping
Deterministic Envelopes for Tamed SGLD: Decoupling Stochastic Gradient Noise and Localizing Taming
Jun 3, 2026Yiwei Zhou, Ziheng ChenStochastic Gradient Descent Langevin Dynamics
DPDL: Towards Differential Privacy Preservation in Decentralized Stochastic Learning on Non-IID Data
Jun 3, 2026Yunsheng Yuan, Xue Xiao, Lina Wang +1Differential Privacy Distributional Learning
Revisiting Privacy Amplification by Subsampling in Selective Release DPSGD
Jun 3, 2026Xiaobo Huang, Fang XieDifferential Privacy Stochastic Gradient Descent
When Do Fewer Coordinates Suffice in DP-SGD?
Jun 3, 2026Huiqi Zhang, Fang XieDifferential Privacy Stochastic Gradient Descent
Bregman meets Lévy: Stochastic mirror descent with heavy-tailed noise in continuous and discrete time
Jun 2, 2026Pierre-Louis Cauvin, Panayotis MertikopoulosMirror Descent Stochastic Gradient Descent
Demystifying Pipeline Parallelism: First Theory for PipeDream
Jun 2, 2026Ivan Ilin, Peter RichtárikParallel Large Models
Beyond $\ell_2$ -norm and $\ell_\infty$ -norm: A Curvature-Inspired $\ell_p$ -Norm Scheme for Deep Neural Networks
Jun 1, 2026Jianhao Xu, Zhuang YangNeural Network Optimization Stochastic Gradient Descent
Adaptive Sharpness-Aware Minimization with a Polyak-type Step size: A Theory-Grounded Scheduler
Jun 1, 2026Dimitris Oikonomou, Nicolas LoizouStochastic Gradient Descent Sharpness-Aware Minimization
What Makes a Strong Model? A Unified Spectral Analysis of Knowledge Transfer over High-dimensional Linear Regression
May 31, 2026Wendao Wu, Fangqing Zhang, Haihan Zhang +1Knowledge Distillation Kernel Ridge Regression
In-Expectation Convergence of Stochastic Gradient Methods under Heavy-Tailed Noise
May 30, 2026Zijian LiuStochastic Smooth Convex Optimization Stochastic Gradient Descent
Accurate Large-sample Uncertainty Quantification using Stochastic Gradient Markov Chain Monte Carlo
May 29, 2026Yu Wang, Jie Ding, Jonathan H. HugginsReliable Uncertainty Quantification Stochastic Gradient Descent
Dynamics of Stochastic Momentum with Sparse Updates in High Dimensions
May 27, 2026Katie Everett, Elliot PaquetteStochastic Gradient Descent Stochastic Dynamics
Stochastic Gradient Descent with Momentum is Algorithmically Stable
May 27, 2026Yunwen Lei, Zimeng Wang, Xiaoming YuanStochastic Gradient Descent Generalization Bounds
Worker Disagreement Reveals Sharp Directions in Local SGD
May 26, 2026Tolga Dimlioglu, Kristi Topollai, Anna ChoromanskaCondition Number Stochastic Gradient Descent
When Muon Optimizer Meets Adversarial Training: A Theoretical and Empirical Study
May 26, 2026Jun Yan, Weiquan Huang, Jiankai Zuo +4Muon Optimizer Adversarial Training
From Privacy to Generalization: Linear Max-Information Bounds for DP-SGD
May 25, 2026Christoph H. Lampert, Hossein ZakeriniaDifferential Privacy Generalization Bounds
Statistical Inference for Stochastic Gradient Descent Beyond Finite Variance
May 25, 2026Jose Blanchet, Peter Glynn, Wenhao YangStochastic Gradient Descent Confidence Intervals
Efficient DP-SGD for LLMs with Randomized Clipping
May 24, 2026Enayat Ullah, Sai Aparna Aketi, Devansh Gupta +2Differential Privacy Gradient Clipping
From One-Pass SGD to Data Reuse: Mini-Batch Scaling Laws in Sketched Linear Regression
May 23, 2026Ziyan Chen, Zhongzhu Zhou, Ding-Xuan ZhouMini-Batch Stochastic Gradient Descent Stochastic Gradient Descent
Why SGD is not Brownian Motion: A New Perspective on Stochastic Dynamics
May 21, 2026Igor Ignashin, Anna Radovskaya, Andrew Semenov +7Stochastic Gradient Descent Stochastic Differential Equations
LOSCAR-SGD: Local SGD with Communication-Computation Overlap and Delay-Corrected Sparse Model Averaging
May 20, 2026Yassine Maziane, Ammar Mahran, Artavazd Maranjyan +1Distributed Optimization Federated Learning
SMA-DP: Spectral Memory-Aware Differential Privacy for Deep Learning
May 19, 2026Mohammad Partohaghighi, Roummel MarciaDifferential Privacy Stochastic Gradient Descent
SDM: A Powerful Tool for Evaluating Model Robustness
May 19, 2026Xinlei Liu, Tao Hu, Jichao Xie +3Adversarial Examples Stochastic Gradient Descent
Training Neural Networks with Optimal Double-Bayesian Learning
May 19, 2026Vy Bui, Hang Yu, Karthik Kantipudi +2Learning Rates Stochastic Gradient Descent
From SGD to Muon: Adaptive Optimization via Schatten-p Norms
May 19, 2026Thomas Massena, Corentin Friedrich, Mathieu SerrurierMuon Optimizer Optimizer Design
Increasing Missingness to Reduce Bias: Richardson-SGD with Missing Data
May 19, 2026Ferdinand Genans, Erwan ScornetMissing Data Imputation
Adynamical systems view of training generativemodels and the memorization phenomenon
May 19, 2026Siva Athreya, Chiranjib Bhattacharya, Vivek S. BorkarStochastic Gradient Descent Memorization
OmniISR: A Unified Framework for Centralized and Federated Learning via Intermediate Supervision and Regularization
May 19, 2026Wei-Bin Kou, Guangxu Zhu, Ming Tang +4Federated Learning Heterogeneous Edge Devices
Factor Augmented High-Dimensional SGD
May 19, 2026Shubo Li, Yuefeng Han, Xiufan YuStochastic Gradient Descent Factor
Provably Data-driven Lagrangian Relaxation for Mixed Integer Linear Programming
May 18, 2026Tung Quoc Le, Anh Tuan Nguyen, Viet Anh NguyenMixed-Integer Programming Semidefinite Programming
Perfect Parallelization in Mini-Batch SGD with Classical Momentum Acceleration
May 18, 2026Sachin Garg, Michał DerezińskiMini-Batch Stochastic Gradient Descent Stochastic Gradient Descent
Scale-Invariant Neural Network Optimization: Norm Geometry and Heavy-Tailed Noise
May 18, 2026Jiayu Zhang, Tianyi LinNeural Network Optimization Stochastic Gradient Descent
Adaptive Stochastic Natural Gradient Method for Safe Optimization on Binary Space
May 18, 2026Kento Uchida, Ryoki Hamano, Masahiro Nomura +1Evolutionary Search Stochastic Gradient Descent
High-dimensional Limit of SGD for Diagonal Linear Networks
May 16, 2026Begoña García Malaxechebarría, Courtney Paquette, Maryam Fazel +1Stochastic Gradient Descent Stochastic Differential Equations
Beyond Bounded Variance: Variance-Reduced Normalized Methods for Nonconvex Optimization under Blum-Gladyshev Noise
May 14, 2026Antesh Upadhyay, Arda Fazla, Abolfazl HashemiStochastic Smooth Convex Optimization Convex Optimization
Rescaled Asynchronous SGD: Optimal Distributed Optimization under Data and System Heterogeneity
May 13, 2026Ammar Mahran, Artavazd Maranjyan, Peter RichtárikStochastic Gradient Descent Distributed Optimization
The Mechanism of Weak-to-Strong Generalization: Feature Elicitation from Latent Knowledge
May 13, 2026Ryoya Awano, Taiji SuzukiFeature Learning Weak Supervision
Population Risk Bounds for Kolmogorov-Arnold Networks Trained by DP-SGD with Correlated Noise
May 12, 2026Puyu Wang, Jan Schuchardt, Nikita Kalinin +4Differential Privacy Stochastic Gradient Descent
DP-LAC: Lightweight Adaptive Clipping for Differentially Private Federated Fine-tuning of Language Models
May 11, 2026Haaris Mehmood, Jie Xu, Karthikeyan Saravanan +2Federated Learning Differential Privacy
The Benefits of Temporal Correlations: SGD Learns k-Juntas from Random Walks Efficiently
May 11, 2026Elisabetta Cornacchia, Dan Mikulincer, Elchanan MosselSpatio-Temporal Correlations Stochastic Gradient Descent
Unveiling High-Probability Generalization in Decentralized SGD
May 11, 2026Jiahuan Wang, Ping Luo, Ziqing Wen +2Stochastic Gradient Descent Generalization Bounds
Deep Learning under Fractional-Order Differential Privacy
May 11, 2026Mohammad Partohaghighi, Roummel MarciaDifferential Privacy Stochastic Gradient Descent
Phases of Muon: When Muon Eclipses SignSGD
May 10, 2026Elliot Paquette, Noah Marshall, Lucas Benigni +3Muon Optimizer Muon
Rennala MVR: Improved Time Complexity for Parallel Stochastic Optimization via Momentum-Based Variance Reduction
May 9, 2026Zhirayr Tovmasyan, Artavazd Maranjyan, Peter RichtárikVariance Reduction Stochastic Gradient Descent
INO-SGD: Addressing Utility Imbalance under Individualized Differential Privacy
May 8, 2026Xiao Tian, Jue Fan, Rachael Hwee Ling Sim +1Differential Privacy Class Imbalance
Characterizing and Correcting Effective Target Shift in Online Learning
May 8, 2026Ziyan Li, Naoki HirataniLearning-Augmented Algorithms Online Learning
Robust stochastic first order methods in heavy-tailed noise via medoid mini-batch gradient sampling
May 8, 2026Manojlo Vukovic, Dusan JakoveticStochastic Gradient Descent Zeroth-Order Optimization
SGD for Variational Inference: Tackling Unbounded Variance via Preconditioning and Dynamic Batching
May 8, 2026Hippolyte Labarrière, Cesare Molinari, Silvia Villa +1Variational Inference Mini-Batch Stochastic Gradient Descent
Decentralized Time-Varying Optimization for Streaming Data via Temporal Weighting
May 7, 2026Muhammad Faraz Ul Abrar, Nicolò Michelusi, Erik G. LarssonStochastic Gradient Descent Time-Varying Systems
Locally Near Optimal Piecewise Linear Regression in High Dimensions via Difference of Max-Affine Functions
May 7, 2026Haitham Kanj, Kiryung LeeProximal Gradient Descent Stochastic Gradient Descent

Jun 8, 2026 · cs.LGJ/K move · Enter open · S save

Improved Convergence Analysis of Topology Dependence in Decentralized SGD

Yuki Takezawa, Anastasia Koloskova, Sebastian U. Stich

Decentralized SGD is a fundamental algorithm in decentralized learning, although the influence of an underlying network topology on its convergence behavior is not yet fully understood. Existing convergence analyses have shown that topologies with a small spectral gap significantly deteriorate the convergence rate of Decentralized SGD in both homogeneous and heterogeneous cases. However, many prior papers have reported that indeed the choice of the topology has a significant experimental impact in the heterogeneous case, but has little experimental impact on training behavior in the homogeneous case. In this paper, we present a tighter convergence analysis of Decentralized SGD, offering a more precise understanding of how topologies affect the convergence rate than the prior analysis. Specifically, unlike existing convergence analyses that used only the spectral gap as a property of the topology, our novel analysis shows that all eigenvalues of the mixing matrix affect the convergence rate. Throughout the experiments, we carefully evaluated the convergence behavior of Decentralized SGD and demonstrated that our novel convergence analysis can more accurately describe the effect of topology on the convergence rate.

Convergence Analysis Stochastic Gradient Descent Convergence Decentralization Topological Data Analysis

Open paper Ask about it PDF

Recent momentum

Neighbouring directions

Recent digests

6 new papers

5 new papers

3 new papers

Latest in Stochastic Gradient Descent