Stochastic Gradient Descent

Follow this field Ask about this field

Recent momentum

emerging

0 papers in the last 28 days · 0.0% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this field, kept on the site without email delivery.

Period ending 2026-09-21

21 new papers

A weekly snapshot of new work published in Stochastic Gradient Descent.

Period ending 2026-09-14

17 new papers

A weekly snapshot of new work published in Stochastic Gradient Descent.

Period ending 2026-09-07

12 new papers

A weekly snapshot of new work published in Stochastic Gradient Descent.

Inside this field

Focused directions

Focused topic

Stochastic Gradient Descent

Within Stochastic Gradient Descent

175 papersExplore

Focused topic

Gradient Descent

Within Stochastic Gradient Descent

155 papersExplore

Focused topic

Gradient Flow

Within Stochastic Gradient Descent

78 papersExplore

Focused topic

Gradient-Based Optimization

Within Stochastic Gradient Descent

58 papersExplore

Focused topic

Wasserstein Gradient Flows

Within Stochastic Gradient Descent

55 papersExplore

Focused topic

Gradient Concentration

Within Stochastic Gradient Descent

48 papersExplore

Focused topic

Mini-Batch Stochastic Gradient Descent

Within Stochastic Gradient Descent

24 papersExplore

541 papers

Latest in Stochastic Gradient Descent

Browse all topics →

Gradient descent at the Edge of Stability: free energy model and kinetic description of the two-layer network
Jun 3, 2026Antonin Chodron de CourcelGradient Descent Two-Layer Neural Networks
Need We Teach Foundation Models What is a Generative Image? Gradient-Free Generative Artifact Detection via Analytic Spectral Adaptation
Jun 3, 2026Qiaoyu Chen, Bing ZhangFace Forgery Detection Vision Foundation Models
Deterministic Envelopes for Tamed SGLD: Decoupling Stochastic Gradient Noise and Localizing Taming
Jun 3, 2026Yiwei Zhou, Ziheng ChenStochastic Gradient Descent Langevin Dynamics
When Both Layers Learn: Training Dynamics of Representing Linear Models via ReLU Networks
Jun 3, 2026Berk Tinaz, Changzhi Xie, Mahdi SoltanolkotabiRectified Linear Unit Networks Gradient Descent
An Ensembled Latent Factor Model via Differential Evolution and Gradient Descent Optimization
Jun 3, 2026Rui Zhang, Jinhang Liu, Wenbo ZhangHigh-Dimensional Data Gradient Descent
DPDL: Towards Differential Privacy Preservation in Decentralized Stochastic Learning on Non-IID Data
Jun 3, 2026Yunsheng Yuan, Xue Xiao, Lina Wang +1Differential Privacy Distributional Learning
Revisiting Privacy Amplification by Subsampling in Selective Release DPSGD
Jun 3, 2026Xiaobo Huang, Fang XieDifferential Privacy Stochastic Gradient Descent
When Do Fewer Coordinates Suffice in DP-SGD?
Jun 3, 2026Huiqi Zhang, Fang XieDifferential Privacy Stochastic Gradient Descent
Quadratic integrate-and-fire neurons exhibit less fragmented loss landscapes and outperform leaky integrate-and-fire neurons in spike-based gradient descent
Jun 2, 2026Carlo Wenig, Raoul-Martin Memmesheimer, Christian KlosSpiking Neural Networks Loss Landscape
Beyond Gradient Descent: Adam for Analog Ising Machines
Jun 2, 2026Stijn Van Vooren, Guy Van der Sande, Guy VerschaffeltTwo-Dimensional Ising Model Adam
Online Learning with Gradient-Variation Interval Regret
Jun 2, 2026Yan-Feng Xie, Shuche Wang, Peng Zhao +1Learning-Augmented Algorithms Linear Regret
Bregman meets Lévy: Stochastic mirror descent with heavy-tailed noise in continuous and discrete time
Jun 2, 2026Pierre-Louis Cauvin, Panayotis MertikopoulosMirror Descent Stochastic Gradient Descent
Demystifying Pipeline Parallelism: First Theory for PipeDream
Jun 2, 2026Ivan Ilin, Peter RichtárikParallel Large Models
ARBOR: Online Process Rewards via a Reusable Rubric Buffer for Search Agents
Jun 2, 2026Zheng Liu, Longxiang Zhang, Xintong Wang +8Reliable Medical Process-Level Reward Model Evaluation Framework Rubric-Based Reinforcement Learning
Neural Networks Provably Learn Spectral Representations for Group Composition
Jun 2, 2026Jianliang He, Leda Wang, Fengzhuo Zhang +2Implicit Neural Representations Two-Layer Neural Networks
Scalable Derivative Gaussian Processes via Exact Gradient Reduction
Jun 1, 2026Hyunseok Seung, Matthias KatzfussGaussian Process Gradient Descent
Pseudospectral Bounds for Transient Amplification in Coupled Gradient Descent
Jun 1, 2026Ahanaf Hasan AriqJacobian Gradient Descent
Beyond $\ell_2$ -norm and $\ell_\infty$ -norm: A Curvature-Inspired $\ell_p$ -Norm Scheme for Deep Neural Networks
Jun 1, 2026Jianhao Xu, Zhuang YangNeural Network Optimization Stochastic Gradient Descent
Adaptive Sharpness-Aware Minimization with a Polyak-type Step size: A Theory-Grounded Scheduler
Jun 1, 2026Dimitris Oikonomou, Nicolas LoizouStochastic Gradient Descent Sharpness-Aware Minimization
Fast Generalization after Interpolation via Critically Damped Momentum Optimization
Jun 1, 2026Luca Muscarnera, Silas Ruhrberg Estévez, Yuanzhang Xiao +1Gradient Descent Interpolation
What Makes a Strong Model? A Unified Spectral Analysis of Knowledge Transfer over High-dimensional Linear Regression
May 31, 2026Wendao Wu, Fangqing Zhang, Haihan Zhang +1Knowledge Distillation Kernel Ridge Regression
DAGGER: Gradient-Free Construction of Transiently Amplifying Networks under Hard Connectivity Constraints
May 31, 2026James C. FergusonSparsity Gradient-Based Optimization
Local MixVR: Breaking the Communication-Sample Dependence in Distributed Learning
May 31, 2026Tehila Dahan, Bassel Hamoud, Roie Reshef +2Distributional Learning Mini-Batch Stochastic Gradient Descent
OPD+: Rethinking the Advantage Design for On-Policy Distillation
May 31, 2026Hanyang Zhao, Haoxian Chen, Han Lin +3On-Policy Distillation Rp-Opsd
RADE: Random Add-Drop Edge as a Regularizer
May 30, 2026Danial Saber, Amirali Salehi-AbariRegularization Techniques Mini-Batch Stochastic Gradient Descent
Looped Transformers with Layer Normalization Provably Learn the Power Method
May 30, 2026Lyumin Wu, Chenyang Zhang, Yuan CaoTransformer Layers Transformer Architectures
In-Expectation Convergence of Stochastic Gradient Methods under Heavy-Tailed Noise
May 30, 2026Zijian LiuStochastic Smooth Convex Optimization Stochastic Gradient Descent
GLENS: Global Search via Learning from Solver Iterates with Diffusion Models
May 29, 2026Anjian Li, Bartolomeo Stellato, Ryne BeesonParallel Continuous Local Search Gradient-Based Optimization
Balancing Learning Rates Across Layers: Exact Two-Step Dynamics and Optimal Scaling in Linear Neural Networks
May 29, 2026Tianyu Pang, Vignesh Kothapalli, Shenyang Deng +3Learning Rates Two-Layer Neural Networks
Accurate Large-sample Uncertainty Quantification using Stochastic Gradient Markov Chain Monte Carlo
May 29, 2026Yu Wang, Jie Ding, Jonathan H. HugginsReliable Uncertainty Quantification Stochastic Gradient Descent
Dynamics and Representation Structure of Local Approximations to Gradient-Based Learning in Linear Recurrent Neural Networks
May 29, 2026Ezekiel Williams, Alexandre Payeur, Guillaume LajoieRecurrent Neural Networks Learning Dynamics
A Unifying View of Variational Generative Wasserstein Flows
May 29, 2026Paul Caucheteux, Clément Bonet, Anna KorbaWasserstein Gradient Flows Wasserstein Distance
Gradient Descent with Large Step Size Restores Symmetry in Deep Linear Networks with Multi-Pathway
May 29, 2026Hee-Sung Kim, Sungyoon LeeDeep Network Gradient Flow
Reachability and asymptotics of Gaussian Transformer dynamics
May 29, 2026Albert Alcalde, Zhengping Ji, Enrique ZuazuaTransformer Architectures Probability Measures
Local linear convergence of gradient methods for overparameterized Gaussian mixtures
May 29, 2026Jingxing Wang, Vasileios Charisopoulos, Maryam FazelGaussian Mixture Models Over-Parameterized Neural Networks
A Unifying View of Anchoring via Operator-Side Tikhonov Regularization
May 29, 2026Zihao ChenAlternating Direction Method Of Multiplier Regularization Techniques
Convergence of Steepest Descent and Adam under Non-Uniform Smoothness
May 28, 2026Sharan Vaswani, Yifan Sun, Reza BabanezhadGradient Descent Adam
The Little Book of Generative AI Foundations: An Intuitive Mathematical Primer
May 28, 2026Tianhua ChenGenerative Models Generative Artificial Intelligence
AMDP: Asynchronous Multi-Directional Pipeline Parallelism for Large-Scale Models Training
May 28, 2026Ling Chen, Houming Wu, Wenjie YuParallel Mini-Batch Stochastic Gradient Descent
Constructing efficient channels for ideal observers using the conjugate gradient method
May 28, 2026Weimin ZhouImage Quality Assessment Image Reconstruction
Dynamics of Stochastic Momentum with Sparse Updates in High Dimensions
May 27, 2026Katie Everett, Elliot PaquetteStochastic Gradient Descent Stochastic Dynamics
Learning High-Dimensional Parity Functions with Product Networks using Gradient Descent
May 27, 2026Guillaume Larue, Louis-Adrien Dufrène, Quentin Lampin +2Two-Layer Neural Networks Neural Network
Stochastic Gradient Descent with Momentum is Algorithmically Stable
May 27, 2026Yunwen Lei, Zimeng Wang, Xiaoming YuanStochastic Gradient Descent Generalization Bounds
Gradient Step Plug-and-Play Model for Dental Cone-Beam CT Reconstruction
May 27, 2026Idris Tatachak, Luis Kabongo, Nicolas Papadakis +2Cone-Beam Computed Tomography Image Denoising
Revisiting ML Training under Fully Homomorphic Encryption: Convergence Guarantees, Differential Privacy, and Efficient Algorithms
May 27, 2026Yvonne Zhou, Mingyu Liang, Ivan Brugere +5Homomorphic Encryption Differential Privacy
SkillGrad: Optimizing Agent Skills Like Gradient Descent
May 26, 2026Hanyu Wang, Yifan Lan, Bochuan Cao +2Skill Evolution Large Language Model Agents
Worker Disagreement Reveals Sharp Directions in Local SGD
May 26, 2026Tolga Dimlioglu, Kristi Topollai, Anna ChoromanskaCondition Number Stochastic Gradient Descent
Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservation
May 26, 2026Tianlei Chen, Jiao Ou, Ziyuan Liu +3On-Policy Distillation Large Language Model Post-Training
When Muon Optimizer Meets Adversarial Training: A Theoretical and Empirical Study
May 26, 2026Jun Yan, Weiquan Huang, Jiankai Zuo +4Muon Optimizer Adversarial Training
From Privacy to Generalization: Linear Max-Information Bounds for DP-SGD
May 25, 2026Christoph H. Lampert, Hossein ZakeriniaDifferential Privacy Generalization Bounds
Global Convergence of Wasserstein Policy Gradient for Entropy-Regularized Reinforcement Learning
May 25, 2026Zhaoyu Zhu, Rui Gao, Shuang LiPolicy Gradient Entropy-Regularized Equilibrium Selection
Statistical Inference for Stochastic Gradient Descent Beyond Finite Variance
May 25, 2026Jose Blanchet, Peter Glynn, Wenhao YangStochastic Gradient Descent Confidence Intervals
Learning Permutation from Structure Without Supervision
May 25, 2026Ran Eisenberg, Ofir LindenbaumPermutation Entropy-Regularized Equilibrium Selection
Grow-Prune-Freeze Networks: Adaptive & Continual Learning Technique for Olfactory Navigation
May 24, 2026Kordel K. France, Ovidiu DaescuEfficient Continual Learning Continual Learning Methods
Leveraging Gauge Freedom for Learning Non-Gradient Population Dynamics of Stochastic Systems
May 24, 2026Jules Berman, Tobias Blickhan, Benjamin PeherstorferPopulation Dynamics Wasserstein Gradient Flows
Efficient DP-SGD for LLMs with Randomized Clipping
May 24, 2026Enayat Ullah, Sai Aparna Aketi, Devansh Gupta +2Differential Privacy Gradient Clipping
PILOT: Policy-Informed Learned Optimization for Adaptive Deep Network Training
May 23, 2026Sattam Altuuaim, Lama Ayash, Muhammad Mubashar +1Neural Network Optimization Gradient Concentration
Momentum Streams for Optimizer-Inspired Transformers
May 23, 2026Jingchu Gai, Nai-Chieh Huang, Jiayun WuTransformer Architectures Batch Normalization
From One-Pass SGD to Data Reuse: Mini-Batch Scaling Laws in Sketched Linear Regression
May 23, 2026Ziyan Chen, Zhongzhu Zhou, Ding-Xuan ZhouMini-Batch Stochastic Gradient Descent Stochastic Gradient Descent
Faithfulness as Information Flow: Evaluating and Training Faithful Chain-of-Thought Reasoning
May 22, 2026Jinghan Jia, Joe Benton, Eric EasleyExplanation Faithfulness Faithfulness

Jun 3, 2026 · math.OCJ/K move · Enter open · S save

Gradient descent at the Edge of Stability: free energy model and kinetic description of the two-layer network

Antonin Chodron de Courcel

Ecole Normale Sup´erieure, CNRS, 45 rue d’Ulm, 75005 Paris, France

We study the dynamics of gradient descent in the Edge of Stability regime, where the learning rate is large enough to induce persistent oscillations in the loss and the sharpness. We propose a continuous-time effective model that tracks the evolution of the average trajectory coupled with the time-averaged covariance of its fast oscillations. Our analysis reveals that the natural quantity to monitor in such unstable regimes is an effective free energy, which combines the original risk functional with a curvature-related "entropic" term. Our model allows us to track the envelope of the oscillations even in situations where its dynamics evolve on similar timescales as the averaged weights. Otherwise stated, we can track the spikes that occur during the training of some neural network architectures. For wide two-layer neural networks optimized under stable non-vanishing oscillations, we derive a mean-field limit that results in a novel kinetic equation describing the joint distribution of weights and their fluctuations. We show that this equation can be interpreted as a Wasserstein-2 gradient flow of a macroscopic free energy. Finally, we provide numerical evidence on matrix factorization and deep learning tasks (CIFAR-10) to demonstrate the model's accuracy in capturing the envelope of the oscillations and the predictive power of the effective free energy.

Gradient Descent Two-Layer Neural Networks Wasserstein Gradient Flows Mean-Field Limit Free Energy Principle Oscillatory Dynamics Edge

Open paper Ask about it PDF