Gradient Descent Dynamics

Follow this topic Ask about this topic

Momentum

9 papers in the last four weeks, up 200% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 108

All topics

Stability Annealing Selects the Implicit Bias of Smoothed Sign Descent: A Rate-Indexed Barrier Path on Separable Data
Jul 7, 2026Xiangwu Wang, Chengwei Cao, Yicheng Song +2Sign-Based Optimization Implicit Regularization
1The University of Hong Kong · University of California, San Diego · 3Beijing University of Aeronautics and Astronautics
The Map Behind the Flow: Finite-Step Gradient Descent as a Dynamical System
Jul 6, 2026Thomas HofmannEdge of Stability Gradient Descent
Department of Computer Science, ETH Zurich
Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks
Jul 5, 2026Shokhrukh Ibragimov, Arnulf JentzenDeep Learning Optimization Neural Network Optimization
Applied Mathematics: Institute for Analysis and Numerics, University of Münster, Germany · School of Data Science and School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China
Directional Curvature from Armijo Backtracking: A Low-Cost Sharpness Probe and a Calibration-Free Learning-Rate Safeguard for Adam
Jul 4, 2026Ashmitha R, Jörg FrochteGradient Descent Adaptive Gradient Methods
Department of Artificial Intelligence and Data Science, Sri Ramakrishna Engineering College, Anna University, Coimbatore 641022, Tamil Nadu, India · Interdisciplinary Institute for Applied AI and Data Science Ruhr (AKIS), Department of Electrical Engineering and Computer Science, Bochum University of Applied Sciences, Am Hochschulcampus 1, 44801 Bochum, Germany
Homogenization of $\ell_2$ -Adversarial Training in High-Dimensions: Exact Dynamics under Stochastic Gradient Descent
Jun 30, 2026Fabrizzio SabelliAdversarial Training Adversarial Robustness
Université de Montréal
Direction-Magnitude Decomposition for Low-Rank Matrix Optimization: Faster Convergence and Saddle-to-saddle Dynamics
Jun 30, 2026Yudong Wei, Liang Zhang, Bingcong Li +1Low-Rank Matrix Decomposition Nonconvex Optimization
Department of Mathematics, ETH Zurich, Zurich, Switzerland · Department of Computer Science, ETH Zurich, Zurich, Switzerland
Predictable GRPO: A Closed-Form Model of Training Dynamics
Jun 29, 2026Rajat Ghosh, Datta Nimmaturi, Aryan Singhal +4Group Relative Policy Optimization Policy Optimization
Nutanix San Jose, USA · Unsloth Hyderabad, India · Nutanix2026 San Jose, USA +1
Why can genetic algorithms work in high-dimensional search spaces?
Jun 29, 2026Stephen WhitelamEvolutionary Optimization Gradient Descent Dynamics
Molecular Foundry, Lawrence Berkeley National Laboratory, 1 Cyclotron Road, Berkeley, CA 94720, USA
Muon learns balanced solutions in matrix factorization without slow saddle-to-saddle dynamics
Jun 29, 2026Mark Rhee, Jamie Simon, Dhruva KarkadaMuon Optimizer Matrix Optimization
UC Berkeley · Imbue
How Complexity Contributes to Learning Opacity in Machine Learning
Jun 23, 2026Joachim Stein, Eric RaidlNeural Network Training Dynamics Gradient Descent Dynamics
1Heidelberger Akademie der Wissenschaften · 2Universität Tübingen, Cluster of Excellence: Machine Learning – New
Asymptotic Signal Subspace Recovery in Softmax Attention Models
Jun 21, 2026Lan V. TruongSoftmax Attention Gradient Descent Dynamics
Faculty of Computer Science and Engineering Ho Chi Minh City University of Technology (HCMUT) · Vietnam National University Ho Chi Minh City (VNU-HCM), Vietnam
Gradient-Descent Steps to Success over Mean Accuracy: A Paradigm Shift for ML
Jun 20, 2026Riccardo Poli, Ahmet YilmazAutomated Machine Learning Hyperparameter Optimization
School of Computer Science and Electronic Engineering University of Essex Wivenhoe Park, Colchester, CO4 3SQ, UK · Department of Computer Engineering Karamanoglu Mehmetbey University2026 Karaman, 70100, Turkey
Finite-Sample Performance of Gradient Descent in Logistic Regression with Gaussian Design
Jun 19, 2026Junren Chen, Arya MazumdarLogistic Regression Gradient Descent
Department of Mathematics, University of Maryland, College Park. · Halıcıoğlu Data Science Institute, UC San Diego.
Global Convergence of Gradient Descent for Score Matching in Gaussian Mixtures via Reverse Fisher Divergence
Jun 18, 2026Alexander TyurinScore Matching Optimization Convergence Analysis
AXXX, Moscow, Russia · Applied AI Institute, Moscow, Russia
Edge Flow: A Tractable and Predictive Continuous-Time Model for Gradient Descent at the Edge of Stability
Jun 16, 2026Pierre MarionEdge of Stability Neural Network Training Dynamics
Inria, École Normale Supérieure, PSL Research University
Conservation Laws for Modern Neural Architectures
Jun 16, 2026Viet-Hoang Tran, Vinh Khanh Bui, Tan Lai Ngoc +3Neural Network Training Dynamics Multi-Head Attention
National University of Singapore · Center for AI Research, VinUniversity · Independent Researcher +1
A Link between Shock-wave Theory and Symmetry-reduced Stochastic Gradient Descent for Artificial Neural Networks
Jun 16, 2026Taiki MiyagawaNeural Network Training Dynamics Gradient Descent Dynamics
NEC Corporation
SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation
Jun 15, 2026Junghun Oh, Sungyong Baik, Kyoung Mu LeeLow-Rank Adaptation Parameter-Efficient Fine-Tuning
Dept. of ECE, ASRI Seoul National University · Dept. of Artificial Intelligence Dept. of Data Science Hanyang University
A Bifurcation Theory Framework for Gradient Descent on the Edge of Stability
Jun 14, 2026Eric GanDynamical Systems Edge of Stability
Independent Researcher.
Mirror Descent Beyond Euclidean Stability: An Exponential Separation in Initialization Sensitivity
Jun 9, 2026Shira Vansover-Hager, Matan Schliserman, Ofir Schlisselberg +1Algorithmic Stability Mirror Descent
Blavatnik School of Computer Science and AI, Tel Aviv University · Google Research
Learning Dynamics Reveal a Hierarchy of Weight-Induced Layerwise Gram Metrics
Jun 8, 2026Claudio NordioBackpropagation ReLU Neural Networks
The Spectral Dynamics and Noise Geometry of Muon
Jun 7, 2026Pierfrancesco Beneventano, Mahmoud Abdelmoneum, Tomaso PoggioMuon Optimizer Matrix Optimization
Massachusetts Institute of Technology
Generalization in Deep Neural Networks: Minimax Rates for Gradient Methods
Jun 4, 2026Junyu Zhou, Puyu Wang, Yunwen Lei +2Neural Network Generalization Gradient Descent Dynamics
Mathematical Institute for Machine Learning and Data Science, Catholic University of Eichstätt-Ingolstadt · Department of Computer Science, RPTU Kaiserslautern-Landau · Department of Mathematics, The University of Hong Kong +1
Flatland: The Adventures of Gradient Descent with Large Step Sizes
Jun 4, 2026Leonardo Galli, Curtis Fox, Wiebke Bartolomaeus +2Edge of Stability Gradient Descent
LMU Munich · Munich Center for Machine Learning · University of British Columbia +1
A prism hierarchy of learning regimes in large linear autoencoders
Jun 3, 2026Eugene Golikov, Yaroslav Gusev, Dmitry YarotskyAutoencoders Neural Network Training Dynamics
Applied AI Institute Moscow, Russia · Steklov Mathematical Institute of Russsian Academy of Sciences Moscow, Russia
Mean--Fluctuation Dynamics at the Edge of Stability
Jun 3, 2026Antonin Chodron de CourcelEdge of Stability Gradient Descent Dynamics
Ecole Normale Sup´erieure, CNRS, 45 rue d’Ulm, 75005 Paris, France
When Both Layers Learn: Training Dynamics of Representing Linear Models via ReLU Networks
Jun 3, 2026Berk Tinaz, Changzhi Xie, Mahdi SoltanolkotabiReLU Neural Networks Neural Network Optimization
Department of Electrical and Computer Engineering University of Southern California Los Angeles, CA, USA · Department of Computer Science University of Southern California Los Angeles, CA, USA
Edge of Stability Selectively Shapes Learning Across the Data Distribution
Jun 2, 2026Shauna Kwag, Anakha Ganesh, Tomaso Poggio +1Edge of Stability Gradient Descent Dynamics
MIT
Denoise First, Orthogonalize Later: Understanding Momentum in Muon via Spectral Filtering
Jun 2, 2026Xianliang Li, Zihan Zhang, Weiyang Liu +1Deep Learning Optimization Momentum Methods
1The Institute of Statistical Mathematics · 2The Graduate Institute for Advanced Studies, SOKENDAI · 3National Institute of Informatics +3
Pseudospectral Bounds for Transient Amplification in Coupled Gradient Descent
Jun 1, 2026Ahanaf Hasan AriqStochastic Approximation Two-Timescale Stochastic Approximation
Ideal School and College

Jul 7, 2026, cs.LGJ/K move · Enter open · S save

Stability Annealing Selects the Implicit Bias of Smoothed Sign Descent: A Rate-Indexed Barrier Path on Separable Data

Xiangwu Wang, Chengwei Cao, Yicheng Song, Ran Bi+1

1The University of Hong Kong · University of California, San Diego · 3Beijing University of Aeronautics and Astronautics

Sign-Based Optimization Implicit Regularization Gradient Descent Dynamics

Open paper AskPDF arXiv

Adaptive gradient methods can favor max-margin separators that differ from gradient descent, yet a fixed positive numerical stability constant eventually changes the update geometry again. This paper studies the rate-controlled middle case for full-batch linear classification on separable data. For memoryless stability-annealed smoothed-sign descent with weighted exponential loss, we prove that the normalized iterates converge to the minimizer of a convex Burg-type barrier over a margin slice. The proof rewrites the dynamics exactly as entropic mirror ascent on a concave dual objective, controls the dual gap by a KL recursion, and yields an explicit S_t^{-1/2} normalized-iterate envelope. The static barrier geometry is fully characterized, including KKT conditions and both endpoint limits. Experiments validate the exact dual identities to floating-point error, illustrate the predicted path and rate diagram, and show an empirical fixed-epsilon crossover scaling in cumulative time. We further report robustness and boundary diagnostics for logistic tails, fixed-epsilon crossover, and adaptive-method variants, delineating the scope of the proved smoothed-sign theory.

Momentum

Related topics

Latest papers 108