Gradient Descent Dynamics

Follow this topic Ask about this topic

Momentum

9 papers in the last four weeks, up 200% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 108

All topics

PRAXIS: Learning Dynamics of Self-Improving Models with Symbolic Archives
Oct 8, 2026Venkat Margapuri, Mustafa TeberSelf-Improving Agents Adaptive Learning
Department of Computing Sciences Villanova University, Villanova, PA
Rare Gate Disagreements Can Limit Plasticity: When Gradient Flow Mispredicts Finite-Batch SGD
Oct 8, 2026Ruoyu Zhao, Mingxuan Zhang, Jianbo Dai +3Neural Network Training Dynamics Stochastic Gradient Descent
City University of Hong Kong · Microsoft · Copula Lab +1
Finite-Sample Approximation of Hessian-Guided Perturbed Wasserstein Gradient Flows
Oct 7, 2026Ryotaro Kawata, Atsushi Nitanda, Taiji SuzukiWasserstein Gradient Flows Gradient Descent Dynamics
The University of Tokyo · Agency for Science, Technology and Research (A*STAR) · Nanyang Technological University +1
Global Exponential Convergence of Two-Layer Linear Network Training
Oct 7, 2026Stephen Y Zhang, Gabriel PeyréNeural Network Training Dynamics Gradient Descent Dynamics
Flatiron Institute & NYU · CNRS, ENS, PSL Univ.
Improved Convergence of Large Stepsize Gradient Descent for Logistic Regression
Oct 5, 2026Xiaochuan Gong, Ang LiLogistic Regression Gradient Descent
University of Maryland, College Park
The Birkhoff Geometry of Manifold-Constrained Hyper-Connections: Two Channels, Vertex Viscosity, and Sinkhorn as a Retraction
Oct 5, 2026Xiaoyu Li, Zhizhou Sha, Chiwun YangHyper-Connections Constrained Optimization
University of New South Wales · University of Texas at Austin · City University of Hong Kong
Universality and Convergence of Generative Flows
Oct 4, 2026Leo BrunswicFlow Matching Generative Flow Networks
LLM Suite Team, JP Morgan Chase & Co.
Ideal Paths for Approximating Logistic Gradient Descent Trajectories at Large Initialization
Oct 2, 2026Junjie Xiao, Huiwen JiaLogistic Regression Neural Network Initialization
Department of Mathematics, Peking University · Department of Industrial Engineering and Operations Research, University of California, Berkeley
Tight Transition Time Bounds for Separable Logistic Regression at the Edge of Stability
Oct 1, 2026Haodong Wen, Kaiyue Wen, Jiaye TengEdge of Stability Logistic Regression
Stanford University · Shanghai University of Finance and Economics
Least-time Gradient Flow
Oct 1, 2026Alessandro Betti, Marco Gori, Stefano Melacci +1Optimal Control Gradient Descent Dynamics
Department of Mathematics “Tullio Levi-Civita”, University of Padua, Italy. · Department of Information Engineering and Mathematics, University of Siena, Italy. · University of Florence, Italy.
On Parameter Symmetries and Conservation Laws in Gradient Flow
Sep 28, 2026Khang Nguyen, Guido MontúfarNeural Network Training Dynamics Gradient Descent Dynamics
UCLA · UCLA & MPI MiS
On the Two Faces of Adam in Separable Linear Classification
Sep 27, 2026Chen Fan, Csaba SzepesváriImplicit Bias Classification
University of Alberta
Geometry of learning dynamics: Gradient descent versus natural gradient on the ridge of optimization
Sep 15, 2026Akira TamamoriInformation Geometry Gradient Descent
Faculty of Information Science, Aichi Institute of Technology 1247 Yachigusa, Yakusa-cho, Toyota-shi, Aichi 470-0392, Japan
How I learned to stop worrying and love StopGrads: Stationarity, Convergence, and a case study on Flow Map Learning
Sep 14, 2026Max W. Shen, Mark Goldstein, Zichu Wang +2Flow Map Learning Stochastic Optimization Convergence
Flatiron Institute · NYU
A Theoretical Analysis of Generalization Dynamics in Neural Networks under Gradient Descent with Weight Decay
Sep 7, 2026Yuqing Wang, Ioannis G. Kevrekidis, Mikhail BelkinNeural Network Generalization Neural Network Training Dynamics
Johns Hopkins University · University of California San Diego
The Multiple Timescales of Gradient Descent on the Edge of Stability: A Perturbative Derivation of the Central Flow
Sep 1, 2026Raphaël BerthierDynamical Systems Edge of Stability
Sorbonne Université, Inria, Centre Inria de Sorbonne Université, Paris, France
Hard-ReLU Gradient Descent Selects an Event-Free Sensitivity Limit
Aug 31, 2026Xiaoyang Li, Runni ZhouDynamical Systems Automatic Differentiation
College of Medicine and Biological Information Engineering Northeastern University, Shenyang, China
Reciprocity Separates Gradient Flow from Rotation in Conservative Physical Learning
Aug 31, 2026Ruiwu Niu, Xiaowen Bi, Michaël Antonie van WykGradient Descent Dynamics
Department of Data Science and Digital Innovation, Hong Kong Shue Yan University · Faculty of Science and Technology, Beijing Normal-Hong Kong Baptist University · School of Electrical and Information Engineering, University of the Witwatersrand, Johannesburg, South Africa
Post-Grokking Collapse at the Representation-Readout Interface in Muon-Trained Transformers
Aug 7, 2026Ali Janati, Kaoutar El Maghraoui, Anass BelfatmiTransformer Muon Optimizer
Data Science Institute, Columbia University, New York, NY, USA · Department of Computer Science, Columbia University, New York, NY, USA · CentraleSupélec, Gif-sur-Yvette, France
Stochastic gradient descent with discontinuity across a manifold
Aug 7, 2026Vivek S. BorkarStochastic Approximation Gradient Descent Dynamics
Department of Electrical Engineering, Indian Institute of Technology Bombay, Powai, Mumbai 400076, INDIA.
Non-asymptotic implicit bias of logistic regression at early-stage gradient descent dynamics
Aug 5, 2026Han BaoLogistic Regression Implicit Bias
The Institute of Statistical Mathematics · Tohoku University · RIKEN AIP
When Do Surrogate Updates Improve Decisions? A Local Theory of Trajectory-Wise Transfer
Aug 2, 2026Yuyang ShenSurrogate Modeling Decision-Focused Learning
The Chinese University of Hong Kong, Shenzhen
Quotient Dynamics, Effective Curvature, and Implicit Bias in Positive Quadratic Networks
Jul 28, 2026Pengcheng ChengRiemannian Optimization Implicit Regularization
School of Mathematics, Jilin University Changchun, 130012, China
Grokking on the Weight-Decay Clock: A Rate Hierarchy from Softly Broken Symmetries
Jul 27, 2026Taeyoung KimNeural Network Generalization Weight Decay
Center for AI and Natural Sciences Korea Institute for Advanced Study Seoul 02455, Republic of Korea
Mini-batch Noise Lowers Sharpness via Dominant-Subspace Fluctuations
Jul 25, 2026Junho So, Dongwook ShinNeural Network Training Dynamics Gradient Descent Dynamics
Department of Mathematics, Ajou University, South Korea
Gradient Flow Dynamics and Implicit Bias of Diagonal Linear Networks under Infinitesimal Initialization
Jul 14, 2026Jiajie Zhao, Jianxing Wang, Junjie Yang +2Deep Linear Networks Implicit Bias
School of Mathematical Sciences, Shanghai Jiao Tong University · Institute of Natural Sciences, MOE-LSC, Shanghai Jiao Tong University · School of Artificial Intelligence, Shanghai Jiao Tong University
LayerNorm as Implicit Gain Control in Looped Transformers
Jul 12, 2026Matthias M. M. BuehlmaierRecurrent Transformers Layer Normalization
Faculty of Business and Economics, The University of Hong Kong · Pokfulam Road, Hong Kong
Understanding Schedule-Free Methods in Nonconvex Optimization: Rate Guarantees and Escaping Saddles
Jul 10, 2026Jiseok Chae, Donghwan KimLearning Rate Scheduling Optimization Convergence Analysis
Natural Science Research Institute KAIST Daejeon, Republic of Korea · Department of Mathematical Sciences KAIST Daejeon, Republic of Korea
Dynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minima
Jul 9, 2026Lachlan Ewen MacDonald, René VidalGradient Descent Flat Minima
Innovation in Data Engineering and Science (IDEAS) · University of Pennsylvania Pennsylvania, PA 19104
Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networks
Jul 8, 2026Yedi Zhang, Peter E. Latham, Leena Chennuru Vankadara +1Learning Rate Scheduling Deep Linear Networks
Gatsby Computational Neuroscience Unit, University College London · Sainsbury Wellcome Centre, University College London

Oct 8, 2026, cs.LGJ/K move · Enter open · S save

PRAXIS: Learning Dynamics of Self-Improving Models with Symbolic Archives

Venkat Margapuri, Mustafa Teber

Department of Computing Sciences Villanova University, Villanova, PA

Self-Improving Agents Adaptive Learning Gradient Descent Dynamics

Open paper AskPDF arXiv

Self-improving learning systems adapt data selection, optimization, and auxiliary symbolic components, inducing nonstationary objectives outside standard learning assumptions. We introduce \textsc{PRAXIS}, a co-evolutionary framework that models generators, learners, and symbolic archives as interacting dynamical processes. We prove that KL-constrained generator updates and controlled archive-weight movement bound one-step objective drift, that archive updates suppress a program relative to any fixed comparator with a persistent cumulative utility advantage under sub-Gaussian noise, and that stochastic gradient descent achieves an average-stationarity guarantee whose degradation is governed by cumulative objective drift. Experiments across visual robustness, relational graph reasoning, and algorithmic graph reasoning exhibit generator stabilization, decreasing learner loss, and archive concentration consistent with these theoretical mechanisms.

Momentum

Related topics

Latest papers 108