Neural Network Training Dynamics

Follow this topic Ask about this topic

It examines how weights, gradients, activations, and predictions evolve during training, including why models learn some features before others, generalize, forget, or become unstable. Researchers analyze gradient-flow and stochastic-gradient dynamics using neural tangent kernels, mean-field limits, random matrix theory, and loss-landscape tools, alongside mechanistic experiments on tasks such as modular arithmetic and language-model pretraining. This work informs optimizer design, learning-rate and weight-decay schedules, architectures, and training diagnostics, with phenomena such as grokking, edge-of-stability behavior, and representation collapse receiving close attention. A central challenge is connecting tractable theories and early-warning signals to causal, predictive accounts of large finite networks, where optimizer, data, architecture, and noise interact.

Latest papers

520 papers77 papers in the last four weeks, up 93% on the four weeks before. 0.4% of all new papers.

Never Saddle for Reparameterized Steepest Descent as Mirror Flow
Mar 2, 2026Tom Jacobs, Chao Zhou, Rebekka BurkholzNeural Network Training Dynamics Gradient Descent Dynamics
CISPA Helmholtz Center for Information Security, Saarbrücken, Germany
Quantitative Convergence of Wasserstein Gradient Flows of Kernel Mean Discrepancies
Mar 2, 2026Lénaïc Chizat, Maria Colombo, Roberto Colombo +1Wasserstein Gradient Flows Maximum Mean Discrepancy
Uniform-in-time concentration in two-layer neural networks via transportation inequalities
Mar 2, 2026Arnaud Guillin, Boris Nectoux, Paul StosNeural Network Training Dynamics Stochastic Gradient Descent
LMBP · Université Clermont-Auvergne, CNRS UMR 6620, LMBP, Clermont-Ferrand, France
A Basin-Selection Perspective on Grokking via Singular Learning Theory
Mar 1, 2026Ben Cullen, Sergio Estan-Ruiz, Riya Danait +1Grokking Neural Network Training Dynamics
Department of Computer Science, University of Pisa, Italy · Department of Mathematics, Imperial College London, United Kingdom · Mathematical Institute, University of Oxford, United Kingdom +1
Scaling of learning time for high dimensional inputs
Mar 1, 2026Carlos Stein BritoNeural Network Training Dynamics Scaling Laws
NightCity Labs, Lisbon, Portugal
When Does Margin Clamping Affect Training Variance? Dataset-Dependent Effects in Contrastive Forward-Forward Learning
Mar 1, 2026Joshua SteierNeural Network Training Dynamics Supervised Contrastive Learning
Independent Researcher
To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters
Feb 28, 2026Sara Dragutinović, Yedi Zhang, Rajesh RanganathNeural Network Generalization Muon Optimizer
Courant Institute School of Mathematics, Computing, and Data Science, New York University · Gatsby Computational Neuroscience Unit, University College London
On Emergences of Non-Classical Statistical Characteristics in Classical Neural Networks
Feb 27, 2026Hanyu Zhao, Yang Wu, Yuexian HouNeural Network Training Dynamics Neural Network Generalization
Tianjin University, China
Optimizer-Induced Low-Dimensional Drift and Transverse Dynamics in Transformer Training
Feb 27, 2026Yongzhong XuTransformer Training Dynamics Neural Network Training Dynamics
Don't stop me now: How Validation Criteria Affect Checkpoint Selection and Early Stopping
Feb 25, 2026Andrea Apicella, Francesco Isgrò, Andrea Pollastro +1Early Stopping Neural Network Training Dynamics
Department of Information Engineering, Electrical Engineering, and Applied Mathematics (DIEM), University of Salerno, Via Giovanni Paolo II, 132, Fisciano (Salerno), 84084, Italy. · Department of Electrical Engineering and Information Technology, University of Naples Federico II, Via Claudio 21, Naples, 80125, Italy.
The Error of Deep Operator Networks Is the Sum of Its Parts: Branch-Trunk and Mode Error Decompositions
Feb 25, 2026Alexander Heinlein, Johannes TarazDeep Operator Networks Neural Operators
Delft Institute of Applied Mathematics, Delft University of Technology, Delft, 2628CD, The Netherlands
When Learning Hurts: Fixed-Pole RNN for Real-Time Online Training
Feb 25, 2026Alexander Morgan, Ummay Sumaya Khan, Lingjia Liu +1Recurrent Neural Networks Neural Network Training Dynamics
dept. of ECE, Virginia Tech, Alexandria, VA, USA
Motivation is Something You Need
Feb 24, 2026Mehdi Acheli, Walid GaaloulNeural Network Training Dynamics Conditional Computation
SAMOVAR, Telecom SudParis Institut Polytechnique de Paris, Palaiseau, France
Path-conditioned training: a principled way to rescale ReLU neural networks
Feb 23, 2026Arthur Lebeurrier, Titouan Vayer, Rémi GribonvalReLU Neural Networks Neural Network Training Dynamics
ENS de Lyon, CNRS, Inria, Université Claude Bernard Lyon 1, LIP, UMR 5668, 69342, Lyon cedex 07, France · Inria, Rennes, France · Inria, CNRS, ENS de Lyon, Université Claude Bernard Lyon 1, LIP, UMR 5668, 69342, Lyon cedex 07, France
The Confusion is Real: GRAPHIC -- A Network Science Approach to Confusion Matrices in Deep Learning
Feb 23, 2026Johanna S. Fröhlich, Bastian Heinlein, Jan U. Claar +3Neural Network Interpretability Explainable Artificial Intelligence
Friedrich-Alexander-Universität Erlangen-Nürnberg Erlangen, Germany · Technical University of Darmstadt Darmstadt, Germany
Leap+Verify: Regime-Adaptive Speculative Weight Prediction for Accelerating Neural Network Training
Feb 23, 2026Jeremy McEntireNeural Network Training Dynamics
Grokking Finite-Dimensional Algebra
Feb 23, 2026Pascal Jr Tikeng Notsawo, Guillaume Dumas, Guillaume RabusseauGrokking Neural Network Generalization
Université de Montréal · Mila, Montréal · CHU Sainte-Justine Research Center, Montréal +1
Incremental Learning of Sparse Attention Patterns in Transformers
Feb 22, 2026Oğuz Kaan Yüksel, Rodrigo Alvarez Lucendo, Nicolas FlammarionSelf-Attention Transformer Training Dynamics
TML Lab, EPFL
Learning from Complexity: Exploring Dynamic Sample Pruning of Spatio-Temporal Training
Feb 22, 2026Wei Chen, Junle Chen, Yuqian Wu +2Data-Efficient Learning Dataset Pruning
HKUST(GZ) · HKUST
Global Low-Rank, Local Full-Rank: The Holographic Encoding of Learned Algorithms
Feb 20, 2026Yongzhong XuNeural Network Training Dynamics Grokking
The Geometry of Multi-Task Grokking: Transverse Instability, Superposition, and Weight Decay Phase Structure
Feb 19, 2026Yongzhong XuGrokking Neural Network Training Dynamics
Convergence Analysis of Two-Layer Neural Networks under Gaussian Input Masking
Feb 19, 2026Afroditi Kolomvaki, Fangshuo Liao, Evan Dramko +2Neural Network Training Dynamics ReLU Neural Networks
Computer Science Dept., Rice University, Houston, TX, USA
On the Mechanism and Dynamics of Modular Addition: Fourier Features, Lottery Ticket, and Grokking
Feb 18, 2026Jianliang He, Leda Wang, Siyu Chen +1Lottery Ticket Hypothesis Modular Addition
Department of Statistics and Data Science, Yale University
Learning under noisy supervision is governed by a feedback-truth gap
Feb 18, 2026Elan Schonfeld, Elias WisniaLearning with Noisy Labels Neural Network Training Dynamics
Department of Biology, Columbia University, New York, NY, USA · Department of Chemistry, Columbia University, New York, NY, USA
Optimizer choice matters for the emergence of Neural Collapse
Feb 18, 2026Jim Zhao, Tin Sum Cheng, Wojciech Masarczyk +1Neural Collapse Neural Network Training Dynamics
University of Basel · Warsaw University of Technology IDEAS Research Institute
Low-Dimensional and Transversely Curved Optimization Dynamics in Grokking
Feb 18, 2026Yongzhong XuGrokking Neural Network Training Dynamics
On the Emergence of Implicit Curriculum in RLVR Learning Dynamics
Feb 16, 2026Yu Huang, Zixin Wen, Yuejie Chi +4RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
Department of Statistics and Data Science, Wharton School, University of Pennsylvania · Machine Learning Department, Carnegie Mellon University · Department of Statistics and Data Science, Yale University +1
Scale redundancy and soft gauge fixing in positively homogeneous neural networks
Feb 16, 2026Rodrigo Carmo TerinNeural Network Optimization Neural Network Training Dynamics
King Juan Carlos University, Facuty of Experimental Sciences and Technology, Department of Applied Physics, Av. del Alcalde de Móstoles, 28933, Madrid, Spain
ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scaling
Feb 9, 2026Yilang Zhang, Bingcong Li, Niao He +1Residual Learning Neural Network Optimization
ML Research, Morgan Stanley, New York, NY 10019 · Department of CS, ETH Zürich, 8092 Zürich, Switzerland · Department of ECE, University of Minnesota, Minneapolis, MN 55455
Adaptive Momentum and Nonlinear Damping for Neural Network Training
Jan 30, 2026Aikaterini Karoni, Rajit Rajpal, Benedict Leimkuhler +1Momentum Methods Stochastic Optimization Convergence
School of Mathematics University of Bristol · School of Mathematics University of Edinburgh · CERMICS, CNRS, École des Ponts, Institut Polytechnique de Paris; Inria Paris, France

Mar 2, 2026J/K move · Enter open · S save

Never Saddle for Reparameterized Steepest Descent as Mirror Flow

Tom Jacobs, Chao Zhou, Rebekka Burkholz

CISPA Helmholtz Center for Information Security, Saarbrücken, Germany

Neural Network Training Dynamics Gradient Descent Dynamics Adaptive Gradient Methods Gradient Descent Mirror Descent Representation Learning

Open paper AskPDF arXiv

How does the choice of optimization algorithm shape a model's ability to learn features? To address this question for steepest descent methods --including sign descent, which is closely related to Adam --we introduce steepest mirror flows as a unifying theoretical framework. This framework reveals how optimization geometry governs learning dynamics, implicit bias, and sparsity and it provides two explanations for why Adam and AdamW often outperform SGD in fine-tuning. Focusing on diagonal linear networks and deep diagonal linear reparameterizations (a simplified proxy for attention), we show that steeper descent facilitates both saddle-point escape and feature learning. In contrast, gradient descent requires unrealistically large learning rates to escape saddles, an uncommon regime in fine-tuning. Empirically, we confirm that saddle-point escape is a central challenge in fine-tuning. Furthermore, we demonstrate that decoupled weight decay, as in AdamW, stabilizes feature learning by enforcing novel balance equations. Together, these results highlight two mechanisms how steepest descent can aid modern optimization.