Neural Network Training Dynamics

Overview

It examines how weights, gradients, activations, and predictions evolve during training, including why models learn some features before others, generalize, forget, or become unstable. Researchers analyze gradient-flow and stochastic-gradient dynamics using neural tangent kernels, mean-field limits, random matrix theory, and loss-landscape tools, alongside mechanistic experiments on tasks such as modular arithmetic and language-model pretraining. This work informs optimizer design, learning-rate and weight-decay schedules, architectures, and training diagnostics, with phenomena such as grokking, edge-of-stability behavior, and representation collapse receiving close attention. A central challenge is connecting tractable theories and early-warning signals to causal, predictive accounts of large finite networks, where optimizer, data, architecture, and noise interact.

Latest papers

CardsList

553 papers45 papers in the last four weeks, down 46% on the four weeks before. 0.3% of all new papers.

  1. Decomposing SGD Dynamics in Neural Networks: Teacher-Induced Spikes and Variance Inflation

    Nov 18, 2026Samet Demir, Zafer DoganNeural Network GeneralizationNeural Network Training Dynamics

  2. Bias-Only Training Dynamics in ReLU Networks

    Nov 18, 2026Shreyas Chaudhari, Jose M. F. MouraNeural Network Training DynamicsReLU Neural Networks

  3. Learning Trajectories Leave Decodable Traces in Neural Network Weight Space and Internal Representations

    Nov 18, 2026Patrick Krauss, Achim Schilling, Andreas Maier +2Neural Network Training DynamicsWeight-Space Learning

  4. A Systematic Analysis of a Small Optimal Subspace for Physics-Informed Neural Networks: nn-DAS ⊂Rp\subset \mathbb{R}^p

    Nov 18, 2026Ahmad AliAhmad, Khemraj ShuklaSubspace OptimizationNeural Network Optimization

  5. Does Learning Order Explain Forgetting? The Lie Bracket as an Early-Warning Signal for Catastrophic Forgetting Without Old Data

    Nov 15, 2026Ephraim Kofi Ayi KwapongCatastrophic ForgettingNeural Network Training Dynamics

  6. High-dimensional limit theorems for adaptive variants of SGD I: Ballistic limits

    Nov 14, 2026Aukosh Jagannath, Varnan SarangianSGD with MomentumAdaptive Gradient Methods

  7. Deep Neural Networks Training Efficiency: Does Grokking Modify its Dynamics?

    Nov 13, 2026Eduardo Cueto-Mendoza, John D. KelleherGrokkingNeural Network Training Dynamics

  8. The balance between feature learning and collapse in dynamical systems-based generative models

    Nov 9, 2026Julian Brandon, Bruno Loureiro, N Alex Cayco-Gajic +1Generative ModelingMode Collapse

  9. Neural Activity Synchronization-Based Fast Local Learnable Spiking Neural Network Architecture

    Nov 8, 2026Pranith Dutta, Sk Wasim Rajjak, Siddhanta Ray +2Spiking Neural NetworksLocal Learning

  10. Deep Learning Theory through the Lens of Statistical Physics: A Topical Review

    Nov 6, 2026Wei Huang, Youjin DengStatistical Physics of LearningNeural Network Generalization

  11. Task-Contribution Dynamics of Adam in Physics-Informed Neural Networks: Slow-Manifold Analysis and Gradient Modulation

    Nov 6, 2026Enzhan Zheng, Xiaoliang SongNeural Network Training DynamicsAdaptive Gradient Methods

  12. Symmetry Breaking at the Edge of Stability via Central Flow Analysis

    Nov 4, 2026Naima Elosegui Borras, Georgios ArvanitidisNeural Network Training DynamicsGradient Descent Dynamics

  13. Finite-width shift of the edge of chaos in deep neural networks

    Nov 4, 2026Arn CaesarFinite-Width Neural Network TheoryNeural Network Initialization

  14. Towards an Optimal Control Foundation of Early Exits - Turnpikes and ResNets

    Nov 3, 2026Jens Püttschneider, Simon Heilig, Asja Fischer +1Early-Exit Neural NetworksNeural Network Training Dynamics

  15. Curriculum unlearning in humans and neural networks

    Nov 3, 2026Jirko Rubruck, Alice Zhang, Andrew Saxe +1Curriculum LearningNeural Network Training Dynamics

  16. Bridging Neural Tangent Kernel Analysis and Reinforcement Learning for Deep Function Approximation

    Nov 2, 2026William St-Arnaud, Margarida Carvalho, Golnoosh FarnadiNeural Tangent KernelNeural Network Training Dynamics

  17. Susceptibilities for Neural Networks Learning from Physical Data

    Nov 2, 2026Rohan Hitchcock, Gary W. Delaney, Jonathan H. Manton +2Statistical Physics of LearningNeural Network Training Dynamics

  18. Unified dynamics of training, learning, inference, and reinforcement in neural networks

    Oct 31, 2026Mian WangNeural Network Training DynamicsEmergent Abilities in Language Models

  19. Solvable dynamical model of in-context learning

    Oct 31, 2026Ruoran Bai, Haiping HuangIn-Context LearningTransformer Training Dynamics

  20. Stochastic learning flow: moment equations for stochastic gradient descent at finite step size

    Oct 30, 2026Christian Schmid, James M MurrayNeural Network Training DynamicsStochastic Gradient Descent

  21. Gradient Descent on Two ReLU Neurons: Global Landscape and Bifurcation Dynamics

    Oct 30, 2026Binghua Li, Mengzhe Li, Denny Wu +1Neural Network Training DynamicsNeural Network Optimization

  22. Infinite ResNets are Gaussian Mean ODEs

    Oct 30, 2026Louis-Pierre Chaintron, Lénaïc Chizat, Javier MaassNeural Network Training DynamicsGradient Descent Dynamics

  23. Softmax Routing Dynamics in Mixture-of-Experts Models: Collapse and Recovery

    Oct 29, 2026Shengze Xu, Maolin Deng, Tieyong ZengMixture of ExpertsMixture-of-Experts Models

  24. Representation Complexity Along Learning Trajectories: Trading Width for Precision and Time

    Oct 28, 2026Ryan Singh, Miguel De Llanza, Christopher L. BuckleyOverparameterized Neural NetworkNeural Network Training Dynamics