Neural Network Training Dynamics

Ask about this topic

Overview

It examines how weights, gradients, activations, and predictions evolve during training, including why models learn some features before others, generalize, forget, or become unstable. Researchers analyze gradient-flow and stochastic-gradient dynamics using neural tangent kernels, mean-field limits, random matrix theory, and loss-landscape tools, alongside mechanistic experiments on tasks such as modular arithmetic and language-model pretraining. This work informs optimizer design, learning-rate and weight-decay schedules, architectures, and training diagnostics, with phenomena such as grokking, edge-of-stability behavior, and representation collapse receiving close attention. A central challenge is connecting tractable theories and early-warning signals to causal, predictive accounts of large finite networks, where optimizer, data, architecture, and noise interact.

Latest papers

CardsList

520 papers77 papers in the last four weeks, up 93% on the four weeks before. 0.4% of all new papers.

  1. Why β1=β2β_1 = β_2 Is Dynamically Special in Adam

    Jan 29, 2026Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre +2Momentum MethodsAdaptive Gradient Methods

    Universitat Politècnica de València Valencia, Spain · Universitat Jaume I Castelló de la Plana, Spain

  2. Linearized subspace refinement framework to expose hidden accuracy in trained neural networks

    Jan 20, 2026Wenbo Cao, Weiwei ZhangSubspace OptimizationNeural Network Optimization

    Institute of AI for Industries, Chinese Academy of Sciences, Nanjing 211135, China · Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China · School of Aeronautics, Northwestern Polytechnical University, Xi’an 710072, China +2

  3. The Impact of Anisotropic Covariance Structure on the Training Dynamics and Generalization Error of Linear Networks

    Jan 11, 2026Taishi Watanabe, Ryo Karakida, Jun-nosuke TeramaeNeural Network GeneralizationNeural Network Training Dynamics

    Graduate School of Informatics, Kyoto University · Artificial Intelligence Research Center, AIST · RIKEN AIP

  4. Understanding and inverse design of implicit bias in stochastic learning: a geometric perspective

    Jan 10, 2026Nicola Aladrah, Emanuele Ballarin, Matteo Biagetti +3Implicit BiasNeural Network Training Dynamics

    Department of Mathematics, Informatics and Geoscience, University of Trieste, Via Valerio 12/1, 34127 Trieste, Italy · Area Science Park, Padriciano, 34149 Trieste, Italy · McGovern Institute, MIT, Main Street, Cambridge, MA 02139, USA

  5. Architecture Before the Formula: Training-Dependent Computational Semantics in Transformers

    Jan 10, 2026Luis F. Rosario FreytesTransformer Training DynamicsTransformer Attention

  6. Deep Neural Networks as Discrete Dynamical Systems: Implications for Physics-Informed Learning

    Jan 1, 2026Abhisek Ganguly, Santosh Ansumali, Sauro SucciDynamical SystemsNeural PDE Solvers

    Engineering Mechanics Unit, Jawaharlal Nehru Centre for Advanced Scientific Research, Jakkur, Bangalore, 560064, Karnataka, India · Italian Institute of Technology, Viale Regina Elena 291, 00161 Rome, Italy · University of Roma Tre, Italy +2

  7. Gradient Descent as Implicit EM in Distance-Based Neural Models

    Dec 31, 2025Alan OurslandExpectation-MaximizationNeural Network Training Dynamics

  8. Quantitative Understanding of PDF Fits and their Uncertainties

    Dec 30, 2025Amedeo Chiefa, Luigi Del Debbio, Richard KenwayUncertainty QuantificationHigh-Energy Physics

    The Higgs Centre for Theoretical Physics, School of Physics and Astronomy, The University of Edinburgh, Peter Guthrie Tait Road, Edinburgh EH9 3FD, United Kingdom

  9. Dropout Neural Network Training Viewed from a Percolation Perspective

    Dec 15, 2025Finley Devlin, Jaron SandersNeural Network Training Dynamics

    Second Foundation, Czechia · Eindhoven University of Technology, The Netherlands

  10. Random matrix theory of sparse neuronal networks with heterogeneous timescales

    Dec 14, 2025Thiparat Chotibut, Oleg Evnin, Weerawit HorinouchiRandom Matrix TheoryRecurrent Neural Networks

    Chula Intelligent and Complex Systems, Department of Physics, Faculty of Science, Chulalongkorn University, Bangkok, Thailand · High Energy Physics Research Unit, Faculty of Science, Chulalongkorn University, Bangkok, Thailand · Theoretische Natuurkunde, Vrije Universiteit Brussel (VUB) and International Solvay Institutes, Brussels, Belgium +2

  11. Learnability Window in Gated Recurrent Neural Networks

    Dec 5, 2025Lorenzo LiviGated Recurrent UnitRecurrent Neural Networks

    OPIT – Open Institute of Technology

  12. Mean-Field Model for Two-Layer Neural Networks Trained with Consensus-Based Optimization

    Nov 26, 2025William De Deyn, Michael Herty, Giovanni SamaeyNeural Network OptimizationMean-Field Theory

    Institut für Geometrie und Praktische Mathematik, RWTH Aachen University, Germany · Department of Computer Science, KU Leuven, Belgium · Extraordinary Professor, Department of Mathematics and Applied Mathematics, University of Pretoria, South Africa

  13. Understanding the Staged Dynamics of Transformers in Learning Latent Structure

    Nov 24, 2025Rohan Saha, Farzane Aminmansour, Alona FysheTransformer Training DynamicsDecoder-Only Language Models

    Department of Computing Science, University of Alberta, Edmonton, Canada · Department of Computing Science and Psychology, University of Alberta, Edmonton, Canada

  14. Fast Generalized Neural Tangent Kernel Statistics via Trace Estimation

    Nov 13, 2025James Hazelden, Balaaji Reddy Nagireddy, Eric Shea-BrownNeural Network Training DynamicsNeural Tangent Kernel

    Applied Mathematics, University of Washington & The Allen Institute, Seattle, WA.

  15. Can Stationary Distributions of Scale-Invariant Neural Networks Be Described by the Thermodynamics of an Ideal Gas?

    Nov 10, 2025Ildus Sadrtdinov, Ekaterina Lobacheva, Ivan Klimov +3Statistical Physics of LearningNeural Network Training Dynamics

    Constructor University · Mila – Quebec AI Institute · Université de Montréal +2

  16. Nonlinear Dynamics In Optimization Landscape of Shallow Neural Networks with Tunable Leaky ReLU

    Oct 29, 2025Jingzhou LiuShallow Neural NetworksDynamical Systems

  17. Why Do We Need Warm-up? A Theoretical Perspective

    Oct 3, 2025Foivos Alimisis, Rustem Islamov, Aurelien LucchiDeep Learning OptimizationLearning Rate Scheduling

    University of Basel, Switzerland

  18. Quantifying How Training Gradient Sparsity Affect Spiking Neural Network Accuracy And Robustness

    Sep 28, 2025Nhan Trong Luu, Duong Trung LuuNeural Network GeneralizationNeural Network Robustness

    College of Information and Communication Technology, Can Tho University, Vietnam · College of Engineering and Computer Science, VinUniversity, Hanoi, Vietnam · Center of Digital Transformation and Communication, Can Tho University, Vietnam

  19. Toward a Physics of Deep Learning and Brains

    Sep 26, 2025Arsham Ghavasieh, Meritxell Vila-Minana, Akanksha Khurd +3Phase Transitions in Neural NetworksStatistical Physics of Learning

    Center for Complex Networks and Systems Research, Luddy School of Informatics, Computing, and Engineering, Indiana University, Bloomington, Indiana 47408, USA · Department of Physics, Indiana University, Bloomington, Indiana 47405, USA · Institute for Advanced Study, Princeton, NJ 08540, USA

  20. Uncertain but Useful: Leveraging CNN Training Variability into Data Augmentation

    Sep 5, 2025Inés Gonzalez-Pepe, Vinuyan Sivakolunthu, Yohan Chatelain +1Image SegmentationData Augmentation

    Department of Computer Science Concordia University Montreal, QC H3G 1M8, Canada · Department of Computer Science Concordia University2026 Montreal, QC H3G 1M8, Canada · Krembil Centre for Neuroinformatics Centre for Addiction and Mental Health Toronto, ON M5T 0S8, Canada +1

  21. Optimal Condition for Initialization Variance in Deep Neural Networks: An SGD Dynamics Perspective

    Aug 18, 2025Hiroshi Horii, Sothea HasFokker–Planck EquationNeural Network Initialization

    SU · KHM

  22. Dynamic gain neuromodulation attenuates stability gaps during continual joint training

    Jul 18, 2025Alejandro Rodriguez-Garcia, Anindya Ghosh, Srikanth RamaswamyContinual LearningIncremental Learning

    Neural Circuits Laboratory, Biosciences Institute, Newcastle University, Newcastle upon Tyne, NE2 4HH, United Kingdom

  23. Understanding Two-Layer Neural Networks with Smooth Activation Functions

    Jul 11, 2025Changcun HuangNeural Network Activation FunctionsNeural Network Approximation Theory

    Shuitu Institute of Applied Mathematics, Chongqing 400700, P.R.C

  24. A statistical physics framework for optimal learning

    Jul 10, 2025Francesca Mignacco, Francesco MoriStatistical Physics of LearningNeural Network Training Dynamics

    1Joseph Henry Laboratories of Physics, Princeton University, Princeton, NJ 08544, USA · 2Graduate Center, City University of New York, New York, NY 10016, USA · Center of Mathematical Sciences and Applications, Harvard University, Cambridge, MA 02138, USA

  25. Random Matrix Theory for Deep Learning: Beyond Eigenvalues of Linear Models

    Jun 16, 2025Zhenyu Liao, Michael W. MahoneyRandom Matrix TheoryNeural Network Generalization

    School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan, China · ICSI, LBNL, and Department of Statistics, University of California, Berkeley, USA

  26. KITINet: KInetic Theory Inspired Inter-Channel Information Exchange for Residual Networks

    May 23, 2025Mingquan Feng, Yifan Fu, Tongcheng Zhang +3Feature Interaction ModelingResidual Learning

    Shanghai Jiao Tong University

  27. KO: Kinetics-inspired Neural Optimizer with PDE Simulation Approaches

    May 20, 2025Mingquan Feng, Yixin Huang, Yifan Fu +2Neural Network OptimizationNeural Network Training Dynamics

    Shanghai Jiao Tong University

  28. Silent Neuron Theory and Plasticity Preservation for Deep Reinforcement Learning in Adaptive Video Streaming

    May 2, 2025Zhiqiang He, Zhi LiuNon-Stationary RLNeural Network Training Dynamics

    Department of Computer and Network Engineering, the University of Electro-Communications, Japan.

  29. The Dynamics of Generalization in Deep Learning

    Apr 23, 2025Rubing Yang, Pratik ChaudhariNeural Network GeneralizationNeural Network Training Dynamics

    Department of Applied Mathematics and Computational Sciences, University of Pennsylvania Philadelphia, PA, 19104, USA · Department of Electrical and Systems Engineering, University of Pennsylvania Philadelphia, PA, 19104, USA

  30. Hyperflux: Pruning Reveals Importance

    Apr 6, 2025Eugen Barbulescu, Antonio Alexoaie, Lucian BusoniuSparse Neural NetworksNeural Network Training Dynamics

    Department of Computer Science Technical University of Cluj-Napoca · Department of Automation Technical University of Cluj-Napoca