Neural Network Training Dynamics

Ask about this topic

Overview

It examines how weights, gradients, activations, and predictions evolve during training, including why models learn some features before others, generalize, forget, or become unstable. Researchers analyze gradient-flow and stochastic-gradient dynamics using neural tangent kernels, mean-field limits, random matrix theory, and loss-landscape tools, alongside mechanistic experiments on tasks such as modular arithmetic and language-model pretraining. This work informs optimizer design, learning-rate and weight-decay schedules, architectures, and training diagnostics, with phenomena such as grokking, edge-of-stability behavior, and representation collapse receiving close attention. A central challenge is connecting tractable theories and early-warning signals to causal, predictive accounts of large finite networks, where optimizer, data, architecture, and noise interact.

Latest papers

CardsList

520 papers77 papers in the last four weeks, up 93% on the four weeks before. 0.4% of all new papers.

  1. Preconditioned Attention: Enhancing Efficiency in Transformers

    Mar 28, 2026Hemanth SaratchandranAttention MechanismsNeural Network Optimization

    Australian Institute for Machine Learning (AIML), Adelaide University

  2. A Systematic Empirical Study of Grokking: Depth, Architecture, Activation, and Regularization

    Mar 26, 2026Shalima Binta Manir, Anamika Paul RupaGrokkingNeural Network Training Dynamics

    Howard University

  3. Hebbian Attractor Networks for Robot Locomotion

    Mar 23, 2026Alexander Dittrich, Fuda van Diggelen, Dario FloreanoHebbian LearningNeural Network Training Dynamics

    Laboratory of Intelligent Systems, EPFL, Switzerland

  4. dynActivation: A Trainable Activation Family for Adaptive Nonlinearity

    Mar 23, 2026Alois BachmannNeural Network Activation FunctionsNeural Network Training Dynamics

    Ruprecht-Karls-Universität Heidelberg

  5. Quotient Geometry, Effective Curvature, and Implicit Bias in Simple Shallow Neural Networks

    Mar 23, 2026Hang-Cheng Dong, Pengcheng ChengShallow Neural NetworksNeural Network Training Dynamics

    School of Instrumentation Science and Engineering, Harbin Institute of Technology Harbin, 150001, China · Harbin Institute of Technology Suzhou Research Institute Suzhou, 215000, China · School of Mathematics, Jilin University Changchun, 130012, China

  6. Rethinking Plasticity in Deep Reinforcement Learning

    Mar 22, 2026Zhiqiang HeLoss of PlasticityNon-Stationary RL

    The University of Electro-Communications, Tokyo, Japan

  7. Model Evolution Under Zeroth-Order Optimization: A Neural Tangent Kernel Perspective

    Mar 22, 2026Chen Zhang, Yuxin Cheng, Chenchen Ding +5Zeroth-Order OptimizationNeural Network Training Dynamics

    The University of Hong Kong · The Hong Kong University of Science and Technology

  8. Intelligence Inertia: Physical Isomorphism and Applications

    Mar 22, 2026Jipeng HanNeural Network Training Dynamics

    OpenImmortal Technology Co., Ltd. Beijing 101100, China

  9. On the Dynamics & Transferability of Latent Generalization during Memorization

    Mar 20, 2026Simran Ketha, Venkatakrishnan RamaswamyNeural Network GeneralizationNeural Network Memorization

    Anuradha & Prashanth Palakurthi Centre for Artificial Intelligence Research, Department of Computer Science & Information Systems, Birla Institute of Technology & Science Pilani, Hyderabad 500078, India.

  10. Two-Time-Scale Learning Dynamics: A Population View of Neural Network Training

    Mar 20, 2026Giacomo Borghi, Hyesung Im, Lorenzo PareschiNeural Network Training DynamicsHyperparameter Optimization

    Maxwell Institute for Mathematical Sciences and School of Mathematical and Computer Sciences Heriot–Watt University, Edinburgh, UK · Department of Mathematics and Computer Science University of Ferrara, Italy

  11. ResNets of All Shapes and Sizes: Convergence of Training Dynamics in the Large-scale Limit

    Mar 18, 2026Louis-Pierre Chaintron, Lénaïc Chizat, Javier MaassDynamical Mean-Field TheoryNeural Network Training Dynamics

    École Polytechnique Fédérale de Lausanne (EPFL), Institute of Mathematics, 1015 Lausanne, Switzerland.

  12. Grokking as a Variance-Limited Phase Transition: Spectral Gating and the Epsilon-Stability Threshold

    Mar 16, 2026Pratyush Acharya, Habish DhakalGrokkingNeural Network Training Dynamics

  13. Controlled Langevin Dynamics for Sampling of Feedforward Neural Networks Trained with Minibatches

    Mar 16, 2026Alessandro Zambon, Francesca Caruso, Riccardo Zecchina +1Stochastic Gradient Langevin DynamicsNeural Network Training Dynamics

    Department of Physics, Università degli Studi di Milano and INFN, via Celoria 16, 20133 Milano, Italy · Department of Computing Sciences and Bocconi Institute for Data Science and Analytics (BIDSA), Bocconi University, 20136 Milano, Italy

  14. Neural Networks as Local-to-Global Computations

    Mar 16, 2026Vicente Bosca, Robert GhristSheaf Neural NetworksFeedforward Neural Networks

    Department of Mathematics, University of Pennsylvania

  15. Spectral Edge Dynamics of Training Trajectories: Signal--Noise Geometry Across Scales

    Mar 14, 2026Yongzhong XuTransformer Training DynamicsNeural Network Training Dynamics

  16. Fractional Rotation, Full Potential? Investigating Performance and Convergence of Partial RoPE

    Mar 12, 2026Mohammad Aflah Khan, Krishna P. Gummadi, Manish Gupta +1Rotary Positional EmbeddingsPositional Encoding

    Max Planck Institute for Software Systems · Microsoft, Hyderabad

  17. On the Learning Dynamics of Two-layer Linear Networks with Label Noise SGD

    Mar 11, 2026Tongcheng Zhang, Zhanpeng Zhou, Mingze Wang +4Two-Layer Neural NetworksNeural Network Training Dynamics

    Sch. of Computer Science and Zhiyuan College, Shanghai Jiao Tong University · Peking University · University of Sydney +4

  18. When Learning Rates Go Wrong: Early Structural Signals in PPO Actor-Critic

    Mar 10, 2026Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre +3Reinforcement LearningProximal Policy Optimization

    Universitat Polit`ecnica de Val`encia, Valencia, Spain · Universitat Jaume I, Castell´o de la Plana, Spain · Openchip & Software Technologies, Barcelona, Spain

  19. A Variational Latent Equilibrium for Learning in Neuronal Circuits

    Mar 10, 2026Simon Brandt, Paul Haider, Walter Senn +2Temporal Credit AssignmentNeural Network Training Dynamics

    Department of Physiology, University of Bern, Bühlplatz 5, 3012, Bern, Switzerland

  20. A Gaussian Comparison Theorem for Training Dynamics in Machine Learning

    Mar 10, 2026Ashkan PanahiDynamical Mean-Field TheoryNeural Network Training Dynamics

    Department of Computer Science and Engineering, Chalmers University, Gothenburg, Sweden

  21. Marginals Before Conditionals

    Mar 10, 2026Mihir SahasrabudheNeural Network Training DynamicsPhase Transitions in Neural Networks

    School of Information Sciences, University of Illinois Urbana-Champaign

  22. Norm-Hierarchy Transitions in Representation Learning: When and Why Neural Networks Abandon Shortcuts

    Mar 7, 2026Truong Xuan Khanh, Truong Quynh HoaNeural Network Training DynamicsRepresentation Learning

    H&K Research Studio, Clevix LLC, Hanoi, Vietnam

  23. NerVE: Nonlinear Eigenspectrum Dynamics in LLM Feed-Forward Networks

    Mar 6, 2026Nandan Kumar Jha, Brandon ReagenTransformer FFNsNeural Network Training Dynamics

    New York University

  24. The Geometric Inductive Bias of Grokking: Bypassing Phase Transitions via Architectural Topology

    Mar 5, 2026Alper YıldırımGrokkingNeural Network Training Dynamics

    Independent Researcher

  25. Does Sparse Connectivity Improve Generalization? Convolutional Networks Below the Edge of Stability

    Mar 5, 2026Tongtong Liang, Esha Singh, Rahul Parhi +2Neural Network GeneralizationGeneralization Bounds

    UC San Diego

  26. Implicit Bias and Loss of Plasticity in Matrix Completion: Depth Promotes Low-Rankness

    Mar 5, 2026Baekrok Shin, Chulhee YunImplicit BiasNeural Network Training Dynamics

    Kim Jaechul Graduate School of AI, KAIST

  27. Why Do Neural Networks Forget: A Study of Collapse in Continual Learning

    Mar 4, 2026Yunqin Zhu, Jun JinCatastrophic ForgettingIncremental Learning

    Department of Electrical and Computer Engineering, University of Alberta, Edmonton, Canada

  28. Riemannian Optimization in Modular Systems

    Mar 4, 2026Christian Pehle, Jean-Jacques SlotineContraction TheoryRiemannian Optimization

    Cold Spring Harbor Laboratory Cold Spring Harbor, NY 11724 · Massachusetts Institute of Technology Cambridge, MA 02139

  29. Heterogeneous Time Constants Improve Stability in Equilibrium Propagation

    Mar 3, 2026Yoshimasa Kubo, Suhani Pragnesh Modi, Smit PatelEquilibrium PropagationNeural Network Training Dynamics

    Department of Computer Science, Lakehead University

  30. On the Rate of Convergence of GD in Non-linear Neural Networks: An Adversarial Robustness Perspective

    Mar 2, 2026Guy Smorodinsky, Sveta Gimpleson, Itay SafranNeural Network Training DynamicsGradient Descent Dynamics

    Faculty of Computer and Information Science, Ben-Gurion University of the Negev, Israel · Department of Computer Science, Reichman University, Israel