Neural Network Training Dynamics

Ask about this topic

Overview

It examines how weights, gradients, activations, and predictions evolve during training, including why models learn some features before others, generalize, forget, or become unstable. Researchers analyze gradient-flow and stochastic-gradient dynamics using neural tangent kernels, mean-field limits, random matrix theory, and loss-landscape tools, alongside mechanistic experiments on tasks such as modular arithmetic and language-model pretraining. This work informs optimizer design, learning-rate and weight-decay schedules, architectures, and training diagnostics, with phenomena such as grokking, edge-of-stability behavior, and representation collapse receiving close attention. A central challenge is connecting tractable theories and early-warning signals to causal, predictive accounts of large finite networks, where optimizer, data, architecture, and noise interact.

Latest papers

CardsList

520 papers77 papers in the last four weeks, up 93% on the four weeks before. 0.4% of all new papers.

  1. Mode Dynamics: A Framework for Neural Network Learning

    Oct 29, 2026Hossein HonarvarNeural Network Training DynamicsGradient Descent Dynamics

    Independent Researcher

  2. Softmax Routing Dynamics in Mixture-of-Experts Models: Collapse and Recovery

    Oct 29, 2026Shengze Xu, Maolin Deng, Tieyong ZengMixture of ExpertsMixture-of-Experts Models

    The Chinese University of Hong Kong · Beijing Normal-Hong Kong Baptist University · Guangzhou Nanfang College

  3. Representation Complexity Along Learning Trajectories: Trading Width for Precision and Time

    Oct 28, 2026Ryan Singh, Miguel De Llanza, Christopher L. BuckleyOverparameterized Neural NetworkNeural Network Training Dynamics

    Department of Engineering and Informatics, University of Sussex

  4. Does LLM Pre-Training Typically Occur at the Edge of Stability?

    Oct 27, 2026Yuhang Cai, Haodong Wen, Haofeng Huang +3Edge of StabilityNeural Network Training Dynamics

    UC Berkeley · Tsinghua University · ByteDance Seed

  5. Dynamical coarse-graining of neural-network training

    Oct 26, 2026Lin-Juan Wang, Da Li, Huiling Duan +1Neural Network Training Dynamics

    School of Astronautics, Beihang University, Beijing 100191, China · School of Mechanics and Engineering Science, Peking University, Beijing 100871, China · CAPT-HEDPS, and IFSA Collaborative Innovation Center of MoE, Peking University, Beijing 100871, China

  6. Scaling and Renormalization in Neural Networks: An Effective Renormalization-Group Description of Grokking

    Oct 26, 2026Min Chen, Junqi Wang, Zeguan Wu +2GrokkingPhase Transitions in Neural Networks

    Department of Computer Science, University of Pittsburgh, Pittsburgh, PA, USA

  7. The proton structure one mode at a time: a Neural Tangent Kernel analysis of a DIS fit

    Oct 26, 2026Amedeo Chiefa, Luigi Del Debbio, Richard KenwayNeural Network Training DynamicsNeural Tangent Kernel

    The Higgs Centre for Theoretical Physics, School of Physics and Astronomy, The University of Edinburgh, Peter Guthrie Tait Road, Edinburgh EH9 3FD, United Kingdom

  8. On the stability of learning

    Oct 26, 2026John J. VastolaNeural Network StabilityNeural Network Training Dynamics

    Department of Neurobiology, Harvard Medical School · Department of Psychology and Center for Brain Science, Harvard University · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University

  9. A theory of noisy linear recurrent neural network learning dynamics

    Oct 26, 2026John J. Vastola, Kanaka RajanRecurrent Neural NetworksNeural Network Training Dynamics

    Department of Neurobiology, Harvard Medical School · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University

  10. Depth Improves Training-Time Scaling for Difficult Tasks in Deep Linear Networks

    Oct 26, 2026Taishi Watanabe, Ryo Karakida, Jun-nosuke TeramaeScaling LawsNeural Network Training Dynamics

    Kyoto University · AIST · Institute of Science Tokyo

  11. Subspace Crystallization in Grokking: Rank Compression Tracks the Phase Transition

    Oct 26, 2026Shlok Mehendale, Paras ChopraGrokkingNeural Network Training Dynamics

    Lossfunk

  12. The Symmetry of Differential Neural Tangent Kernel under the Orthogonal Group

    Oct 25, 2026Bangti Jin, Longjun WuNeural Tangent KernelNeural Network Training Dynamics

    Department of Mathematics, The Chinese University of Hong Kong, Shatin, N.T., Hong Kong, P.R. China

  13. Neural LoFi with Backward Coupling: A Spectral Theory of Cross-Layer Feature Learning

    Oct 23, 2026Yatin Dandi, Matteo Vilucchio, Hugo Tabanelli +1Neural Network Training DynamicsBackpropagation

    Information, Learning, and Physics Laboratory (IDEPHICS) EPFL, Lausanne, Switzerland

  14. Early learning in large residual networks

    Oct 22, 2026Vladimir Narovlansky, Boris HaninNeural Network Training DynamicsResidual Learning

    ORFE Department, Princeton University, Princeton, NJ 08544, USA · Racah Institute of Physics, The Hebrew University of Jerusalem, Jerusalem 91904, Israel

  15. Width and Depth Limits Commute During Training in Linear Residual Networks

    Oct 22, 2026Ruijia Zhang, Soufiane HayouNeural Network Training DynamicsDeep Linear Networks

    Department of Applied Mathematics and Statistics, Johns Hopkins University

  16. The role of locality and weight sharing in convolutional neural networks

    Oct 21, 2026Francesco Mori, Melanie WeberConvolutional Neural NetworksNeural Network Training Dynamics

    Center of Mathematical Sciences and Applications, Harvard University · John A. Paulson School of Engineering and Applied Sciences, Harvard University

  17. Silent Alignment in Two-Layer Linear Networks: A High-Dimensional Perspective

    Oct 21, 2026Simon Martin, Francis Bach, Giulio BiroliTwo-Layer Neural NetworksNeural Network Training Dynamics

    DI-ENS, LPENS, SIERRA · DI-ENS, SIERRA · LPENS

  18. Asymmetry Tunneling Beyond Adaptive Equivariance Relaxation

    Oct 21, 2026Eunho Jeong, Joon Jang, Hyeonjin KimEquivariant Neural NetworksEquivariant Representation Learning

    Seoul National University.

  19. Dynamics of Hidden Learning Structure

    Oct 21, 2026Qinyou WangNeural Network Training DynamicsDynamic Neural Networks

  20. Physics-Inspired Optimizers beyond the Edge of Stability: Period-Two Orbits and Delayed Instability

    Oct 19, 2026Alexander ItinGradient Descent DynamicsNeural Network Training Dynamics

    Independent researcher, Hamburg, Germany

  21. Dynamical Batching: How Class-Cycling Training Shapes the Internal Structure of Neural Networks

    Oct 19, 2026Nicolás Ratier-Werbin, Samuel Lozano, Miguel Ruiz-GarcíaNeural Network Training DynamicsNeural Network Optimization

    Departamento de Estructura de la Materia, Física Térmica y Electrónica, Universidad Complutense de Madrid, 28040 Madrid, Spain · Grupo Interdisciplinar de Sistemas Complejos, Universidad Complutense de Madrid, 28040 Madrid, Spain

  22. A "feature ODE" describing the learning behavior of shallow MLPs on simple functions

    Oct 19, 2026Joseph Turnbull, Berkan Ottlik, Jamie SimonNeural Network Training DynamicsShallow Neural Networks

    UC Berkeley · University of Pennsylvania · Imbue

  23. Sampling and Curvature in First-Exit Comparisons for Momentum SGD

    Oct 18, 2026Haiyi LiStochastic Gradient DescentNeural Network Training Dynamics

  24. Global Convergence in Deep Networks via the Second Law of Thermodynamics

    Oct 17, 2026Matan Tsipory, Ofir Gaash, Itamar Harel +2Neural Network Training DynamicsStochastic Optimization Convergence

    Technion - Israel Institute of Technology, The Andrew and Erna Viterbi Faculty of Electrical & Computer Engineering, Technion City, Haifa 3200003, Israel. · Technion - Israel Institute of Technology, Faculty of Data and Decision Sciences, Technion City, Haifa 3200003, Israel. · Toyota Technological Institute at Chicago, Chicago, IL, USA.

  25. Representations Have Trajectories: Characterizing Temporal Representation Dynamics in Neural Networks

    Oct 17, 2026Shivaprasad B. Gowda, Sparsh MittalLow-Rank Representation LearningNeural Representation Geometry

    Indian Institute of Information Technology, Nagpur · Indian Institute of Technology, Roorkee

  26. Mesoscopic learning dynamics of extensive-width neural networks

    Oct 16, 2026Shu Zhou, Jean Barbier, Sebastian GoldtNeural Network Training DynamicsRepresentation Learning

    International School for Advanced Studies (SISSA) Via Bonomea 265, 34136 Trieste, Italy · The Abdus Salam International Centre for Theoretical Physics (ICTP) Strada Costiera 11, 34151 Trieste, Italy

  27. Neural Expansion: A Unified Mechanism for How Deep Neural Networks Generalize

    Oct 16, 2026Chashi Mahiul Islam, Samuel Jacob Chacko, Mao Nishino +2Neural Network GeneralizationOverparameterized Neural Network

    Department of Computer Science, Florida State University · Department of Mathematics, Florida State University · Independent

  28. Understanding Stability of Outer Nesterov Momentum on Stochastic Pseudo-Gradients

    Oct 15, 2026Nishant Sharma, Laura Gomezjurado Gonzalez, Ganesh Talluri +2Neural Network StabilityNeural Network Training Dynamics

    Chandar Research Lab · Mila – Quebec AI Institute · Université de Montréal +4

  29. Flat Loss, Evolving Features: Grokking in Modular Addition

    Oct 15, 2026Akash Kumar, Karthik SankaranarayananNeural Network Training DynamicsNeural Network Generalization

    University of California, San Diego · Microsoft AI

  30. Learning Changes the World and Then What Was Learned has Changed: Training and Deploying Neural Networks in the Control Loop

    Oct 15, 2026Vittorio LippiDistribution ShiftFeedback Control

    Calejo Hybrid Intelligence AB, Mellansjö 109, 92592 Tärnaby, Sweden