Neural Network Training Dynamics

Ask about this topic

Overview

It examines how weights, gradients, activations, and predictions evolve during training, including why models learn some features before others, generalize, forget, or become unstable. Researchers analyze gradient-flow and stochastic-gradient dynamics using neural tangent kernels, mean-field limits, random matrix theory, and loss-landscape tools, alongside mechanistic experiments on tasks such as modular arithmetic and language-model pretraining. This work informs optimizer design, learning-rate and weight-decay schedules, architectures, and training diagnostics, with phenomena such as grokking, edge-of-stability behavior, and representation collapse receiving close attention. A central challenge is connecting tractable theories and early-warning signals to causal, predictive accounts of large finite networks, where optimizer, data, architecture, and noise interact.

Latest papers

CardsList

520 papers77 papers in the last four weeks, up 93% on the four weeks before. 0.4% of all new papers.

  1. Decomposing the Depth Profile of Fine-Tuning

    Apr 19, 2026Jayadev BillaFine-TuningRepresentation Drift

  2. A Mechanism Study of Delayed Loss Spikes in Batch-Normalized Linear Models

    Apr 18, 2026Peifeng Gao, Wenyi Fang, Yang Zheng +1Neural Network OptimizationNeural Network Training Dynamics

    School of Computing and Data Science, The University of Hong Kong . · RAMS Technologies Lab, Huawei Technologies Ltd.

  3. Curvature-Aligned Probing for Local Loss-Landscape Stabilization

    Apr 16, 2026Nikita Kiselev, Andrey GrabovoyNeural Network OptimizationNeural Network Training Dynamics

    Department of Intelligent Systems Moscow Institute of Physics and Technology

  4. Zeroth-Order Optimization at the Edge of Stability

    Apr 16, 2026Minhak Song, Liang Zhang, Bingcong Li +3Edge of StabilityZeroth-Order Optimization

    KAIST · KRAFTON · ETH Zurich +2

  5. Learning Inference Concurrency in DynamicGate MLP Structural and Mathematical Justification

    Apr 15, 2026Yongil ChoiDynamic Neural NetworksOnline Learning

    Sorynorydotcom Co., Ltd./AI Open Research Lab

  6. Distinct mechanisms underlying in-context learning in transformers

    Apr 14, 2026Cole Gibson, Wenping Cui, Gautam ReddyIn-Context LearningTransformer Training Dynamics

    Lewis-Sigler Institute for Integrative Genomics, Princeton University, Princeton, NJ 08544, USA · Department of Physics, Princeton University, Princeton, NJ 08544, USA

  7. VISTA: Validation-Informed Trajectory Adaptation via Self-Distillation

    Apr 13, 2026Eli Corn, Daphna WeinshallNeural Network GeneralizationDeep Learning Optimization

    Department of Computer Science, Hebrew University of Jerusalem, Jerusalem, Israel.

  8. Spectral Entropy Collapse as a Phase Transition in Delayed Generalisation: An Interventional and Predictive Framework for Grokkin

    Apr 13, 2026Truong Xuan Khanh, Truong Quynh Hoa, Luu Duc Trung +1GrokkingNeural Network Training Dynamics

    H&K Research Studio, Clevix LLC, Hanoi, Vietnam · Banking Academy of Vietnam, Hanoi, Vietnam

  9. Subcritical Signal Propagation at Initialization in Normalization-Free Transformers

    Apr 13, 2026Sergey AlekseevTransformerNeural Network Initialization

    Department of Physics and Astronomy Stony Brook University Stony Brook, NY 11794

  10. Training single-electron and single-photon stochastic physical neural networks

    Apr 12, 2026Tong Dou, Shiro Kumara, Josh Burns +6Handwritten Digit RecognitionPhysical Neural Networks

    School of Engineering and Technology, UNSW Canberra, Campbell, ACT 2600, Australia · Sussex Centre for Quantum Technologies, University of Sussex, Brighton BN1 9RH, United Kingdom · Department of Engineering Science, University of Oxford, Oxford OX1 3PJ, United Kingdom +2

  11. Heterogeneous Connectivity in Sparse Networks: Fan-in Profiles, Gradient Hierarchy, and Topological Equilibria

    Apr 12, 2026Nikodem TomczakDynamic Sparse TrainingSparse Neural Networks

    Thulge Labs, Singapore

  12. The Phase Is the Gradient: Equilibrium Propagation for Frequency Learning in Kuramoto Networks

    Apr 11, 2026Mani Rash AhmadiEquilibrium PropagationOscillatory Neural Networks

    Calibur Labs

  13. Sharp description of local minima in the loss landscape of high-dimensional two-layer ReLU neural networks

    Apr 10, 2026Jie Huang, Bruno Loureiro, Stefano Sarao MannelliOverparameterizationLoss Landscape

    Physics, Chalmers University of Technology and University of Gothenburg · Ecole Normale Superieure, PSL & CNRS · Data Science and AI, Computer Science and Engineering, Chalmers University of Technology and University of Gothenburg +1

  14. Statistical Properties of the King Wen Sequence: An Anti-Habituation Structure That Does Not Improve Neural Network Training

    Apr 10, 2026Augustin ChanPermutation TestingNeural Network Optimization

    Independent Researcher

  15. Mathematical analysis of one-layer neural network with fixed biases, a new activation function and other observations

    Apr 9, 2026Fabricio Macià, Shu NakamuraShallow Neural NetworksNeural Network Training Dynamics

    M2ASAI, Universidad Politécnica de Madrid. ETSI Navales. Avda. de la Memoria, 4. 28040 Madrid, Spain. · Department of Mathematics, Faculty of Sciences, Gakushuin University, 1-5-1, Mejiro, Toshima, Tokyo, 171-8588 Japan

  16. Training Deep Visual Networks Beyond Loss and Accuracy Through a Dynamical Systems Approach

    Apr 8, 2026Hai La Quang, Hassan Ugail, Newton Howard +3Neural Network Training Dynamics

    Posts and Telecommunications Institute of Technology Vietnam · Centre for Visual Computing and Intelligent Systems University of Bradford United Kingdom · School of Individualized Study Rochester Institute of Technology United States

  17. The Lifecycle of the Spectral Edge: From Gradient Learning to Weight-Decay Compression

    Apr 8, 2026Yongzhong XuNeural Network Training DynamicsGrokking

  18. On Dominant Manifolds in Reservoir Computing Networks

    Apr 7, 2026Noa Kaplan, Alberto Padoan, Anastasia BizyaevaTime Series ForecastingReservoir Computing

    Department of Computer Science at Cornell University, Ithaca, NY, USA 14853 · Department of Electrical and Computer Engineering, University of British Columbia, Vancouver, BC V6T 1Z4, Canada · Sibley School of Mechanical and Aerospace Engineering at Cornell University, Ithaca, NY, USA, 14853

  19. Spectral Edge Dynamics Reveal Functional Modes of Learning

    Apr 6, 2026Yongzhong XuNeural Network Training DynamicsGrokking

  20. Muon Dynamics as a Spectral Wasserstein Flow

    Apr 6, 2026Gabriel PeyréNeural Network Training DynamicsWasserstein Distance

    CNRS and ENS, PSL Université

  21. Dimensional Criticality at Grokking Across MLPs and Transformers

    Apr 6, 2026Ping WangNeural Network Training DynamicsPhase Transitions in Neural Networks

    Institute of High Energy Physics, Chinese Academy of Science, 100049 Beijing, China

  22. Grokking as Dimensional Phase Transition in Neural Networks

    Apr 6, 2026Ping WangPhase Transitions in Neural NetworksGrokking

    Institute of High Energy Physics, Chinese Academy of Science, 100049 Beijing, China

  23. Topological Sensitivity in Connectome-Constrained Neural Networks

    Apr 5, 2026Nalin DhimanBrain Network AnalysisNeural Network Training Dynamics

    School of Computing and Electrical Engineering, Indian Institute of Technology, Mandi, India

  24. The Spectral Lifecycle of Transformer Training: Transient Compression Waves, Persistent Spectral Gradients, and the Q/K--V Asymmetry

    Apr 3, 2026Yi LiuTransformer Training DynamicsTransformer

  25. Revealing the Learning Dynamics of Long-Context Continual Pre-training

    Apr 3, 2026Yupu Liang, Shuang Chen, Guanwei Zhang +2Continued PretrainingNeural Network Training Dynamics

    Tencent Hunyuan, Beijing, China

  26. Plateaus, Optima, and Overfitting in Multi-Layer Perceptrons: A Saddle-Saddle-Attractor Scenario

    Apr 2, 2026Alex Alì Maleknia, Yuzuru SatoNeural Network Training DynamicsMultilayer Perceptrons

    Department of Mathematics, Hokkaido University, Kita 12 Nishi 7, Kita-ku, Sapporo, Hokkaido 060-0812, Japan · Present address: IMAG, IROKO, Univ Montpellier, Inria, CNRS · RIES, Hokkaido University, Kita 12 Nishi 7, Kita-ku, Sapporo, Hokkaido 060-0812, Japan

  27. Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Threshold

    Mar 31, 2026Anamika Paul RupaNeural CollapseNeural Network Training Dynamics

    Department of Electrical Engineering and Computer Science, Howard University, Washington, DC, 20059, USA

  28. From Density Matrices to Phase Transitions in Deep Learning: Spectral Early Warnings and Interpretability

    Mar 31, 2026Max Hennick, Guillaume CorlouerPhase Transitions in Neural NetworksNeural Network Training Dynamics

  29. Spectral Edge Dynamics: An Analytical-Empirical Study of Phase Transitions in Neural Network Training

    Mar 30, 2026Yongzhong XuPhase Transitions in Neural NetworksNeural Network Training Dynamics

  30. Critical Damping as a Momentum Schedule: Multi-Seed Validation, a Hybrid Recipe, and an Exhaustive Negative Result on Surgical Layer Selection

    Mar 30, 2026Ivan PasichnykGradient-Based AttributionMomentum Methods

    We Label Data Inc.