Neural Network Training Dynamics

Ask about this topic

Overview

It examines how weights, gradients, activations, and predictions evolve during training, including why models learn some features before others, generalize, forget, or become unstable. Researchers analyze gradient-flow and stochastic-gradient dynamics using neural tangent kernels, mean-field limits, random matrix theory, and loss-landscape tools, alongside mechanistic experiments on tasks such as modular arithmetic and language-model pretraining. This work informs optimizer design, learning-rate and weight-decay schedules, architectures, and training diagnostics, with phenomena such as grokking, edge-of-stability behavior, and representation collapse receiving close attention. A central challenge is connecting tractable theories and early-warning signals to causal, predictive accounts of large finite networks, where optimizer, data, architecture, and noise interact.

Latest papers

CardsList

520 papers77 papers in the last four weeks, up 93% on the four weeks before. 0.4% of all new papers.

  1. Navigating by Old Maps: The Pitfalls of Static Mechanistic Localization in LLM Post-Training

    May 7, 2026Hang Chen, Jiaying Zhu, Hongyang Chen +3Supervised Fine-TuningTransformer Interpretability

    School of Computer Science and Technology Xi’an Jiaotong University · School of Computer Science and Engineering The Chinese University of Hong Kong · Shaanxi Co., Ltd(Xi’an 710077, China) China Mobile Group +1

  2. Spectral Lens: Activation and Gradient Spectra as Diagnostics of LLM Optimization

    May 7, 2026Andy Zeyi Liu, Elliot Paquette, John SousRepresentation LearningDecoder-Only Language Models

    Department of Applied Physics, Yale University, New Haven, CT 06511, United States · Department of Mathematics and Statistics, McGill University · Department of Applied Physics, Yale University, New Haven, Connecticut 06511, USA +1

  3. Why Geometric Continuity Emerges in Deep Neural Networks: Residual Connections and Rotational Symmetry Breaking

    May 6, 2026Kyungwon Jeong, Won-Gi Paeng, Honggyo SuhSymmetry BreakingGeometric Representation Learning

    Hyntel

  4. SPHERE: Mitigating the Loss of Spectral Plasticity in Mixture-of-Experts for Deep Reinforcement Learning

    May 6, 2026Lirui Luo, Guoxi Zhang, Hongming Xu +2Loss of PlasticityReinforcement Learning

    State Key Lab of General AI, School of Intelligence Science and Technology, Peking University · State Key Laboratory of General Artificial Intelligence, BIGAI · Institute for Artificial Intelligence, Peking University

  5. Gradient Scaling Effects in Adaptive Spectral PINNs for Stiff Nonlinear ODEs

    May 6, 2026Isabela M. Yepes, Pavlos ProtopapasDynamical SystemsNeural Network Optimization

    Harvard University

  6. Demystifying Manifold Constraints in LLM Pre-training

    May 6, 2026Kang An, Jiaxiang Li, Donald Goldfarb +1Language Model PretrainingRiemannian Optimization

    Rice University · Independent Researcher · Columbia University

  7. Critical Windows of Complexity Control: When Transformers Decide to Reason or Memorize

    May 6, 2026Sarwan AliDeep Learning OptimizationWeight Decay

    Columbia University, Irving Medical Center, NY, USA

  8. Information Plane Analysis of Binary Neural Networks

    May 5, 2026Maximilian Nothnagel, Bernhard C. GeigerNeural Network GeneralizationBinary Neural Networks

    Graz University of Technology, Graz, Austria · Know Center Research GmbH, Graz, Austria · Signal Processing and Speech Communication Laboratory, Graz University of Technology, Graz, Austria +1

  9. Learning reveals invisible structure in low-rank RNNs

    May 5, 2026Yoav Ger, Omri BarakNeural ProcessesRecurrent Neural Networks

    Ruth and Bruce Rappaport Faculty of Medicine and Network Biology Research Laboratory Technion – Israel Institute of Technology, Haifa, Israel

  10. Learning Dynamics of Zeroth-Order Optimization: A Kernel Perspective

    May 5, 2026Zhe Li, Bicheng Ying, Zidong Liu +1Zeroth-Order OptimizationLLM Fine-Tuning

    Department of Computing and Information Sciences Ph.D., Rochester Institute of Technology, Rochester, NY, USA · Google Inc., Los Angeles, CA, USA · ComboCurve Inc., Houston, TX, USA

  11. Trust, but Verify: Peeling Low-Bit Transformer Networks for Training Monitoring

    May 4, 2026Arian Eamaz, Farhang Yeganegi, Mojtaba SoltanalianDeep Learning OptimizationNeural Network Training Dynamics

    Department of Electrical and Computer Engineering University of Illinois Chicago Chicago, IL, USA

  12. Deciphering Shortcut Learning from an Evolutionary Game Theory Perspective

    May 4, 2026Xiayang Li, Kuo Gai, Shihua ZhangNeural Network Training DynamicsShortcut Learning

    State Key Laboratory of Mathematical Sciences Academy of Mathematics and Systems Science School of Mathematical Sciences University of Chinese Academy of Science Beijing, 100049, China · Shanghai Institute for Mathematics and Interdisciplinary Sciences Shanghai, 200433, China

  13. NeuroViz: Real-time Interactive Visualization of Forward and Backward Passes in Neural Network Training

    May 3, 2026Tanvi Sharma, Reza RawassizadehScientific VisualizationData Visualization

    Boston University · Boston, MA, USA

  14. A Theory of Saddle Escape in Deep Nonlinear Networks

    May 2, 2026Divit Rawal, Michael R. DeWeeseNeural Network OptimizationNeural Network Training Dynamics

    Department of Physics University of California, Berkeley Berkeley, CA 94720

  15. Focus and Dilution: The Multi-stage Learning Process of Attention

    May 2, 2026Zheng-An Chen, Pengxiao Lin, Zhi-Qin John Xu +1Self-AttentionTransformer Attention

    School of Mathematical Sciences, Shanghai Jiao Tong University. · Institute of Natural Sciences, Shanghai Jiao Tong University. · MOE-LSC, Shanghai Jiao Tong University. +2

  16. A Theory of Generalization in Deep Learning

    May 2, 2026Elon Litman, Gabe GuoNeural Network GeneralizationBenign Overfitting

    Stanford University

  17. Dendritic Neural Networks with Equilibrium Propagation

    May 1, 2026Yoshimasa KuboEquilibrium PropagationNeural Network Training Dynamics

    Department of Computer Science Lakehead University Thunder Bay, Canada

  18. Better Models, Faster Training: Sigmoid Attention for single-cell Foundation Models

    Apr 29, 2026Vijay Sadashivaiah, Georgios Dasoulas, Judith Mueller +1Softmax AttentionSelf-Attention

    Merck & Co., Inc., Cambridge, MA, USA

  19. Monitoring Neural Training with Topology: A Footprint-Predictable Collapse Index

    Apr 28, 2026Alexander KalinowskiRepresentation CollapseNeural Representation Geometry

    SUNY Empire

  20. Feature Repulsion and Spectral Lock-in: An Empirical Study of Two-Layer Network Grokking

    Apr 28, 2026Yongzhong XuNeural Network Training DynamicsModular Arithmetic

  21. Self-Abstraction Learning for Effective and Stable Training of Deep Neural Networks

    Apr 27, 2026Wonyong Cho, Taemin Kim, Jungmin Kim +2Neural Network GeneralizationNeural Network Optimization

    Department of Mathematics, University of Seoul, Seoul 02504, South Korea · Department of Applied Artificial Intelligence, Hansung University, Seoul 02876, South Korea

  22. When Does Removing LayerNorm Help? Activation Bounding as a Regime-Dependent Implicit Regularizer

    Apr 25, 2026Lucky VermaNeural Network Activation FunctionsLayer Normalization

    Independent Researcher

  23. GFlowState: Visualizing the Training of Generative Flow Networks Beyond the Reward

    Apr 23, 2026Florian Holeczek, Andreas Hinterreiter, Alex Hernandez-Garcia +2Data VisualizationGenerative Flow Networks

    Johannes Kepler University Linz · Mila – Quebec AI Institute · Eidgenössische Technische Hochschule Zürich

  24. There Will Be a Scientific Theory of Deep Learning

    Apr 23, 2026Jamie Simon, Daniel Kunin, Alexander Atanasov +11Neural Network Training DynamicsStatistical Learning Theory

    UC Berkeley and Imbue · UC Berkeley · Harvard University +6

  25. SGD at the Edge of Stability: The Stochastic Sharpness Gap

    Apr 22, 2026Fangshuo Liao, Afroditi Kolomvaki, Anastasios KyrillidisEdge of StabilityNeural Network Optimization

    Rice CS · Rice CS, Ken Kennedy Institute

  26. Too Sharp, Too Sure: When Calibration Follows Curvature

    Apr 22, 2026Alessandro Morosini, Matea Gjika, Tomaso Poggio +1Loss Landscape SharpnessMargin-Based Learning

    Massachusetts Institute of Technology, Cambridge, MA, USA.

  27. The Origin of Edge of Stability

    Apr 22, 2026Elon LitmanEdge of StabilityGradient Descent

    Stanford University

  28. Neuro-evolutionary stochastic architectures in gauge-covariant neural fields

    Apr 22, 2026Rodrigo Carmo TerinEvolutionary OptimizationEquivariant Neural Networks

    King Juan Carlos University, Facuty of Experimental Sciences and Technology, Department of Applied Physics, Av. del Alcalde de M´ostoles, 28933, Madrid, Spain

  29. Generalization at the Edge of Stability

    Apr 21, 2026Mario Tuci, Caner Korkmaz, Umut Şimşekli +1Intrinsic DimensionalityNeural Network Generalization

    INRIA, CNRS, Département d’Informatique de l’Ecole Normale Supérieure / PSL, France · Department of Computing, Imperial College London, United Kingdom

  30. ZC-Swish: Stabilizing Deep BN-Free Networks for Edge and Micro-Batch Applications

    Apr 21, 2026Suvinava BasakNeural Network StabilityNeural Network Activation Functions

    TU Braunschweig, Universitätspl. 2, 38106 Braunschweig, Germany