Neural Network Training Dynamics

Ask about this topic

Momentum

41 papers in the last four weeks, up 273% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 349

All topics
CardsList

  1. Backpropagation-Free Trunk Training via the Split Forward Gradients

    Jul 18, 2026Tian Qin, Wei-Min HuangBackpropagationGradient Descent

    Math department Lehigh University Bethlehem, PA 18015, USA

  2. Exact Network Surgery: Functional Invariance and Gradient Plasticity in Reactive Computational Graphs

    Jul 18, 2026Abdallah KhemaisNeural Network Training Dynamics

    ISITCOM, University of Sousse

  3. Certifying Residual Architectures from Their Primitives: A Sharp Stability Threshold

    Jul 16, 2026Hyemin Gu, Michael Tyrrell, Tuhin Sahai +1Neural Network VerificationResidual Learning

    University of Massachusetts Amherst · SRI International

  4. Adaptive Runge-Kutta Step Control Buys Training Loss, Not Generalization: An Honest Compute-Matched Study of RK-Adam Optimizers

    Jul 16, 2026Akhilesh GogikarNeural Network OptimizationAdaptive Gradient Methods

    1Independent Researcher.

  5. How the Hessian-Spectrum of Neural Networks Depends on Data

    Jul 15, 2026Jasraj Singh, Enea Monzio Compagnoni, Antonio OrvietoNeural Network OptimizationDeep Linear Networks

    ELLIS Institute, Tübingen, Germany · Universität Basel, Basel, Switzerland · MPI-IS, Tübingen, Germany

  6. Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks

    Jul 13, 2026Tiberiu Musat, Tiago Pimentel, Nicolas Zucchet +1In-Context LearningCircuits in Language Models

    ETH Zurich · Stanford University

  7. How to Tame Grokking: Representation Geometry as a Control Signal

    Jul 13, 2026Maksim A KazanskiiNeural Network GeneralizationSpectral Regularization

    Independent Researcher

  8. Backpropagation as a Nilpotent Linear System

    Jul 13, 2026Ahmed BoughammouraBackpropagationNeural Network Training Dynamics

    Higher Institute of Informatics and Mathematics of Monastir, University of Monastir, 5000 Monastir, Tunisia.

  9. Infrared Organization and Critical Cognitive Field Formation in Transformer Dynamics

    Jul 12, 2026Byung Gyu ChaeTransformerNeural Network Training Dynamics

    Electronics and Telecommunications Research Institute, 218 Gajeong-ro, Yuseong-gu, Daejeon 34129, Republic of Korea

  10. Singular perturbations and hierarchical learning in two-layer neural networks

    Jul 12, 2026Cédric Gerbelot, Jean-Christophe MourratMultilayer PerceptronsNeural Network Training Dynamics

    Unité de Mathématiques Pures et Appliquées (UMPA), ENS Lyon · CNRS

  11. LayerNorm as Implicit Gain Control in Looped Transformers

    Jul 12, 2026Matthias M. M. BuehlmaierRecurrent TransformersLayer Normalization

    Faculty of Business and Economics, The University of Hong Kong · Pokfulam Road, Hong Kong

  12. Interpreting learning dynamics of autoencoders: Transient scaling and emerging concepts of the Ising model

    Jul 11, 2026Max Weinmann, Miriam KlopotekAutoencodersUnsupervised Representation Learning

    1WIN-Kolleg of the Young Academy | Heidelberg Academy of Sciences and Humanities, Karlstraße 4, 69117 Heidelberg, Germany · 2Stuttgart Center for Simulation Science, Cluster of Excellence EXC 2075, University of Stuttgart, Universitätsstraße 32, 70569 Stuttgart, Germany

  13. The Differential Neural Tangent Kernel and Its Positivity

    Jul 11, 2026Bangti Jin, Longjun WuNeural PDE SolversPhysics-Informed ML

    Department of Mathematics, The Chinese University of Hong Kong, Shatin, N.T., Hong Kong, P.R. China

  14. How are linear representations learned? Exact solutions to the dynamics of abstraction

    Jul 9, 2026William W. Yang, Andrew M. Saxe, Peter E. LathamLinear Representation HypothesisDeep Linear Networks

    Gatsby Computational Neuroscience Unit, University College London, London W1T 4JG, United Kingdom. · Sainsbury Wellcome Centre, University College London, London W1T 4JG, United Kingdom.

  15. The Silent Freeze: Predicting When Low-Precision Training Stops Learning

    Jul 9, 2026Zekai ShangNeural Network Training DynamicsLow-Bit Quantization

    University of Illinois at Urbana-Champaign, Champaign, IL, USA

  16. The Anatomy of Implicit Bias: Information Allocation in Neural Network Training

    Jul 8, 2026Zhang Gongyue, Wang Zhiyong, Liu Donghan +3Neural Network OptimizationImplicit Bias

    1State Key Laboratory of Robotics and Systems, Harbin Institute of Technology Shenzhen, Shenzhen, 518055, China.

  17. On the Principles of Deep Feedforward ReLU Networks

    Jul 8, 2026Changcun HuangReLU Neural NetworksNeural Network Interpretability

    Shuitu Institute of Applied Mathematics, Chongqing 400700, P.R.C

  18. Cross-Trajectory Chimera Interventions Reveal Dissociable Roles of Weight Magnitude and Direction in Grokking

    Jul 7, 2026Truong Xuan KhanhNeural Network Training DynamicsCausal Intervention

    H&K Research Studio, Clevix LLC, Hanoi, Vietnam

  19. Broken Ergodicity and the Violation of the Fluctuation-Dissipation Theorem Lead to Generalization Beyond Overfitting in Machine Learning

    Jul 5, 2026Chan Li, Nigel GoldenfeldDouble DescentStatistical Physics of Learning

    Department of Physics, University of California San Diego, 9500 Gilman Drive, La Jolla, California 92093, USA. · Halıcıoğlu Data Science Institute, University of California San Diego, 9500 Gilman Drive, La Jolla, California 92093, USA

  20. Implicit Bias of SGD in Multivariate ReLU Networks: Effective Width Collapse

    Jul 3, 2026Shuang Liang, Tom Jacobs, Guido MontúfarShallow Neural NetworksReLU Neural Networks

    Department of Statistics & Data Science University of California, Los Angeles Los Angeles, CA 90095, USA · CISPA Helmholtz Center for Information Security · Departments of Mathematics and Statistics & Data Science University of California, Los Angeles; and Max Planck Institute for Mathematics in the Sciences

  21. Muon as a Residual Connection

    Jul 1, 2026Hao HuangNeural Network OptimizationMuon Optimizer

    College of Computer Science and Technology Zhejiang University Hangzhou, Zhejiang, China

  22. Radial Suppression Accelerates Algorithmic Generalization: A Geometric Analysis of Delayed Generalization

    Jun 30, 2026Srijan Tiwari, Aditya Chauhan, Manjot SinghNeural Network GeneralizationNeural Representation Geometry

    Indian Institute of Technology Roorkee

  23. Review Residuals: Update-Conditioned Residual Gating for Transformers

    Jun 30, 2026Kyle KramerTransformerResidual Learning

    NeraTech LLC

  24. SGD at the Edge of Stability: Stochastic Stabilization with Large Learning Rates

    Jun 29, 2026Konstantinos Emmanouilidis, Lachlan MacDonald, Salma Tarmoun +1Edge of StabilityStochastic Optimization Convergence

    Department of CIS1 · IDEAS4 · University of Pennsylvania +2

  25. Predictable GRPO: A Closed-Form Model of Training Dynamics

    Jun 29, 2026Rajat Ghosh, Datta Nimmaturi, Aryan Singhal +4Group Relative Policy OptimizationPolicy Optimization

    Nutanix San Jose, USA · Unsloth Hyderabad, India · Nutanix2026 San Jose, USA +1

  26. SGD Provably Prioritizes a Shortcut Spurious Feature in the XOR Model

    Jun 29, 2026Tyler LaBonte, Vidya MuthukumarRepresentation LearningReLU Neural Networks

    H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology · School of Electrical and Computer Engineering, Georgia Institute of Technology

  27. Scalar Representations of Neural Network Training Dynamics

    Jun 29, 2026Pedro Jiménez-González, Miguel C. Soriano, Lucas LacasaNeural Network OptimizationChaotic Dynamical Systems

    Institute for Cross-Disciplinary Physics and Complex Systems (IFISC, CSIC-UIB), Campus UIB, 07122 Palma de Mallorca, Spain

  28. Characterizing Optimizer-Dependent Training Dynamics Through Hessian Eigenvector Displacement and Localization

    Jun 29, 2026Marcelina Marjankowska, Valerio Modugno, Paolo BaruccaSpectral MethodsAdaptive Gradient Methods

    Financial Computing and Analytics Group, University College London · Robot Perception and Learning Lab, University College London

  29. What Drives the Inlier-Memorization Effect? A Theory of Outlier Detection via Early Training Dynamics

    Jun 29, 2026Kunwoong Kim, Dongha KimUnsupervised Anomaly DetectionNeural Network Training Dynamics

    KAIST AI · Department of Statistics Center for Data Science Sungshin Women’s University

  30. Learning as Observable Matrix Dynamics: Diffusive Relaxations versus Phase Transitions

    Jun 29, 2026Igor HalperinNeural Representation GeometryNeural Network Training Dynamics

    All calculations, numerical analysis, and manuscript preparation were performed by Claude Code with Opus 4.7 working as an AI assistant under author's supervision.