Neural Network Training Dynamics

Ask about this topic

Momentum

41 papers in the last four weeks, up 273% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 349

All topics
CardsList

  1. On Parameter Symmetries and Conservation Laws in Gradient Flow

    Sep 28, 2026Khang Nguyen, Guido MontúfarNeural Network Training DynamicsGradient Descent Dynamics

    UCLA · UCLA & MPI MiS

  2. LLN: Learnable Lens Networks for Parameter-Efficient Long-Horizon Dynamical Prediction

    Sep 28, 2026Binbin Yong, Zhao Su, Lan Guo +3Invertible Neural NetworksResidual Learning

    Lanzhou University · Monash University · University of Wollongong

  3. Scalable Attribution and Control of Model Behavior During Training

    Sep 27, 2026Sleem AbdelghafarGradient-Based AttributionCausal Interventions in Language Models

    Rice University

  4. Learning Dynamics of Continual Learning: A Unified View of Data Attribution, Forgetting, and Plasticity Loss

    Sep 27, 2026Yi Ren, Wenlong Deng, Guanzhe Hong +2Continual Learning for LLMsLoss of Plasticity

    OATML · University of Oxford · UBC +1

  5. Tracking States or Tracking Cosets? An Algebraic Account of Learned State Tracking

    Sep 24, 2026Zhiyu Zhang, Yupeng LiTransformerRecurrent Neural Networks

    Conflux Labs Ltd · Michigan State University

  6. An Analytical Theory of Auxiliary Learning

    Sep 24, 2026Federico Milanesio, Alessandro Ingrosso, Matteo OsellaStatistical Physics of LearningNeural Network Generalization

    Department of Physics and INFN University of Turin Via Giuria 1, 10125 Turin, Italy · Donders Centre for Neuroscience Radboud University Heyendaalseweg 135, 6525 AJ Nijmegen, The Netherlands

  7. The Dynamics of Quasiregular Neural Learning

    Sep 22, 2026Matthia SabatelliNeural Network Training Dynamics

    Department of Artificial Intelligence, University of Groningen Groningen, The Netherlands

  8. An Exploratory Replica-Overlap Probe of the Grokking Transition

    Sep 22, 2026A. C. Opus, J. Q. LuStatistical Physics of LearningNeural Network Training Dynamics

    Department of Physics, University of Puerto Rico, Mayagüez, PR 00680, USA

  9. Position Paper: Neurotransmitters as a Missing Dimension in Artificial Neural Networks

    Sep 17, 2026Yupei Li, Manuel Milling, Berrak Sisman +1Neural Network Training Dynamics

    Department of Computing and Chair of Health Informatics Imperial College London and Technical University of Munich London and Munich, United Kingdom and Germany · Chair of Health Informatics Technical University of Munich Munich, Germany · Department of Electrical and Computer Engineering Johns Hopkins Whiting School of Engineering Baltimore, United States +1

  10. Learning-Induced Dynamical Transition in Recurrent Neural Networks

    Sep 16, 2026Varun VaidyaDynamical SystemsRecurrent Neural Networks

    Department of Physics, University of South Dakota, Vermillion 57069, USA

  11. Beyond Quadratic Loss: The Stability Phase Diagram of Adam

    Sep 16, 2026Gaoxiang Tang, Huanran Chen, Ziming LiuAdaptive Gradient MethodsNeural Network Training Dynamics

    IIIS, Tsinghua University · College AI, Tsinghua University · Shanghai Qizhi Institute +1

  12. Information Geometric Self-Organization at the Edge of Stability in High-Capacity Kernel Associative Memories

    Sep 15, 2026Akira TamamoriInformation GeometryEdge of Stability

    Faculty of Information Science, Aichi Institute of Technology, Aichi, 470-0392, Japan

  13. Dimension-Corrected Hitting Times for Heavy-Tailed Spectral Emergence in Neural Optimizer Dynamics

    Sep 14, 2026Zongmin LiuHeavy-Tailed DistributionsNeural Network Training Dynamics

    Stanford University

  14. A Full Adam Theorem for Spectral Heavy-Tail Onset

    Sep 14, 2026Zongmin LiuAdaptive Gradient MethodsNeural Network Training Dynamics

    Stanford University

  15. Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking

    Sep 12, 2026Anish KatariaMultilayer PerceptronsNeural Network Memorization

    Princeton University, Princeton, NJ, USA.

  16. Learning Orthogonal Multi-Index Models Beyond Small Initialization: Incremental Learning, Competitive Dynamics and Symmetry

    Sep 11, 2026Mo Zhou, Weihang Xu, Simon S. Du +1Shallow Neural NetworksOrthogonal Representation Learning

    University of Washington · Amazon, Inc.

  17. Teacher Geometry Shapes Learnability in Teacher-Student Networks

    Sep 10, 2026Kai J. Sandbrink, Flavio Martinelli, Alexander van Meegen +2Neural Network OptimizationNeural Network Training Dynamics

    Department of Experimental Psychology, University of Oxford · Department of Physics, RWTH Aachen

  18. Training Trajectories Determine Circuit Removability in Annealable Soft-Prior Transformers

    Sep 9, 2026Zonglin Yang, Ziming Zhao, Wei Tang +5Transformer InterpretabilityAttention Mechanisms

    Criminal Science and Technology, Guangdong Police College, China · Cybersecurity and Law Enforcement, Guangdong Police College, China · Traffic Management Engineering, Guangdong Police College, China

  19. A Function-Space Approach to the Statistical Mechanics of Learning Dynamics

    Sep 9, 2026Yizhou Zhang, Weichen Wu, Lun Du +1Statistical Physics of LearningRepresentation Learning

    Variational AI · The Voleon Group · Independent Researcher +1

  20. When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay

    Sep 8, 2026Hasan Amin, Wei-Kai Chang, Rajiv KhannaLearning Rate SchedulingWeight Decay

    Purdue University

  21. Speed Limit for Information Acquisition in Stochastic Learning Dynamics

    Sep 8, 2026Shuta Kobayashi, Andreas DechantNeural Network Training DynamicsInformation-Theoretic Lower Bounds

    Department of Physics, Kyoto University, Kyoto 606-8502, Japan

  22. A Theoretical Analysis of Generalization Dynamics in Neural Networks under Gradient Descent with Weight Decay

    Sep 7, 2026Yuqing Wang, Ioannis G. Kevrekidis, Mikhail BelkinNeural Network GeneralizationNeural Network Training Dynamics

    Johns Hopkins University · University of California San Diego

  23. High-Dimensional Learning Dynamics of Attention-Indexed Models

    Sep 3, 2026Yizhou Xu, Margarita Sagitova, Lenka Zdeborová +1Implicit BiasAttention Mechanisms

    1Information, Learning and Physics Laboratory, École Polytechnique Fédérale de Lausanne (EPFL) · 2Statistical Physics of Computation Laboratory, École Polytechnique Fédérale de Lausanne (EPFL)

  24. Measuring Optimal Transport in Transformer Depth

    Sep 1, 2026Alexandre QuemyTransformerNeural Network Training Dynamics

    Hother Labs

  25. How Temporal Correlations Shape Memory in Linear Recurrent Neural Networks

    Aug 31, 2026Arnol Manuel Fokam, Fasseu Sieyondji Akpevwoghene, Edem Fiifi DawsonRecurrent Neural NetworksLinear RNNs

    Independent Researcher, United Kingdom · Department of Electrical and Electronic Engineering, University of Buea, Cameroon · minoHealth AI Labs, Ghana

  26. Sparse Competition during Training For the Emergence of Specialized Modules

    Aug 31, 2026Baptiste Rossigneux, Karim HarounSparse Neural NetworksNeural Network Training Dynamics

    Univ. Rennes · Inria, IRISA · University of Paris 8

  27. Context Staircase: Signature-Aligned Dynamics of Token Embeddings under Small Initialization

    Aug 31, 2026Junjie Yao, Liangkai Hang, Zhi-Qin John XuLanguage Model PretrainingWord Embeddings

    School of Mathematical Sciences Shanghai Jiao Tong University Shanghai 200240, China · Institute of Natural Sciences, School of Mathematical Sciences Shanghai Jiao Tong University Shanghai 200240, China

  28. Mapping the Emergence of Regularization-Driven Dynamics in Grokking

    Aug 26, 2026Yiming Lin, Yuxuan WangNeural Network GeneralizationNeural Network Training Dynamics

    School of Artificial Intelligence, University of Chinese Academy of Sciences

  29. Functional compatibility as a determinant of persistent neural learning

    Aug 23, 2026Hossein JavidniaStability-Plasticity DilemmaContinual Learning

    School of Computing Dublin City University Dublin, Ireland

  30. Adam at the Edge of Stability: Adaptive Feedback, Provable Oscillation, and Gradient Reversal

    Aug 21, 2026Yiman Fong, Heng YangEdge of StabilityAdaptive Gradient Methods

    School of Engineering and Applied Sciences, Harvard University.