Neural Network Training Dynamics

Follow this topic Ask about this topic

Momentum

41 papers in the last four weeks, up 273% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 349

All topics

The Weight Norm Sets the Grokking Timescale: A Causal Delay Law
Jun 11, 2026Truong Xuan Khanh, Doan Hoang Viet, Luu Duc Trung +1Neural Network Generalization Neural Network Training Dynamics
H&K Research Studio / Clevix LLC, Hanoi, Vietnam · Bac A Bank, Hanoi · Banking Academy of Vietnam
Different Layers, Different Manifolds: Module-Wise Weight-Space Geometry in Transformer Optimization
Jun 11, 2026Kirato YoshiharaStiefel Manifold Optimization Neural Network Optimization
School of Engineering Science, The University of Osaka, Osaka, Japan.
Circuit Synchronization Precedes Generalization: A Causal Precursor to Grokking
Jun 11, 2026Achyuthan SivasankarWeight Decay Frequency-Domain Feature Learning
New York University
Weibull Weight-Scale Parameter Evolution under AdamW Training Dynamics
Jun 11, 2026Tiexin DingWeight-Space Analysis Adaptive Gradient Methods
Independent Researcher.
Phase Transitions in Attention: A Bayesian Theory of Copy Head Emergence
Jun 10, 2026Itay Lavie, Kirsten Fischer, Andrey Lekov +3Softmax Attention Phase Transitions
Racah Institute of Physics, Hebrew University of Jerusalem, Jerusalem 91904, Israel · John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02138, USA · Institute for Advanced Simulation (IAS-6), Computational and Systems Neuroscience, Jülich Research Center, Jülich 52428, Germany +3
Overcoming Rank Collapse in Feedback Alignment
Jun 9, 2026Gauthier Boeshertz, Razvan Pascanu, Claudia ClopathBackpropagation Neural Network Optimization
Department of Bioengineering Imperial College London · Mila
Conservation Laws from Data Symmetry in Neural Networks
Jun 9, 2026Jakob Galley, Vahid Shahverdi, Axel FlinthNeural Network Optimization Neural Network Training Dynamics
Umeå University, Umeå, Sweden
Towards Critical Branching Mechanism in Recurrent Neural Networks
Jun 9, 2026Feixiang Ren, Ling FengRecurrent Neural Networks Long Short-Term Memory Networks
Department of Physics, National University of Singapore Singapore · Institute of High Performance Computing Agency for Science, Technology and Research (A*STAR) Singapore
Rank Collapse, Fixed Points, and the Renormalization Group Structure of MLP Residual Networks
Jun 9, 2026Parviz Haggi-Mani, Irina RishMultilayer Perceptrons Neural Network Training Dynamics
Universit´e de Montr´eal · Mila - Quebec AI Institute
Learning Entropy and Spatial Adaptation Dynamics of Multilayer Perceptrons for Structural Point Extraction
Jun 8, 2026Jan Glaser, Ivo Bukovsky, Marcel JirinaMultilayer Perceptrons Neural Network Training Dynamics
Czech Technical University in Prague, Prague, Czech Republic · University of South Bohemia in Ceske Budejovice, Ceske Budejovice, Czech Republic
Preserving Plasticity in Continual Learning via Dynamical Isometry
Jun 8, 2026Andries Rosseau, Robert Müller, Ann NowéLoss of Plasticity Continual Learning
AI Lab, Vrije Universiteit Brussel · Aganthos
Learning Dynamics Reveal a Hierarchy of Weight-Induced Layerwise Gram Metrics
Jun 8, 2026Claudio NordioBackpropagation ReLU Neural Networks
Integrating Out, Twice:The Open-System Case That Neural-Network Ensemble Theory Is Missing
Jun 8, 2026Jin LeiEnsemble Learning Neural Network Training Dynamics
Department of Physics, Tongji University, Shanghai 200092, China · Southern Center for Nuclear-Science Theory (SCNT), Institute of Modern Physics, Chinese Academy of Sciences, Huizhou 516000, Guangdong Province, China
Hybridizing Equilibrium Propagation with Ising Machines for Efficient Energy-Based Learning
Jun 8, 2026Chen-Rui Fan, Bo Lu, Xing-Yu Wu +2Equilibrium Propagation Hopfield Networks
School of Artificial Intelligence, Beijing Normal University, Beijing 100875, China · Laboratory for Advanced Computing and Intelligence Engineering, Information Engineering University, Zhengzhou 450001, China · School of Physical Science and Technology, Beijing University of Posts and Telecommunications, Beijing 100876, China
Beyond Neural Collapse: Task-Intrinsic Geometry Governs Neural Representations in Modular Arithmetic
Jun 8, 2026Hu Tan, Kuo Gai, Shihua ZhangNeural Representation Geometry Classification
Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China; and School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China · Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS), Shanghai, China · Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China; School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China; and Key Laboratory of Systems Health Science of Zhejiang Province, School of Life Science, Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Chinese Academy of Sciences, Hangzhou 310024, China
Between Amnesia and Chaos: A Memory Stability Expressivity Trilemma for Trainable Dissipative Oscillator Networks
Jun 7, 2026Caleb MunigetyDynamical Systems Lyapunov Stability
Independent Researcher
Fourier fractal dimension to predict the generalization of deep neural networks
Jun 6, 2026Joao B. Florindo, Davi Wanderley MisturiniNeural Network Generalization Neural Network Training Dynamics
Institute of Mathematics, Statistics and Scientific Computing - University of Campinas Rua Sérgio Buarque de Holanda, 651, Cidade Universitária “Zeferino Vaz” - Distr. Barão Geraldo, CEP 13083-859, Campinas, SP, Brasil
Breaking the Bubble: Asynchronous Pipeline Parallel Training with Bounded Weight Inconsistency
Jun 5, 2026Itay Elam, Eliron Rahimi, Avi Mendelson +1Neural Network Training Dynamics Pipeline Parallelism
Department of Computer Science Technion - Israel Institute of Technology · School of Electrical and Computer Engineering Ben-Gurion University of the Negev
Second-Order Path Kernel Interpolation Formulas in Machine Learning
Jun 5, 2026Jin Guo, Roy Y. He, Jean-Michel MorelNeural Network Training Dynamics
Department of Mathematics, City University of Hong Kong, Hong Kong · Division of Industrial Data Science, Lingnan University, Hong Kong
Beyond Linear and Overcomplete Regimes: A Mean-Field Analysis of Bottleneck Autoencoders
Jun 5, 2026Santanu Das, Ramyak Bilas, Pascal Esser +1Autoencoders Mean-Field Theory
STCS department Tata Institute Of Fundamental Research Mumbai, India-400005 · Department of Mathematics Indiana University Bloomington Indiana, USA-47405 · Department of Mathematics Ludwig-Maximilians-Universität München Germany, München-80799 +1
Quantifying Uncertainty In Wide Two-Layer Neural Networks: On The Law Of The Limiting Fluctuation Process
Jun 4, 2026Arnaud Descours, Arnaud Guillin, Geoffrey Lacour +3Uncertainty Quantification Neural Network Training Dynamics
LSAF · LMBP · MaIAGE +1
Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reduction
Jun 4, 2026Hu Tan, Kuo Gai, Shihua ZhangReLU Neural Networks Deep Linear Networks
1State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China · School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China · 3Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS), Shanghai, China +1
A prism hierarchy of learning regimes in large linear autoencoders
Jun 3, 2026Eugene Golikov, Yaroslav Gusev, Dmitry YarotskyAutoencoders Neural Network Training Dynamics
Applied AI Institute Moscow, Russia · Steklov Mathematical Institute of Russsian Academy of Sciences Moscow, Russia
Position: Don't Just "Fix it in Post": A Science of AI Must Study Training Dynamics
Jun 3, 2026Stella Biderman, Mohammad Aflah Khan, Niloofar Mireshghallah +3Neural Network Training Dynamics
1EleutherAI · 2Max Planck Institute for Software Systems · 3Carnegie Mellon University +3
When Both Layers Learn: Training Dynamics of Representing Linear Models via ReLU Networks
Jun 3, 2026Berk Tinaz, Changzhi Xie, Mahdi SoltanolkotabiReLU Neural Networks Neural Network Optimization
Department of Electrical and Computer Engineering University of Southern California Los Angeles, CA, USA · Department of Computer Science University of Southern California Los Angeles, CA, USA
Spectral Scaling Laws of Muon
Jun 2, 2026Gagik Magakyan, Pablo Parrilo, Asuman OzdaglarNewton-Schulz Iteration Momentum Methods
MIT
Neural Networks Provably Learn Spectral Representations for Group Composition
Jun 2, 2026Jianliang He, Leda Wang, Fengzhuo Zhang +2Representation Learning Neural Network Training Dynamics
Department of Statistics and Data Science, Yale University
Beyond $\ell_2$ -norm and $\ell_\infty$ -norm: A Curvature-Inspired $\ell_p$ -Norm Scheme for Deep Neural Networks
Jun 1, 2026Jianhao Xu, Zhuang YangNeural Network Optimization Nonconvex Stochastic Optimization
School of Computer Science and Technology, Soochow University, Suzhou, 215006, China
Looped Transformers with Layer Normalization Provably Learn the Power Method
May 30, 2026Lyumin Wu, Chenyang Zhang, Yuan CaoRecurrent Transformers Layer Normalization
School of Computing & Data Science, The University of Hong Kong
On the Difficulty of Learning a Meta-network for Training Data Selection
May 30, 2026Zilin Du, Junqi Zhao, Boyang Albert LiMeta-Learning Training Data Selection
College of Computing and Data Science, Nanyang Technological University, Singapore.

Jun 11, 2026, cs.LGJ/K move · Enter open · S save

The Weight Norm Sets the Grokking Timescale: A Causal Delay Law

Truong Xuan Khanh, Doan Hoang Viet, Luu Duc Trung, Phan Thanh Duc

H&K Research Studio / Clevix LLC, Hanoi, Vietnam · Bac A Bank, Hanoi · Banking Academy of Vietnam

Neural Network Generalization Neural Network Training Dynamics Grokking

Open paper AskPDF arXiv

Grokking is the delayed onset of generalization in neural networks, arising long after they fit the training data. Whether the weight norm causes this delay is disputed: some studies report a critical norm at the transition, others observe grokking with no fixed norm at all. We settle this by intervening on the norm during training rather than only observing it. Under free training with weight decay, networks grok when the weight norm reaches a value Wc that varies little across seeds and learning rates (CV 1 to 2 percent) and grows with the modular base as a power law. When we instead clamp the norm to a fixed multiple rho of Wc and hold it there, the network still groks, but the delay follows T_grok proportional to exp(alpha rho). One exponent, alpha near 7.5, fits this delay across four moduli (R^2 = 0.996). Over the swept ranges the held norm moves the delay by about 19x and the learning rate by only about 2x, and holding the norm above Wc slows grokking rather than preventing it. A final LayerNorm removes the dependence by decoupling weight scale from the network function; without it the exponential law returns. This pinned-norm delay is the exponential counterpart to the logarithmic delay predicted for a freely contracting norm.

Momentum

Related topics

Latest papers 349