Neural Network Training Dynamics

Follow this topic Ask about this topic

Momentum

41 papers in the last four weeks, up 273% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 349

All topics

The Implicit Bias of Depth: From Neural Collapse to Softmax Codes
May 21, 2026Connall Garrod, Jonathan P. Keating, Christos ThrampoulidisDeep Linear Networks Implicit Bias
Mathematical Institute, University of Oxford · Department of Electrical and Computer Engineering, University of British Columbia
Anytime Training with Schedule-Free Spectral Optimization
May 21, 2026Anuj Apte, Pranav Deshpande, Niraj Kumar +2Deep Learning Optimization Learning Rate Scheduling
Global Technology Applied Research, JPMorganChase New York, NY 10001, USA
Human-Centered Learning Mechanics: A Dynamical Framework for Entropy-Regulated Representation Learning
May 21, 2026Kim Phuc TranRepresentation Learning Neural Network Training Dynamics
Univ. Lille, ENSAIT, ULR 2461 – GEMTEX – Génie et Matériaux Textiles, F-59000 Lille, France · International Chair in DS & XAI, International Research Institute for Artificial Intelligence and Data Science, Dong A University, Danang, Vietnam
Why SGD is not Brownian Motion: A New Perspective on Stochastic Dynamics
May 21, 2026Igor Ignashin, Anna Radovskaya, Andrew Semenov +7Neural Network Training Dynamics Gradient Descent Dynamics
Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · P.N. Lebedev Physical Institute of the Russian Academy of Sciences · Innopolis University
AMUSE: Anytime Muon with Stable Gradient Evaluation
May 21, 2026Jueun Kim, Baekrok Shin, Jihun Yun +3Deep Learning Optimization Muon Optimizer
KAIST · KRAFTON · Seoul National University, KRAFTON
A Boundary-Layer Mechanism for One-Third Scaling in Online Softmax Classification
May 21, 2026Marcel Kühn, Yoon Thelge, Bernd RosenowStatistical Physics of Learning Multiclass Classification
Institute for Theoretical Physics, Leipzig University · 2ScaDS.AI Dresden/Leipzig
Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks
May 21, 2026Margalit Glasgow, Joan BrunaShallow Neural Networks Mean-Field Theory
Massachusetts Institute of Technology · Courant Institute School of Mathematics, Computing and Data Science, New York University
Thermodynamic Irreversibility of Training Algorithms
May 21, 2026Liu Ziyin, Yuanjie Ren, Adam Levine +1Neural Network Training Dynamics Gradient Descent Dynamics
1Massachusetts Institute of Technology · 2NTT Research
Large-Step Training Dynamics of a Two-Factor Linear Transformer Model
May 20, 2026Krishnakumar BalasubramanianTransformer Phase Transitions
Department of Statistics, University of California, Davis
The Devil is in the Condition Numbers: Why is GLU Better than non-GLU Structure?
May 20, 2026Xingyu Lyu, Qianqian Xu, Zhiyong Yang +2Neural Network Optimization Neural Network Training Dynamics
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China · School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 101408, China · Beijing Academy of Artificial Intelligence (BAAI), Beijing, China
Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnostics
May 19, 2026Lucky VermaWeight Decay Neural Network Memorization
Independent Researcher
StableGrad: Backward Scale Control without Batch Normalization
May 19, 2026Jose I. Mestre, Alberto Fernández-Hernández, Cristian Pérez-Corral +2Deep Learning Optimization Neural Network Training Dynamics
Universitat Politècnica de València · Universitat Jaume I
Adynamical systems view of training generativemodels and the memorization phenomenon
May 19, 2026Siva Athreya, Chiranjib Bhattacharya, Vivek S. BorkarNeural Network Memorization Memorization in Generative Models
International Institute for Theoretical Sciences, Survey No. 151, Hesarghatta Road, Shivakote Hobli, Bengaluru 560089, India · Department of Computer Science and Automation, Indian Institute of Science, Bengaluru 560012, India · Department of Electrical Engineering, Indian Institute of Technology Bombay, Powai, Mumbai 400076, India
Implicit Bias of Mirror Flow in Homogeneous Neural Networks: Sparse and Dense Feature Learning
May 19, 2026Tom Jacobs, Guido MontufarImplicit Bias Neural Network Training Dynamics
CISPA Helmholtz Center
Understanding Dynamics of Adam in Zero-Sum Games: An ODE Approach
May 19, 2026Yi Feng, Weiming Ou, Xiao WangZero-Sum Games Min-Max Optimization
Aarhus University, Aarhus, Denmark · Shanghai University of Finance and Economics, Shanghai, China · MoE Key Laboratory of Interdisciplinary Research of Computation and Economics, Shanghai University of Finance and Economics, Shanghai, China
Canonical Regularisation of Wide Feature-Learning Neural Networks
May 18, 2026George Whittle, Pranav Vaidhyanathan, Juliusz Ziomek +2Kernel Ridge Regression Representation Learning
Department of Engineering Science University of Oxford
Feature Learning in Linear-Width Two-Layer Networks: Two vs. One Step of Gradient Descent
May 18, 2026Behrad Moniri, Hamed HassaniRepresentation Learning Neural Network Training Dynamics
University of Pennsylvania, Philadelphia, PA, USA
How does feature learning reshape the function space?
May 18, 2026João Lobo, Bruno Loureiro, Long Tran-Than +1Representation Learning Kernel Methods
Department of Computer Science, University of Warwick, United Kingdom · Departement d’Informatique, École Normale Supérieure, PSL & CNRS · School of Mathematical Sciences, Institute of Natural Sciences and MOE-LSC, Shanghai Jiao Tong University, China. Part of work was done at Department of Computer Science, and Centre for Discrete Mathematics and its Applications (DIMAP), University of Warwick, United Kingdom
Training Infinitely Deep and Wide Transformers
May 17, 2026Raphaël Barboni, Maarten V. de Hoop, Takashi Furuya +1Transformer Wasserstein Gradient Flows
Bocconi University · Rice University · Doshisha University, RIKEN AIP +1
Bug or Feature $^2$ : Weight Drift, Activation Sparsity and Spikes
May 17, 2026Egor Shvetsov, Aleksandr Serkov, Shokorov Viacheslav +3ReLU Neural Networks Activation Sparsity
The Neural Tangent Kernel for Classification
May 17, 2026Jonathan Plenk, Sergio Calvo-Ordonez, Alvaro Cartea +3Classification Neural Network Training Dynamics
Mathematical Institute, University of Oxford · Oxford-Man Institute of Quantitative Finance, University of Oxford · OATML, University of Oxford +2
Von Economo neurons enable reliable social skill acquisition in recurrent spiking neural networks: a computational account with clinical predictions
May 17, 2026Esila KeskinNeural Processes Recurrent Neural Networks
School of Computing and Creative Technologies, University of the West of England, Bristol, UK
A Two-Parameter Weibull Framework for Diagnosing Transformer Weight Distributions
May 17, 2026Tiexin DingTransformer Weight-Space Analysis
Independent Researcher
High-dimensional Limit of SGD for Diagonal Linear Networks
May 16, 2026Begoña García Malaxechebarría, Courtney Paquette, Maryam Fazel +1Stochastic Optimization Convergence Neural Network Training Dynamics
University of Washington · McGill University · Google DeepMind +2
DynMuon: A Dynamic Spectral Shaping View of Muon
May 16, 2026Fangzhou Wu, Rikhav Shah, Sandeep Silwal +1Neural Network Optimization Muon Optimizer
University of Wisconsin–Madison · MIT · Elorian AI
A Fourier perspective on the learning dynamics of neural networks: from sample complexities to mechanistic insights
May 16, 2026Fabiola Ricci, Claudia Merger, Sebastian GoldtSample Complexity Frequency-Domain Feature Learning
International School of Advanced Studies (SISSA), Trieste, Italy
Does Weight Decay Enhance Training Stability?
May 15, 2026Marius Saether, Amir Kolic, Tomaso Poggio +1Edge of Stability Deep Learning Optimization
NTNU, MIT · MIT
Characterizing Learning in Deep Neural Networks using Tractable Algorithmic Complexity Analysis
May 15, 2026Pedram Bakhtiarifard, Sophia N. Wilson, Mahmoud Afifi +2Neural Network Generalization Neural Network Training Dynamics
Department of Computer Science, University of Copenhagen, Denmark · The American University in Cairo, Egypt
Rethinking Neural Network Learning Rates: A Stackelberg Perspective
May 15, 2026Sihan Zeng, Sujay Bhatt, Sumitra GaneshDeep Learning Optimization Neural Network Optimization
JPMorgan AI Research, United States.
On the Stability of Growth in Structural Plasticity
May 14, 2026Lute Lillo, Nick CheneyDynamic Neural Networks Neural Network Training Dynamics
University of Vermont

May 21, 2026, cs.LGJ/K move · Enter open · S save

The Implicit Bias of Depth: From Neural Collapse to Softmax Codes

Connall Garrod, Jonathan P. Keating, Christos Thrampoulidis

Mathematical Institute, University of Oxford · Department of Electrical and Computer Engineering, University of British Columbia

Deep Linear Networks Implicit Bias Neural Collapse Neural Network Training Dynamics

Open paper AskPDF arXiv

Neural collapse (NC) describes the structured geometry that emerges in the features and weights of trained classifiers. Recent theory suggests NC can be suboptimal in deep architectures, attributing this to an explicit low-rank bias from L2 regularization. We study the deep unconstrained feature model (UFM)-equivalent to a deep linear network with orthogonal inputs-trained without regularization, to isolate how gradient descent and depth alone shape NC. We show that depth induces an implicit low-rank bias: low-rank matrices propagate norm more efficiently through successive multiplications, promoting low-rank alternatives to NC. These alternatives, we argue, correspond to softmax codes: max-margin solutions previously found in width-bottlenecked networks. Analyzing training dynamics under spectral initialization, we identify an early-time repulsion among singular values that drives low-rank emergence, and characterize how depth shrinks NC's basin of attraction. Finally, we show that some effects act in the opposite direction: for randomly initialized networks, increasing width biases training toward higher-rank solutions. Our results provide the first asymptotic and dynamic characterization of implicit bias in deep UFMs trained with unregularized multiclass cross-entropy.

Momentum

Related topics

Latest papers 349