Neural Network Training Dynamics

Follow this topic Ask about this topic

Momentum

41 papers in the last four weeks, up 273% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 349

All topics

A Mechanism Study of Delayed Loss Spikes in Batch-Normalized Linear Models
Apr 18, 2026Peifeng Gao, Wenyi Fang, Yang Zheng +1Neural Network Optimization Neural Network Training Dynamics
School of Computing and Data Science, The University of Hong Kong . · RAMS Technologies Lab, Huawei Technologies Ltd.
Curvature-Aligned Probing for Local Loss-Landscape Stabilization
Apr 16, 2026Nikita Kiselev, Andrey GrabovoyNeural Network Optimization Neural Network Training Dynamics
Department of Intelligent Systems Moscow Institute of Physics and Technology
Zeroth-Order Optimization at the Edge of Stability
Apr 16, 2026Minhak Song, Liang Zhang, Bingcong Li +3Edge of Stability Zeroth-Order Optimization
KAIST · KRAFTON · ETH Zurich +2
VISTA: Validation-Informed Trajectory Adaptation via Self-Distillation
Apr 13, 2026Eli Corn, Daphna WeinshallNeural Network Generalization Deep Learning Optimization
Department of Computer Science, Hebrew University of Jerusalem, Jerusalem, Israel.
Statistical Properties of the King Wen Sequence: An Anti-Habituation Structure That Does Not Improve Neural Network Training
Apr 10, 2026Augustin ChanNeural Network Optimization Curriculum Learning
Independent Researcher
On Dominant Manifolds in Reservoir Computing Networks
Apr 7, 2026Noa Kaplan, Alberto Padoan, Anastasia BizyaevaTime Series Forecasting Reservoir Computing
Critical Damping as a Momentum Schedule: Multi-Seed Validation, a Hybrid Recipe, and an Exhaustive Negative Result on Surgical Layer Selection
Mar 30, 2026Ivan PasichnykGradient-Based Attribution Momentum Methods
We Label Data Inc.
To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters
Feb 28, 2026Sara Dragutinović, Yedi Zhang, Rajesh RanganathNeural Network Generalization Muon Optimizer
Courant Institute School of Mathematics, Computing, and Data Science, New York University · Gatsby Computational Neuroscience Unit, University College London
Don't stop me now: How Validation Criteria Affect Checkpoint Selection and Early Stopping
Feb 25, 2026Andrea Apicella, Francesco Isgrò, Andrea Pollastro +1Early Stopping Neural Network Training Dynamics
Department of Information Engineering, Electrical Engineering, and Applied Mathematics (DIEM), University of Salerno, Via Giovanni Paolo II, 132, Fisciano (Salerno), 84084, Italy. · Department of Electrical Engineering and Information Technology, University of Naples Federico II, Via Claudio 21, Naples, 80125, Italy.
Incremental Learning of Sparse Attention Patterns in Transformers
Feb 22, 2026Oğuz Kaan Yüksel, Rodrigo Alvarez Lucendo, Nicolas FlammarionSelf-Attention Transformer Attention
TML Lab, EPFL
On the Emergence of Implicit Curriculum in RLVR Learning Dynamics
Feb 16, 2026Yu Huang, Zixin Wen, Yuejie Chi +4RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
Department of Statistics and Data Science, Wharton School, University of Pennsylvania · Machine Learning Department, Carnegie Mellon University · Department of Statistics and Data Science, Yale University +1
ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scaling
Feb 9, 2026Yilang Zhang, Bingcong Li, Niao He +1Residual Learning Neural Network Optimization
University of Minnesota · ETH Zurich
Adaptive Momentum and Nonlinear Damping for Neural Network Training
Jan 30, 2026Aikaterini Karoni, Rajit Rajpal, Benedict Leimkuhler +1Momentum Methods Stochastic Optimization Convergence
School of Mathematics University of Bristol · School of Mathematics University of Edinburgh · CERMICS, CNRS, École des Ponts, Institut Polytechnique de Paris; Inria Paris, France
Why $β_1 = β_2$ Is Dynamically Special in Adam
Jan 29, 2026Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre +2Momentum Methods Adaptive Gradient Methods
Universitat Politècnica de València Valencia, Spain · Universitat Jaume I Castelló de la Plana, Spain
Linearized subspace refinement framework to expose hidden accuracy in trained neural networks
Jan 20, 2026Wenbo Cao, Weiwei ZhangNeural Network Optimization Neural Network Approximation Theory
Institute of AI for Industries, Chinese Academy of Sciences, Nanjing 211135, China · Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China · School of Aeronautics, Northwestern Polytechnical University, Xi’an 710072, China +2
Understanding and inverse design of implicit bias in stochastic learning: a geometric perspective
Jan 10, 2026Nicola Aladrah, Emanuele Ballarin, Matteo Biagetti +3Implicit Bias Neural Network Training Dynamics
Department of Mathematics, Informatics and Geoscience, University of Trieste, Via Valerio 12/1, 34127 Trieste, Italy · Area Science Park, Padriciano, 34149 Trieste, Italy · McGovern Institute, MIT, Main Street, Cambridge, MA 02139, USA
Deep Neural Networks as Discrete Dynamical Systems: Implications for Physics-Informed Learning
Jan 1, 2026Abhisek Ganguly, Santosh Ansumali, Sauro SucciDynamical Systems Neural PDE Solvers
Engineering Mechanics Unit, Jawaharlal Nehru Centre for Advanced Scientific Research, Jakkur, Bangalore, 560064, Karnataka, India · Italian Institute of Technology, Viale Regina Elena 291, 00161 Rome, Italy · University of Roma Tre, Italy +2
Gradient Descent as Implicit EM in Distance-Based Neural Models
Dec 31, 2025Alan OurslandExpectation-Maximization Neural Network Training Dynamics
Quantitative Understanding of PDF Fits and their Uncertainties
Dec 30, 2025Amedeo Chiefa, Luigi Del Debbio, Richard KenwayUncertainty Quantification High-Energy Physics
The Higgs Centre for Theoretical Physics, School of Physics and Astronomy, The University of Edinburgh, Peter Guthrie Tait Road, Edinburgh EH9 3FD, United Kingdom
Dropout Neural Network Training Viewed from a Percolation Perspective
Dec 15, 2025Finley Devlin, Jaron SandersNeural Network Training Dynamics
Second Foundation, Czechia · Eindhoven University of Technology, The Netherlands
Random matrix theory of sparse neuronal networks with heterogeneous timescales
Dec 14, 2025Thiparat Chotibut, Oleg Evnin, Weerawit HorinouchiRandom Matrix Theory Recurrent Neural Networks
Chula Intelligent and Complex Systems, Department of Physics, Faculty of Science, Chulalongkorn University, Bangkok, Thailand · High Energy Physics Research Unit, Faculty of Science, Chulalongkorn University, Bangkok, Thailand · Theoretische Natuurkunde, Vrije Universiteit Brussel (VUB) and International Solvay Institutes, Brussels, Belgium +2
Learnability Window in Gated Recurrent Neural Networks
Dec 5, 2025Lorenzo LiviRecurrent Neural Networks Neural Network Training Dynamics
OPIT – Open Institute of Technology
Mean-Field Model for Two-Layer Neural Networks Trained with Consensus-Based Optimization
Nov 26, 2025William De Deyn, Michael Herty, Giovanni SamaeyNeural Network Optimization Mean-Field Theory
Institut für Geometrie und Praktische Mathematik, RWTH Aachen University, Germany · Department of Computer Science, KU Leuven, Belgium · Extraordinary Professor, Department of Mathematics and Applied Mathematics, University of Pretoria, South Africa
Understanding the Staged Dynamics of Transformers in Learning Latent Structure
Nov 24, 2025Rohan Saha, Farzane Aminmansour, Alona FysheDecoder-Only Language Models Neural Network Training Dynamics
Department of Computing Science University of Alberta Edmonton, Canada · Department of Computing Science and Psychology University of Alberta Edmonton, Canada
Fast Generalized Neural Tangent Kernel Statistics via Trace Estimation
Nov 13, 2025James Hazelden, Balaaji Reddy Nagireddy, Eric Shea-BrownNeural Network Training Dynamics Neural Tangent Kernel
Applied Mathematics, University of Washington & The Allen Institute, Seattle, WA.
Can Stationary Distributions of Scale-Invariant Neural Networks Be Described by the Thermodynamics of an Ideal Gas?
Nov 10, 2025Ildus Sadrtdinov, Ekaterina Lobacheva, Ivan Klimov +3Statistical Physics of Learning Neural Network Training Dynamics
Constructor University · Mila – Quebec AI Institute · Université de Montréal +2
Nonlinear Dynamics In Optimization Landscape of Shallow Neural Networks with Tunable Leaky ReLU
Oct 29, 2025Jingzhou LiuShallow Neural Networks Dynamical Systems
Why Do We Need Warm-up? A Theoretical Perspective
Oct 3, 2025Foivos Alimisis, Rustem Islamov, Aurelien LucchiDeep Learning Optimization Learning Rate Scheduling
University of Basel, Switzerland
Quantifying How Training Gradient Sparsity Affect Spiking Neural Network Accuracy And Robustness
Sep 28, 2025Nhan Trong Luu, Duong Trung LuuNeural Network Generalization Neural Network Robustness
College of Information and Communication Technology, Can Tho University, Vietnam · College of Engineering and Computer Science, VinUniversity, Hanoi, Vietnam · Center of Digital Transformation and Communication, Can Tho University, Vietnam
Uncertain but Useful: Leveraging CNN Training Variability into Data Augmentation
Sep 5, 2025Inés Gonzalez-Pepe, Vinuyan Sivakolunthu, Yohan Chatelain +1Image Segmentation Data Augmentation
Department of Computer Science Concordia University Montreal, QC H3G 1M8, Canada · Department of Computer Science Concordia University2026 Montreal, QC H3G 1M8, Canada · Krembil Centre for Neuroinformatics Centre for Addiction and Mental Health Toronto, ON M5T 0S8, Canada +1

Apr 18, 2026, stat.MLJ/K move · Enter open · S save

A Mechanism Study of Delayed Loss Spikes in Batch-Normalized Linear Models

Peifeng Gao, Wenyi Fang, Yang Zheng, Difan Zou

School of Computing and Data Science, The University of Hong Kong . · RAMS Technologies Lab, Huawei Technologies Ltd.

Neural Network Optimization Neural Network Training Dynamics

Open paper AskPDF arXiv

Delayed loss spikes have been reported in neural-network training, but existing theory mainly explains earlier non-monotone behavior caused by overly large fixed learning rates. We study one stylized hypothesis: normalization can postpone instability by gradually increasing the effective learning rate during otherwise stable descent. To test this hypothesis at theorem level, we analyze batch-normalized linear models. Our flagship result concerns whitened square-loss linear regression, where we derive explicit no-rising-edge and delayed-onset conditions, bound the waiting time to directional onset, and show that the rising edge self-stabilizes within finitely many iterations. Combined with a square-loss decomposition, this yields a concrete delayed-spike mechanism in the whitened regime. For logistic regression, under highly restrictive active-margin assumptions, we prove only a supporting finite-horizon directional precursor in a knife-edge regime, with an optional appendix-only loss lower bound under an extra non-degeneracy condition. The paper should therefore be read as a stylized mechanism study rather than a general explanation of neural-network loss spikes. Within that scope, the results isolate one concrete delayed-instability pathway induced by batch normalization.

Momentum

Related topics

Latest papers 349