Neural Network Training Dynamics

Follow this topic Ask about this topic

It examines how weights, gradients, activations, and predictions evolve during training, including why models learn some features before others, generalize, forget, or become unstable. Researchers analyze gradient-flow and stochastic-gradient dynamics using neural tangent kernels, mean-field limits, random matrix theory, and loss-landscape tools, alongside mechanistic experiments on tasks such as modular arithmetic and language-model pretraining. This work informs optimizer design, learning-rate and weight-decay schedules, architectures, and training diagnostics, with phenomena such as grokking, edge-of-stability behavior, and representation collapse receiving close attention. A central challenge is connecting tractable theories and early-warning signals to causal, predictive accounts of large finite networks, where optimizer, data, architecture, and noise interact.

Latest papers

520 papers77 papers in the last four weeks, up 93% on the four weeks before. 0.4% of all new papers.

AYLA: Architecting a loss landscape in shallow neural networks to accelerate feature recovery
Apr 2, 2025Behnam Gheshlaghi, Shahin AtakishiyevShallow Neural Networks Representation Learning
Derivation of effective gradient flow equations and dynamical truncation of training data in Deep Learning
Jan 13, 2025Thomas ChenNeural Network Interpretability Neural Network Training Dynamics
Improving the adaptive and continuous learning capabilities of artificial neural networks: Lessons from multi-neuromodulatory dynamics
Jan 12, 2025Jie Mei, Alejandro Rodriguez-Garcia, Daigo Takeuchi +4Continual Learning Neural Network Training Dynamics
IT:U Interdisciplinary Transformation University Austria, Linz, Austria · International Research Center for Neurointelligence, The University of Tokyo, Tokyo, Japan · Department of Anatomy, University of Quebec in Trois-Rivieres, Trois-Rivieres, QC, Canada +8
Network Dynamics-Based Framework for Understanding Deep Neural Networks
Jan 5, 2025Yuchen Lin, Yong Zhang, Sihan Feng +1Neural Network Generalization Neural Network Training Dynamics
Department of Physics, Xiamen University, Xiamen 361005, China · 2Lanzhou Center for Theoretical Physics, Lanzhou University, Lanzhou 730000, China · 3Shenzhen Angel FOF Management Co., Ltd., Shenzhen 518000, China
ExpTest: Loss-Curve Hypothesis Testing for Autonomous Learning-Rate Selection in Deep Neural Networks
Nov 25, 2024Zan Chaudhry, Naoko MizunoLearning Rate Scheduling Neural Network Training Dynamics
Laboratory of Structural Cell Biology, National Institutes of Health, Bethesda, MD, 20892 USA · Ragon Institute, Cambridge, MA, 02139
A Mechanistic Study of Transformers Training Dynamics
Oct 31, 2024Ambroise Odonnat, Wassim Bouaziz, Vivien CabannesTransformer Interpretability Attention Mechanisms
Inria, Univ. Rennes 2 · Mistral AI; work done while at Meta. · FAIR at Meta.
How Learning Dynamics Drive Adversarially Robust Generalization?
Oct 10, 2024Yuelin Xu, Xiao ZhangAdversarial Training PAC-Bayesian Generalization Bounds
CISPA Helmholtz Center for Information Security, Saarbrücken, Germany
A Functional-Space Mean-Field Theory of Partially-Trained Three-Layer Neural Networks
Oct 28, 2022Zhengdao Chen, Eric Vanden-Eijnden, Joan BrunaNeural Network Generalization Neural Network Optimization
Google Research Mountain View, CA 94043 · Courant Institute, New York University New York, NY 10003
Deep Residual Networks Learn the Geodesic Curve in the Wasserstein Space
Feb 18, 2021Kuo Gai, Shihua ZhangResidual Learning Neural Network Training Dynamics
Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS) Shanghai 200433, China · Academy of Mathematics and Systems Science Chinese Academy of Sciences Beijing 100190, China · School of Mathematical Sciences2026 University of Chinese Academy of Sciences Beijing 100049, China
Theory of the Frequency Principle for General Deep Neural Networks
Jun 21, 2019Tao Luo, Zheng Ma, Zhi-Qin John Xu +1Frequency-Domain Feature Learning Neural Network Training Dynamics
Department of Mathematics, Zheng Ma Purdue University · Department of Mathematics, Purdue University · New York University Abu Dhabi and Courant Institute

Apr 2, 2025J/K move · Enter open · S save

AYLA: Architecting a loss landscape in shallow neural networks to accelerate feature recovery

Behnam Gheshlaghi, Shahin Atakishiyev

Shallow Neural Networks Representation Learning Neural Network Optimization Neural Network Training Dynamics

Open paper AskPDF arXiv

Feature learning in shallow neural networks exhibits rich yet fragile dynamics, including prolonged plateaus, abrupt phase transitions, and sensitivity to optimization hyperparameters. While recent theoretical work has characterized these behaviors through the geometry of loss landscapes, saddle escape mechanisms, and emergent scaling laws, practical methods for actively shaping these dynamics remain limited. In this paper, we introduce AYLA, a principled loss reparameterization framework that dynamically modulates gradient magnitudes during training without altering the location of stationary points or optimal solutions. AYLA applies a smooth, sigmoid-controlled power-law transformation to empirical loss, yielding a state-dependent effective learning rate that accelerates descent in flat or saddle-dominated regions while stabilizing late-stage optimization. Crucially, AYLA preserves all critical points of the original objective, acting solely as a monotone transformation that reshapes optimization trajectories rather than objectives. We evaluate AYLA in controlled teacher student settings using two-layer tanh networks trained on synthetic Gaussian data. Across stochastic gradient descent and multiple loss-exponent schedules, AYLA consistently improves feature recovery. This evidence is observed in terms of weight alignment, per-neuron cosine similarity, hidden-activation correlation, and spectral properties of learned representations, while AYLA maintains competitive or faster loss convergence. Spectral analyses further demonstrate that AYLA mitigates rank collapse and promotes richer internal representations, signaling a transition from lazy to active feature-learning regimes. AYLA offers a lightweight, theoretically grounded way to improve shallow-network optimization, especially in resource-limited or noise-sensitive settings.