Neural Network Training Dynamics

Follow this topic Ask about this topic

Momentum

41 papers in the last four weeks, up 273% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 349

Jul 11, 2025cs.LG

Understanding Two-Layer Neural Networks with Smooth Activation Functions

This paper aims to understand the training solution, which is obtained by the back-propagation algorithm, of two-layer neural networks whose hidden layer is composed of the units with smooth activation functions, including the usual sigmoid type most commonly used before the advent of ReLUs. The mechanism contains four main principles: construction of Taylor series expansions, strict partial order of knots, smooth-spline implementation and smooth-continuity restriction. The universal approximation for arbitrary input dimensionality is proved and the explanation of training solutions is given. Through the principles proposed, the mystery of ``black box'' of the solution space is largely revealed. The new proofs employed also enrich approximation theory.

Changcun Huang

Shuitu Institute of Applied Mathematics, Chongqing 400700, P.R.C

Neural Network Activation Functions Neural Network Approximation Theory Neural Network Training Dynamics Neural Network Expressivity

Jul 10, 2025cond-mat.dis-nn

A statistical physics framework for optimal learning

Learning is a complex dynamical process shaped by a range of interconnected decisions. Careful design of hyperparameter schedules for artificial neural networks or efficient allocation of cognitive resources by biological learners can dramatically affect performance. Yet, theoretical understanding of optimal learning strategies remains sparse, especially due to the intricate interplay between evolving metaparameters and nonlinear learning dynamics. The search for optimal protocols is further hindered by the high dimensionality of the learning space, often resulting in predominantly heuristic, difficult to interpret, and computationally demanding solutions. Here, we combine statistical physics with control theory in a unified theoretical framework to identify optimal learning protocols in prototypical neural network models. In the high-dimensional limit, we derive closed-form ordinary differential equations that track online stochastic gradient descent through low-dimensional order parameters. We formulate the design of learning protocols as an optimal control problem directly on the dynamics of the order parameters with the goal of minimizing the generalization error. This formulation encompasses a variety of learning scenarios, optimization constraints, and control budgets. We apply it to representative cases, including optimal curricula, adaptive dropout regularization and noise schedules in denoising autoencoders. We find nontrivial yet interpretable strategies highlighting how optimal protocols mediate learning trade-offs. Our results establish a principled foundation for understanding and designing optimal protocols and suggest a path toward a theory of meta-learning grounded in statistical physics.

Francesca Mignacco, Francesco Mori

1Joseph Henry Laboratories of Physics, Princeton University, Princeton, NJ 08544, USA · 2Graduate Center, City University of New York, New York, NY 10016, USA · Center of Mathematical Sciences and Applications, Harvard University, Cambridge, MA 02138, USA

Statistical Physics of Learning Neural Network Training Dynamics Statistical Learning Theory Optimal Control Hyperparameter Optimization

Jun 16, 2025stat.ML

Random Matrix Theory for Deep Learning: Beyond Eigenvalues of Linear Models

Modern Machine Learning (ML) and Deep Neural Networks (DNNs) often operate on high-dimensional data and rely on overparameterized models, where classical low-dimensional intuitions break down. In particular, the proportional regime where the data dimension, sample size, and number of model parameters are all large and comparable, gives rise to novel and sometimes counterintuitive behaviors. This paper extends traditional Random Matrix Theory (RMT) beyond eigenvalue-based analysis of linear models to address the challenges posed by nonlinear ML models such as DNNs in this regime. We introduce the concept of High-dimensional Equivalent, which unifies and generalizes both Deterministic Equivalent and Linear Equivalent, to systematically address three technical challenges: high dimensionality, nonlinearity, and the need to analyze generic eigenspectral functionals. Leveraging this framework, we provide precise characterizations of the training and generalization performance of linear models, nonlinear shallow networks, and deep networks. Our results capture rich phenomena, including scaling laws, double descent, and nonlinear learning dynamics, offering a unified perspective on the theoretical understanding of deep learning in high dimensions.

Zhenyu Liao, Michael W. Mahoney

School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan, China · ICSI, LBNL, and Department of Statistics, University of California, Berkeley, USA

Random Matrix Theory Neural Network Generalization Neural Network Training Dynamics Neural Scaling Laws

May 23, 2025cs.LG

KITINet: KInetic Theory Inspired Inter-Channel Information Exchange for Residual Networks

Residual connections are foundational to modern neural networks, yet the standard additive update provides no explicit mechanism for structured inter-channel interaction. We introduce KITINet (KInetics Theory Inspired Network), a training-time module with no additional trainable parameters that augments residual updates with stochastic pairwise feature interactions inspired by collision sampling in kinetic theory. KITINet reshapes channel groups as particles and mixes their features according to relative distance and velocity, while leaving the underlying residual layer trainable in the usual way. The module is disabled at inference, so the deployed model is identical to the original backbone and incurs no additional inference cost. Across language pre-training and fine-tuning, image classification, and PDE operator learning, KITINet demonstrates broad and robust performance gains across diverse tasks and architectures. We further observe enhanced parameter condensation across several settings and develop a simplified stochastic analysis that provides a mechanistic account of how collision-inspired interactions can promote condensation dynamics.

Mingquan Feng, Yifan Fu, Tongcheng Zhang +3

Shanghai Jiao Tong University

Feature Interaction Modeling Residual Learning Neural Network Training Dynamics

May 20, 2025cs.LG

KO: Kinetics-inspired Neural Optimizer with PDE Simulation Approaches

The design of effective optimization algorithms for neural networks remains a fundamental challenge, and most existing methods rely on heuristic extensions of gradient-based updates. We introduce KO (Kinetics-inspired Optimizer), a plug-and-play optimization module grounded in kinetic theory and partial differential equations. KO models parameter dynamics as a particle system, augmenting standard gradient updates with stochastic interactions induced by a discretization of the Boltzmann transport equation. This mechanism naturally promotes parameter diversity and mitigates weight condensation, the tendency of parameters to collapse into low-dimensional subspaces, a phenomenon closely associated with degraded generalization. We provide both a rigorous theoretical analysis and a physical interpretation, showing that KO provably increases parameter diversity while preserving convergence guarantees. Extensive experiments on image classification benchmarks (CIFAR-10/100, ImageNet) and large-scale language model pretraining demonstrate that KO consistently improves accuracy over competitive baselines with negligible additional computational cost.

Mingquan Feng, Yixin Huang, Yifan Fu +2

Shanghai Jiao Tong University

Neural Network Optimization Neural Network Training Dynamics

May 2, 2025cs.LG

Silent Neuron Theory and Plasticity Preservation for Deep Reinforcement Learning in Adaptive Video Streaming

Adaptive video streaming optimizes Quality of Experience (QoE) metrics by selecting appropriate bitrates according to varying network bandwidth and user demands. In practice, however, real-world network bandwidth often exhibits heterogeneity relative to training environments. Current methods predominantly tackle this problem through learning-based approaches designed to improve generalization performance. While our systematic investigation reveals a critical limitation: neural networks suffer from plasticity loss, significantly impeding their ability to adapt to heterogeneous network conditions. Through theoretical analysis of neural propagation mechanisms, we demonstrate that existing dormant neuron metrics inadequately characterize neural plasticity loss. To address this limitation, we have developed the Silent Neuron theory, which provides a more comprehensive framework for understanding plasticity degradation. Based on these theoretical insights, we propose the Reset Silent Neuron (ReSiN), which preserves neural plasticity through strategic neuron resets guided by both forward and backward propagation states. Moreover, we establish a tighter performance bound for ReSiN under non-stationary network conditions. In our implementation of an adaptive video streaming system, ReSiN has shown significant improvements over existing solutions, achieving up to 168% higher bitrate and 108% better quality of experience (QoE) while maintaining comparable smoothness. Furthermore, ReSiN consistently outperforms in stationary environments, demonstrating its robust adaptability across different network conditions.

Zhiqiang He, Zhi Liu

Department of Computer and Network Engineering, the University of Electro-Communications, Japan.

Non-Stationary RL Neural Network Training Dynamics

Apr 23, 2025cs.LG

The Dynamics of Generalization in Deep Learning

We derive a differential equation that governs the evolution of the generalization gap when a model is trained by gradient descent-based methods. This differential equation is driven by two key quantities, a contraction factor that brings together trajectories corresponding to slightly different datasets, and a perturbation factor that accounts for them training on different datasets. The coupled decay of contraction and perturbation guarantees a controlled accumulation of generalization gap during training. We analyze this differential equation to show that the generalization gap is given by a quadratic form that consists of an ``effective Gram matrix'' that depends upon the training trajectory and a certain residual of the predictor at initialization. Our framework is applicable to general deep networks and smooth loss functions. In numerical experiments on different neural network architectures, datasets and sample sizes, we show that this quadratic form accurately captures the actual generalization gap. We also show how to instantiate our framework in a number of examples via analytical calculations. For example, for high-dimensional linear regression, our framework matches existing calculations of generalization gap in the literature exactly in under-parameterized, over-parameterized and critical regimes.

Rubing Yang, Pratik Chaudhari

Department of Applied Mathematics and Computational Sciences, University of Pennsylvania Philadelphia, PA, 19104, USA · Department of Electrical and Systems Engineering, University of Pennsylvania Philadelphia, PA, 19104, USA

Neural Network Generalization Neural Network Training Dynamics Statistical Learning Theory Gradient Descent Dynamics

Apr 6, 2025stat.ML

Hyperflux: Pruning Reveals Importance

Network pruning is used to reduce inference latency and power consumption in large neural networks. However, most methods focus on empirical results at the expense of understanding the pruning process. We introduce Hyperflux, a novel

L_0

method which models pruning as a continuously evolving system determined by flux, the gradient response to a weight's removal, and pressure, a global regularization driving weights toward pruning. By exploiting this model, Hyperflux's pruning behavior becomes understandable at both microscopic (weight regrowth/pruning) and macroscopic (sparsity convergence, etc.) levels. We also introduce a novel pressure scheduler that reliably targets desired sparsities. Hyperflux achieves competitive results with ResNet-50, VGG-19 and DeiT-T/S on CIFAR-10, CIFAR-100 and ImageNet datasets.

Eugen Barbulescu, Antonio Alexoaie, Lucian Busoniu

Department of Computer Science Technical University of Cluj-Napoca · Department of Automation Technical University of Cluj-Napoca

Sparse Neural Networks Neural Network Training Dynamics Neural Network Pruning

Apr 2, 2025cs.LG

AYLA: Architecting a loss landscape in shallow neural networks to accelerate feature recovery

Feature learning in shallow neural networks exhibits rich yet fragile dynamics, including prolonged plateaus, abrupt phase transitions, and sensitivity to optimization hyperparameters. While recent theoretical work has characterized these behaviors through the geometry of loss landscapes, saddle escape mechanisms, and emergent scaling laws, practical methods for actively shaping these dynamics remain limited. In this paper, we introduce AYLA, a principled loss reparameterization framework that dynamically modulates gradient magnitudes during training without altering the location of stationary points or optimal solutions. AYLA applies a smooth, sigmoid-controlled power-law transformation to empirical loss, yielding a state-dependent effective learning rate that accelerates descent in flat or saddle-dominated regions while stabilizing late-stage optimization. Crucially, AYLA preserves all critical points of the original objective, acting solely as a monotone transformation that reshapes optimization trajectories rather than objectives. We evaluate AYLA in controlled teacher student settings using two-layer tanh networks trained on synthetic Gaussian data. Across stochastic gradient descent and multiple loss-exponent schedules, AYLA consistently improves feature recovery. This evidence is observed in terms of weight alignment, per-neuron cosine similarity, hidden-activation correlation, and spectral properties of learned representations, while AYLA maintains competitive or faster loss convergence. Spectral analyses further demonstrate that AYLA mitigates rank collapse and promotes richer internal representations, signaling a transition from lazy to active feature-learning regimes. AYLA offers a lightweight, theoretically grounded way to improve shallow-network optimization, especially in resource-limited or noise-sensitive settings.

Behnam Gheshlaghi, Shahin Atakishiyev

Shallow Neural Networks Representation Learning Neural Network Optimization Neural Network Training Dynamics

Jan 13, 2025cs.LG

Derivation of effective gradient flow equations and dynamical truncation of training data in Deep Learning

We derive explicit equations governing the cumulative biases and weights in Deep Learning with ReLU activation function, based on gradient descent for the Euclidean loss in the input layer, and under the assumption that the weights are, in a precise sense, adapted to the coordinate system distinguished by the activations. We show that gradient descent corresponds to a dynamical process in the input layer, whereby clusters of data are progressively reduced in complexity ("truncated") at an exponential rate that increases with the number of data points that have already been truncated. We provide a detailed discussion of several types of solutions to the gradient flow equations. A main motivation for this work is to shed light on the interpretability question in supervised learning.

Thomas Chen

Neural Network Interpretability Neural Network Training Dynamics Gradient Descent Dynamics

Jan 12, 2025q-bio.NC

Improving the adaptive and continuous learning capabilities of artificial neural networks: Lessons from multi-neuromodulatory dynamics

Continuous adaptive learning, the ability to adapt to the environment and keep improving performance, is a hallmark of natural intelligence. Biological organisms excel in acquiring, transferring, and retaining knowledge while adapting to volatile environments, making them a source of inspiration for artificial neural networks (ANNs). This study explores how neuromodulation, a building block of learning in biological systems, can help address catastrophic forgetting and enhance the robustness of ANNs in continual learning. Driven by neuromodulators including dopamine (DA), acetylcholine (ACh), serotonin (5-HT) and noradrenaline (NA), neuromodulatory processes in the brain operate at multiple scales, facilitating dynamic responses to environmental changes through mechanisms ranging from local synaptic plasticity to global network-wide adaptability. Importantly, the relationship between neuromodulators and their interplay in modulating sensory and cognitive processes is more complex than previously expected, demonstrating a "many-to-many" neuromodulator-to-task mapping. To inspire neuromodulation-aware learning rules, we highlight (i) how multi-neuromodulatory interactions enrich single-neuromodulator-driven learning, (ii) the impact of neuromodulators across multiple spatio-temporal scales, and correspondingly, (iii) strategies for approximating and integrating neuromodulated learning processes in ANNs, and (iv) an architectural-general formulation of multi-neuromodulatory dynamics. We also present a conceptual study to showcase how neuromodulation-inspired mechanisms, such as DA-driven reward processing and NA-based cognitive flexibility, can enhance ANN performance in a Go/No-Go task. Though multi-scale neuromodulation, we aim to bridge the gap between biological and artificial learning, paving the way for ANNs with greater flexibility, robustness, and adaptability.

Jie Mei, Alejandro Rodriguez-Garcia, Daigo Takeuchi +4

IT:U Interdisciplinary Transformation University Austria, Linz, Austria · International Research Center for Neurointelligence, The University of Tokyo, Tokyo, Japan · Department of Anatomy, University of Quebec in Trois-Rivieres, Trois-Rivieres, QC, Canada +8

Continual Learning Neural Network Training Dynamics Adaptive Learning Catastrophic Forgetting

Jan 5, 2025cs.LG

Network Dynamics-Based Framework for Understanding Deep Neural Networks

Advancements in artificial intelligence call for a deeper understanding of the fundamental mechanisms underlying deep learning. In this work, we propose a theoretical framework to analyze learning dynamics through the lens of dynamical systems theory. We redefine the notions of linearity and nonlinearity in neural networks by introducing two fundamental transformation units at the neuron level: order-preserving transformations and non-order-preserving transformations. Different transformation modes lead to distinct collective behaviors in weight vector organization, different modes of information extraction, and the emergence of qualitatively different learning phases. Transitions between these phases may occur during training, accounting for key phenomena such as grokking. To further characterize generalization and structural stability, we introduce the concept of attraction basins in both sample and weight spaces. The distribution of neurons with different transformation modes across layers, along with the structural characteristics of the two types of attraction basins, forms a set of core metrics for analyzing the performance of learning models. Hyperparameters such as depth, width, learning rate, and batch size act as control variables for fine-tuning these metrics. Our framework not only sheds light on the intrinsic advantages of deep learning, but also provides a novel perspective for optimizing network architectures and training strategies.

Yuchen Lin, Yong Zhang, Sihan Feng +1

Department of Physics, Xiamen University, Xiamen 361005, China · 2Lanzhou Center for Theoretical Physics, Lanzhou University, Lanzhou 730000, China · 3Shenzhen Angel FOF Management Co., Ltd., Shenzhen 518000, China

Neural Network Generalization Neural Network Training Dynamics

Oct 31, 2024cs.LG

A Mechanistic Study of Transformers Training Dynamics

Large-scale pretraining of transformers has been central to the success of foundation models. However, the scale of those models limits our understanding of the mechanisms at play during optimization. In this work, we study the training dynamics of transformers in a controlled and interpretable setting. On the sparse modular addition task, we demonstrate that specialized attention circuits, called clustering heads, can be implemented during gradient descent to solve the problem. Our experiments show that such pathways naturally emerge during training. By monitoring the evolution of tokens via a visual sandbox, we uncover a two-stage learning and the occurrences of loss spikes due to the high curvature of normalization layers. Our findings provide several insights into patterns observed in more practical settings, such as the pretraining of large language models.

Ambroise Odonnat, Wassim Bouaziz, Vivien Cabannes

Inria, Univ. Rennes 2 · Mistral AI; work done while at Meta. · FAIR at Meta.

Transformer Interpretability Attention Mechanisms Neural Network Training Dynamics Attention Head Specialization

Oct 10, 2024cs.LG

How Learning Dynamics Drive Adversarially Robust Generalization?

Despite being widely adopted as a canonical framework for learning robust models, adversarial training suffers from robust overfitting. Existing empirical and theoretical explorations fail to provide a satisfactory mechanistic interpretation of the phenomenon. By modeling adversarial training with momentum SGD as a discrete-time dynamical system, we propose a PAC-Bayesian analytical framework that proves time-resolved robust generalization bounds. Specifically, our framework tracks the closed-form evolution of the posterior mean and covariance under both stationary and non-stationary transient regimes, connecting the model's robust generalization performance to learning rate, local loss geometry, and mini-batch stochastic gradients. By estimating the key quantities associated with the bound, we illustrate the underlying mechanism of robust overfitting. Our framework also shows how adversarial weight perturbation reduces robust generalization gaps by suppressing dominant loss-curvature modes, while suggesting that excessive penalization can be sub-optimal for optimization.

Yuelin Xu, Xiao Zhang

CISPA Helmholtz Center for Information Security, Saarbrücken, Germany

Adversarial Training PAC-Bayesian Generalization Bounds Neural Network Robustness Adversarial Robustness Neural Network Training Dynamics

Oct 28, 2022cs.LG

A Functional-Space Mean-Field Theory of Partially-Trained Three-Layer Neural Networks

To understand the training dynamics of neural networks, prior studies have considered the mean-field limit of two-layer neural networks as the width tends to infinity, establishing theoretical guarantees for its convergence under gradient flow training as well as approximation and generalization capabilities. In this work, we study the infinite-width limit of a type of three-layer neural network where the first-layer weights are randomly sampled and untrained. To rigorously define the limiting model, we extend the mean-field theory by lifting the representation of neurons from Euclidean to functional spaces. This allows us to establish the mean-field training dynamics as a functional gradient flow with a time-varying kernel that remains positive-definite under suitable assumptions, thus proving a linear-rate convergence of its training loss. Furthermore, we define novel function spaces that contain the solutions obtained through the mean-field training dynamics and prove Rademacher complexity bounds for these spaces. Notably, our analysis applies to a range of scaling choices of the model, resulting in two distinct regimes of the mean-field limit that both exhibit feature learning through training.

Zhengdao Chen, Eric Vanden-Eijnden, Joan Bruna

Google Research Mountain View, CA 94043 · Courant Institute, New York University New York, NY 10003

Neural Network Generalization Neural Network Optimization Mean-Field Theory Neural Network Training Dynamics

Feb 18, 2021cs.LG

Deep Residual Networks Learn the Geodesic Curve in the Wasserstein Space

Recent studies revealed the mathematical connection between deep neural networks (DNNs) and dynamic systems. However, the specific dynamics that DNNs, especially deep residual networks (ResNets), tend to learn during training remain insufficiently characterized. To this end, we model the forward propagation of deep residual networks using continuity equations, in which the measure is conserved and infinite curves in the measure space connect the input distribution to the output one of a ResNet. We find ResNets with

L_2

regularization attempt to learn the geodesic curve in the Wasserstein space, induced by the optimal transport map. Compared with plain networks, ResNets can better approximate the geodesic curve, which explains why ResNets can be optimized and generalize better. Numerical experiments show that the data tracks of a ResNet tend to be line-shaped in terms of the line-shape score, and the map learned by a ResNet is closer to the optimal transport map in terms of the optimal transport score. In a word, we conclude that ResNets learn the geodesic curve in the Wasserstein space and discretely engineer the data transformation in high-dimensional spaces.

Kuo Gai, Shihua Zhang

Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS) Shanghai 200433, China · Academy of Mathematics and Systems Science Chinese Academy of Sciences Beijing 100190, China · School of Mathematical Sciences2026 University of Chinese Academy of Sciences Beijing 100049, China

Residual Learning Neural Network Training Dynamics Optimal Transport

Jun 21, 2019cs.LG

Theory of the Frequency Principle for General Deep Neural Networks

Along with fruitful applications of Deep Neural Networks (DNNs) to realistic problems, recently, some empirical studies of DNNs reported a universal phenomenon of Frequency Principle (F-Principle): a DNN tends to learn a target function from low to high frequencies during the training. The F-Principle has been very useful in providing both qualitative and quantitative understandings of DNNs. In this paper, we rigorously investigate the F-Principle for the training dynamics of a general DNN at three stages: initial stage, intermediate stage, and final stage. For each stage, a theorem is provided in terms of proper quantities characterizing the F-Principle. Our results are general in the sense that they work for multilayer networks with general activation functions, population densities of data, and a large class of loss functions. Our work lays a theoretical foundation of the F-Principle for a better understanding of the training process of DNNs.

Tao Luo, Zheng Ma, Zhi-Qin John Xu +1

Department of Mathematics, Zheng Ma Purdue University · Department of Mathematics, Purdue University · New York University Abu Dhabi and Courant Institute

Frequency-Domain Feature Learning Neural Network Training Dynamics

Date pendingcs.LG

ExpTest: Loss-Curve Hypothesis Testing for Autonomous Learning-Rate Selection in Deep Neural Networks

Hyperparameter tuning remains a significant challenge in the training of deep neural networks (DNNs), requiring manual search or time-intensive grid searches that increase resource costs and limit the accessibility of machine learning. The global initial learning rate is among the most consequential of these hyperparameters. Adaptive and scheduling-based methods manage the learning rate during training but still require manual selection of an initial global value; learning-rate-free alternatives remove this selection at the cost of performance or stability on non-convex problems. We present ExpTest, an autonomous learning-rate controller that treats the training loss curve as an online signal and performs sequential statistical tests on theoretically motivated windows to detect convergent behavior and trigger learning-rate reductions. The framework combines a covariance-based initial learning-rate estimate, curvature-motivated window sizing, and two-phase test-driven decay, drawing on the approximately exponential decay behavior predicted under linearized network dynamics. We provide a mathematical motivation for ExpTest and evaluate it on regression, classification, forecasting, and natural-language tasks across fully-connected, convolutional, transformer-based, and pretrained architectures. Across these tasks, ExpTest achieves competitive performance relative to hand-tuned SGD-based baselines and recent learning-rate-free methods, without manual initial learning-rate selection or predefined scheduling.

Zan Chaudhry, Naoko Mizuno

Laboratory of Structural Cell Biology, National Institutes of Health, Bethesda, MD, 20892 USA · Ragon Institute, Cambridge, MA, 02139

Learning Rate Scheduling Neural Network Training Dynamics Hyperparameter Optimization

Date pendingcs.LG

On the Residual Scaling of Looped Transformers: Stability and Transferability

Looped (weight-tied) Transformers apply a shared residual block

N

times (

h \leftarrow h + \varepsilon\,f(h)

, same

f

at each step), increasing effective depth without adding parameters. Prior depth-scaling analyses prescribe

\varepsilon = 1/\!\sqrt{L}

for depth-

L

residual networks. We show that this is insufficient for looped architectures: weight sharing makes residual updates correlated across iterations, requiring the stronger scaling

\varepsilon = 1/N

. For multi-layer blocks (

L

unique layers looped

N

times), we derive a factored parameterization

\varepsilon = \lambda/(N\!\sqrt{L})

that separates the two sources of growth:

1/N

controls the within-layer loop correlation, and

1/\!\sqrt{L}

controls the across-layer variance. A key consequence is that the optimal learning rate depends only on the number of unique layers

L

, not on the loop count

N

, enabling direct hyperparameter transfer from small to large

N

without retuning. Experiments on looped Transformers confirm that

1/N

scaling improves trainability and yields better loss than

1/\!\sqrt{N}

scaling across loop counts.

Shaowen Wang, Bingrui Li, Ge Zhang +3

Tsinghua University · ByteDance Seed · M-A-P

Residual Learning Hyperparameter Transfer Recurrent Transformers Neural Network Training Dynamics