We establish convergence of the training dynamics of residual neural networks (ResNets) to their joint infinite depth L, hidden width M, and embedding dimension D limit. Specifically, we consider ResNets with two-layer perceptron blocks in the maximal local feature update (MLU) regime and prove that, after a bounded number of training steps, the error between the ResNet and its large-scale limit is O(1/L + sqrt(D/(L M)) + 1/sqrt(D)). This error rate is empirically tight when measured in embedding space. For a budget of P = Theta(L M D) parameters, this yields a convergence rate O(P^(-1/6)) for the scalings of (L, M, D) that minimize the bound. Our analysis exploits in an essential way the depth-two structure of residual blocks and applies formally to a broad class of state-of-the-art architectures, including Transformers with bounded key-query dimension. From a technical viewpoint, this work completes the program initiated in the companion paper [Chi25] where it is proved that for a fixed embedding dimension D, the training dynamics converges to a Mean ODE dynamics at rate O(1/L + sqrt(D)/sqrt(L M)). Here, we study the large-D limit of this Mean ODE model and establish convergence at rate O(1/sqrt(D)), yielding the above bound by a triangle inequality. To handle the rich probabilistic structure of the limit dynamics and obtain one of the first rigorous quantitative convergence for a DMFT-type limit, we combine the cavity method with propagation of chaos arguments at a functional level on so-called skeleton maps, which express the weight updates as functions of CLT-type sums from the past.
Figures & tables
Figure 1: Comparison of the experimental convergence rate of the hidden representation hkL with the theoretical upper-bound ∥[MLαD,Dβ]∥2 from Theorem 1.1 with α=0.8 and β=2.5 manually adjusted to fit observations (plain lines). The y-axis shows RMS error (averaged over 5 random repetitions and over the dataset) on the last hidden state hkL after k=15 GD steps.
Figure 2: Comparison of the experimental convergence rate of the output yk with the conjectured rate ∥[MLαD,Dβ]∥2 (which is smaller than the rate from Theorem 1.1 ) with coefficients α=0.15 and β=0.9 manually adjusted to fit observations (plain lines). The y-axis shows RMS error (averaged over 5 random repetitions and over the dataset) on the output yk after k=15 GD steps.
Figure 3: Histograms for the coordinates of the forward pass in a finite linear Resnet ( L=200 , M=D=1000 ), compared to the limit model, at training iterations k∈{0,4,8} . The red lines represent the pdf of the Gaussian random variable Hk(ℓ/L) . The right plot shows values trajectories across depth of a few embedding coordinates d , comparing them to the coupled samples of the limit stochastic process ℓ↦Hkd(ℓ/L) .
Figure 4: Forward Pass RMS error after k=5 GD steps, Δ5h , in the linear case ρ(s)=s , between finite ResNets of variable size, and the true limit model. Dotted lines display the function ∥[0.67MLD,0.44D1]∥2 obtained from fitting our error rate to the data. The error term O(L1) is negligible in our regime. The y-axis shows RMS error (averaged over 60 random repetitions with a single datapoint). In the last plot, we overlay the predicted scaling of Θ(P−1/6)
Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.
Deep neural networks (DNNs) have significantly advanced machine learning, with model depth playing a central role in their successes. The dynamical system modeling approach has recently emerged as a powerful framework, offering new mathematical insights into the structure and learning behavior of DNNs. In this work, we establish generalization error bounds for both discrete- and continuous-time residual networks (ResNets) by combining Rademacher complexity, flow maps of dynamical systems, and the convergence behavior of ResNets in the deep-layer limit. The resulting bounds are of order O(1/S) with respect to the number of training samples S, and include a structure-dependent negative term, yielding depth-uniform and asymptotic generalization bounds under milder assumptions. These findings provide a unified understanding of generalization across both discrete- and continuous-time ResNets, helping to close the gap in both the order of sample complexity and assumptions between the discrete- and continuous-time settings.
Jinshu Huang, Mingfei Sun, Chunlin Wu
School of Mathematical Sciences, Nankai University, Tianjin, China
We analyze the universal approximation constraints of narrow Residual Neural Networks (ResNets) both theoretically and numerically. For deep neural networks without input space augmentation, a central constraint is the inability to represent critical points of the input-output map. We prove that this has global consequences for target function approximations and show that the manifestation of this defect is typically a shift of the critical point to infinity, which we call the ``tunnel effect'' in the context of classification tasks. While ResNets offer greater expressivity than standard multilayer perceptrons (MLPs), their capability strongly depends on the signal ratio between the skip and residual channels. We establish quantitative approximation bounds for both the residual-dominant (close to MLP) and skip-dominant (close to neural ODE) regimes. These estimates depend explicitly on the channel ratio and uniform network weight bounds. Low-dimensional examples further provide a detailed analysis of the different ResNet regimes and how architecture-target incompatibility influences the approximation error.
Christian Kuehn, Sara-Viola Kuntz, Tobias Wöhrer
Technical University of Munich, School of Computation, Information and Technology, Department of Mathematics, Boltzmannstraße 3, 85748 Garching, Germany · Munich Data Science Institute (MDSI), Garching, Germany · Munich Center for Machine Learning (MCML), München, Germany +1
Training dynamics is central to understanding neural networks, yet its theoretical analysis remains difficult even for simple architectures and becomes substantially more challenging for general modern architectures. In this paper, we propose a convergence framework for analyzing gradient descent (GD) dynamics under a broad family of neural network architectures and datasets beyond the neural tangent kernel (NTK) regime. The framework is formulated at the level of network blocks and covers architectures including pre-normalized multi-layer transformers. More precisely, under mild assumptions, we prove that for almost all initializations, GD with regular learning rates converges to the neighbourhood of a stationary point. This is mainly proved by establishing an iterate-dependent PL-type inequality through analyticity and measure-zero arguments, and by proving Lipschitz smoothness along the GD trajectory through polynomial generalized smoothness and a local relaxed dissipative condition. We further interpret the theorem under Xavier initialization and practical architectural scaling, showing that the learning rate scale depends on the depth and effective bottleneck dimensions rather than the largest width. Finally, we derive structural nondegeneracy implications for residual connections and function composition, and provide a generic characterization of global minimizers within our framework.