Width and Depth Limits Commute During Training

Ruijia Zhang; Soufiane Hayou

cs.LGOct 22, 2026

Width and Depth Limits Commute During Training

Authors: Ruijia Zhang, Soufiane Hayou

Abstract

Recent work established that the infinite-width and infinite-depth limits of residual networks commute at initialization when residual branches are scaled by $1/\sqrt{depth}$ . In particular, the network output converges to the same limit whether width and depth tend to infinity sequentially or jointly. We extend this commutativity result to training in linear residual networks. Under the same $1/\sqrt{depth}$ residual-branch scaling (Depth- $μ$ P) and the $μ$ P width parameterization, we prove that, for any fixed number of gradient-descent steps with gradient clipping, the network output converges in $L^2$ along every joint width--depth growth path as both width and depth tend to infinity. Consequently, the infinite-width and infinite-depth limits commute throughout any finite training horizon. Without gradient clipping, the corresponding $L^2$ convergence does not hold in general, but we establish the weaker convergence in probability.

Figures & tables

Figure 1: Commutativity of the width and depth limits. Illustration of three relative order regimes for $(n_{r},L_{r})$ as $n_{r},L_{r}\to\infty$ . For the same fixed input and any fixed training iteration $k$ , all three paths converge to the same trained output limit $f_{\infty}^{(k)}$ .

Figure 2: Commutativity of the width and depth limits during training. Each curve corresponds to a fixed width–depth scaling path and is plotted against the joint scale

m=\min(n,L)

. The top row shows the synthetic regression output at training iterations

k=1,5,10,15

, and the bottom row shows the HIGGS logit margin at four training checkpoints. For each task, all configurations are evaluated at the same fixed input. Curves show the mean across 5 model seeds, with shaded regions denoting

\pm 1

standard deviation.

Figure 3: Width–depth convergence rates. (a) RMS output error as width

n

increases with depth fixed at

L=12{,}288

. (b) RMS output error as depth

L

increases with width fixed at

n=12{,}288

. Curves in (a)–(b) correspond to training iterations

k\in\{1,5,10,15\}

. (c) Empirical convergence exponents for width and depth over training. The width rate remains close to

n^{-1/2}

, while the depth rate approaches an empirical

L^{-1}

rate.

Figure 4: Width–depth commutativity with nonlinear activations. Rows correspond to activation functions and columns to training iterations. Each panel shows the output at synthetic training input

86

along the five width–depth scaling paths in Section 5.1 . Curves show the mean over five model seeds, with shaded regions denoting

\pm 1

standard deviation.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

$k$	$\alpha_{n}(k)$	$95\%$ CI	$\alpha_{L}(k)$	$95\%$ CI
$1$	$0.427$	$[0.360,0.492]$	$0.738$	$[0.671,0.807]$
$2$	$0.441$	$[0.368,0.508]$	$0.789$	$[0.749,0.837]$
$3$	$0.434$	$[0.369,0.493]$	$0.898$	$[0.870,0.929]$
$4$	$0.437$	$[0.364,0.497]$	$0.979$	$[0.943,1.017]$
$5$	$0.425$	$[0.350,0.498]$	$0.899$	$[0.866,0.936]$
$6$	$0.416$	$[0.335,0.497]$	$0.932$	$[0.898,0.971]$

Appendix

Table 1: Empirical convergence exponents across training. Confidence intervals are

95\%

model-seed bootstrap intervals.

Figure 5: Synthetic width–depth commutativity across all saved iterations. Each panel shows the output for training input

86

at one iteration

k=0,\ldots,15

. Curves correspond to the five width–depth scaling paths in Section 5.1 and show means over five model seeds, with shaded regions denoting

\pm 1

standard deviation.

Figure 6: Synthetic width–depth commutativity at additional training inputs. Rows correspond to four fixed training inputs, and columns show the representative iterations in Section 5.1 . Curves correspond to the same five width–depth scaling paths and show means over five model seeds, with shaded regions denoting

\pm 1

standard deviation.

Figure 7: HIGGS width–depth commutativity across all saved checkpoints. Each panel shows the class-

1

versus class-

0

logit margin for the fixed test example used in Section 5.1 , from initialization through two training epochs. Curves show means over five model seeds, with shaded regions denoting

\pm 1

standard deviation, over the common scales

m\in\{16,32,64,128,256\}

Figure 8: HIGGS width–depth commutativity at additional test examples. Rows correspond to three fixed test examples, and columns show the representative checkpoints in Section 5.1 . Curves correspond to the same five width–depth scaling paths and show means over five model seeds, with shaded regions denoting

\pm 1

standard deviation.

Figure 9: Width–depth commutativity in the single-datum setting. Each panel shows the trained scalar output at one training iteration along the five width–depth scaling paths. Curves show the mean over five model seeds, with shaded regions denoting

\pm 1

standard deviation. The paths exhibit substantial finite-scale differences at small

m

, but these differences diminish as both width and depth increase.

Experiment	Maximum $(n,L)$	Backend	Dense memory	Peak memory
Single datum	$(61{,}440,3{,}072)$	lazy	$46.4$ TB	$31.7$ GB
Synthetic commutativity	$(61{,}440,3{,}072)$	lazy	$46.4$ TB	$46.9$ GB
Linear rate sweeps	$(12{,}288,12{,}288)$	lazy	$7.42$ TB	$25.4$ GB
HIGGS commutativity	$(5{,}120,256)$	dense	$26.8$ GB	$26.9$ GB
Nonlinear activations	$(5{,}120,256)$	dense	$26.8$ GB	$27.6$ GB

Appendix

Table 2: Memory usage for the reported experiments. Dense memory is the float32 storage required to materialize all residual matrices; peak memory is the maximum recorded CUDA allocation.

Neural Network Training Dynamics

Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.

See all 507 papers

Mar 18, 2026

ResNets of All Shapes and Sizes: Convergence of Training Dynamics in the Large-scale Limit

We establish convergence of the training dynamics of residual neural networks (ResNets) to their joint infinite depth L, hidden width M, and embedding dimension D limit. Specifically, we consider ResNets with two-layer perceptron blocks in the maximal local feature update (MLU) regime and prove that, after a bounded number of training steps, the error between the ResNet and its large-scale limit is O(1/L + sqrt(D/(L M)) + 1/sqrt(D)). This error rate is empirically tight when measured in embedding space. For a budget of P = Theta(L M D) parameters, this yields a convergence rate O(P^(-1/6)) for the scalings of (L, M, D) that minimize the bound. Our analysis exploits in an essential way the depth-two structure of residual blocks and applies formally to a broad class of state-of-the-art architectures, including Transformers with bounded key-query dimension. From a technical viewpoint, this work completes the program initiated in the companion paper [Chi25] where it is proved that for a fixed embedding dimension D, the training dynamics converges to a Mean ODE dynamics at rate O(1/L + sqrt(D)/sqrt(L M)). Here, we study the large-D limit of this Mean ODE model and establish convergence at rate O(1/sqrt(D)), yielding the above bound by a triangle inequality. To handle the rich probabilistic structure of the limit dynamics and obtain one of the first rigorous quantitative convergence for a DMFT-type limit, we combine the cavity method with propagation of chaos arguments at a functional level on so-called skeleton maps, which express the weight updates as functions of CLT-type sums from the past.

Louis-Pierre Chaintron, Lénaïc Chizat, Javier Maass

École Polytechnique Fédérale de Lausanne (EPFL), Institute of Mathematics, 1015 Lausanne, Switzerland.

Dynamical Mean-Field Theory Neural Network Training Dynamics Residual Dynamics Learning

Ask PDF

Oct 22, 2026

Early learning in large residual networks

We study the early training dynamics of nonlinear residual networks with blocks of zero hidden layers as a model of more general residual networks. We analyze networks in a sequential limit where first width and depth diverge, then the number of training data points diverges, and finally the input dimension diverges. This is a more refined setup compared to the typical mean-field theory analysis in which depth and width diverge but the number of training datapoints and input dimension are fixed. Our main technical result is a diagrammatic computation of both the network output and the hidden representations after the few steps of gradient descent in terms of the statistics of inner products of inputs from the training set. One of our principle conceptual findings is a negative result: in the limit of large input dimension the network output can learn only a linear function of its inputs early in training since higher learned monomials are more and more suppressed in the input dimension and vanish in the large dimension limit. This suggests that mean-field limits of ResNets are surprisingly rigid in the regime of high input dimension and may not be sufficient toy models for understanding feature learning. Finally, we investigate how to ensure hyperparameter transfer for the learning rate as a function of dataset size and input dimension. We test our predictions numerically and find excellent agreement with our theory.

Vladimir Narovlansky, Boris Hanin

ORFE Department, Princeton University, Princeton, NJ 08544, USA · Racah Institute of Physics, The Hebrew University of Jerusalem, Jerusalem 91904, Israel

Neural Network Training Dynamics Residual Learning Representation Learning

Ask PDF

Sep 3, 2026

Correlated initialization of deep residual networks

We study the large-depth behavior of residual networks whose weights are correlated across layers at initialization. Our results confirm and extend a conjecture of Marion et al. [2025], according to which correlated initializations should interpolate continuously between the Brownian stochastic differential equation arising from independent initialization and the ordinary differential equation arising from perfectly correlated initialization. When the initialization is obtained from the application of a feature function to a stationary Gaussian sequence with regularly varying correlation, we prove that there exists a unique critical scaling such that the infinite-depth limit is the solution of a Young differential equation driven by a Hermite process. Hermite processes reduce to the fractional Brownian motion if the feature function generating the initialization has Hermite rank one, which is the case for the identity function, for example. We show that the critical scaling and asymptotic limit are uniquely determined by the decay of correlations together with the Hermite rank of the feature function. Consequently, the correlation structure and Hermite rank of the initialization represent meaningful hyperparameters in the asymptotic regime. By contrast, under finite-variance iid initialization, the asymptotic driver is universally Brownian up to normalization regardless of the choice of distribution. Our proofs rely on a collection of novel results establishing a robust stability theory for Young differential equations in Banach spaces.

Felix Benning, Ivan Nourdin, Giovanni Peccati

Neural Network Initialization Residual Learning

Ask PDF

Abstract

Neural Network Training Dynamics

Explore similar work

ResNets of All Shapes and Sizes: Convergence of Training Dynamics in the Large-scale Limit

Early learning in large residual networks

Correlated initialization of deep residual networks