Width and Depth Limits Commute During Training

Ruijia Zhang; Soufiane Hayou

cs.LGOct 22, 2026

Width and Depth Limits Commute During Training

Authors: Ruijia Zhang, Soufiane Hayou

Abstract

Recent work established that the infinite-width and infinite-depth limits of residual networks commute at initialization when residual branches are scaled by $1/\sqrt{depth}$ . In particular, the network output converges to the same limit whether width and depth tend to infinity sequentially or jointly. We extend this commutativity result to training in linear residual networks. Under the same $1/\sqrt{depth}$ residual-branch scaling (Depth- $μ$ P) and the $μ$ P width parameterization, we prove that, for any fixed number of gradient-descent steps with gradient clipping, the network output converges in $L^2$ along every joint width--depth growth path as both width and depth tend to infinity. Consequently, the infinite-width and infinite-depth limits commute throughout any finite training horizon. Without gradient clipping, the corresponding $L^2$ convergence does not hold in general, but we establish the weaker convergence in probability.

Figures & tables

Figure 1: Commutativity of the width and depth limits. Illustration of three relative order regimes for $(n_{r},L_{r})$ as $n_{r},L_{r}\to\infty$ . For the same fixed input and any fixed training iteration $k$ , all three paths converge to the same trained output limit $f_{\infty}^{(k)}$ .

Figure 2: Commutativity of the width and depth limits during training. Each curve corresponds to a fixed width–depth scaling path and is plotted against the joint scale

m=\min(n,L)

. The top row shows the synthetic regression output at training iterations

k=1,5,10,15

, and the bottom row shows the HIGGS logit margin at four training checkpoints. For each task, all configurations are evaluated at the same fixed input. Curves show the mean across 5 model seeds, with shaded regions denoting

\pm 1

standard deviation.

Figure 3: Width–depth convergence rates. (a) RMS output error as width

n

increases with depth fixed at

L=12{,}288

. (b) RMS output error as depth

L

increases with width fixed at

n=12{,}288

. Curves in (a)–(b) correspond to training iterations

k\in\{1,5,10,15\}

. (c) Empirical convergence exponents for width and depth over training. The width rate remains close to

n^{-1/2}

, while the depth rate approaches an empirical

L^{-1}

rate.

Figure 4: Width–depth commutativity with nonlinear activations. Rows correspond to activation functions and columns to training iterations. Each panel shows the output at synthetic training input

86

along the five width–depth scaling paths in Section 5.1 . Curves show the mean over five model seeds, with shaded regions denoting

\pm 1

standard deviation.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

$k$	$\alpha_{n}(k)$	$95\%$ CI	$\alpha_{L}(k)$	$95\%$ CI
$1$	$0.427$	$[0.360,0.492]$	$0.738$	$[0.671,0.807]$
$2$	$0.441$	$[0.368,0.508]$	$0.789$	$[0.749,0.837]$
$3$	$0.434$	$[0.369,0.493]$	$0.898$	$[0.870,0.929]$
$4$	$0.437$	$[0.364,0.497]$	$0.979$	$[0.943,1.017]$
$5$	$0.425$	$[0.350,0.498]$	$0.899$	$[0.866,0.936]$
$6$	$0.416$	$[0.335,0.497]$	$0.932$	$[0.898,0.971]$

Appendix

Table 1: Empirical convergence exponents across training. Confidence intervals are

95\%

model-seed bootstrap intervals.

Figure 5: Synthetic width–depth commutativity across all saved iterations. Each panel shows the output for training input

86

at one iteration

k=0,\ldots,15

. Curves correspond to the five width–depth scaling paths in Section 5.1 and show means over five model seeds, with shaded regions denoting

\pm 1

standard deviation.

Figure 6: Synthetic width–depth commutativity at additional training inputs. Rows correspond to four fixed training inputs, and columns show the representative iterations in Section 5.1 . Curves correspond to the same five width–depth scaling paths and show means over five model seeds, with shaded regions denoting

\pm 1

standard deviation.

Figure 7: HIGGS width–depth commutativity across all saved checkpoints. Each panel shows the class-

1

versus class-

0

logit margin for the fixed test example used in Section 5.1 , from initialization through two training epochs. Curves show means over five model seeds, with shaded regions denoting

\pm 1

standard deviation, over the common scales

m\in\{16,32,64,128,256\}

Figure 8: HIGGS width–depth commutativity at additional test examples. Rows correspond to three fixed test examples, and columns show the representative checkpoints in Section 5.1 . Curves correspond to the same five width–depth scaling paths and show means over five model seeds, with shaded regions denoting

\pm 1

standard deviation.

Figure 9: Width–depth commutativity in the single-datum setting. Each panel shows the trained scalar output at one training iteration along the five width–depth scaling paths. Curves show the mean over five model seeds, with shaded regions denoting

\pm 1

standard deviation. The paths exhibit substantial finite-scale differences at small

m

, but these differences diminish as both width and depth increase.

Experiment	Maximum $(n,L)$	Backend	Dense memory	Peak memory
Single datum	$(61{,}440,3{,}072)$	lazy	$46.4$ TB	$31.7$ GB
Synthetic commutativity	$(61{,}440,3{,}072)$	lazy	$46.4$ TB	$46.9$ GB
Linear rate sweeps	$(12{,}288,12{,}288)$	lazy	$7.42$ TB	$25.4$ GB
HIGGS commutativity	$(5{,}120,256)$	dense	$26.8$ GB	$26.9$ GB
Nonlinear activations	$(5{,}120,256)$	dense	$26.8$ GB	$27.6$ GB

Appendix

Table 2: Memory usage for the reported experiments. Dense memory is the float32 storage required to materialize all residual matrices; peak memory is the maximum recorded CUDA allocation.

Neural Network Training Dynamics

Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.

See all 507 papers

ResNets of All Shapes and Sizes: Convergence of Training Dynamics in the Large-scale Limit
Mar 18, 2026Louis-Pierre Chaintron, Lénaïc Chizat, Javier MaassDynamical Mean-Field Theory Neural Network Training Dynamics
École Polytechnique Fédérale de Lausanne (EPFL), Institute of Mathematics, 1015 Lausanne, Switzerland.
Early learning in large residual networks
Oct 22, 2026Vladimir Narovlansky, Boris HaninNeural Network Training Dynamics Residual Learning
ORFE Department, Princeton University, Princeton, NJ 08544, USA · Racah Institute of Physics, The Hebrew University of Jerusalem, Jerusalem 91904, Israel
Correlated initialization of deep residual networks
Sep 3, 2026Felix Benning, Ivan Nourdin, Giovanni PeccatiNeural Network Initialization Residual Learning

Mar 18, 2026J/K move · Enter open · S save

ResNets of All Shapes and Sizes: Convergence of Training Dynamics in the Large-scale Limit

Louis-Pierre Chaintron, Lénaïc Chizat, Javier Maass

École Polytechnique Fédérale de Lausanne (EPFL), Institute of Mathematics, 1015 Lausanne, Switzerland.

Dynamical Mean-Field Theory Neural Network Training Dynamics Residual Dynamics Learning

Open paper AskPDF arXiv

We establish convergence of the training dynamics of residual neural networks (ResNets) to their joint infinite depth L, hidden width M, and embedding dimension D limit. Specifically, we consider ResNets with two-layer perceptron blocks in the maximal local feature update (MLU) regime and prove that, after a bounded number of training steps, the error between the ResNet and its large-scale limit is O(1/L + sqrt(D/(L M)) + 1/sqrt(D)). This error rate is empirically tight when measured in embedding space. For a budget of P = Theta(L M D) parameters, this yields a convergence rate O(P^(-1/6)) for the scalings of (L, M, D) that minimize the bound. Our analysis exploits in an essential way the depth-two structure of residual blocks and applies formally to a broad class of state-of-the-art architectures, including Transformers with bounded key-query dimension. From a technical viewpoint, this work completes the program initiated in the companion paper [Chi25] where it is proved that for a fixed embedding dimension D, the training dynamics converges to a Mean ODE dynamics at rate O(1/L + sqrt(D)/sqrt(L M)). Here, we study the large-D limit of this Mean ODE model and establish convergence at rate O(1/sqrt(D)), yielding the above bound by a triangle inequality. To handle the rich probabilistic structure of the limit dynamics and obtain one of the first rigorous quantitative convergence for a DMFT-type limit, we combine the cavity method with propagation of chaos arguments at a functional level on so-called skeleton maps, which express the weight updates as functions of CLT-type sums from the past.

Abstract

Neural Network Training Dynamics

Explore similar work

ResNets of All Shapes and Sizes: Convergence of Training Dynamics in the Large-scale Limit

Early learning in large residual networks

Correlated initialization of deep residual networks