cond-mat.dis-nnNov 4, 2026

Finite-width shift of the edge of chaos in deep neural networks

Authors: Arn Caesar

Organizations: Blackett Laboratory and Centre for Complexity Science, Imperial College London, London SW7 2AZ, United Kingdom

Abstract

Deep neural networks train best when they are initialised at the edge of chaos, the critical line in the plane of weight and bias variances on which the distance between two inputs neither grows nor shrinks with depth. The line is known for infinitely wide networks; at finite width it moves, and how it moves has been disputed. We treat the logarithm of the distance between two inputs through a network of width NN as a random walk and compute its drift and variance to order 1/N1/N. The edge of chaos moves to a larger weight variance, by Δσw2=−a1/(N ∂σw2ln⁡χ1)Δσ_w^2=-a_1/(N\,\partial_{σ_w^2}\lnχ_1), about 8/N8/N for tanh⁡\tanh networks at a bias variance of 0.050.05, confirmed by simulations of the exact finite-width recursion. The shift describes a typical network; the average over many networks is dominated by rare networks in which the distance grows, and its critical point moves the other way, which explains why calculations and simulations have disagreed. Backpropagation follows the same random walk, so the shift applies to gradients unchanged. Along the edge of chaos, the information about the similarity of two inputs is constant until the compression over the depth exceeds about one nat, which fixes where to sit on the line. The results give an initialisation rule, Crinit, that places a network on its finite-width edge of chaos without a search. On MNIST and Fashion-MNIST at width 6464 it is more accurate than the standard rules at their usual settings, by a small margin in shallow networks and a large one in deep networks, where most standard rules fail; it trains in fewer steps, and it matches or beats the standard rules with a tuned gain.

Figures & tables

Explore similar work

Aug 8, 2026

Correlation flow governs learning at criticality

The initialization of deep neural networks determines whether information and gradients can propagate across depth, yet a unified theory connecting these properties to learning dynamics remains elusive. Combining mean-field theory and random matrix theory, we establish a direct link between correlation propagation and the Neural Tangent Kernel (NTK) that governs learning in the sequential limit of infinitely wide, infinitely deep networks. Correlation propagation to infinite depth is possible only at a single, critical point in the weight-bias variance plane. At this point, we leverage the algebraic decay of the end-to-end Jacobian with depth to prove that the NTK becomes exactly proportional to the output correlation at infinite depth, tying together information propagation and learning dynamics. We further show that orthogonal initialization suppresses the leading finite-size corrections present under Gaussian initialization, clarifying the respective roles of the two initialization ensembles in this limit. These theoretical predictions are validated quantitatively on finite-width, finite-depth networks. Together, these results demonstrate that orthogonal initialization and criticality are required to control the asymptotic dynamics of deep learning.
Sep 8, 2026

Critical initialization destabilizes higher input derivatives in wide scalar-input networks

The edge-of-chaos condition preserves first-order input perturbations in wide randomly initialized networks, but physics-informed losses, score matching and derivative regularization depend on higher input derivatives. For smooth scalar-input fully connected networks, using a joint Gaussianity of the finite derivative jet that holds in the infinite-width limit at each fixed depth, we derive mean-field recursions through third order that are exact at the variance fixed point, with finite-depth corrections that decay geometrically. At criticality, the first-derivative variance is depth-invariant, whereas the second-derivative variance grows linearly whenever the activation has nonzero curvature. The resulting third-order system closes on mean-field susceptibilities. For residual networks with branch scale L^{-1/2}, we prove that every fixed finite derivative order has uniformly bounded variance under explicit regularity assumptions. Simulations verify the critical growth laws, the residual bound, and the closed recursion. The results concern initialization, not trained-network performance.
May 21, 2026

Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks

We consider one-hidden layer neural networks trained in the feature-learning regime using gradient descent, and relate the output of the finite-width network fρ^tmf_{\hatρ_t^m} to its infinite-width counterpart fρtMFf_{ρ_t^{MF}}, which evolves in the mean-field dynamics. While constant-time horizon bounds for ∥fρtMF−fρ^tm∥\|f_{ρ_t^{MF}} - f_{\hatρ_t^m}\| may be obtained via standard Grönwall estimates, the long-time behavior of the fluctuation is a more delicate matter. Uniform-in-time bounds often rely on (local) strong convexity in the landscape or Logarithmic Sobolev inequalities present in noisy gradient dynamics. In this work, we establish non-asymptotic weak propagation-of-chaos that holds uniformly in time, obtained by exploiting instead the convergence rate of the mean-field deterministic Wasserstein-gradient-flow dynamics. Specifically, denoting by LtL_t the mean-field excess MSE loss at time tt and mm the number of neurons, under standard regularity assumptions and the condition ∫0∞Lt1/2dt=O(log⁡d)\int_0^\infty L_t^{1/2} dt =O(\log d), we obtain the uniform in time bound ∥fρtMF−fρ^tm∥2≲poly(d)m−min⁡(1,c/6)\|f_{ρ_t^{MF}}- f_{\hatρ_t^m}\|^2 \lesssim \text{poly}(d) m^{-\min(1,c/6)} whenever Lt≲t−cL_t \lesssim t^{-c}. Our result holds in a noiseless setting and does not make any assumptions on the geometry of the landscape near the optimum, and extends seamlessly to other forms of discretization, including finite number of samples and time discretization. A key takeaway of our result is that whenever the convergence rate of the mean-field, population-loss dynamics is faster than t−2t^{-2}, we can attain a loss of εε with only poly(d/ε)\text{poly}(d/ε) neurons, training samples, and GD steps.