Finite-Width Neural Network Theory

Latest papers

11 papers3 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Nov 16, 2026

Learning from Duality in Neural Network Field Theories

We develop neural realizations of positive-level Wess-Zumino-Witten models using bosonic Wakimoto and fermionic Fourier-Berezin fields. Level-rank duality connects the charged sectors of the fermionic construction, where explicit neural observables recover conformal block data and enable the transfer of information about unknown correlators between paired theories. We also show that distinct neural presentations of the same continuum current algebra exhibit different finite-width behavior, with approximation accuracy depending on architecture, regulator, and parameter distribution. These results connect the exact mathematical structure of conformal field theory with the expressive and computational properties of neural field theories.
Nov 15, 2026

Barron Approximation and locally optimal weight densities for shallow neural networks

Mean field theories provide optimization results for shallow neural networks by analyzing the weight distribution in infinite width limits. Corresponding results for finite sized networks are obtained by particle approximations, for which sharp quantitative bounds are still an open problem. In this paper, we consider a modified mean field loss, which allows a more fine grained control over finite sized networks. We prove convexity and equidistribution properties, which directly lead to Barron type approximation results for networks sampled from local loss minimizers. We demonstrate that particle approximations of the new loss function naturally lead to gradient descent methods with dropout type regularization.
Nov 4, 2026

Finite-width shift of the edge of chaos in deep neural networks

Deep neural networks train best when they are initialised at the edge of chaos, the critical line in the plane of weight and bias variances on which the distance between two inputs neither grows nor shrinks with depth. The line is known for infinitely wide networks; at finite width it moves, and how it moves has been disputed. We treat the logarithm of the distance between two inputs through a network of width NN as a random walk and compute its drift and variance to order 1/N1/N. The edge of chaos moves to a larger weight variance, by Δσw2=−a1/(N ∂σw2ln⁡χ1)Δσ_w^2=-a_1/(N\,\partial_{σ_w^2}\lnχ_1), about 8/N8/N for tanh⁡\tanh networks at a bias variance of 0.050.05, confirmed by simulations of the exact finite-width recursion. The shift describes a typical network; the average over many networks is dominated by rare networks in which the distance grows, and its critical point moves the other way, which explains why calculations and simulations have disagreed. Backpropagation follows the same random walk, so the shift applies to gradients unchanged. Along the edge of chaos, the information about the similarity of two inputs is constant until the compression over the depth exceeds about one nat, which fixes where to sit on the line. The results give an initialisation rule, Crinit, that places a network on its finite-width edge of chaos without a search. On MNIST and Fashion-MNIST at width 6464 it is more accurate than the standard rules at their usual settings, by a small margin in shallow networks and a large one in deep networks, where most standard rules fail; it trains in fewer steps, and it matches or beats the standard rules with a tuned gain.
Oct 27, 2026

Para-Gaussian Approximations of Random Deep Neural Networks

It is known that randomly initialized neural networks have Gaussian behaviour in the n→∞n\to \infty limit. Unfortunately, this approximation can be poor for finite nn and gets worse as one goes deeper into the network. In this work, we develop a more accurate approximation that is slightly non-Gaussian, which we call the para-Gaussian distribution and is constructed from the Hermite polynomials. We show how this para-Gaussian approximation can lead to a practical recursive method to approximate the deeper layers of the network. Finally, we apply this theory to linear neural networks (where things can be solved exactly) and to ReLU neural networks (where things can be approximated).
Aug 21, 2026

Neural Network Field Theory at Finite Width

Under mild assumptions, any quantum mechanical (QM) model or quantum field theory (QFT) admits a representation in terms of an ensemble of neural networks with countably many random parameters. We investigate the features of NN-QM and NN-FT models with finitely many parameters, such as a feedforward network of width N<∞N < \infty. We find that, generically, such models must violate one of the properties of conventional Euclidean QFTs, such as reflection positivity or cluster decomposition. We present several complementary ways of understanding which features can and cannot be preserved at finite NN, both in QM and in QFT.
Jul 7, 2026

Quantitative Gaussian-Process limits of Tensor Programs

We study the infinite-width Gaussian-process limit of random neural networks through the lens of tensor programs, and we provide a quantitative convergence theory in Wasserstein distance. Our main result gives explicit finite-width error bounds, of order inverse square-root of the widths between finite-network executions and their Gaussian-process limits. The framework is architecture-agnostic and covers feed-forward models together with weight-sharing schemes relevant for recurrent and transformer-type architectures.
May 28, 2026

Kernel Renormalization in Bayesian Deep Neural Networks: the Equivalent Wishart Ansatz in the Proportional Regime

The scaling limit where both the size of the training set PP and the width NN of a deep neural network grow at the same rate, the so-called proportional-width regime, has been intensely studied for shallow, single-hidden-layer networks. However, extending these non-perturbative results from shallow architectures to deep non-linear networks has proven very challenging. Here we present an effective approximate approach to predict the generalization performance of Bayesian multi-layer perceptrons (MLPs) of fixed depth LL on arbitrary high-dimensional data. We propose an equivalent Wishart Ansatz to capture the dominant stochastic fluctuations of the hierarchical empirical kernels of MLPs. This allows us to perform a large deviation analysis for the partition function of MLPs in the proportional limit, expressed in terms of a renormalized NNGP kernel. In this description, even strong representation learning in the proportional limit is encoded in at most LL scalar order parameters, determined self-consistently. Extending the approach to convolutional architectures (CNNs), we identify a hierarchical local kernel renormalization mechanism, which allows to quantify more complex data-dependent transformations of the large-width kernel in CNNs due to finite-width effects. We test our effective theory against sampling experiments from the Bayesian posterior of finite deep neural networks with depths L∼O(10)L \sim O(10) and P∼O(103)P\sim O(10^3) on classic benchmark datasets, finding overall very good agreement together with two distinct types of systematic deviations.
May 22, 2026

Optimal Non-Asymptotic Edgeworth Expansions for Multivariate Neural Network Outputs

Finite-width fully connected neural networks with Gaussian-initialized weights deviate from their infinite-width Gaussian limit, exhibiting non-vanishing higher-order cumulants. We approximate these deviations, for a neural network evaluated in a finite number of inputs, using multidimensional Edgeworth expansions of arbitrary order 4m−14m-1, with m∈Nm\in\mathbb{N}. Assuming that the corresponding Gaussian limit has an invertible covariance matrix and that the activation function is polynomially bounded, we establish a bound of order n−mn^{-m} on the total variation distance between the law of the true network output and its Edgeworth approximation, with matching lower bounds. As an application, we quantify the error in Bayesian posterior distributions when the prior is replaced by its Edgeworth expansion. Our results are more general and also apply to sequences of conditionally Gaussian vectors converging to a Gaussian vector with invertible covariance.
May 7, 2026

Criticality and Saturation in Orthogonal Neural Networks

It has been known for a long time that initializing weight matrices to be orthogonal instead of having i.i.d. Gaussian components can improve training performance. This phenomenon can be analyzed using finite-width corrections, where the infinite-width statistics are supplemented by a power series in 1/width1/\mathrm{width}. In particular, recent empirical results by Day et al. show that the tensors appearing in this treatment stabilize for large depth, as opposed to the tensors of i.i.d.-initialized networks. In this article, we derive explicit layer-wise recursion relations for the tensors appearing in the finite-width expansion of the network statistics in the case of orthogonal initializations. We also provide an extension of recently-introduced Feynman diagrams for the corresponding recursions in the i.i.d.-case which are valid to all orders in 1/width1/\mathrm{width}. Finally, we show explicitly that the recursions we derive reproduce the stability of the finite-width tensors which was observed for activation functions with vanishing fixed point. This work therefore provides a theoretical explanation for the stability of nonlinear networks of finite width initialized with orthogonal weights, closing a long-standing gap in the literature. We validate our theoretical results experimentally by showing that numerical solutions of our recursion relations and their analytical large-depth expansions agree excellently with Monte-Carlo estimates from network ensembles.
Apr 17, 2026

Collective Kernel EFT for Pre-activation ResNets

In finite-width deep neural networks, the empirical kernel GG evolves stochastically across layers. We develop a collective kernel effective field theory (EFT) for a simplified pre-activation residual model. Exact conditional Gaussianity of the residual increment gives finite-width discrete identities for GG and its sigma-kernel source. A projected GG-only closure and a continuous-depth approximation define reference equations for the mean kernel K0K_0, the kernel covariance V4V_4, and the next-to-leading mean correction K1K_1. Numerical tests show that this closure captures the leading mean-kernel evolution, whereas its covariance and next-to-leading accuracy depend on sigma-kernel modes omitted from the GG-only state. For the scalar (N=1N=1) model with tanh⁡\tanh activation, a Hermite--Galerkin extension agrees with the extrapolated K1K_1 value and reduces the V4V_4 endpoint error by 8585--93%93\% relative to the GG-only reference. These results clarify the range of validity of the equal-time Markov GG-only description and identify the additional collective information required beyond it.
May 16, 2025

Benign Overfitting of Two-Layer ReLU Networks in Fixed Dimension through Localized Initialization

Benign overfitting occurs when a model fits noisy training data exactly and still generalizes well. We establish this behavior for finite-width two-layer networks with the standard ReLU activation, trained by gradient flow in fixed dimension. Our data-independent initialization combines Gaussian weights with neurons initially active only on shrinking spherical caps. For uniform spherical inputs and any fixed bounded conditional response law, training and excess risks vanish in probability uniformly over all sufficiently late times. The parameters also converge, with probability tending to one, to an interpolating limit whose excess risk vanishes in probability. We use one schedule depending only on sample size and dimension. Our proof connects the network to a kernel flow whose limit combines ridge regression with localized spikes that fit the training residuals. We give explicit sample-size, width and training-time bounds.