cs.LGNov 15, 2026

Barron Approximation and locally optimal weight densities for shallow neural networks

Authors: G. Welper

Organizations: School of Mathematical and Statistical Sciences, University of Texas Rio Grande Valley, Brownsville, TX 78520, USA

Abstract

Mean field theories provide optimization results for shallow neural networks by analyzing the weight distribution in infinite width limits. Corresponding results for finite sized networks are obtained by particle approximations, for which sharp quantitative bounds are still an open problem. In this paper, we consider a modified mean field loss, which allows a more fine grained control over finite sized networks. We prove convexity and equidistribution properties, which directly lead to Barron type approximation results for networks sampled from local loss minimizers. We demonstrate that particle approximations of the new loss function naturally lead to gradient descent methods with dropout type regularization.

Figures & tables

Explore similar work

May 21, 2026

Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks

We consider one-hidden layer neural networks trained in the feature-learning regime using gradient descent, and relate the output of the finite-width network fρ^tmf_{\hatρ_t^m} to its infinite-width counterpart fρtMFf_{ρ_t^{MF}}, which evolves in the mean-field dynamics. While constant-time horizon bounds for ∥fρtMF−fρ^tm∥\|f_{ρ_t^{MF}} - f_{\hatρ_t^m}\| may be obtained via standard Grönwall estimates, the long-time behavior of the fluctuation is a more delicate matter. Uniform-in-time bounds often rely on (local) strong convexity in the landscape or Logarithmic Sobolev inequalities present in noisy gradient dynamics. In this work, we establish non-asymptotic weak propagation-of-chaos that holds uniformly in time, obtained by exploiting instead the convergence rate of the mean-field deterministic Wasserstein-gradient-flow dynamics. Specifically, denoting by LtL_t the mean-field excess MSE loss at time tt and mm the number of neurons, under standard regularity assumptions and the condition ∫0∞Lt1/2dt=O(log⁡d)\int_0^\infty L_t^{1/2} dt =O(\log d), we obtain the uniform in time bound ∥fρtMF−fρ^tm∥2≲poly(d)m−min⁡(1,c/6)\|f_{ρ_t^{MF}}- f_{\hatρ_t^m}\|^2 \lesssim \text{poly}(d) m^{-\min(1,c/6)} whenever Lt≲t−cL_t \lesssim t^{-c}. Our result holds in a noiseless setting and does not make any assumptions on the geometry of the landscape near the optimum, and extends seamlessly to other forms of discretization, including finite number of samples and time discretization. A key takeaway of our result is that whenever the convergence rate of the mean-field, population-loss dynamics is faster than t−2t^{-2}, we can attain a loss of εε with only poly(d/ε)\text{poly}(d/ε) neurons, training samples, and GD steps.
Sep 30, 2026

Minimax rates for learning spectral Barron functions by deep ReLU neural networks

We study how well deep neural networks approximate and learn spectral Barron functions. Recent studies have shown that these function classes can be efficiently approximated by shallow neural networks without suffering from the curse of dimensionality. We complement these results by providing new approximation bounds for deep networks with ReLU activation and establishing the minimax rates for learning these function classes. Specifically, we show that dd-dimensional spectral Barron functions with smoothness index s>0s>0 can be approximated by deep ReLU neural networks with approximation rate O~(S−12−sd)\widetilde{\mathcal{O}} (S^{-\frac{1}{2}-\frac{s}{d}}), where SS denotes the number of nonzero parameters in the network. Using this approximation result, we further show that deep ReLU neural networks can learn spectral Barron functions in a fast rate n−d+2s2d+2sn^{-\frac{d+2s}{2d+2s}} with nn training samples. Finally, we prove that this convergence rate is minimax optimal up to logarithmic factors.
May 11, 2026

On the global convergence of gradient descent for wide shallow models with bounded nonlinearities

A surprising phenomenon in the training of neural networks is the ability of gradient descent to find global minimizers of the training loss despite its non-convexity. Following earlier works, we investigate this behavior for wide shallow networks. Existing results essentially cover the case of ReLU activations and the case of sigmoid activations with scalar output weights. We study a large class of models that includes multi-head attention layers and two-layer sigmoid networks with vector output weights. Building upon [Chizat and Bach, 2018], we prove that all non-global minimizers of the training loss are unstable under gradient descent dynamics. Thus, when the initial distribution of the parameters has full support (which includes the popular Gaussian case), and in the many hidden neurons or attention heads limit, continuous-time gradient descent can only converge to global minimizers. Establishing the instability of non-global minimizers corresponds to the construction of an ``escaping active set'' -- we complete the proof of [Chizat and Bach, 2018] to construct this set for models with bounded nonlinearities and scalar output weights. We also extend this construction to new cases for models with vector output weights. Finally, we show the well-posedness and the stability with respect to discretization of the mean field training dynamic for sub-Gaussian initializations.