Shallow Neural Networks

Follow this topic Ask about this topic

Shallow neural networks—typically models with one hidden layer—are studied for what functions they can represent, how they learn, and when their compact structure is useful. Approximation theory uses tools such as Fourier and spherical-harmonic analysis to give error and generalization guarantees, while optimization research examines gradient descent, loss landscapes, symmetry, and mean-field dynamics. Practical approaches include changing activation functions, regularizing parameter norms, and making individual units more expressive, with applications ranging from differential equations to compact classifiers. A central tension is that width and richer units can improve approximation, but depth may be essential for some tasks, and representational capacity alone does not ensure that training finds a good solution.

Latest papers

57 papers11 papers in the last four weeks, up 120% on the four weeks before. 0.1% of all new papers.

Jun 17, 2026

Convex training of Lipschitz-regularized shallow neural networks

In this work, we introduce a training procedure for shallow neural networks that promotes robustness against adversarial attacks. We solve a non-convex Lipschitz-regularized training program by introducing a convex restriction that can be efficiently solved to global optimality. Our approach can be employed as a post-processing step by taking a pre-trained network as an initial solution to then solving the convex program whose optimal network is guaranteed to be no worse than the initial one. We illustrate the improvements of our training procedure with experiments using real world datasets for regression tasks under an adversarial setting. We show numerically that solving our proposed convex program yields networks with lower objective values on the Lipschitz-regularized program compared to existing methods. Additionally, we show that on certain datasets, networks obtained using our convex training program are both more accurate and robust with respect to adversarial attacks.

Chao Yin, Antoine Lesage-Landry

Polytechnique Montr´eal, GERAD & Mila, Montr´eal, QC, Canada, H3T 1J4.

Adversarial Training Shallow Neural Networks Neural Network Optimization Neural Network Robustness Adversarial Robustness Lipschitz Neural Networks

May 21, 2026

Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks

We consider one-hidden layer neural networks trained in the feature-learning regime using gradient descent, and relate the output of the finite-width network

f_{\hatρ_t^m}

to its infinite-width counterpart

f_{ρ_t^{MF}}

, which evolves in the mean-field dynamics. While constant-time horizon bounds for

\|f_{ρ_t^{MF}} - f_{\hatρ_t^m}\|

may be obtained via standard Grönwall estimates, the long-time behavior of the fluctuation is a more delicate matter. Uniform-in-time bounds often rely on (local) strong convexity in the landscape or Logarithmic Sobolev inequalities present in noisy gradient dynamics. In this work, we establish non-asymptotic weak propagation-of-chaos that holds uniformly in time, obtained by exploiting instead the convergence rate of the mean-field deterministic Wasserstein-gradient-flow dynamics. Specifically, denoting by

L_t

the mean-field excess MSE loss at time

t

and

m

the number of neurons, under standard regularity assumptions and the condition

\int_0^\infty L_t^{1/2} dt =O(\log d)

, we obtain the uniform in time bound

\|f_{ρ_t^{MF}}- f_{\hatρ_t^m}\|^2 \lesssim \text{poly}(d) m^{-\min(1,c/6)}

whenever

L_t \lesssim t^{-c}

. Our result holds in a noiseless setting and does not make any assumptions on the geometry of the landscape near the optimum, and extends seamlessly to other forms of discretization, including finite number of samples and time discretization. A key takeaway of our result is that whenever the convergence rate of the mean-field, population-loss dynamics is faster than

t^{-2}

, we can attain a loss of

ε

with only

\text{poly}(d/ε)

neurons, training samples, and GD steps.

Margalit Glasgow, Joan Bruna

Massachusetts Institute of Technology · Courant Institute School of Mathematics, Computing and Data Science, New York University

Shallow Neural Networks Mean-Field Theory Neural Network Training Dynamics Feedforward Neural Networks Gradient Descent Dynamics

May 19, 2026

Machine-Learning-Enhanced Non-Invasive Testing for MASLD Fibrosis: Shallow-Deep Neural Networks Versus FIB-4, Tabular Foundation Models, and Large Language Models

Advanced fibrosis is a major determinant of liver-related morbidity in metabolic dysfunction-associated steatotic liver disease (MASLD). FIB-4 is widely used as a first-line non-invasive test, but its fixed formula may underuse diagnostic information contained in age, aspartate aminotransferase, alanine aminotransferase, and platelet count. We evaluated whether machine-learning-enhanced non-invasive testing (MLE-NIT) can improve advanced fibrosis detection while preserving this FIB-4 variable space. We used three biopsy-confirmed MASLD cohorts from China, Malaysia, and India (n=784). The Chinese cohort was split into 486 training and 54 internal validation/tuning patients; final performance was reported only on the Malaysian and Indian external cohorts. Models used five variables: age, FIB-4, aspartate aminotransferase, platelet count, and alanine aminotransferase. We compared FIB-4 with a shallow-deep neural network (s-DNN), TabPFN, and gpt-4o-2024-08-06. FIB-4 achieved external ROC-AUCs of 0.75 and 0.60 in Malaysia and India, respectively. TabPFN achieved 0.69 and 0.66, fine-tuned GPT-4o achieved 0.75 and 0.63, and the s-DNN achieved 0.77 and 0.67, respectively. The s-DNN contained only 354 trainable parameters, compared with 7,244,554 for TabPFN, yet provided a more balanced external operating profile. Calibration showed s-DNN Brier scores of 0.18 and 0.22, and permutation importance identified AST and FIB-4 as dominant variables. Compact non-linear MLE-NITs may enhance FIB-4-based fibrosis assessment without increasing clinical data requirements.

Athanasios Angelakis, Gabriele De Vito, Eleni-Myrto Trifylli +1

BioML Lab, RI CODE, UniBw, Munich, Germany · Department of Epidemiology and Data Science, Amsterdam UMC, Amsterdam, Netherlands · Alpha Indicium, Rijswijk, Netherlands +2

Shallow Neural Networks Tabular ML Clinical Prediction Clinical Risk Prediction

May 18, 2026

Shallow ReLU $^s$ Networks in $L^p$ -Type and Sobolev Spaces: Approximation and Path-Norm Controlled Generalization

This paper studies approximation by shallow ReLU

^s

networks,

σ_s(t)=\max\{0,t\}^s

, together with their generalization behavior under

\ell_1

path-norm control. For the

L^p

-type integral spaces

\widetilde{\mathcal{F}}_{p,τ_d,s}

,

1\le p\le2

, spherical harmonic analysis yields approximation bounds for shallow networks. In particular, when

τ_d

is the uniform measure and

1\le p<2

, the approximation rate is

O\!\left(m^{-\frac{p(2s+2d+1)-2d}{2dp}}\right)

for

1\le p\le p^*

and

O\!\left(m^{-\frac{p(4s+3d-1)-2d+2}{4dp}}\right)

for

p^*<p<2

, where

p^*=\frac{2d+2}{d+3}

. Approximation bounds for Sobolev spaces

W^{α,p}

,

1\le p<2

, are obtained through embeddings into spectral Barron spaces. For nonparametric regression with sub-Gaussian noise, path-norm-regularized shallow ReLU

^s

networks achieve minimax-optimal rates

O\!\left(n^{-\frac{d+2s+1}{2d+2s+1}}\log n\right)

over

\mathscr{B}_s

and

O\!\left(n^{-\frac{2α}{2α+d}}\log n\right)

over

W^{α,\infty}

, with matching lower bounds up to logarithmic factors.

Weizhao Li, Fanghui Liu, Lei Shi

School of Mathematical Sciences Fudan University Shanghai, China · School of Mathematical Sciences Institute of Natural Sciences and MOE-LSC2026 Shanghai Jiao Tong University Shanghai, China · School of Mathematical Sciences Shanghai Key Laboratory for Contemporary Applied Mathematics25 Fudan University Shanghai, China

Shallow Neural Networks Nonparametric Regression Neural Network Approximation Theory Generalization Bounds

May 14, 2026

Learning with Shallow Neural Networks on Cluster-Structured Features

The success of deep learning in high-dimensional settings is often attributed to the presence of low-dimensional structure in real-world data. While standard theoretical models typically assume that this structure lies in the target function, projecting unstructured inputs onto a low-dimensional subspace, data such as images, text or genomic sequences exhibit strong spatial correlations within the input space itself. In this paper, we propose a tractable model to study how these correlations affect the sample complexity of learning with gradient descent on shallow neural networks. Specifically, we consider targets that depend on a small number of latent Boolean variables, and input features grouped into clusters and correlated with the latent variables. Under an identifiability assumption, we show that for a layerwise gradient-descent variant, the sample complexity scales with the number of hidden variables and, when the signal-to-noise ratio is sufficiently high, is independent of the input dimension, up to logarithmic terms. We empirically test our theoretical findings on both synthetic and real data.

Elisabetta Cornacchia, Laurent Massoulié

INRIA, DI/ENS, PSL.

Shallow Neural Networks Neural Network Generalization Sample Complexity Statistical Learning Theory

May 11, 2026

On the global convergence of gradient descent for wide shallow models with bounded nonlinearities

A surprising phenomenon in the training of neural networks is the ability of gradient descent to find global minimizers of the training loss despite its non-convexity. Following earlier works, we investigate this behavior for wide shallow networks. Existing results essentially cover the case of ReLU activations and the case of sigmoid activations with scalar output weights. We study a large class of models that includes multi-head attention layers and two-layer sigmoid networks with vector output weights. Building upon [Chizat and Bach, 2018], we prove that all non-global minimizers of the training loss are unstable under gradient descent dynamics. Thus, when the initial distribution of the parameters has full support (which includes the popular Gaussian case), and in the many hidden neurons or attention heads limit, continuous-time gradient descent can only converge to global minimizers. Establishing the instability of non-global minimizers corresponds to the construction of an ``escaping active set'' -- we complete the proof of [Chizat and Bach, 2018] to construct this set for models with bounded nonlinearities and scalar output weights. We also extend this construction to new cases for models with vector output weights. Finally, we show the well-posedness and the stability with respect to discretization of the mean field training dynamic for sub-Gaussian initializations.

Romain Petit, Clarice Poon, Gabriel Peyré

CNRS and ENS, PSL Universit´e · Mathematics Institute, University of Warwick

Shallow Neural Networks Gradient Descent Neural Network Training Dynamics Gradient Descent Dynamics

May 11, 2026

Parameterized Complexity of Stationarity Testing for Piecewise-Affine Functions and Shallow CNN Losses

We study the parameterized complexity of testing approximate first-order stationarity at a prescribed point for continuous piecewise-affine (PA) functions, a basic task in nonsmooth optimization. PA functions form a canonical model for nonsmooth stationarity testing and capture the local polyhedral geometry that appears in ReLU-type training losses. Recent work by Tian and So (SODA 2025) shows that testing approximate stationarity notions for PA functions is computationally intractable in the worst case, and identifies fixed-dimensional tractability as an open direction. We address this direction from the viewpoint of parameterized complexity, with the ambient dimension

d

as the parameter. In this paper, we give XP algorithms in fixed dimension for the tractable sides, and prove W[1]-hardness for the complementary sides. Moreover, lower bounds under the Exponential Time Hypothesis rule out algorithms running in time

ρ(d)\size^{o(d)}

for any computable function

ρ

, where

\size

denotes the total binary encoding length of the stationarity-testing instance. As a further consequence, our results yield the corresponding parameterized complexity picture for testing local minimality of continuous PA functions. We further extend our hardness results to a family of shallow ReLU CNN training losses, with stationarity tested in the trainable weight space. Thus, the same parameterized-complexity picture also appears for simple CNN training losses.

Yuhan Ye

MIT

Shallow Neural Networks Nonconvex Optimization Parameterized Complexity Nonsmooth Convex Optimization

Apr 15, 2026

A Complete Symmetry Classification of Shallow ReLU Networks

Parameter space is not function space for neural network architectures. This fact, investigated as early as the 1990s under terms such as reverse engineering," or parameter identifiability", has led to the natural question of parameter space symmetries\textemdash the study of distinct parameters in neural architectures which realize the same function. Indeed, the quotient space obtained by identifying parameters giving rise to the same function, called the neuromanifold, has been shown in some cases to have rich geometric properties, impacting optimization dynamics. Thus far, techniques towards complete classifications have required the analyticity of the activation function, notably excising the important case of ReLU. Here, in contrast, we exploit the non-differentiability of the ReLU activation to provide a complete classification of the symmetries in the shallow case.

Pranavkrishnan Ramakrishnan

Shallow Neural Networks Neural Representation Geometry Parameter Identifiability ReLU Neural Networks

Apr 10, 2026

Sharp description of local minima in the loss landscape of high-dimensional two-layer ReLU neural networks

We study the population loss landscape of two-layer ReLU networks of the form

\sum_{k=1}^K \mathrm{ReLU}(w_k^\top x)

in a realisable teacher-student setting with Gaussian covariates. We show that local minima admit an exact low-dimensional representation in terms of summary statistics, yielding a sharp and interpretable characterisation of the landscape. We further establish a direct link with one-pass SGD: local minima correspond to attractive fixed points of the dynamics in summary statistics space. This perspective reveals a hierarchical organisation of minima into discrete families and shows how overparameterisation changes their stability and reachability under gradient-based dynamics. In this overparameterised regime, global minima become increasingly accessible, attracting the dynamics and reducing convergence to spurious solutions. Overall, our results reveal intrinsic limitations of common simplifying assumptions, which may miss essential features of the loss landscape even in minimal neural network models.

Jie Huang, Bruno Loureiro, Stefano Sarao Mannelli

Physics, Chalmers University of Technology and University of Gothenburg · Ecole Normale Superieure, PSL & CNRS · Data Science and AI, Computer Science and Engineering, Chalmers University of Technology and University of Gothenburg +1

Overparameterization Loss Landscape Neural Network Optimization ReLU Neural Networks Loss Landscape Geometry Shallow Neural Networks+1

Apr 9, 2026

Tensor-Augmented Convolutional Neural Networks: Enhancing Expressivity with Generic Tensor Kernels

Convolutional Neural Networks (CNNs) excel at extracting local features hierarchically, but their performance in capturing complex correlations hinges heavily on deep architectures, which are usually computationally demanding and difficult to interpret. To address these issues, we propose a physically-guided shallow model: tensor-augmented CNN (TACNN), which replaces conventional convolution kernels with generic tensors to enhance representational capacity. This choice is motivated by the fact that an order-

N

tensor naturally encodes an arbitrary quantum superposition state in the Hilbert space of dimension

d^N

, where

d

is the local physical dimension, thus offering substantially richer expressivity. Furthermore, in our design the convolution output of each layer becomes a multilinear form capable of capturing high-order feature correlations, thereby equipping a shallow multilayer architecture with an expressive power competitive to that of deep CNNs. On the Fashion-MNIST benchmark, TACNN demonstrates clear advantages over conventional CNNs, achieving remarkable accuracies with only a few layers. In particular, a TACNN with only two convolution layers attains a test accuracy of 93.7

\%

, surpassing or matching considerably deeper models such as VGG-16 (93.5

\%

) and GoogLeNet (93.7

\%

). These findings highlight TACNN as a promising framework that strengthens model expressivity while preserving architectural simplicity, paving the way towards more interpretable and efficient deep learning models.

Chia-Wei Hsing, Wei-Lin Tu

blueqat Inc., 2-24-12-39F, Shibuya, Shibuya-ku, Tokyo 150-6139, Japan · Center for Quantum Science and Engineering, National Taiwan University, Taipei 10617, Taiwan · Graduate School of Science and Technology, Keio University, Yokohama, Kanagawa 223-8522, Japan +1

Convolutional Neural Networks Neural Network Expressivity Shallow Neural Networks Image Classification

Apr 9, 2026

Mathematical analysis of one-layer neural network with fixed biases, a new activation function and other observations

We analyze a simple one-hidden-layer neural network with ReLU activation functions and fixed biases, with one-dimensional input and output. We study both continuous and discrete versions of the model, and we rigorously prove the convergence of the learning process with the

L^2

squared loss function and the gradient descent procedure. We also prove the spectral bias property for this learning process. Several conclusions of this analysis are discussed; in particular, regarding the structure and properties that activation functions should possess, as well as the relationships between the spectrum of certain operators and the learning process. Based on this, we also propose an alternative activation function, the full-wave rectified exponential function (FReX), and we discuss the convergence of the gradient descent with this alternative activation function.

Fabricio Macià, Shu Nakamura

M2ASAI, Universidad Politécnica de Madrid. ETSI Navales. Avda. de la Memoria, 4. 28040 Madrid, Spain. · Department of Mathematics, Faculty of Sciences, Gakushuin University, 1-5-1, Mejiro, Toshima, Tokyo, 171-8588 Japan

Shallow Neural Networks Neural Network Training Dynamics Neural Network Optimization ReLU Neural Networks Neural Network Activation Functions Spectral Bias

Apr 1, 2026

Towards Initialization-dependent and Non-vacuous Generalization Bounds for Overparameterized Shallow Neural Networks

Overparameterized neural networks often show a benign overfitting property in the sense of achieving excellent generalization behavior despite the number of parameters exceeding the number of training examples. A promising direction to explain benign overfitting is to relate generalization to the norm of distance from initialization, motivated by the empirical observations that this distance is often significantly smaller than the norm itself. However, the existing initialization-dependent complexity analyses measure the distance from initialization by the Frobenius norm, and often imply vacuous bounds in practice for overparamterized models. In this paper, we develop initialization-dependent complexity bounds for shallow neural networks with general Lipschitz activation functions. Our bounds depend on the path-norm of the distance from initialization, which are derived by introducing a new peeling technique to handle the challenge along with the initialization-dependent constraint. We also develop a lower bound tight up to a constant factor. Finally, we conduct empirical comparisons and show that our generalization analysis implies non-vacuous bounds for overparameterized networks.

Yunwen Lei, Yufeng Xie

Department of Mathematics, The University of Hong Kong

Generalization Bounds Overparameterized Neural Network Overparameterization Shallow Neural Networks Benign Overfitting Neural Network Initialization

Mar 27, 2026

Expressibility of neural quantum states: a Walsh-complexity perspective

Neural quantum states are powerful variational wavefunctions, but it remains unclear which many-body states can be represented efficiently by modern additive architectures. We introduce Walsh complexity, a basis-dependent measure of how broadly a wavefunction is spread over parity patterns. States with an almost uniform Walsh spectrum require exponentially large Walsh complexity from any good approximant. We show that shallow additive feed-forward networks cannot generate such complexity in the tame regime, e.g. polynomial activations with subexponential parameter scaling. As a concrete example, we construct a simple dimerized state prepared by a single layer of disjoint controlled-

Z

gates. Although it has only short-range entanglement and a simple tensor-network description, its Walsh complexity is maximal. Full-cube fits across system size and depth are consistent with the complexity bound: for polynomial activations, successful fitting appears only once depth reaches a logarithmic scale in

N

, whereas activation saturation in

\tanh

produces a sharp threshold-like jump already at depth

3

. Walsh complexity therefore provides an expressibility axis complementary to entanglement and clarifies when depth becomes an essential resource for additive neural quantum states.

Taige Wang

Department of Physics, Massachusetts Institute of Technology, Cambridge, MA 02139, USA · Department of Physics, Harvard University, Cambridge, MA 02138, USA

Neural Quantum States Neural Network Expressivity Shallow Neural Networks Neural Network Approximation Theory

Mar 23, 2026

Quotient Geometry, Effective Curvature, and Implicit Bias in Simple Shallow Neural Networks

Overparameterized shallow neural networks admit substantial parameter redundancy: distinct parameter vectors may represent the same predictor due to hidden-unit permutations, rescalings, and related symmetries. As a result, geometric quantities computed directly in the ambient Euclidean parameter space can reflect artifacts of representation rather than intrinsic properties of the predictor. In this paper, we develop a differential-geometric framework for analyzing simple shallow networks through the quotient space obtained by modding out parameter symmetries on a regular set. We first characterize the symmetry and quotient structure of regular shallow-network parameters and show that the finite-sample realization map induces a natural metric on the quotient manifold. This leads to an effective notion of curvature that removes degeneracy along symmetry orbits and yields a symmetry-reduced Hessian capturing intrinsic local geometry. We then study gradient flows on the quotient and show that only the horizontal component of parameter motion contributes to first-order predictor evolution, while the vertical component corresponds purely to gauge variation. Finally, we formulate an implicit-bias viewpoint at the quotient level, arguing that meaningful complexity should be assigned to predictor classes rather than to individual parameter representatives. Our experiments confirm that ambient flatness is representation-dependent, that local dynamics are better organized by quotient-level curvature summaries, and that in underdetermined regimes, implicit bias is most naturally described in quotient coordinates.

Hang-Cheng Dong, Pengcheng Cheng

School of Instrumentation Science and Engineering, Harbin Institute of Technology Harbin, 150001, China · Harbin Institute of Technology Suzhou Research Institute Suzhou, 215000, China · School of Mathematics, Jilin University Changchun, 130012, China

Shallow Neural Networks Neural Network Training Dynamics

Mar 22, 2026

Gradient Descent with Projection Finds Over-Parameterized Neural Networks for Learning Low-Degree Polynomials with Nearly Minimax Optimal Rate

We study the problem of learning a low-degree spherical polynomial of degree

k_0 = Θ(1) \ge 1

defined on the unit sphere in

\RR^d

by training an over-parameterized two-layer neural network with augmented feature in this paper. Our main result is the significantly improved sample complexity for learning such low-degree polynomials. We show that, for any regression risk

\eps \in (0, Θ(d^{-k_0})]

, an over-parameterized two-layer neural network trained by a novel Gradient Descent with Projection (GDP) requires a sample complexity of

n \asymp Θ( \log(4/δ) \cdot d^{k_0}/\eps)

with probability

1-δ

for

δ\in (0,1)

, in contrast with the representative sample complexity

Θ(d^{k_0} \max\set{\eps^{-2},\log d})

. Moreover, such sample complexity is nearly unimprovable since the trained network renders a nearly optimal rate of the nonparametric regression risk of the order

\log({4}/δ) \cdot Θ(d^{k_0}/{n})

with probability at least

1-δ

. On the other hand, the minimax optimal rate for the regression risk with a kernel of rank

Θ(d^{k_0})

is

Θ(d^{k_0}/{n})

, so that the rate of the nonparametric regression risk of the network trained by GDP is nearly minimax optimal. In the case that the ground truth degree

k_0

is unknown, we present a novel and provable adaptive degree selection algorithm which identifies the true degree and achieves the same nearly optimal regression rate. To the best of our knowledge, this is the first time that a nearly optimal risk bound is obtained by training an over-parameterized neural network with a popular activation function (ReLU) and algorithmic guarantee for learning low-degree spherical polynomials. Due to the feature learning capability of GDP, our results are beyond the regular Neural Tangent Kernel (NTK) limit.

Yingzhen Yang, Ping Li

School of Computing and Augmented Intelligence, Arizona State University, Tempe, AZ 85281, USA · VecML Inc., Bellevue, WA 98004, USA

Low-Degree Polynomial Method Neural Network Generalization Nonparametric Regression Shallow Neural Networks Gradient Projection

Mar 18, 2026

A Dual Certificate Approach to Sparsity in Infinite-Width Shallow Neural Networks

In this paper, we study total variation (TV)-regularized training of infinite-width shallow ReLU neural networks, formulated as a convex optimization problem over measures on the unit sphere. Our approach leverages the duality theory of TV-regularized optimization problems to establish rigorous guarantees on the sparsity of the solutions to the training problem. Our analysis further characterizes how and when this sparsity persists in a low noise regime and for small regularization parameter. The key observation that motivates our analysis is that, for ReLU activations, the associated dual certificate is piecewise linear in the weight space. Its linearity regions, which we name dual regions, are determined by the activation patterns of the data via the induced hyperplane arrangement. Taking advantage of this structure, we prove that, on each dual region, the dual certificate admits at most one extreme value. As a consequence, the support of any minimizer is finite, and its cardinality can be bounded from above by a constant depending only on the geometry of the data-induced hyperplane arrangement. Then, we further investigate sufficient conditions ensuring uniqueness of such sparse solution. Finally, under a suitable non-degeneracy condition on the dual certificate along the boundaries of the dual regions, we prove that in the presence of low label noise and for small regularization parameter, solutions to the training problem remain sparse with the same number of Dirac deltas. Additionally, their location and the amplitudes converge, and, in case the locations lie in the interior of a dual region, the convergence happens with a rate that depends linearly on the noise and the regularization parameter.

Leonardo Del Grande, Christoph Brune, Marcello Carioni

Department of Applied Mathematics, University of Twente, 7500AE Enschede, The Netherlands

TV Regularization Shallow Neural Networks Sparse Neural Networks

Mar 13, 2026

Scaling Laws and Pathologies of Single-Layer PINNs: Network Width and PDE Nonlinearity

We establish empirical scaling laws for Single-Layer Physics-Informed Neural Networks on canonical nonlinear PDEs. We identify a dual optimization failure: (i) a baseline pathology, where the solution error fails to decrease with network width, even at fixed nonlinearity, falling short of theoretical approximation bounds, and (ii) a compounding pathology, where this failure is exacerbated by nonlinearity. We provide quantitative evidence that a simple separable power law is insufficient, and that the scaling behavior is governed by a more complex, non-separable relationship. This failure is consistent with the concept of spectral bias, where networks struggle to learn the high-frequency solution components that intensify with nonlinearity. We show that optimization, not approximation capacity, is the primary bottleneck, and propose a methodology to empirically measure these complex scaling effects.

Faris Chaudhry

Department of Computer Science, Imperial College London

Physics-Informed ML Neural Scaling Laws Neural Network Optimization Shallow Neural Networks

Mar 5, 2026

How Does the ReLU Activation Affect the Implicit Bias of Gradient Descent on High-dimensional Neural Network Regression?

Overparameterized ML models, including neural networks, typically induce underdetermined training objectives with multiple global minima. The implicit bias refers to the limiting global minimum that is attained by a common optimization algorithm, such as gradient descent (GD). In this paper, we characterize the implicit bias of GD for training a shallow ReLU model with the squared loss on high-dimensional random features. Prior work (Vardi and Shamir, 2021) showed that the implicit bias does not exist in the worst-case, or corresponds exactly to the minimum-

\ell_2

-norm interpolating solution under exactly orthogonal data (Boursier et al., 2022). Our work interpolates between these two extremes and shows that, for sufficiently high-dimensional random data, the implicit bias approximates the minimum-

\ell_2

-norm solution with high probability with a gap on the order

Θ(\sqrt{n/||λ||_1})

, where

n

is the number of training examples and

λ

denotes the spectrum of the data covariance matrix. Our results are obtained through a novel primal-dual analysis that carefully tracks the evolution of predictions, data-span coefficients, as well as their interactions, and show that the ReLU activation pattern quickly stabilizes with high probability over random data.

Kuo-Wei Lai, Guanghui Wang, Molei Tao +1

School of Electrical and Computer Engineering, Georgia Institute of Technology · College of Computing, Georgia Institute of Technology · School of Mathematics, Georgia Institute of Technology +1

High-Dimensional Linear Regression Random Feature Methods Implicit Bias ReLU Neural Networks Gradient Descent Shallow Neural Networks

Feb 26, 2026

From Shallow Bayesian Neural Networks to Gaussian Processes: General Convergence, Identifiability and Scalable Inference

In this work, we study scaling limits of shallow Bayesian neural networks (BNNs) via their connection to Gaussian processes (GPs), with an emphasis on statistical modeling, identifiability, and scalable inference. We first establish a general convergence result from BNNs to GPs by relaxing assumptions used in prior formulations, and we compare alternative parameterizations of the limiting GP model. Building on this theory, we propose a new covariance function defined as a convex mixture of components induced by four widely used activation functions, and we characterize key properties including positive definiteness and both strict and practical identifiability under different input designs. For computation, we develop a scalable maximum a posterior (MAP) training and prediction procedure using a Nyström approximation, and we show how the Nyström rank and anchor selection control the cost-accuracy trade-off. Experiments on controlled simulations and real-world tabular datasets demonstrate stable hyperparameter estimates and competitive predictive performance at realistic computational cost.

Gracielle Antunes de Araújo, Flávio B. Gonçalves

Universidade do Estado de Minas Gerais (UEMG), FaEnge – Faculdade de Engenharia. · Universidade Federal de Minas Gerais (UFMG), ICEx – Instituto de Ciências Exatas.

Nyström Method Bayesian Neural Networks Shallow Neural Networks Gaussian Processes Gaussian Process Regression

Feb 23, 2026

Smoothness Adaptivity in Constant-Depth Neural Networks: Optimal Rates via Smooth Activations

Smooth activation functions are ubiquitous in modern deep learning, yet their theoretical advantages over non-smooth counterparts remain poorly understood. In this work, we study both approximation and statistical properties of neural networks with smooth activations for learning functions in the Sobolev space

W^{s,\infty}([0,1]^d)

with

s>0

. We prove that constant-depth networks equipped with smooth activations achieve smoothness adaptivity: increasing width alone suffices to attain the minimax-optimal approximation and estimation error rates (up to logarithmic factors). In contrast, for non-smooth activations such as ReLU, smoothness adaptivity is fundamentally limited by depth: the attainable approximation order is bounded by depth, and higher-order smoothness requires proportional depth growth. These results identify activation smoothness as a fundamental mechanism, complementary to depth, for achieving optimal rates over Sobolev function classes. Technically, our analysis is based on a multi-scale approximation framework that yields explicit neural network approximators with controlled parameter norms and model size. This complexity control ensures statistical learnability under empirical risk minimization (ERM) and avoids the impractical

\ell^0

-sparsity constraints commonly required in prior analyses.

Yuhao Liu, Zilin Wang, Lei Wu +1

Department of Mathematical Sciences, Tsinghua University · School of Mathematical Sciences, Peking University · Center for Machine Learning Research, Peking University +1

Neural Network Approximation Theory Neural Network Activation Functions Shallow Neural Networks Statistical Learning Theory

Feb 19, 2026

From Approximation Rates to Loss-Landscape Barrier Decay in Shallow ReLU Networks

We study pathwise connectivity of sublevel sets for one-hidden-layer ReLU networks with constrained first-layer weights and an

\ell_1

penalty on the output layer. The data term is assumed convex and globally Lipschitz in the scalar logit. We first give a finite-width construction that connects any two points of a common sublevel through a path controlled by a loss-consistent compression functional and a first-order perturbation term. The proof replaces the quadratic perturbation estimate in the Freeman--Bruna mechanism by a direct Lipschitz bound. Positive homogeneity is then used in a direction that is compatible with the penalty: every active atom is moved monotonically from the unit ball to the unit sphere while its output coefficient is reduced. Sphere covering and cluster merging consequently give

O(m^{-1/(n-1)})

fixed-level thickening for

n\ge2

, while the one-dimensional two-ray dictionary gives exact connectivity for every

m\ge4

. We also prove internally that the regularized approximation values satisfy

e(l)-e_\infty=O(l^{-1/2})

. More generally, a rate

O(l^{-s})

transfers to a near-optimal barrier rate

O(m^{-s/((n-1)s+1)})

; under the standing assumptions, this yields the explicit rate

O(m^{-1/(n+1)})

. A theorem-aligned finite-distribution experiment complements the analysis. The primary Huber run yields a maximal best certified upper gap

1.66\times10^{-5}

over 720 recorded pairs at widths

m\ge16

; a matched binary-cross-entropy rerun and a 720-endpoint dense-representation stress test probe loss robustness and the active cluster-merging mechanism.

Saveliy Baturin

Independent Researcher, Russia.

Shallow Neural Networks ReLU Neural Networks Mode Connectivity Neural Network Approximation Theory

Nov 26, 2025

SUPN: Shallow Universal Polynomial Networks for Efficient Function Approximation

Deep neural networks (DNNs) and Kolmogorov-Arnold networks (KANs) are flexible function approximators, but their accuracy can require substantial network depth, width, and trainable parameter counts. Beyond making the resulting networks less transparent, large parameter spaces can increase training cost and produce solutions that are sensitive to optimization and parameter initialization, leading to variability in generalization error. For these reasons, we propose Shallow Universal Polynomial Networks (SUPNs), which retain a single hidden layer while increasing the expressivity of each unit through a learnable multivariate polynomial inside each activation, which we refer to as a polynomial pre-activation. This construction trades architectural depth and width for greater expressivity within each hidden unit, reducing parameter counts and training cost while preserving a shallow computational graph. We establish two complementary universal approximation results: SUPNs are universal by increasing network width at fixed polynomial complexity, as in classical shallow-network approximation, and by increasing polynomial complexity at fixed finite width. We further prove that finite SUPNs can approach the best approximation attainable in a prescribed polynomial space to arbitrary relative tolerance, derive explicit quasi-optimal parameters, and establish corresponding convergence rates. We empirically evaluate SUPNs against DNNs, KANs, and polynomial projection on smooth, nonsmooth, and discontinuous function-approximation problems in one, two, and ten dimensions using more than 16,000 trained models. Across the target functions studied, SUPNs frequently achieve lower approximation error and variability than DNNs and KANs at comparable parameter counts while retaining lower training cost, supporting their use as efficient function approximators when the target structure is not known a priori.

Zachary Morrow, Michael Penwarden, Brian Chen +3

Polynomial Neural Networks Neural Network Approximation Theory Shallow Neural Networks Universal Approximation

Oct 29, 2025

Nonlinear Dynamics In Optimization Landscape of Shallow Neural Networks with Tunable Leaky ReLU

In this work, we study the nonlinear dynamics of a shallow neural network trained with mean-squared loss and leaky ReLU activation. Under Gaussian inputs and equal layer width k, (1) we establish, based on the equivariant gradient degree, a theoretical framework, applicable to any number of neurons k>= 4, to detect bifurcation of critical points with associated symmetries from global minimum as leaky parameter

α

varies. Typically, our analysis reveals that a multi-mode degeneracy consistently occurs at the critical number 0, independent of k. (2) As a by-product, we further show that such bifurcations are width-independent, arise only for nonnegative

α

and that the global minimum undergoes no further symmetry-breaking instability throughout the engineering regime

α

in range (0,1). An explicit example with k=5 is presented to illustrate the framework and exhibit the resulting bifurcation together with their symmetries.

Jingzhou Liu

Shallow Neural Networks Dynamical Systems Neural Network Optimization Neural Network Activation Functions Neural Network Training Dynamics

Oct 5, 2025

Configuration-Dependent Lower Bounds for Approximation by Shallow ReLU $^k$ Networks on the Sphere

We establish two related but logically distinct results for shallow ReLU

^k

neural networks on the unit sphere

\SS^d

. First, for an arbitrary set of inner neural-network parameters, the best

\mathcal{L}^2(\SS^d)

approximation of a fixed target function with smoothness

r>\tfrac{d+2k+1}{2}

admits an asymptotic lower bound given by a constant multiple of

n^{-1/2}\underline{h}^{k+1/2}

, where

\underline{h}

denotes the antipodal separation distance of the normalized inner-parameter set. This lower bound depends explicitly on the parameter configuration through

\underline{h}

and applies without additional assumptions on the parameters. Second, for antipodally quasi-uniform parameters,

\underline{h}\simeq n^{-1/d}

, and the lower bound establishes the exact saturation order

n^{-\frac{d+2k+1}{2d}}

for such parameter families: a target function with regularity greater than

\frac{d+2k+1}{2}

and satisfying the required parity condition can be approximated at this rate, whereas approximation at any strictly faster rate forces the target function to be zero. Our results therefore place linearized neural-network approximation within the classical saturation framework and show that, although ReLU

^k

network spaces can outperform finite elements of the same degree, this advantage is intrinsically limited.

Tong Mao, Jinchao Xu

Shenzhen Loop Area Institute, Shenzhen 518000, P. R. China · King Abdullah University of Science and Technology, Thuwal 23955, Saudi Arabia.

Shallow Neural Networks Neural Network Approximation Theory Neural Network Expressivity Function Approximation

Apr 3, 2025

On shallow feedforward neural networks with inputs from a topological space

We study feedforward neural networks with inputs from a topological space (TFNNs). We prove a universal approximation theorem for shallow TFNNs, which demonstrates their capacity to approximate any continuous function defined on this topological space. As an application, we obtain an approximative version of Kolmogorov's superposition theorem for compact metric spaces.

Vugar Ismailov

Institute of Mathematics and Mechanics, Baku, Azerbaijan · Center for Mathematics and its Applications, Khazar University, Baku, Azerbaijan

Feedforward Neural Networks Neural Network Approximation Theory Shallow Neural Networks Universal Approximation

Apr 2, 2025

AYLA: Architecting a loss landscape in shallow neural networks to accelerate feature recovery

Feature learning in shallow neural networks exhibits rich yet fragile dynamics, including prolonged plateaus, abrupt phase transitions, and sensitivity to optimization hyperparameters. While recent theoretical work has characterized these behaviors through the geometry of loss landscapes, saddle escape mechanisms, and emergent scaling laws, practical methods for actively shaping these dynamics remain limited. In this paper, we introduce AYLA, a principled loss reparameterization framework that dynamically modulates gradient magnitudes during training without altering the location of stationary points or optimal solutions. AYLA applies a smooth, sigmoid-controlled power-law transformation to empirical loss, yielding a state-dependent effective learning rate that accelerates descent in flat or saddle-dominated regions while stabilizing late-stage optimization. Crucially, AYLA preserves all critical points of the original objective, acting solely as a monotone transformation that reshapes optimization trajectories rather than objectives. We evaluate AYLA in controlled teacher student settings using two-layer tanh networks trained on synthetic Gaussian data. Across stochastic gradient descent and multiple loss-exponent schedules, AYLA consistently improves feature recovery. This evidence is observed in terms of weight alignment, per-neuron cosine similarity, hidden-activation correlation, and spectral properties of learned representations, while AYLA maintains competitive or faster loss convergence. Spectral analyses further demonstrate that AYLA mitigates rank collapse and promotes richer internal representations, signaling a transition from lazy to active feature-learning regimes. AYLA offers a lightweight, theoretically grounded way to improve shallow-network optimization, especially in resource-limited or noise-sensitive settings.

Behnam Gheshlaghi, Shahin Atakishiyev

Shallow Neural Networks Representation Learning Neural Network Optimization Neural Network Training Dynamics

Nov 10, 2024

Overcoming a Learning Bottleneck in Impartial Games: A Circuit Complexity Approach

Why can neural networks struggle to learn optimal play in a game with a simple, efficient solution algorithm? Motivated by empirical difficulties with parity in impartial games, we use NIM to study a representational bottleneck and three ways around it. We show that a restricted model of shallow, fixed-scale quantised networks lies in

\mathrm{AC}^0

, a circuit class unable to compute parity on arbitrarily large inputs. This prevents optimal single-position play across all board sizes within the model. The obstruction also guides architectural design: two-frame inputs allow restoration from a known zero-nim-sum position; complementary rollout policies reveal the missing invariant; and persistent recurrence accumulates it row by row. Each route retains shallow network components while supplying additional information or processing time. Supervised experiments illustrate the benefits of two-frame restoration and multiple rollout heads; recurrence is a theoretical construction. These results establish representational possibilities, not guarantees of successful training:

\mathrm{AC}^0

does not characterise practical learnability. Core mathematical results and constructions have been checked in Lean, with classical complexity results as explicit external inputs. The interactive edition and complete Lean sources are available online at https://sr123.github.io/impartial-games-learning-lean/.

Søren Riis

Centre for Fundamentals of AI and Computational Theory, Queen Mary University of London, London, United Kingdom.

Circuit Complexity Neural Network Expressivity Shallow Neural Networks