Neural Network Approximation Theory

Follow this topic Ask about this topic

Momentum

15 papers in the last four weeks, up 15% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 163

Aug 9, 2026cs.LG

Approximation Rates for Metaplectic Neural Networks

In this paper we develop quantitative approximation results for shallow neural networks constructed using a dictionary based on metaplectic operators. First, we extend the concept of Barron spaces by considering a symplectically motivated extension of the Fourier transform, known as the metaplectic transform. Then, after establishing embedding between metaplectic Barron spaces and Sobolev spaces we consider a neural metaplectic dictionary and we prove Monte-Carlo approximation bounds for metaplectic Barron functions using finite linear combinations of atoms of the dictionary. Finally, we validate the introduction of the neural metaplectic dictionary by devising a deep neural network architecture that uses as building blocks the atoms of the dictionary. We test it to approximate solutions of time-dependent Schrödinger equations, demonstrating better performance compared to classical phyisics informed neural networks architectures.

Ahmed Abdeljawad, Marcello Carioni, Elena Cordero

Johann Radon Institute of Computational and Applied Mathematics (RICAM) Austrian Academy of Sciences Altenberger Straße 69, A-4040 Linz, Austria · Department of Applied Mathematics University of Twente 7500AE Enschede, The Netherlands · Dipartimento di Matematica Università degli Studi di Torino via Carlo Alberto 10, 10123 Torino, Italy

Shallow Neural Networks Neural Network Approximation Theory

Aug 8, 2026math.PR

A cylindrical neural approximation theorem for conditional laws of McKean-Vlasov equations with common noise

We introduce conditional cylindrical neural networks for approximating functionals of conditional laws in McKean-Vlasov equations with common noise. Fourier moments of the initial law and truncated signatures of the time augmented common noise are mapped by a mixture density network to a Gaussian mixture approximation of the conditional law. A cylindrical neural network then evaluates the target functional through analytic integrals against this predicted measure. Rough path well posedness and stability provide a conditional law map that is continuous in the initial distribution and the rough driver and agrees almost surely with the classical conditional law at the Itô Brownian lift. Combining this continuity with Fourier separation, signature uniqueness, Wasserstein density of Gaussian mixtures, and neural universal approximation, we prove an

L^2

universal approximation theorem for continuous square integrable functionals. The numerical study implements the resulting two stage procedure on six examples, including non Gaussian initial laws, nonlinear drift, multiplicative common noise, and a two dimensional state. Independent particle references are used when no closed form law is available. The learned conditional law and functional approximations consistently improve on the empirical particle plug in, and additional experiments examine feature sensitivity, training from one terminal observation per common noise scenario, and Itô--Stratonovich consistency.

Nacira Agram, Reda Hmioui, Jan Rems

Department of Mathematics, KTH Royal Institute of Technology, 100 44 Stockholm, Sweden. · Digital Futures, Sweden. · Grenoble INP – Ensimag, Université Grenoble Alpes, Grenoble, France. +1

Mean-Field Theory Neural Network Approximation Theory

Aug 7, 2026math.NA

Optimal Neural Network Approximation via Empirical Least Squares with Deterministic Samples

We develop a rigorous theory of discrete residual least-squares approximation for elliptic spectral equations

\mathfrak L_βu=f

using linearized ReLU

^k

neural networks on the sphere, where

\mathfrak L_β

is a positive elliptic spectral multiplier of order

β

. Given a parameter set

Θ_n=\{θ_{j}^*\}_{j=1}^n\subset\mathbb S^d

, we approximate

u

in the linearized network space

L_n^k(Θ_n)

by the discrete residual on the collocation points

\{η_i^*\}_{i=1}^m

\begin{equation*} u_{n,m}\in\arg\min_{v_n\in L_n^k(Θ_n)}\frac1m\sum_{i=1}^m\left(f(η_i^)-\mathfrak L_βv_n(η_i^)\right)^2. \end{equation*} With

k>\frac{d-1}{2}+β

, for antipodally quasi-uniform network parameter sets and any quasi-uniform collocation points with

m\gtrsim n

, we prove that \begin{equation*} |u-u_{n,m}|{\mathcal H^β(\mathbb S^d)}\eqsim|f-\mathfrak L_βu{n,m}|{\mathcal L^2(\mathbb S^d)}\lesssim n^{-\frac{r}{d}} \begin{cases} |f|{\mathcal W^{r,p}(\mathbb S^d)},&\frac{d}{p}<r\leq \frac{d}{2},~p>2,\ |f|{\mathcal H^r(\mathbb S^d)},&r>\frac{d}{2}. \end{cases} \end{equation*} We also establish a high-probability residual estimate, up to a logarithmic factor and an arbitrarily small smoothness loss, for i.i.d.\ uniformly distributed collocation points. The key analytical ingredient is a Bernstein inequality for linearized ReLU $^k$ network spaces. If $\underline h$ denotes the antipodal separation distance of the network parameters, then \begin{equation*} |v_n|{\mathcal H^r(\mathbb S^d)}\lesssim\underline h^{-(r-s)}|v_n|_{\mathcal H^s(\mathbb S^d)},\qquad 0\leq s<r<k+\tfrac12. \end{equation*}

Xinliang Liu, Tong Mao, Jinchao Xu

School of Mathematical Sciences, Ocean University of China, Qingdao 266100, China · Shenzhen Loop Area Institute, Shenzhen 518000, P. R. China

Neural PDE Solvers ReLU Neural Networks Neural Network Approximation Theory

Aug 5, 2026cs.LG

Fixed and Adaptive Topological DeepONets: Functional Measurements on Hausdorff Locally Convex Spaces

Deep Operator Networks (DeepONets; arXiv:1910.03193) typically encode an input function through point values on a fixed discretization. Building on the Topological DeepONet framework of Ismailov (arXiv:2603.11972), we replace point samples by continuous linear functionals drawn from the continuous dual of a Hausdorff locally convex space

({V},\{p_α\}_{α\in A})

, whose topology is generated by a point-separating family of seminorms rather than a single norm, and develop fixed and adaptive functional measurement systems. Measurements are combined with the coefficient-space Two-Step procedure of Lee and Shin (arXiv:2309.01020), while a training-only decoder and regularization stabilize the adaptive coordinates. We derive a discrete error decomposition separating measurement, output-basis, and neural-approximation errors, together with a Barron-rate refinement. The framework is evaluated on the antiderivative operator, a non-normable locally convex input space, heterogeneous Darcy flow, a controlled operator, and fixed-time and time-evolving Navier-Stokes vorticity operators. In the heterogeneous Darcy problem, the functional models retain nearly resolution-independent errors of 5.5-5.6% on unseen grids, while in the controlled problem adaptive measurements reduce the mean error below 1.2%. For the fixed-time Navier-Stokes problem, the Adaptive Topological DeepONet is the most accurate DeepONet-based model, attaining a mean relative

L^2

error of 1.685% +/- 0.017% using 128 functional coordinates. A comparably sized Fourier neural operator (FNO; arXiv:2010.08895) achieves the lower error 0.832% +/- 0.172%, but requires the full 64x64 input field, twice the training time, and 10.7x greater peak GPU memory. The formulation provides compact, interpretable, and discretization-portable coordinates in the continuous dual

V'

, including for non-normable input spaces.

Khemraj Shukla, George Em Karniadakis

Division of Applied Mathematics, Brown University Providence, Rhode Island 02912, USA

PDE Operator Learning Neural Network Approximation Theory Deep Operator Networks

Aug 2, 2026cs.LG

Do Neural Networks Really Beat the Curse of Dimensionality? A Bit-Complexity View

Traditional approximation theory measures convergence rates in terms of the number of parameters or degrees of freedom. However, practical computation operates under finite precision: parameters must be encoded using a finite number of bits. Therefore, approximation efficiency should be evaluated in terms of computational bit complexity, which is intrinsically connected to the metric entropy of the underlying function class. In this work, we develop a unified approximation framework based on binary encoding and metric entropy. We analyze classical methods (including polynomial approximation, sparse grids, and finite elements) as well as shallow and deep neural networks, and compare their approximation rates for function classes with comparable metric entropy. We observe that, when evaluated in terms of bits, most classical methods are in general suboptimal relative to the intrinsic limits dictated by metric entropy, while neural network methods may exhibit different behaviors. We show that when complexity is measured in bits rather than parameters, no method fundamentally exceeds the approximation order achieved by classical approaches. Our results also indicate that many seeming advantages of neural networks, including dimension-independent rates and superconvergence phenomena, stem from differences in function class complexity rather than intrinsic architectural superiority. In this sense, the traditional curse of dimensionality can be misleading; the fundamental limitation is instead a curse of bit complexity, governed by metric entropy.

Tong Mao, Jinchao Xu

King Abdullah University of Science and Technology, Thuwal 23955, Saudi Arabia

Neural Network Approximation Theory

Jul 30, 2026math.AP

Fractional Parabolic Partial Differential Equations in Anisotropic Spectral Barron Spaces: Regularity and Neural Approximation

We study fractional parabolic initial-value problems with lower-order drift and potential terms in anisotropic spectral Barron spaces, defined by weighted space--time Fourier

L^1

norms adapted to parabolic scaling. We prove existence, uniqueness, and maximal regularity with a gain of one derivative in time and

γ

derivatives in space, where

γ>0

is the order of the fractional Laplacian. The evolution is defined only for

t\geq0

, whereas the finite-time norm requires a global extension with sufficient temporal Fourier decay. We construct a finite reflected semigroup extension using a Vandermonde system to match derivatives at

t=0

, obtaining temporal Fourier estimates uniform in the semigroup parameter. Combined with Fourier multiplier estimates for the damped principal operator, it yields maximal regularity. Dimension-independent multiplication estimates support a finite regularity bootstrap, while interpolation and sufficient damping absorb the lower-order terms in the base estimate. The a priori estimate and the method of continuity yield maximal regularity without smallness assumptions on the lower-order coefficients. A frequency-localized counterexample shows that a uniform-in-time spatial Barron bound on the forcing does not imply the corresponding two-derivative solution bound, even for the one-dimensional heat equation. Using this regularity, Fourier sampling yields

n^{-1/2}

approximation rates for the solution in mixed space--time Sobolev norms using shallow networks with suitable activations. Sampling in a product Hilbert space yields a population-level PINN consistency estimate for shallow cosine networks on a bounded cylinder. There exists a single width-

n

network for which the sum of the squared mixed-Sobolev solution error, the squared

L^2

-norm of the residual for the whole-space fractional equation, and the squared initial-data error is

O(n^{-1})

.

Jae-Hwan Choi, Hyojae Lim, Jinsol Seo +2

Neural PDE Solvers PDE Solving Neural Network Approximation Theory Physics-Informed ML

Jul 30, 2026stat.ML

Error Analysis of Neural-Network-Based Engression

Engression (Shen and Meinshausen, 2024) learns a conditional distribution by fitting a generative model

Y = f(X,\varepsilon)

under the energy score, a strictly proper scoring rule. We provide a theoretical error analysis of engression implemented with deep neural networks. We decompose the excess risk into three components: the approximation error, the stochastic error, and the Monte Carlo error. Based on this decomposition, we establish convergence rates under the assumption that the target conditional generator admits a compositional smoothness structure.

Juntong Chen, Zijian Guo, Xinwei Shen

School of Mathematical Sciences, Xiamen University · Center for Data Science, Zhejiang University · Department of Statistics, University of Washington

Conditional Distribution Estimation Neural Network Approximation Theory Statistical Learning Theory

Jul 29, 2026cs.LG

Universality and Approximation Rates of Graph Neural Networks with Random Features

We investigate message-passing graph neural networks with random node features. Random node features are known to enhance the expressiveness of graph neural networks (GNNs) both theoretically and empirically. Here, we establish a novel universality result focusing on permutation-equivariant neural networks (PENNs), a class of GNNs built from feedforward neural network components that subsumes many prominent GNN architectures. We show that PENNs, combined with partially random node features, can approximate arbitrarily well in probability any measurable permutation-invariant or permutation-equivariant function on directed graphs of fixed size with multidimensional node and edge features. For

k

-times continuously differentiable functions,

k\geq 2

, we also derive upper bounds on the approximation rates, relating the complexity of the feedforward components of a PENN in terms of layer depth and number of nonzero weights to the desired approximation accuracy.

Lukas Gonon, Thilo Meyer-Brandis, Niklas Weber

School of Computer Science University of St. Gallen St. Gallen, 9000, Switzerland · Department of Mathematics LMU Munich Munich, 80333, Germany

Graph Neural Networks Equivariant Neural Networks GNN Expressivity Message Passing Neural Networks Neural Network Approximation Theory

Jul 28, 2026cs.LG

Algorithmic Separation between Constant-Depth and Logarithmic-Depth Neural Networks

Despite the empirical advantages of deep networks over shallow ones, theoretical depth separations largely concern approximation power, while algorithmic results are mostly limited to comparisons between two- and three-layer networks. In this work, we prove the first algorithmic separation between constant-depth and logarithmic-depth networks. Specifically, we identify a class of Boolean functions with hierarchically structured Fourier spectra that logarithmic-depth networks can learn efficiently using layerwise coordinate descent by reconstructing the spectra hierarchically and adaptively. We also exhibit a subclass for which every constant-depth, polynomial-width network with sufficiently regular activations and controlled spectral norms must incur constant

L^2

approximation error under the uniform distribution over the hypercube.

Yunwei Ren, Zihao Wang, Jason D. Lee

Princeton University · Stanford University · University of California, Berkeley

Boolean Function Learning Neural Network Approximation Theory Feedforward Neural Networks

Jul 25, 2026stat.ML

Operator Neural Jump ODEs: $L^2$ -optimal prediction in function spaces

In this paper, we study the extension of Neural Jump ODEs to infinite-dimensional function spaces. In particular, the underlying process

X

now takes values in

L^2(Ξ, \mathbb{R}^{d_X})

instead of

\mathbb{R}^{d_X}

and the Operator NJ-ODE approximates the optimal predictor of this process by producing a representative of the conditional expectation. The NJ-ODE model is a framework for online learning the optimal prediction of continuous-time stochastic processes, given discrete, possibly irregular and incomplete past observations. In a series of works, this model has been extended to deal with generic path-dependent processes, with observation noise and dependent observations, with long-term predictions, and with input-output systems. However, throughout all of these works, the underlying processes were restricted to be finite-dimensional. In particular, function-valued problems, like yield curve or volatility surface predictions, could only be handled through discretization, which inherently leads to a loss of information. In this work, we build on ideas from Neural Operator methods that allow us to extend the NJ-ODE framework to an infinite-dimensional output process. To prove convergence of the NJ-ODE to the optimal prediction process, we develop a new approximation strategy that also generalizes previous works in the finite-dimensional setting by considerably weakening the assumptions.

Florian Krach, Oliver Löthgren, Josef Teichmann

Department of Mathematics ETH Zurich

Time Series Forecasting Neural Network Approximation Theory Neural ODEs

Jul 22, 2026cs.LG

Shallower ReLU Network Representations via Exact Linear Algebra

We study the depth required by ReLU networks to exactly represent piecewise linear functions, focusing specifically on the maximum function. This problem has recently received significant attention in both the ML and TCS literature. We prove that

\max_n(x)=\max\{x_1,\ldots,x_n\}

is exactly representable with two hidden layers for every

n\leq 12

. Previously, this was only known up to

n\leq5

[Bakaev, Brunck, Hertrich, Stade, Yehudayoff, STOC'26]. We obtain our constructions through an exact computer-assisted search within a space of candidate solutions: After a symmetry reduction, we obtain a finite system of linear equations over

\mathbb{Q}

such that any solution yields a valid representation of the maximum function. The resulting constructions have a structured first hidden layer, which enables recursive substitution into deeper networks. This yields an exact ReLU representation of

\max_n

with at most

\lceil \log_6(n/2) \rceil+1

hidden layers. Consequently, every continuous piecewise-linear function on

\mathbb{R}^d

admits an exact representation with at most

\lceil\log_6((d+1)/2)\rceil+1

hidden layers; in particular, two hidden layers suffice for

d\leq 11

. Again, these results improve upon [Bakaev, Brunck, Hertrich, Stade, Yehudayoff, STOC'26], who proved analogous logarithmic bounds with base three.

Kilian Rueß, Gennadiy Averkov, Florestan Brunck +7

University of Technology Nuremberg · Brandenburg University of Technology Cottbus–Senftenberg · University of Copenhagen +3

Shallow Neural Networks ReLU Neural Networks Neural Network Approximation Theory

Jul 22, 2026cs.LG

Local Stability and Gaussian Smoothing of Quantized Neural Networks

We study Gaussian averaging as a smooth surrogate for quantized neural models. Under bounded local oscillation, we derive a local dimension-dependent bound on |f-g|, linking Gaussian smoothing to the stability analysis of discontinuous networks. We compute closed-form Gaussian averages of the rectified linear unit (ReLU) and sign activation functions, and illustrate the mechanism on a high-dimensional binary perceptron, where layer-preactivation aggregation under an explicit quantization-noise surrogate yields the Gaussian envelope used in inference-side smoothing and training-side smooth surrogate gradients.

Sergey Salishev, Anton Makarov, Oleg Granichin

St. Petersburg State University, 199034, Saint Petersburg, Russia

Neural Network Activation Functions Neural Network Approximation Theory Neural Network Quantization

Jul 21, 2026math.NA

Boundary-Adapted PINNs for Elliptic Dirichlet Problems: $H^2(Ω)$ A Priori Error Bounds with Application to Mean Escape Time Computation

Motivated by the numerical computation of the Mean Escape Time (MET)

τ:Ω\to\mathbb{R}

of a stochastic process from a bounded domain

Ω\subseteq\mathbb{R}^d

, we study elliptic Dirichlet boundary value problems (BVPs) using boundary-enforced Physics-Informed Neural Networks (PINNs), in which the Dirichlet condition is imposed exactly by multiplying the network output with a predefined distance-to-boundary approximation

ρ

. Combining approximation-theoretic and statistical-learning arguments for Rectified Quadratic Unit (ReQU) and hyperbolic tangent (tanh) networks, we derive a priori error bounds that make explicit the dependence on

ρ

. In particular, we show that exact boundary enforcement alone is not enough for

H^2(Ω)

error bounds, and that a sufficient and essentially necessary condition is for

ρ

to be a smooth distance approximation

\textit{normalized to first order}

, of the kind constructed in arXiv:2104.08426 [math.NA]. We thereby identify this subclass of

\textit{boundary-adapted}

PINNs as the appropriate neural network ansatz for solving Dirichlet BVPs. Numerical experiments support the theory, showing that appropriate choices of

ρ

improve accuracy and convergence, while poorly chosen distance functions can substantially degrade the solution. Our proof also yields new VC-dimension bounds for hypothesis spaces of higher-order derivatives of ReQU and tanh networks, together with new approximation bounds for shallow ReQU networks in higher-order Sobolev norms, all of which are of important independent interest.

Nathanael Tepakbong, Jun Fan, Xiang Zhou +1

Department of Data Science, City University of Hong Kong · Department of Mathematics, Hong Kong Baptist University · Department of Mathematics, City University of Hong Kong +1

PDE Solving Neural Network Approximation Theory Physics-Informed ML Generalization Bounds

Jul 21, 2026math.NA

Neural network realization of binary refinement iterates via a two-chart atlas selector

Refinement operators generate many functions used in wavelet constructions, subdivision schemes, and geometric modeling. Their finite iterates can develop rapidly increasing numbers of linear pieces, making them a natural test case for the expressive power of deep neural networks. Earlier work showed that, for scalar binary refinement with a finitely supported mask, every compactly supported continuous piecewise linear seed has finite refinement iterates that admit exact ReLU realizations of fixed width and depth growing linearly with the number of refinement steps. The present paper gives a new construction of this known theorem. The difficulty is that the refinement cascade is driven by discontinuous binary digit choices, whereas ReLU networks produce continuous piecewise linear maps. We represent the residual dynamics on a polygonal model of the circle and describe each residual position in two overlapping coordinate systems, one ordinary and one shifted by one half. Their discontinuities occur at different points. The network switches between the two descriptions only where both are valid and the corresponding fixed linear cascade updates agree, so the switch is exact and requires no multiplication by a variable selector. The construction also gives exact readout of every continuous piecewise linear circle function satisfying the natural endpoint compatibility condition. Localized seeds are handled by a two-pass network, and translation covariance, finite decomposition, and gluing extend the result to arbitrary compactly supported continuous piecewise linear seeds in a preserved support window.

Tsogtgerel Gantumur

McGill University, Montr´eal, QC, Canada · National University of Mongolia, Ulaanbaatar, Mongolia · Mongolian Academy of Sciences, Institute of Mathematics and Digital Technology

ReLU Neural Networks Neural Network Approximation Theory Iterative Refinement Neural Network Expressivity

Jul 21, 2026cs.LG

Functional Equivalence and Geometric Diversity in Neural Network Approximations: An Empirical Characterization

The Universal Approximation Theorem states that a neural network with a single hidden layer is sufficient to approximate any continuous univariate function on a compact domain to arbitrary error. However, the uniqueness of such neural network representations is not guaranteed, raising questions about practical identifiability. In this work, we address this concern by analyzing functional equivalence and geometric diversity of neural network approximations to a few elementary mathematical functions. The analysis includes an extensive study of single-layer neural networks and multilayer perceptrons under noisy and noise-free conditions. Beyond just network capacity, we study the geometric properties through the lens of sloppiness, characterized by the eigen spectrum of the Hessian of the cost function and the effective rank to quantify the dimensionality of parameter space. The study reveals large equivalence classes of functionally indistinguishable yet geometrically diverse networks that consistently exhibit low effective rank and structural redundancy. Finally, a model select criterion is proposed for identifying optimal models based on parsimony, ease of estimation, and inference efficiency.

Anuragine S A, Prem Jagadeesan

Amrita School of Artificial Intelligence Amrita Vishwa Vidyapeetham, Coimbatore, India

Multilayer Perceptrons Neural Representation Geometry Parameter Identifiability Neural Network Approximation Theory

Jul 19, 2026math.AG

Expressivity of Shallow Neural Networks Over Finite Fields

We study the expressivity of shallow polynomial neural networks (PNNs) with monomial activation functions over finite fields. For a given architecture, we define a neuromanifold as the image of the map from all possible network weights into the product of polynomial rings. We quantify the expressivity by the cardinality of the neuromanifold, and derive a natural lower and upper bound. This leads to counting rational points over finite fields, a problem closely linked to the Weil conjectures. Finally, we present an architecture that exhibits a striking difference in the neuromanifolds when considered over a characteristic zero versus a finite-characteristic field, illustrating the critical role of field characteristic in the notion of expressivity.

Maksym Zubkov, Carol Wu, Shiwei Yang +2

Department of Mathematics, University of British Columbia, 1984 Mathematics Rd, Vancouver, V6T 1Z2, BC, Canada

Shallow Neural Networks Polynomial Neural Networks Neural Network Approximation Theory Neural Network Expressivity

Jul 17, 2026math.NA

Non-Asymptotic Variational Learning for Monotone Nonlinear Multiscale Elliptic Equations: Scale-Robust Primal-Dual Bounds and Strong-Form Statistical Ill-Conditioning

We develop a non-asymptotic approximation, sampling, and finite-iteration optimization theory for variational physics-informed approximation of uniformly monotone nonlinear multiscale elliptic equations. For boundary-compatible neural feature classes, the population error splits into approximation, empirical quadrature, and projected-gradient terms, with all non-approximation constants uniform in the microscopic scale

\varepsilon

. Assuming a quantitative corrected

H^1

-estimate, a two-scale state class yields

\mathcal A_m^\varepsilon \le C\bigl(\varepsilon+Φ_{0,m_0}^2+Φ_{1,m_1}^2\bigr)

in arbitrary dimension. We further introduce a convex primal-dual physics loss whose population value is a computable upper certificate for the state error. With additional flux-corrector regularity, a divergence-compatible two-scale flux class gives a certified state-flux bound combining

O(\varepsilon)

approximation, state and flux feature errors, empirical sampling error, and an

O(K^{-1})

optimization term. In contrast, for general periodic nonlinear fluxes satisfying a natural nondegeneracy condition, the empirical Rademacher complexities of strong-residual and squared-residual classes are bounded below by constant multiples of

(\varepsilon\sqrt N)^{-1}

and

(\varepsilon^2\sqrt N)^{-1}

, respectively. These optimizer-independent lower bounds hold in every spatial dimension. Numerical experiments confirm the predicted

\varepsilon

- and

N

-scalings for nonlinear fluxes in

d=1,2,3

, validate every computed primal-dual certificate, and show that corrector-enriched classes substantially reduce energy and

H^1

errors as the microscopic scale is refined.

Ronald Katende

Department of Mathematics Kabale University Kikungiri Hill, Katuna Road, Kabale, Uganda

Primal-Dual Optimization Neural PDE Solvers Neural Network Approximation Theory Physics-Informed ML Statistical Learning Theory

Jul 15, 2026math.PR

NeuralChaos: Optimal Adapted Approximation of Square Integrable Predictable Processes

We address fundamental challenges in representing and computing

\mathbb{R}^{d}

-valued predictable square-integrable processes over

[0,T]

, collected in the space

\mathcal{H}^2_T(\mathbb{R}^{d})

. These processes are central to continuous-time stochastic control, reinforcement learning, and mathematical finance. Although Wiener-chaos expansions offer strong theoretical tools, traditional computational methods are hindered by the need for large chaos dictionaries and high-order iterated integrals. To overcome these obstacles, we introduce NeuralChaos -- a neural operator architecture that produces elements of

\mathcal{H}^2_T(\mathbb{R}^{d})

using only finitely many evaluations of the driving Brownian motion, while preserving predictability and square-integrability. We prove that NeuralChaos is dense in

\mathcal{H}^2_T(\mathbb{R}^{d})

and achieves the best

N

-term chaoslet approximation rates for compressible and Malliavin--Sobolev regular processes. Moreover, compressibility is shown to be typical for processes from

\mathcal{H}^2_T(\mathbb{R}^{d})

under non-degenerate sub-Gaussian sampling. In contrast, we show that finite-dimensional Markovian neural SDE models constitute a meagre and Gaussian-null subset in

\mathcal{H}^2_T(\mathbb{R}^{d})

, regardless of discretization, whereas compressible processes are generic. Numerical experiments on a stochastic optimal control problem and dynamic hedging highlight the practical effectiveness of our approach. Our results enable more efficient and expressive modelling in stochastic analysis and mathematical finance.

Anastasis Kratsios, Giulia Livieri, Philipp Schmocker

Neural Network Approximation Theory Neural Operators

Jul 15, 2026math.NA

Approximation of solutions of parameter-dependent problems by residual neural networks

We develop a convergent scheme to train neural networks involving analytic activation functions based on gradient flows. Convergence properties are guaranteed by Lojasiewicz theory. The main advantage of this approach is its simplicity of implementation. The coefficients of the network are approximated by solving a system of ordinary differential equations. We test the method by constructing residual neural network approximations of solutions of parametric problems. The dependence of the solutions of simple ordinary differential equations on a few parameters is correctly reproduced. The solutions of inverse problems involving wave constraints which depend on a few parameters can be reasonably approximated, even in regions in which the problem is severely ill posed.

Ana Carpio

Universidad Complutense de Madrid, Spain

Neural Surrogate Modeling Neural Network Optimization Neural Network Approximation Theory

Jul 12, 2026stat.ML

Approximation of Analytic Functions by ReLU Neural Networks with Adjustable Depth and Width

In contrast to most studies on neural network approximation theory that characterize results through a single parameter, such as the total number of network parameters, \cite{shen2020deep} pioneered the characterization of approximation rates as a joint function of the width parameter

N

and the depth parameter

L

, thereby granting greater architectural flexibility. Existing works using the

(N,L)

-characterization focus on function classes with finite smoothness

s

, establishing a typical approximation rate of

\mathcal{O}\left(N^{-2s/d}L^{-2s/d}\right)

with

d

denoting the input dimension, which indicates that network depth and width play symmetric roles for these classes. In contrast, this paper establishes upper bounds for the approximation of analytic functions, which possess infinite smoothness, via ReLU networks under the

(N,L)

-characterization. Specifically, we derive approximation rates of

\mathcal{O}\left(N^{-C L^τ}\right)

, where

C>0

is some constant and

τ>0

is a parameter influenced by the relation between

L

and

N

. In particular,

τ=1

if

N

scales roughly as

L^d

. Our findings reveal that depth plays a more critical role than width in the context of analytic function approximation. The main technical difficulty of obtaining such upper bounds lies in the trade-off between the smoothness parameters and the approximation accuracy. To overcome this difficulty, we employ refined constructions of several ReLU networks to approximate power functions, multivariate multiplication, and polynomials, which may be of independent interest.

Yanming Lai, Defeng Sun, Yang Wang

Department of Applied Mathematics, The Hong Kong Polytechnic University, Hung Hom, Hong Kong, China · Department of Mathematics, The University of Hong Kong, Pokfulam, Hong Kong, China

ReLU Neural Networks Neural Network Approximation Theory

Jul 10, 2026cs.LG

All you need is SAMPAT

The current state of the art in AI/ML rests on deep neural architectures, which, in general, suffer from a lack of interpretability. Interpretability is crucial to gleaning insights while analyzing experimental data, where quantitative predictions may not be adequate for a scientist. We present a three layer neural architecture, SAMPAT (Smooth Approximation via Multivariate Polynomials and Analytic Transformations), that can provably learn a continuous, everywhere differentiable function, that can approximate any smooth function arbitrarily closely. SAMPAT's approximant can be expressed as a closed and compact algebraic, analytic expression, providing complete interpretability. Experiments on synthetic and benchmark datasets indicate that SAMPAT yields competitive performance with simpler representations. For many tasks, a two layer SAMPAT suffices. By imposing restrictions on the connectivity between neurons, SAMPAT may be used to provide a range of approximants, including regular and trigonometric polynomials, rational expressions, Gaussians, mixtures of Gaussians, as well as arbitrary combinations of the same; without restrictions, it learns a suitable structure. SAMPAT may be used to factorize polynomials and model nonlinear systems. With the addition of skip connections, a 4 to 6 layer SAMPAT is adequate to represent a substantive range of methods widely used in AI/ML, allowing the choice of a model's family, not just its parameters, to also be optimized as part of the learning process.

Jayadeva, Madhur Aswani

Department of Electrical Engineering, Indian Institute of Technology, Delhi

Shallow Neural Networks Interpretable ML Neural Network Interpretability Neural Network Approximation Theory Neural Network Expressivity

Jul 8, 2026cs.LG

An optimal control approach for neural network architecture adaptation with a posteriori error estimation

This work presents a novel approach for adapting neural network architecture along the depth based on a posteriori error estimation. By formulating neural network training as a continuous-time optimal control problem, we derive rigorous error estimates that quantify how approximation error distributes across network layers. This error decomposition enables a principled depth adaptation strategy: new layers are inserted at locations of maximum estimated error, allowing the network to efficiently capture complex, nonlinear variations in the underlying problem. Our framework introduces a novel network architecture that treats weights and biases as piecewise linear functions varying across layers, with the error estimator bounding the discrepancy between this discrete representation and the true continuous optimal control solution. The approach leverages dual weighted residual methodology from finite element analysis to derive computable upper bounds on the functional error. A key theoretical contribution is the derivation of explicit error bounds that decompose the total approximation error into interval-wise contributions, providing a rigorous basis for targeted architecture refinement. We demonstrate the effectiveness of our method on scientific datasets, including learning the observable-to-parameter map for the Navier-Stokes equation. Numerical results reveal that our approach consistently outperforms existing architecture adaptation methods in terms of generalization performance.

C G Krishnanunni, Thomas Scott, Tan Bui-Thanh

Department of Aerospace Engineering & Engineering Mechanics, UT Austin · Oden Institute for Computational Engineering and Sciences, UT Austin.

Neural Network Approximation Theory

Jul 7, 2026cs.LG

On Explicit Super-Expressive Approximation for Neural Networks

In this work, we investigate the fixed-architecture neural network approximation with explicit parameter bounds and elementary activations. While prior work demonstrated super-expressive approximation using fixed-size networks, they lack quantitative and non-asymptotic characterizations of parameter magnitude with respect to the approximation error. We resolve this issue by introducing the Chinese Remainder Theorem as a constructive encoding mechanism. For Lipschitz continuous functions on

[0,1]^D

, we construct a width-

\max\{D,4\}

, depth-

5

network with explicit parameter-error trade-offs. For Hölder-smooth functions in

C^{r,γ}_A\left([0,1]^D\right)

, our fixed network of width

\max\{2D,\ D+5N+1\}

and depth

r + 9

achieves the parameter magnitude

\mathcal{P}

bounded by

\log_2 \mathcal{P}=\mathcal{O}\bigl(\varepsilon^{-2D/(r+γ)}\log(1/\varepsilon)\bigr)

. This is the dual result compared to those in the parameter-bounded and architecture-unbounded paradigm.

Feng-Lei Fan, Ze-Yu Li, Chen-Yu Wang +1

Department of Data Science, City University of Hong Kong, 83 Tat Chee Avenue, Kowloon Tong, Kowloon, Hong Kong. · Department of Mathematics, Chinese University of Hong Kong, University Avenue, Shatin, N.T., Hong Kong. · School of Civil Engineering and Architecture, Guangxi Minzu University, Nanning 530006, China.

Neural Network Approximation Theory Neural Network Expressivity

Jul 7, 2026stat.ML

A Function-Space Dichotomy for Compositional Learning: Exponential Sub-Optimality of the Neural Tangent Kernel

A persistent empirical observation is that trained neural networks outperform their neural tangent kernel (NTK) limit on tasks with compositional structure, yet a quantitative account of

\textbf{when}

and

\textbf{by how much}

has been lacking. Working on the unit circle, we give such an account through a dichotomy between two complexity measures of the target: its

\textbf{Fourier complexity}

, which controls NTK kernel regression, and its

\textbf{architectural complexity}

, which controls learning over depth-

L

, width-

w

ReLU networks with the variation norm of the weights bounded by

R

. We first characterize the minimax rate of the architecture class

\mathcal{C}_{L,w,R}

, pinning it down up to a single factor of

L

: between

Ω(Lw^2R^2/n)

and

\tilde{O}(L^2w^2R^2/n)

. We then show the NTK estimator sits

\textbf{exponentially}

above this floor whenever the two complexities decouple: for the depth-

L

iterated sawtooth, NTK regression needs

Ω(4^L)

samples while the minimax floor is polynomial in

L

. Numerical experiments confirm the theoretical claims: on bandlimited smooth targets, the NTK is competitive or better, while on the hypercube sparse-parity model, a standard two-layer network beats the NTK by four to six orders of magnitude in test error. The gap is thus a function-space property, a mismatch between the kernel's smoothness bias and the target's compositional structure, rather than a generic kernel-versus-network phenomenon.

Arkaprabha Ganguli, Emil Constantinescu

Mathematics & Computer Science Division, Argonne National Laboratory, Lemont, IL, USA

Neural Network Generalization Kernel Regression ReLU Neural Networks Neural Network Approximation Theory Neural Tangent Kernel

Jul 6, 2026stat.ML

Deep Neural Variation Spaces: A Unifying Perspective on Depth and Complexity

We develop a unified function space theory of deep fully connected neural networks. Functions in our spaces are defined recursively as

\ell^1

-bounded linear combinations of activated functions from preceding layers, with a dictionary of affine functions at the first layer. Unlike existing theories that are largely specialized to homogeneous activations such as the ReLU, our framework provides a meaningful notion of functional complexity for deep networks with a broad range of homogeneous and non-homogeneous activation functions commonly used in practice. This simple construction unites several seemingly disparate ideas from the literature, including norm-based complexity bounds and variational characterizations of depth, and facilitates novel analyses of what kinds of functions deep norm-constrained networks can represent. To this end, we prove a novel representer theorem for our spaces and establish novel function-space complexity bounds showing that the associated function classes remain qualitatively small at arbitrary depth. In the univariate ReLU case, we prove a "depth saturation" result: depth in this setting yields only a small constant rescaling of the function class, with no added functional diversity. As a consequence, we show that deep norm-controlled ReLU functions in any dimension cannot exhibit high frequencies along any direction. This finding reveals that some commonly cited expressivity benefits of depth disappear once network complexity is controlled by an appropriate function space norm, rather than parameter count or other representational costs that permit compounded rescaling across layers. Overall, our results illustrate how a function space perspective yields new structural insights into the relationship between depth and complexity.

Julia Nakhleh, Robert D. Nowak

Department of Computer Science · University of Wisconsin-Madison · Department of Electrical & Computer Engineering

ReLU Neural Networks Neural Network Approximation Theory Neural Network Expressivity

Jul 6, 2026cs.LG

Minimum Block Width for Universal Approximation by Residual Neural Networks with Inner Width One

In this paper, we study the universal approximation property of residual neural networks, and obtain some new results. For input and output dimensions

d_x

and

d_y

, and LeakyReLU, ReLU, ReLU-like activation functions, the upper and lower bounds of the minimum block width are established. To achieve

L^p

approximation

(1\leq p <+\infty)

on any compact domain, we show that the exact minimum block width is

\max\{d_x,d_y\}

when each residual branch has inner width 1. Furthermore, we show that residual neural networks with block width

\min\{d_x+d_y, \max\{2d_x+1,d_y\}\}

can achieve uniform approximation on any compact domain under the constraint that each residual branch has inner width 1. Besides, for any activation function family, we prove that there exist functions that cannot be approximated by residual neural networks with block width less than

\max\{d_x, d_y\}

, both in the

L^p

sense and the uniform sense, regardless of inner width.

Qi Zhou, Xuan Zhou, Xiao-Song Yang

School of Mathematics and Statistics, Huazhong University of Science and Technology, Wuhan, 430074, Hubei, P.R. China. · Hubei Key Laboratory of Engineering Modeling and Scientific Computing, Huazhong University of Science and Technology, Wuhan, 430074, Hubei, P.R. China.

ReLU Neural Networks Residual Learning Universal Approximation Neural Network Approximation Theory Neural Network Expressivity

Jul 4, 2026cs.LG

A Unified Framework for Quantized and Continuous Strong Lottery Tickets

The Strong Lottery Ticket Hypothesis (SLTH) asserts that sufficiently overparameterized, randomly initialized neural networks contain sparse subnetworks that, even without any training, can match the performance of a small trained network on a given dataset. A key mathematical tool in the theoretical study of SLTH has been the Random Subset Sum Problem (RSSP). The SLTH has recently been extended to the quantized setting, where the network weights are sampled from a discrete set rather than from a continuous interval. These new results are however far from those in arbitrary-precision setting in several ways. In this work, we provide an analysis of the RSSP in the discrete setting, and use it to derive tight SLTH guarantees in the quantized case. Our analysis obtain tight bounds on the failure probability of finding a strong lottery ticket in the quantized regime, providing an exponential improvement over previous results. Most importantly, it unifies the literature by showing that both approximate representations in the continuous setting and exact representations in quantized settings naturally emerge as limiting cases of our results. This perspective not only sharpens existing bounds but also provides a cohesive framework that simultaneously handles approximation and rounding errors.

Aakash Kumar, Emanuele Natale

COATI, CNRS, Inria, I3S, Université Coté d’Azur, France

Neural Network Approximation Theory Neural Network Quantization

Jul 4, 2026cs.LG

Foundations of Equivariant Deep Learning: Unifying Graph and Sheaf Neural Networks

Symmetry is everywhere in nature and society. Geometric deep learning exploits symmetries in data to improve the performance and efficiency of deep learning systems. In this paper, we extend geometric deep learning to utilize richer symmetry structures. Specifically, we develop order-equivariant neural networks (OENN), which generalize standard graph message passing and sheaf neural networks via the theory of equivariant bundles over face posets (face categories). We (i) characterize all linear order-equivariant maps, (ii) build OENN layers, and (iii) prove universal approximation theorems (UATs) for continuous order-equivariant maps, which are new results even when restricted to sheaf neural networks (for which no UAT was known before). We illustrate the framework on graph and sheaf models. Our results can also be seen as extending the known UAT for graph neural networks to a more general setting that subsumes sheaf neural networks as well. In addition, we show that OENN can be extended further to CENN, Category-Equivariant Neural Network, which gives the general form of equivariant neural networks as well as of equivariant universal approximation theorems, allowing us to leverage categorical symmetry in data (e.g., non-invertible symmetries on multiple objects with compositional relations on those symmetries).

Yoshihiro Maruyama

Department of Mathematical and Information Sciences, Kyoto University, Kyoto, Japan.

Graph Neural Networks Equivariant Neural Networks Sheaf Neural Networks Neural Network Approximation Theory Equivariant Representation Learning

Jul 4, 2026math.AP

LRX-PINN: A Layer-Resolving XNet Physics-Informed Neural Network with Integrated Cauchy Activations for Convection-Dominated Problems

Convection-dominated convection-diffusion problems often develop thin layers, where the solution has sharp transition profiles and its derivatives are highly localized. This creates a structural mismatch for standard physics-informed neural networks (PINNs), whose trial spaces are not designed to match the value--derivative structure of such layers. We propose a Layer-Resolving XNet Physics-Informed Neural Network (LRX-PINN) based on integrated Cauchy activations. The proposed basis is transition-type at the solution level, while its derivative recovers a localized Cauchy kernel. We show that this structure matches the scaling of convection-dominated layers, inherits the Cauchy approximation mechanism at the derivative-profile level, and identifies

d/\|w\|

as the effective physical width of a ridge neuron. For analytic layer profiles, this yields derivative-stable exponential approximation in the stretched coordinate and a layer-scaled estimate for the strong residual of the singularly perturbed operator. Numerical experiments on several convection-dominated benchmarks show that LRX-PINN achieves higher accuracy than PIKAN and Fourier-feature PINNs while using less than

30\%

of their trainable parameters. On more challenging benchmarks, embedding the proposed representation into hp-VPINN-based frameworks further improves the best results obtained by existing hp-VPINN-based baselines without changing their original loss functionals or stabilization strategies. These results show that neural representations aligned with layer structure provide a compact and effective approach for convection-dominated problems.

Zihao Guo, Xin Li, Zhihong Xia

Institute for Advanced Research, Great Bay University, Dongguan, China · College of Computer Science and Technology, Dongguan University of Technology, Dongguan, China · Department of Mathematics, Northwestern University, Evanston, USA

Neural PDE Solvers PDE Solving Neural Network Activation Functions Neural Network Approximation Theory Physics-Informed ML

Jul 3, 2026cs.LG

Rethinking Neural Nonlinearity as Gating

Activation functions are considered an essential primitive for neural nonlinearity, i.e., they enable neural networks to serve as universal approximators. In this paper, we show that this nonlinearity can also be achieved by input-conditioned threshold gating through branches as a universal primitive. We demonstrate that standard activations -- whether piecewise-linear (ReLU, PReLU, Hardtanh) or smooth (SiLU, Sigmoid, Tanh, GELU) -- are in fact instances of a single Threshold Gating (TG) primitive. For softmax, we show that it admits an exact TG conversion via its equivalent per-element Sigmoid form. We then validate these equivalences by converting pretrained networks across CNNs, transformer-based models, and recurrent architectures, preserving model performance without requiring retraining. Threshold Gating also enables training from scratch that goes beyond replacing existing activations, enabling gains in model compression, performance, and shorter training. We also propose a 'Minimal Branch Theorem' which relates the minimum number of required branches in our primitive to the trainability of general deep neural networks. In terms of hardware implementation, TG maps to a unified implementation in the case of analog in-memory systems, addressing the bottleneck of analog-to-digital and digital-to-analog converters (ADC/DAC) that is known to significantly impact power consumption and on-chip area.

Muhammad Sabih, Frank Hannig, Jürgen Teich

Department of Computer Science, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), Germany

Neural Network Activation Functions Neural Network Approximation Theory