Neural Network Approximation Theory

Follow this topic Ask about this topic

Momentum

15 papers in the last four weeks, up 15% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 163

Jul 1, 2026cs.LG

From Approximation to Emergence: A Theory of Deep Learning

Deep learning has outgrown any single mathematical explanation. From Approximation to Emergence develops a unified, proof-oriented account of modern deep learning theory, tracing a path from the classical foundations of approximation, optimization, and generalization to the contemporary mechanisms of overparameterization, robustness, generative modeling, transformers, in-context learning, scaling laws, interpretability, alignment, and emergence. Rather than presenting isolated results, the book organizes a broad literature into a coherent research narrative: each theory is examined through the object it controls, the assumptions that make it valid, and the phenomena it leaves unexplained. Written for researchers, graduate students, and mathematically trained practitioners, this monograph offers a rigorous map of deep learning theory as it stands today: powerful, incomplete, and increasingly centered on the question of how learned mechanisms arise from scale, data, architecture, and training.

Zhilin Zhao

Sun Yat-sen University

Neural Network Generalization Deep Learning Optimization In-Context Learning Neural Network Approximation Theory Neural Scaling Laws

Jul 1, 2026stat.ML

From Spectral Methods to Sample Complexity Bounds for Fourier Neural Operators

We establish approximation and learning guarantees for Fourier neural operators (FNOs) applied to time-

T

solution operators of dissipative evolution equations. The analysis builds on the premise that FNOs can efficiently approximate and learn solution operators whenever these operators admit stable and accurate spectral discretizations. To formalize this idea, we introduce classes of evolution operators defined through spectral methods and derive FNO approximation bounds and polynomial sample complexity guarantees for these classes. For equations with polynomial nonlinearities, the learning rates depend primarily on the smoothness of the input space and the dimension of the physical domain. Our results hold uniformly over broad families of dissipative equations, rather than for a single fixed PDE, and apply in particular to the Navier--Stokes, Allen--Cahn, and Cahn--Hilliard equations. For equations with non-polynomial smooth nonlinearities, we prove that polynomial sample complexity still holds with rates that now additionally depend on the smoothness of the nonlinear terms and the dissipation strength. Overall, we connect classical spectral approximation theory with modern operator learning and explain when FNOs can learn nonlinear evolution operators efficiently.

Nisha Chandramoorthy, Daniel Sanz-Alonso, Nathan Waniorek

University of Chicago

Dynamical Systems Fourier Neural Operator PDE Operator Learning Neural Network Approximation Theory Neural Operators

Jun 30, 2026math.NA

Domain-Decomposed Randomized Neural Networks for Partial Differential Equations in Unbounded Domains

Partial differential equations on unbounded domains are challenging because the exterior region must be represented without excessive truncation error. Truncation-based methods often require problem-dependent artificial boundary conditions, while global spectral bases may be inefficient for localized structures, irregular geometries, or solutions with different near-field and far-field behaviors. We propose a domain-decomposed randomized neural network framework for such problems. Different randomized subnetworks are assigned to different spatial regimes: a near-field subnetwork captures local and geometric features, whereas a far-field subnetwork represents exterior decay. The subnetworks are coupled by boundary and interface conditions, and only the output-layer coefficients are solved from linear least-squares systems arising from Petrov--Galerkin or collocation formulations. We develop a Petrov--Galerkin method for semi-unbounded elliptic problems and a collocation method for fully unbounded, perforated, and time-dependent problems. A conditional bounded-parameter approximation result is proved in a broken Sobolev norm, together with an error decomposition covering approximation, empirical-consistency/quadrature, and least-squares optimization errors. Numerical experiments for Poisson and time-dependent Schrödinger equations demonstrate the accuracy and flexibility of the proposed method.

Haixin Wang, Haoning Dang, Fei Wang +1

School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, P.R. China. · School of Mathematics and Statistics & State Key Laboratory of Multiphase Flow in Power Engineering, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, China.

Neural PDE Solvers PDE Solving Neural Network Approximation Theory

Jun 29, 2026math.LO

Fast approximation and learning of binary classification tasks in o-minimal structures using ReLU neural networks

We study binary classification problems whose decision sets are given by definable sets in o-minimal expansions of the real field. Motivated by cell decomposition of definable sets, we introduce traceable sets as a classical proxy for definable decision regions and analyze their approximation by ReLU neural networks. Under uniform bounds on the number of connected components and suitable

C^m

extensions for the boundary functions, we prove that characteristic functions of traceable subsets of

[-1/2,1/2]^n

can be approximated in

L^p

to accuracy

\varepsilon>0

by ReLU neural networks of size

\mathcal{O}(\varepsilon^{-p(n-1)/m})

, with depth independent of

\varepsilon

and polynomially bounded weights. This establishes quantitative approximation rates for certain definable collections in o-minimal structures using ReLU neural networks. The same approach also yields the stated approximation rates for a subclass of definable maps

[-1/2,1/2]^n \to \mathbb{R}

. We then combine the approximation capabilities with entropy estimates for ReLU neural network classes to obtain statistical learning rates for empirical risk minimization with hinge loss. For

N

uniformly distributed samples, the resulting classifiers achieve expected misclassification error of order

N^{-m/(m+pn-p)}

up to an arbitrarily small polynomial loss.

Clemens Kinn, Philipp Petersen

Faculty of Mathematics, University of Vienna, 1090 Vienna, Austria

ReLU Neural Networks Binary Classification Classification Neural Network Approximation Theory Statistical Learning Theory

Jun 25, 2026cs.LG

Algorithmic Foundations of Deep Learning: Complexity-Theoretic Rates and a Characterization of Universal Approximation

Feedforward neural network (NN) expressivity is typically studied by emulating optimal basis-expansion schemes. While powerful, this perspective is incomplete: it primarily captures complexity through regularity, and therefore does not distinguish intuitively simple and complicated objects with comparable regularity, such as the square-root function and a typical Brownian path. The guiding message is that neural networks should be viewed not only as flexible basis functions, but also as models of computation. If a function is computable by a real-valued circuit over a prescribed elementary gate language, then it can be computed to comparable accuracy by an NN with explicit depth, width, and non-zero-parameter bounds controlled by the depth, width, gate count, and gate structure. Thus, neural-network complexity is not governed by regularity alone, but also by algorithmic complexity. We then show that any definable NN model satisfying a natural parallelization condition, allowing possibly multivariate non-linearities such as attention or layer normalization, is a universal approximator if and only if it contains a non-affine nonlinearity. The scope of our theory is illustrated by deducing universal approximation guarantees for continuous functions, minimax-optimal approximation guarantees for Besov classes, logarithmic-error complexity for holomorphic functions, and by showing that NNs can emulate numerical algorithms such as Newton-Raphson root finding and power iteration without architecture-specific arguments. Its precision is illustrated by shortest-path computation on

k

-vertex graphs: compiling the tropical dynamic-programming circuit yields NNs with O(log(1/ε)) non-zero parameters, exponentially improving in 1/ε over the generic

O(ε^{-c k^2})

Lipschitz-approximation scale, for a constant c>0.

Anastasis Kratsios, Simone Brugiapaglia, Bum Jun Kim +2

McMaster University & Vector Institute, Toronto, ON, Canada · Concordia University, Montréal, QC, Canada · University of Tokyo, Tokyo, Japan +2

Universal Approximation Neural Network Approximation Theory Feedforward Neural Networks Neural Network Expressivity

Jun 22, 2026cs.LG

EML Trees Are Universal Approximators

The recently introduced EML (Exp-Minus-Log) function acts as continuous analogue of NAND gates, providing a compositional building block capable of representing elementary functions. In this work, we study the expressive power of tree-structured compositions of EML functions. We show that such trees enjoy a universal approximation property for functions in

W^{k, \infty}

for

k \in \mathbb N

, drawing on classical neural network approximation arguments while exploiting the ability to explicitly construct EML trees that mimic polynomial representations. We further propose a learning algorithm for EML-type trees equipped with fitting parameters, and demonstrate its feasibility in practical optimization problems. Our results establish EML trees as a theoretically grounded framework for function approximation.

Joe Germany, Elie Abdo, Joseph Bakarji

Department of Mathematics American University of Beirut Beirut, Lebanon · Department of Mechanical Engineering American University of Beirut Beirut, Lebanon

Universal Approximation Neural Network Approximation Theory Neural Network Expressivity

Jun 19, 2026cs.LG

Expressivity Saturation: Reduced Affine Region Usage Under Increasing Task Complexity

Piecewise-affine neural networks (e.g., with ReLU or LeakyReLU activations) implement continuous piecewise-affine maps, and the number of affine regions provides a natural proxy for expressive capacity. However, the gap between theoretical region capacity and the affine regions realized after training remains insufficiently understood. We study this gap from two complementary perspectives. First, we give a rigorous, architecture-dependent theorem for affine line-segment probes: for multilayer perceptrons with piecewise-affine activations, the number of affine pieces realized along an affine line-segment probe is upper bounded by an explicit product of layer-wise width terms (and activation breakpoint factors). This yields a neuron-threshold lower bound for representing target functions with prescribed one-dimensional piece complexity, formalizing the minimal region budget required for complex signals. Second, we exactly enumerate affine regions realized within bounded 2D and higher-dimensional domains under controlled task complexity. Under fixed architectures and training protocols, increasing input--label complexity yields trained solutions with markedly fewer realized regions in the evaluation domain, even though worst-case architectural capacity is unchanged; we call this reduced region usage expressivity saturation. Moreover, in the most challenging regimes, 2D visualizations show that region-usage collapse often coincides with degraded decision boundaries. Finally, we visualize the training dynamics of affine-region partitions and decision boundaries, revealing a consistent refinement process during optimization.

Xuan Qi, Yi Wei, Fanqi Yu +1

AI for Good (AIGO), Istituto Italiano di Tecnologia · DITEN, University of Genoa · State Key Laboratory of Novel Software Technology +1

Multilayer Perceptrons Neural Network Approximation Theory Neural Network Training Dynamics Neural Network Expressivity

Jun 18, 2026cs.LG

Recurrent neural networks approximate continuous functions

Classical approximation theorems ask for a new neural network whenever the target accuracy is improved. This paper studies the opposite possibility: can the network be chosen once and for all, and can accuracy be bought only by letting it run longer? We prove that this is possible for every continuous function on [-1,1]. More precisely, each such function is uniformly approximated by the time evolution of a single ReLU recurrent neural network with fixed weights and fixed hidden dimension. The mechanism behind the construction is a new intermediate model, the Turing machine with neural units (TMNU). This model retains the algorithmic freedom needed to implement polynomial approximation schemes, while remaining rigid enough to be simulated by RNNs with explicit bounds on hidden dimension and weight magnitude. The resulting convergence rates reflect the underlying polynomial approximation rates. We complement the construction with minimax lower bounds showing that runtime is not merely a proof artifact, but an unavoidable resource in this fixed-network approximation paradigm.

Valentin Abadie, Clemens Hutter, Helmut Bölcskei

ReLU Neural Networks Recurrent Neural Networks Neural Network Approximation Theory

Jun 18, 2026cs.LG

Score Approximation for Diffusion Models on Arbitrary Low-Dimensional Structures

Score-based diffusion models have achieved remarkable empirical success, motivating extensive theoretical work to establish their foundations. However, existing complexity bounds for score approximation, a vital step in diffusion modeling, rely on rigid constraints such as Lipschitz continuous scores or lower bounded densities. This severely limits their applicability to real-world perceptual data, where singularities, sharp boundaries, and disjoint clusters routinely violate such restrictive assumptions. We bridge this gap between theory and practice, presenting the first universal score approximation theorem applicable to any compactly supported distribution in

\mathbb{R}^n

. Using a novel discretization technique that directly models the underlying distribution, we prove that the neural network complexity is governed by the support's upper Minkowski dimension

d

rather than the ambient dimension

n

. Furthermore, by leveraging the inherent smoothing of Gaussian kernels, we show that even for irregular, fractal distributions, an

ε

approximation error can be achieved with

\mathcal{O}(ε^{-{d/(M+1)}})

local Taylor modules each at size of

\mathcal{O}(n^M)

, an

ε

-scaling rate previously achieved only for distributions with

(M+1)

-Hölder smoothness. Thus, we reveal a possible mechanism by which score-based diffusion models represent non-smooth data distributions.

Xinhe Mu, Zaijiu Shang, Zhaoqi Zhou +4

Academy of Mathematics and Systems Sciences Chinese Academy of Sciences Beijing, China · Shanghai Institute for Mathematics and Interdisciplinary Sciences Shanghai, China · Huawei Technologies Co., Ltd. Beijing, China

Neural Network Approximation Theory Score-Based Generative Modeling Diffusion Models

Jun 18, 2026cs.LG

Learning universal approximations for partial differential equations with Physics-Informed Broad Learning System

Partial differential equations (PDEs) play a central role in modeling complex physical, biological, and engineering systems. While traditional numerical solvers are robust, they often incur prohibitive computational costs due to mesh dependencies, whereas recent Physics-Informed Neural Networks (PINNs) offer a mesh-free alternative but frequently suffer from slow convergence and optimization instability. To bridge this gap, this article proposes the Physics-Informed Broad Learning System (PIBLS), a novel backpropagation-free framework that reformulates PDE solving as a direct least-squares optimization. We improved an algorithm within this framework to handle nonlinear PDEs efficiently and provide a rigorous mathematical proof establishing the universal approximation property of PIBLS for these equations. Experiments on linear and nonlinear PDEs demonstrate that PIBLS is one to three orders of magnitude faster than conventional PINNs while achieving significantly higher solution accuracy. This framework provides a computationally efficient paradigm for scientific machine learning, offering a practical, high-speed alternative for real-time simulation and design optimization tasks.

Zhiwen Yu, Derong Yang, Liujian Zhang +5

School of Computer Science and Engineering, South China University of Technology, Guangzhou 510006, China · Peng Cheng Laboratory, Shenzhen 518000, China · School of Future Technology, South China University of Technology, Guangzhou 510006, China +2

PDE Solving Universal Approximation Neural Network Approximation Theory Physics-Informed ML

Jun 16, 2026cs.LG

Monotonic Kolmogorov-Arnold Networks: A Theoretical and Empirical Study of Monotonicity as an Inductive Bias

Monotonicity has been a long-running architectural inductive bias for neural networks, motivated by tabular, scientific, and economic settings where outputs are known to respond monotonically to certain inputs. Existing approaches are MLP- or flow-based and lack per-edge functional transparency; the only Kolmogorov--Arnold Network (KAN) variant with monotonicity, MonoKAN, enforces the constraint only on a restricted parameter subset and requires a projection-style training procedure. We close this gap with \textbf{MKAN}, a KAN with hard monotonicity guaranteed for \emph{all} parameter values via exponential reparameterization of B-spline coefficients, positive edge weights, and a monotone base activation. Training reduces to standard unconstrained gradient descent. Our headline theoretical contribution is a \emph{representation-cost} theorem: any

C^K, K >0

feature extractor inducing a ball-shaped semantic-neighborhood partition admits a monotone realization of the equivalent neighborhood structure at

N' = N^* + k \le 2N^*

, where

k

is the number of non-monotone coordinates of the original. The bound is architecture-agnostic and gives a principled sizing rule for monotone encoders. Empirically, MKAN is competitive with state-of-the-art monotone NNs on the SMM/ICML-2024 benchmark while being the only method that combines hard unconstrained monotonicity with KAN's per-edge functional transparency; the

2N^*

prediction is validated in a self-supervised feature-size sweep on four real datasets, and on a controlled monotone-generative dataset MKAN recovers ground-truth factors with substantially higher Spearman alignment than KAN, MLP, and linear baselines.

Mikhail Krasnov, Blaž Bertalanič, Carolina Fortuna

Jozef Stefan Institute

Representation Learning Kolmogorov-Arnold Networks Neural Network Approximation Theory

Jun 16, 2026cs.LG

Generalization Guarantees for Multi-Input Neural Operator Learning in Sobolev Spaces

We develop approximation and generalization error estimates for multi-input neural operators, with the output error measured in Sobolev norms. In contrast to standard operator-learning settings with a single input function, our framework allows multiple input functions defined on possibly different domains, with different dimensions and Sobolev regularities. The derived rates explicitly quantify the contribution of each input space to the final error bound. In particular, in the balanced regime, the approximation and generalization rates are governed by the interaction between the input dimensions, regularities, and Sobolev orders, while the dependence on the model complexity retains a

\log\log/\log

-type structure. Our analysis provides a general theoretical framework for multi-input operator learning, including Sobolev training, and is applicable to operator learning problems arising from partial differential equations and scientific computing.

Yahong Yang, Zecheng Zhang, Wei Zhu +2

School of Mathematics, Georgia Institute of Technology, 686 Cherry Street, Atlanta, GA 30332-0160, USA. · Department of Applied and Computational Mathematics and Statistics, University of Notre Dame, Notre Dame, IN 46556, USA. · Department of Mathematics, Hong Kong Baptist University, FSC1202, Fong Shu Chuen Building, Hong Kong Baptist University, Kowloon Tong, Hong Kong.

Neural Network Approximation Theory Operator Learning Neural Operators Generalization Bounds

Jun 15, 2026stat.ML

Sobolev Approximation by Fixed-Size Neural Networks with Arbitrary Accuracy

In this work, we investigate new activation functions for achieving arbitrary-accuracy Sobolev approximation by fixed-size neural networks. We first show that any function in

W^{2,\infty}((a,b)^d)

can be approximated with arbitrary accuracy, measured in the

W^{1,\infty}

-norm, by a fixed-size neural network using the Elementary Universal Activation Function (

\mathrm{EUAF}

). To extend this result to

W^{s,\infty}((a,b)^d)

for

s\in\mathbb{N}

, we introduce a smooth activation

\mathrm{DUAF}_{\infty}

from the family of Differentiable Universal Activation Functions (

\mathrm{DUAF}_n

). We prove that any function in

W^{s,\infty}((a,b)^d)

can be approximated with arbitrary accuracy in the

W^{s-1,\infty}

-norm by a fixed-size

\mathrm{DUAF}_{\infty}

-activated network. We further construct sigmoidal variants

\widetilde{\mathrm{DUAF}}_n

and show that, for every

1\leq s\leq n

, fixed-size

\widetilde{\mathrm{DUAF}}_n

-activated networks still approximate any

f\in W^{s,\infty}((a,b)^d)

with arbitrary accuracy in the

W^{s-1,\infty}

-norm. In all these results, the width and depth bounds are computed explicitly, and the proposed activations are elementary.

Baicheng Li, Haizhao Yang, Shijun Zhang

Department of Mathematics, University of Maryland, College Park, MD, USA · Department of Applied Mathematics, The Hong Kong Polytechnic University, Hong Kong, China

Neural Network Activation Functions Universal Approximation Neural Network Approximation Theory

Jun 15, 2026cs.LG

RepNN: Tackling spectral bias in deep neural networks for regression and PDE problems via parameter reparameterization

Deep neural networks (DNNs) have achieved remarkable success in scientific computing, yet they often suffer from spectral bias in capturing oscillatory and multiscale behaviors. In this study, we investigate this limitation by examining the failure of shallow ReLU neural networks in fitting high-frequency functions. This observation identifies two important factors in resolving rapid oscillations: the initial slope scale and the distribution of partition points induced by the networks. Motivated by this analysis, we propose RepNN, a reparameterized neural network model with ReLU or tanh activations designed for high-frequency and multiscale problems. The key idea is to reparameterize the weights and biases in the first hidden layer, which enables effective control of the initial slope scale and provides an appropriate distribution of the initial partition points. Furthermore, treating the reparameterized weights and biases as trainable parameters allows the DNN to achieve adaptive frequency scaling during training. In addition, we derive quantitative estimates for the output and slope magnitudes of the reparameterized DNN to guide the initialization of the proposed method. Numerical experiments, including multiscale one-, two-, and four-dimensional function approximations, forward and inverse PDE problems in combination with physics-informed neural networks (PINNs), and operator learning for an earthquake problem using real data, demonstrate that RepNN improves the predicted accuracy of vanilla DNNs in capturing highly oscillatory features. These results indicate that RepNN provides an effective and flexible approach for overcoming spectral bias and applying DNNs to multiscale problems.

Yong Wang, Tao Zhou, Xuhui Meng

Institute of Interdisciplinary Research for Mathematics and Applied Science, School of Mathematics and Statistics, Huazhong University of Science and Technology, Wuhan 430074, China · Institute of Computational Mathematics, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China

PDE Solving Spectral Bias Neural Network Approximation Theory

Jun 14, 2026cs.LG

The Information-Theoretic Benefit of Shared Representations under Orthogonality Constraints

Modern deep learning architectures are increasingly multi-task and multi-modal, using a pretrained foundation model combined with task-specific, fine-tuned models. Empirically, exploiting similarity across different problems, instead of solving them individually, can significantly improve overall performance. While the generalization and sample complexity properties of multitask learning have been widely studied, the parametric complexity of joint approximation in comparison to separate approximation remains less well understood. The question is particularly relevant in modern deep learning, where models are increasingly required to satisfy structural constraints such as equivariance, conservation laws, or orthogonality. We prove lower and upper bounds on the description-length for separate and joint approximation classes, respectively, in uniform norm. We build a class of orthogonal functions by composing a shared hard feature, realized by a Rademacher-Haar wavelet series, with Sawtooth-Walsh readouts to enforce orthogonality of output coordinates. The dyadic tree structure of the Rademacher-Haar wavelet concentrates the approximation hardness in the common feature component, while the readouts act as task-specific heads. Using an information-theoretic framework, we obtain a sharp gap between the optimal approximation rates achievable by joint and separate coding. Finally, we realize this separation in a neural network model using Heaviside activations via reduction to triangle-wave approximation. Our results show that even under an orthogonality constraint joint approximation requires strictly fewer bits in compositional architectures, provided the tasks share a latent hard feature. This provides theoretical insight into the description-length-efficiency of compositional multi-output architectures and clarifies how neural networks can retain expressivity under geometric constraints.

Thomas Dittrich, Oliver Potocki, Philipp Grohs

Johann Radon Institute of Computational and Applied Mathematics, Austrian Academy of Sciences · Faculty of Mathematics, University of Vienna

Representation Learning Minimum Description Length Multi-Task Learning Neural Network Approximation Theory Information-Theoretic Lower Bounds

Jun 12, 2026math.FA

Representation Costs in Data Science: Foundations and the Quasi-Banach Spaces of Deep Neural Networks

We develop a general framework for analyzing representation costs of parametric data-fitting methods through their parameter-space regularizers. From this abstract perspective, we define representation costs for arbitrary parametric models and reveal their induced (native) function spaces. This unifies recent function-space views of data-fitting methods. We also prove that many natural results hold in this abstract setting, including representer theorems for parametric methods on their native spaces. The framework also rigorously connects parametric methods with their equivalent nonparametric descriptions under sufficient overparameterization. Classical methods and their native spaces, such as kernel methods / reproducing kernel Hilbert spaces, wavelets / Besov spaces, and shallow neural networks / variation spaces emerge as special cases of our abstract framework. A byproduct of "axiomatizing" the study of representation costs is that we also immediately obtain new results for deep neural networks: For depth-

L

feedforward ReLU networks, their induced native spaces are

p

-normable quasi-Banach spaces with

p = 2/L

. This reveals that the inductive bias of deep neural networks (as given by the representation cost) cannot be captured by norms for depths

L > 2

.

Greg Ongie, Rahul Parhi

Marquette University · University of California, San Diego

Neural Network Approximation Theory

Jun 9, 2026cs.LG

Limitations of Learning Tanh Neural Networks with Finite Precision

We investigate limitations of learning

\tanh

neural networks from point evaluations under finite-precision computations and

L^p

accuracy guarantees, building on Berner, Grohs, and Voigtländer (2023). Our approach is based on a novel construction of sharply localized bump functions via iterated

\tanh

activations. Using this mechanism, we show that, in a finite-precision setting, no adaptive randomized algorithm based on

m

samples can achieve a convergence rate higher than the Monte Carlo rate

O(m^{-1/p})

in the

L^p

norm, unless the sampling budget grows exponentially with the size of the network parameters and architecture. The results reveal fundamental limitations imposed by finite precision on the learnability of classes containing localized bump functions, extending previous results for ReLU networks to the

\tanh

setting.

Philipp Grohs, Matěj Trödler

Faculty of Mathematics, University of Vienna · RICAM, Austrian Academy of Sciences

Sample Complexity Neural Network Approximation Theory

Jun 8, 2026math.FA

Weighted universal approximation of differentiable maps on infinite-dimensional manifolds

We generalize the universal approximation theorem for functional input neural networks (FNN) to differentiable maps by including the approximation of the derivatives. A FNN maps the input from a possibly infinite-dimensional weighted manifold to the real-valued hidden layer, on which a non-linear scalar activation function is applied, and then returns the output into a Banach space via some linear readouts. By proving a weighted Nachbin theorem, we establish a universal approximation theorem for differentiable maps, which goes beyond the usual formulation on compact sets and also includes the approximation of the derivatives. This leads us to approximation results for non-anticipative functionals including the horizontal and vertical derivatives. As a further application, we show that linear functions of the signature are able to approximate path space functionals including their directional derivatives.

Philipp Schmocker, Josef Teichmann

Universal Approximation Neural Network Approximation Theory

Jun 7, 2026math.NA

Compositional Approximation Can Strictly Outperform Superpositional Approximation

Many classically studied function classes are known to be approximated optimally by superpositional methods, i.e. with approximants constructed as the linear combination of elements in some dictionary. Here optimality means that the uniform approximation error viewed as a function of the number of parameters used has polynomial decay of the highest order achievable by any parametrized method whose parameters can be encoded as a bit string of length proportional, up to logarithmic factors, to the number of parameters. While compositional methods like neural networks are structurally different, their approximation rates can be made comparable by imposing constraints that ensure such a proportional bit string encoding. In this work we study function classes exhibiting structural properties that limit superpositional approximation rates to be strictly lower than compositional approximation rates. In particular, we construct explicit examples for which there is an arbitrarily large gap.

Dennis Elbrächter, Philipp Petersen

Faculty of Mathematics, University of Vienna, Austria

Neural Network Approximation Theory

Jun 5, 2026cs.LG

Beyond Linear and Overcomplete Regimes: A Mean-Field Analysis of Bottleneck Autoencoders

Autoencoders (AEs) learn low-dimensional representations by mapping data into a latent space while minimizing reconstruction error. Despite their empirical success, theoretical understanding remains limited and largely restricted to linear models or settings without a bottleneck. In this work, we study nonlinear AEs with a fixed finite-dimensional bottleneck in the mean-field (MF) regime. We derive explicit MF learning dynamics for both encoder and decoder, providing a tractable characterization of training in the nonlinear setting. We show that, over finite time horizons, the empirical risk of finite-width networks trained with stochastic gradient descent closely tracks the MF risk trajectory with high probability. At optimality, we further establish that the finite-width risk converges to the MF optimum, demonstrating that finite networks are sufficiently expressive to approximate the infinite-width solution.

Santanu Das, Ramyak Bilas, Pascal Esser +1

STCS department Tata Institute Of Fundamental Research Mumbai, India-400005 · Department of Mathematics Indiana University Bloomington Indiana, USA-47405 · Department of Mathematics Ludwig-Maximilians-Universität München Germany, München-80799 +1

Autoencoders Mean-Field Theory Neural Network Approximation Theory Neural Network Training Dynamics

Jun 4, 2026cs.LG

Mitigating the Curse of Dimensionality in Uniform Convergence of Deep Neural Networks via Smooth Activations

This paper establishes a theoretical framework for the uniform convergence of smoothly activated deep neural network (DNN) estimators. While standard ReLU networks achieve minimax-optimal rates in the

L^2(P)

norm for various nonparametric regression tasks, we establish a theoretical lower bound demonstrating that least-squares ReLU estimators can suffer from the curse of dimensionality in their uniform convergence behavior. Motivated by the need for reliable uniform guarantees in downstream tasks requiring worst-case reliability, we address this limitation by analyzing smoothly activated DNNs (smooth DNNs), encompassing both feedforward and residual structures. We establish novel pseudo-dimension bounds, non-asymptotic approximation guarantees, and Hölder-norm bounds for the approximators of these models. Leveraging these results, we derive non-asymptotic uniform convergence rates for smooth DNN estimators across multiple statistical contexts, including Huber, least-squares, quantile, and logistic regression. We prove that smooth DNNs can mitigate the {curse of dimensionality} in uniform convergence by adaptively exploiting the low-dimensional hierarchical composition structure of the target function. Supported by both simulation studies and a real-world application, our results position smooth DNNs as a theoretically grounded and practically viable alternative to ReLU networks for statistical learning tasks requiring uniform guarantees.

Yizhe Ding, Runze Li, Jia Liu +1

Department of Statistics, The Pennsylvania State University

ReLU Neural Networks Neural Network Approximation Theory Statistical Learning Theory

Jun 2, 2026cs.LG

Rethinking Neural Width for Alternating Current Optimal Power Flow Proxies

Deep learning proxies for Alternating Current Optimal Power Flow (ACOPF) lack systematic methods for determining architectural size. This paper conducts a constructive thought experiment to answer a fundamental inquiry: how wide must a neural network be to almost accurately approximate the ACOPF manifold? We introduce a Loss-Guided Neural Densification (LG-ND) algorithm that incrementally discovers necessary capacity by expanding only when the current deep neural network topology fails to improve further. Empirical results across various IEEE systems show that LG-ND achieves performance parity with literature baselines using up to ten times fewer neurons per layer. Such architectural minimalism is critical for the formal verification required in safety-critical grid operations.

Dhruvi Khandelwal, Anurag Basistha, Ayushi Jolotia +1

Department of Electrical Engineering, National Institute of Technology Kurukshetra, India. · Indraprastha Institute of Information Technology Delhi, India. · Department of Electrical Engineering, Indian Institute of Technology Roorkee, India.

Neural Surrogate Modeling Optimal Power Flow Constrained Optimization Neural Network Approximation Theory Neural Architecture Search Neural Network Expressivity

Jun 1, 2026cs.LG

Hierarchical RBF-KAN and RBF-SKAN Architectures for Multidimensional Function Approximation and Random Field Learning

In this manuscript, we propose and analyze hierarchical Kolmogorov--Arnold neural network architectures employing radial basis functions as activation functions for approximating deterministic functions and random field models. Specifically, we develop a hierarchical radial-basis-function Kolmogorov--Arnold network (hierarchical RBF-KAN) for multidimensional deterministic function approximation and a hierarchical radial-basis-function stochastic Kolmogorov--Arnold network (hierarchical RBF-SKAN) for random field learning. From a theoretical perspective, we establish universal approximation results for both architectures. In particular, we derive quantitative approximation estimates for the hierarchical RBF-KAN, showing that the proposed framework has the potential to partially alleviate the curse of dimensionality in learning high-dimensional functions by reducing the effective dimensionality of the approximation problem. Furthermore, we show that the hierarchical RBF-SKAN can approximate random field models under the Wasserstein-2 metric. Empirically, we show that our proposed radial-basis-function-based neural network structure could effectively learn multivariate functions and random field models.

Mingtao Xia, Qijing Shen

University of Houston, Philip Guthrie Hoffman Hall, 3551 Cullen Blvd, Houston, 77204, Texas, United States of America · University of Birmingham, Watson Building, Birmingham, B15 2TT,, United Kingdom · University of Oxford, Henry Wellcome Building for Molecular Physiology, Old Road, Oxford, OX3 7BN, Oxfordshire, United Kingdom

Kolmogorov-Arnold Networks Neural Network Approximation Theory Radial Basis Function Networks

May 31, 2026stat.ML

Variation Spaces for Encoder--Decoder Neural Operators: Approximation and Generalization

Inspired by the function-space theory of neural networks, we formulate and analyze a variation space for nonlinear operators between Hilbert spaces, defined through vector-valued Borel measures of bounded variation. We characterize its unit ball as the closed convex hull of a vector-valued single-neuron dictionary in Bochner spaces. For the ReLU activation, the bounded linear operators in this space are precisely the Schatten-

1

operators, with equivalent norms. For operators in this space, we establish encoder--decoder approximation bounds in the Bochner

L^q

-norm, where the error decomposes into input and output encoding errors and a finite-width term of order

N^{-1/2}

. Under sub-Gaussian assumptions on the input and noise, we further derive high-probability generalization bounds for empirical least squares over path-norm-constrained encoder--decoder networks; the finite-sample contribution to the squared prediction error is of order

K^{-1/2}

up to logarithmic factors. The finite-width and finite-sample constants are independent of the encoding dimensions and bases, with the latter also independent of the network width. When the encoding errors decay algebraically, these bounds yield algebraic approximation and learning rates, in contrast to the complexity barriers for Lipschitz and Fréchet differentiable operator classes.

Jia-Qi Yang, Lei Shi

School of Mathematical Sciences and Shanghai Key Laboratory for Contemporary Applied Mathematics, Fudan University, Shanghai 200433, China.

Neural Network Approximation Theory Operator Learning Neural Operators

May 30, 2026stat.ML

Taming the Loss Landscape of PINNs with Noisy Feynman-Kac Supervision: Operator Preconditioning and Non-Asymptotic Error Bounds

Physics-Informed Neural Networks (PINNs) often train slowly or fail to converge on challenging partial differential equations (PDEs), a behavior recently linked to severely ill-conditioned loss landscapes inherited from the underlying differential operator. We study PINNs augmented with a pointwise data-fidelity term, added at a few points in the domain to the standard residual and boundary losses. We show that this supervision term acts as an operator-level preconditioner: for suitable weights, our comparison bounds guarantee a substantially smaller condition number than under the standard PINN loss, independently of how the pointwise labels are obtained. For a broad class of PDEs admitting a Feynman-Kac (FK) representation, we generate such labels by Monte Carlo averages of the FK functional, resulting in what we call ``FK-PINNs", and using the excess risk decomposition approach, we derive non-asymptotic

L^2(Ω)

-error bounds for FK-PINNs with

\tanh

activation trained by finitely many steps of gradient descent. Along the way, we establish pseudo-dimension bounds for first- and second-order derivatives of

\tanh

neural networks, which are of independent interest and, to the best of our knowledge, new. Numerical experiments on Poisson, Schrödinger, mean exit time, and committor problems corroborate the theory, and show that FK-PINNs can successfully solve PDEs for which standard PINNs exhibit severe failure modes.

Nathanael Tepakbong, Hanyu Hu, Chengyu Liu +1

Department of Data Science, City University of Hong Kong, 83 Tat Chee Ave, Hong Kong · Department of Mathematics, City University of Hong Kong, 83 Tat Chee Ave, Hong Kong

Neural Network Approximation Theory Physics-Informed ML

May 29, 2026stat.ML

Approximation and learning of anisotropic and mixed smooth functions by deep ReLU neural networks

This paper studies how efficiently deep ReLU neural networks can approximate and learn smooth functions. When the error is measured in

L^p([0,1]^d)

norm and the approximator is a network with width

W

and depth

L

, recent works have proven the supper approximation rate

\mathcal{O}((WL)^{-2s/d})

for Besov space

\mathcal{B}^s_{q,r}([0,1]^d)

under the Sobolev embedding condition

s/d>1/q-1/p

. In order to overcome the curse of dimensionality in this rate, we extent this result to anisotropic and mixed smooth function classes. We establish the approximation rate

\mathcal{O}((WL)^{-2\tilde{s}})

for anisotropic Besov space

\mathcal{B}^{\boldsymbol{s}}_{q,r}([0,1]^d)

with anisotropic smoothness

\boldsymbol{s}=(s_1,\dots,s_d)

under the embedding condition

\tilde{s} > 1/q-1/p

, where the mean smoothness

\tilde{s} = (\sum_{i=1}^d s_i^{-1})^{-1}

. For mixed smooth Besov space

\mathcal{MB}^s_{q,r}([0,1]^d)

with mixed smoothness

s>1/q-1/p

, we show that the approximation rate

\mathcal{O}((WL)^{-2s})

holds up to logarithmic factors. Using these results, we also derive approximation bounds for the composition of anisotropic Besov functions. As an application, it is shown that deep ReLU neural networks can achieve minimax optimal rates up to logarithmic factors for a wide range of smooth function classes.

Yunfei Yang, Jun Fan

School of Mathematics (Zhuhai) and Guangdong Province Key Laboratory of Computational Science, Sun Yat-sen University, Zhuhai, China. · Department of Mathematics, Hong Kong Baptist University, Kowloon, Hong Kong, China.

ReLU Neural Networks Neural Network Approximation Theory

May 27, 2026cs.LG

Expressive Power of Floating-Point Neural Networks with Arbitrary Reduction Orders and Inexact Activation Implementations

Most existing expressivity theories for neural networks assume exact real arithmetic, whereas practical neural networks are executed under finite-precision floating-point arithmetic with implementation-dependent execution semantics. Recent works have begun studying the expressive power of floating-point neural networks, but existing results are limited to highly restricted activation functions and idealized assumptions such as fixed left-to-right reduction orders and correctly rounded activation implementations. In this work, we study the expressive power of floating-point neural networks under generalized floating-point execution semantics, including arbitrary reduction orders and inexact activation implementations with bounded ulp errors. We investigate when floating-point neural networks can represent arbitrary functions between floating-point domains exactly. To this end, we introduce a general distinguishability framework and show that the ability to distinguish every pair of distinct inputs in the first layer is necessary for universal representability. This characterization yields broad classes of activation implementations that are not universal representators, extending previous isolated counterexamples such as the correctly rounded cosine activation. We further prove that a suitable form of distinguishability is also sufficient for universal representability under mild conditions on the activation implementation. Using this framework, we establish universal representability results for a broad class of practical activation functions, including implementations of

\mathrm{Sigmoid}

,

\tanh

,

\mathrm{ReLU}

,

\mathrm{ELU}

,

\mathrm{SeLU}

,

\mathrm{GeLU}

,

\mathrm{Swish}

,

\mathrm{Mish}

, and

\sin

, under significantly more realistic floating-point execution models than previously known.

Yeachan Park, Geonho Hwang, Wonyeol Lee +1

Neural Network Activation Functions Universal Approximation Neural Network Approximation Theory Neural Network Expressivity

May 25, 2026stat.ML

Learning Sparse Compositional Functions with Norm-Constrained Neural Networks

The ability of deep neural networks to learn hierarchical features is widely regarded as a key mechanism underlying their success in high-dimensional learning. Existing theory partially supports this view by establishing approximation rates based on parameter counts and sample complexity guarantees for compositional models without incurring the curse of dimensionality (CoD). To study overparameterized regimes, where the number of parameters exceeds the sample size, we develop a framework that measures complexity via the parameter norm. Within this approach, we establish approximation rates and excess risk bounds for learning sparse compositional functions whose compositional structure is represented by directed acyclic graphs (DAGs), using Frobenius norm-constrained deep neural networks. Our results have broad applicability since every function that is efficiently Turing computable admits sparse compositional representations. In particular, we cover a range of representative models, including multi-index models, binary tree structures, and general compositional architectures. The rates we derive show that deep networks can exploit the compositional structure of the target functions, effectively avoiding the CoD through hierarchical representations.

Shuo Huang, Lorenzo Fiorito, Lorenzo Rosasco +1

MaLGa, DIMA, Università degli Studi di Genova, Genoa, Italy · CBMM, Massachusetts Institute of Technology, Cambridge, MA, USA

Neural Network Generalization Hierarchical Representation Learning Neural Network Approximation Theory Compositional Representation Learning

May 24, 2026math.NA

Random Neural Network Expressivity for Non-Linear Partial Differential Equations

Neural networks with randomly generated hidden weights (RaNNs) have been extensively studied, both as a standalone learning method and as an initialization for fully trainable deep learning methods. In this work, we study RaNN expressivity for learning solutions to non-linear partial differential equations (PDEs). Despite their widespread use in practical applications, a rigorous theoretical understanding of the approximation properties of RaNNs in this context remains limited. Here, we derive error bounds for RaNN approximations to time-dependent Sobolev functions and obtain a dimension-free approximation rate

\frac{1}{2}

for sufficiently regular functions. We apply our results to two important classes of non-linear PDEs: Porous Medium Equations and Compressible Navier-Stokes Equations, showing that RaNNs are capable of efficiently approximating solutions to these complex, non-linear PDEs. Our theoretical analysis is supported by numerical experiments, showing that the obtained convergence rates extend beyond the considered setting.

Muhammed Ali Mehmood, Lukas Gonon

Department of Mathematics Imperial College London UK · School of Computer Science University of St. Gallen Switzerland

Neural PDE Solvers PDE Solving Neural Network Approximation Theory Neural Network Expressivity

May 24, 2026math.NA

IV-Net: A neural network for elliptic PDEs with random and highly varying coefficients

We introduce a novel neural operator architecture designed to approximate solutions of linear elliptic partial differential equations with high-contrast, spatially varying coefficients. The network, termed the Iterated V-shaped Net (IV-Net), realizes a mapping from the input coefficients and righthand side to the corresponding solution field. The architecture of IV-Net is informed by, and closely resembles, a V-cycle multigrid solver. The IV-Net model is parameterized via convolutional layers defined in the physical domain. For coercive problems with highly heterogeneous coefficients, the proposed network exhibits superior performance relative to a proper orthogonal decomposition (POD) approach and several existing neural operator architectures. For low-frequency oscillatory Helmholtz problems with smooth coefficients, its performance is similar to that of a Fourier neural operator. We analyze the approximation error and convergence behavior of IV-Net, its data efficiency, and its dependence on the underlying discretization mesh. Furthermore, we demonstrate the practical effectiveness of the architecture through a series of numerical experiments, including applications to uncertainty quantification, inverse problems, and prediction of quantities of interest.

Shan Zhong, George Biros

Oden Institute for Computational Science and Engineering, The University of Texas at Austin, 201 E 24th St, Austin, 78712, TX, USA · Walker Department of Mechanical Engineering, The University of Texas at Austin, 3012026 E Dean Keeton St, Austin, 78712, TX, USA

Neural PDE Solvers PDE Solving PDE Operator Learning Neural Network Approximation Theory Neural Operators