Neural Network Approximation Theory

Ask about this topic

Overview

The field asks which functions and operators neural networks can represent, how many parameters or layers they need, and how approximation error depends on target smoothness and structure. Researchers prove constructive approximation rates and lower bounds for architectures including ReLU and smooth-activation networks, Kolmogorov-Arnold networks, Transformers, and neural operators, often using tools from harmonic analysis, approximation spaces, and geometry. Applications include nonparametric regression and PDE solution maps, where Sobolev, Barron, Besov, compositional, and symmetry-aware assumptions can yield sharper rates than general worst-case guarantees. A central tension is that favorable rates often rely on strong structure or idealized parameter precision, while optimization, sampling, and finite-precision limits can determine whether those guarantees are useful in practice.

Latest papers

CardsList

242 papers29 papers in the last four weeks, up 21% on the four weeks before. 0.2% of all new papers.

Feb 16, 2026

BEACONS: Bounded-Error, Algebraically-Composable Neural Solvers for Partial Differential Equations

The traditional limitations of neural networks in reliably generalizing beyond the convex hulls of their training data present a significant problem for computational physics, in which one often wishes to solve PDEs in regimes far beyond anything which can be experimentally or analytically validated. In this paper, we show how it is possible to circumvent these limitations by constructing formally-verified neural network solvers for PDEs, with rigorous convergence, stability, and conservation properties, whose correctness can therefore be guaranteed even in extrapolatory regimes. By using the method of characteristics to predict the analytical properties of PDE solutions a priori (even in regions arbitrarily far from the training domain), we show how it is possible to construct rigorous extrapolatory bounds on the worst-case L^inf errors of shallow neural network approximations. Then, by decomposing PDE solutions into compositions of simpler functions, we show how it is possible to compose these shallow neural networks together to form deep architectures, based on ideas from compositional deep learning, in which the large L^inf errors in the approximations have been suppressed. The resulting framework, called BEACONS (Bounded-Error, Algebraically-COmposable Neural Solvers), comprises both an automatic code-generator for the neural solvers themselves, as well as a bespoke automated theorem-proving system for producing machine-checkable certificates of correctness. We apply the framework to a variety of linear and non-linear PDEs, including the linear advection and inviscid Burgers' equations, as well as the full compressible Euler equations, in both 1D and 2D, and illustrate how BEACONS architectures are able to extrapolate solutions far beyond the training data in a reliable and bounded way. Various advantages of the approach over the classical PINN approach are discussed.

Jonathan Gorard, Ammar Hakim, James Juno

Princeton University Princeton, NJ 08544, USA · Princeton Plasma Physics Laboratory Princeton, NJ 08540, USA

Neural PDE SolversPartial Differential EquationsNeural Network GeneralizationNeural Network VerificationNeural Network Approximation Theory
AskPDF
Feb 16, 2026

Divine Benevolence is an x2x^2: GLUs scale asymptotically faster than MLPs

Scaling laws can be understood from ground-up numerical analysis, where traditional function approximation theory can explain shifts in model architecture choices. GLU variants now dominate frontier LLMs and similar outer-product architectures are prevalent in ranking models. The success of these architectures has mostly been left as an empirical discovery. In this paper, we apply the tools of numerical analysis to expose a key factor: these models have an x2x^2 which enables asymptotically faster scaling than MLPs. GLUs have piecewise quadratic functional forms that are sufficient to exhibit quadratic order of approximation. Our key contribution is to demonstrate that the L(P)L(P) scaling slope is L(P)∝P−3L(P)\propto P^{-3} for GLUs but only L(P)=P−2L(P)=P^{-2} for MLPs on function reconstruction problems. We provide a parameter construction and empirical verification of these slopes for 1D function approximation. From the first principles we discover, we make one stride and propose the ``Gated Quadratic Unit'' which has an even steeper L(P)L(P) slope than the GLU and MLP. This opens the possibility of architecture design from first principles numerical theory to unlock superior scaling in large models. Replication code is available at https://github.com/afqueiruga/divine_scaling.

Alejandro Francisco Queiruga

Neural Scaling LawsNeural Network Approximation TheoryMultilayer Perceptrons
AskPDF
Feb 12, 2026

Rational Neural Networks have Expressivity Advantages

We study neural networks with trainable low-degree rational activation functions and show that they are more expressive and parameter-efficient than modern piecewise-linear and smooth activations such as ELU, LeakyReLU, LogSigmoid, PReLU, ReLU, SELU, CELU, Sigmoid, SiLU, Mish, Softplus, Tanh, Softmin, Softmax, and LogSoftmax. For an error target of ε>0\varepsilon>0, we establish approximation-theoretic separations: Any network built from standard fixed activations can be uniformly approximated on compact domains by a rational-activation network with only poly(log⁡log⁡(1/ε))\mathrm{poly}(\log\log(1/\varepsilon)) overhead in size, while the converse provably requires Ω(log⁡(1/ε))Ω(\log(1/\varepsilon)) parameters in the worst case. This exponential gap persists at the level of full networks and extends to gated activations and transformer-style nonlinearities. In practice, rational activations integrate seamlessly into standard architectures and training pipelines, allowing rationals to match or outperform fixed activations under identical architectures and optimizers.

Maosen Tang, Alex Townsend

Center for Applied Mathematics, Cornell University, United States · Department of Mathematics, Cornell University, United States

Neural Network Activation FunctionsNeural Network Approximation TheoryNeural Network Expressivity
AskPDF
Feb 5, 2026

Clifford Kolmogorov-Arnold Networks

We introduce Clifford Kolmogorov-Arnold Network (ClKAN), a flexible and efficient architecture for function approximation in arbitrary Clifford Algebra spaces. We propose the use of Randomized Quasi-Monte Carlo grid generation as a solution to the exponential scaling associated with higher-dimensional algebras. Our ClKAN also introduces new batch normalization strategies to deal with variable domain input. ClKAN finds application in scientific discovery and engineering, and is validated in synthetic and physics-inspired tasks.

Matthias Wolff, Francesco Alesiani, Christof Duhme +1

Dept. of Computer Science University of Münster Münster, Germany · Data Science Div. NEC Labs Europe Heidelberg, Germany

Kolmogorov-Arnold NetworksNeural Network Approximation TheoryScientific ML
AskPDF
Feb 3, 2026

Geometry-Preserving Neural Architectures on Manifolds with Boundary

A growing number of neural architectures have been proposed to enforce geometric constraints, including projection-based networks, exponential-map updates, constrained output layers, and manifold neural ODEs. We provide a unified framework for these geometry-preserving architectures by organizing them according to where and how constraints are enforced, either throughout the intermediate layers or only at the final output. This perspective reveals several gaps in the existing theory. To address these gaps, we prove high-level approximation theorems for projected neural ODEs, intermediate augmented architectures, and final augmented architectures on prox-regular constraint sets, including smooth manifolds with boundary. Numerical experiments on synthetic dynamics over S^2, the disk, SO(3), together with real-world protein backbone data on SE(3), demonstrate exact feasibility for analytic updates and show that the final augmentation have simpler architecture and outperform in most tasks considered. When the constraint set is unknown, we learn projections via small-time heat-kernel limits, showing diffusion/flow-matching can be used as data-based projections. Moreover, we also the demonstrate the usefulness of the architectures that enforce non-convex constraints for path planning on manifolds with boundary.

Karthik Elamvazhuthi, Shiba Biswal, Kian Rosenblum +4

Los Alamos National Laboratory · Boston College

Geometric Representation LearningNeural Network Approximation TheoryNeural ODEs
AskPDF
Jan 30, 2026

Agile Reinforcement Learning through Separable Neural Architecture and Applications

Deep reinforcement learning (RL) is increasingly deployed in resource-constrained environments, yet go-to function approximators - multilayer perceptrons (MLPs) - are often parameter-inefficient due to an imperfect inductive bias for the smooth structure of many value functions. This mismatch can also hinder sample efficiency and slow policy learning in this capacity-limited regime. Although model compression techniques exist, they operate post-hoc and do not improve learning efficiency. Spline-based architectures such as Kolmogorov-Arnold Networks (KANs) have been shown to offer parameter efficiency but are widely reported to exhibit significant computational overhead, especially at scale. In seeking to address these limitations, this work introduces SPAN (SPline-based Adaptive Networks) for RL. SPAN adapts the KHRONOS framework with a learnable preprocessing layer. SPAN is evaluated across discrete (PPO) and high-dimensional continuous (SAC) control tasks, offline settings (Minari/D4RL) and a real-world datacenter HVAC control application. SPAN achieves a 30-50% improvement in sample efficiency and 1.3-9 times higher success rates across benchmarks compared to MLP baselines. Despite incurring a per-step evaluation overhead of 1.2-1.8x, SPAN's superior convergence reliability yields an expected total training cost 1.3-6.3x lower than MLP baselines when accounting for convergence failures. In the HVAC application, SPAN reduces energy consumption in 9 of 12 months relative to MLP while simultaneously achieving a 1.1-3.4x reduction in thermal comfort violations across the evaluation year, demonstrating generalization to real-world engineering control. Furthermore, SPAN demonstrates superior anytime performance and robustness to hyperparameter variations, suggesting it as a viable, high-performance alternative for learning efficient policies in resource-limited settings.

Rajib Mostakim, Reza T. Batley, Sourav Saha

Department of Mechanical Engineering, Bangladesh University of Engineering and Technology, Dhaka, Bangladesh · Kevin T. Crofton Department of Aerospace and Ocean Engineering, Virginia Polytechnic Institute and State University, Blacksburg, VA, USA

RL ControlNeural Network Approximation TheorySample-Efficient RLHeating, Ventilation, and Air Conditioning Control
AskPDF
Jan 23, 2026

Multigrade Neural Network Approximation

We study multigrade deep learning (MGDL) as a principled framework for structured error refinement in deep neural networks. While the approximation power of neural networks is now relatively well understood, training very deep architectures remains challenging due to highly nonconvex and often ill-conditioned optimization landscapes. In contrast, for relatively shallow networks, most notably certain one-hidden-layer ReLU models, training admits convex reformulations with global guarantees under appropriate settings, motivating learning paradigms that improve stability while scaling to depth. MGDL builds on this insight by training deep networks grade by grade: previously learned grades are frozen, and each newly added grade-wise subnetwork is composed on top of the previously learned grades and trained to fit the residual left by the current approximation, yielding a structured and interpretable hierarchical refinement process. We develop an operator-theoretic foundation for MGDL and prove that, for any continuous target function defined on a hypercube, there exists a fixed-width multigrade ReLU scheme whose residuals are pointwise nonincreasing in magnitude and converge uniformly to zero, with strict LpL^p-norm decay at every nontrivial grade for p∈[1,∞)p\in [1,\infty). To the best of our knowledge, this work provides the first rigorous constructive approximation guarantee showing that a grade-wise residual refinement scheme can achieve vanishing error in a fixed-width multigrade ReLU architecture.

Shijun Zhang, Zuowei Shen, Yuesheng Xu

Department of Applied Mathematics, Hong Kong Polytechnic University · Department of Mathematics, National University of Singapore · Department of Mathematics and Statistics, Old Dominion University

Neural Network Approximation TheoryReLU Neural Networks
AskPDF
Jan 20, 2026

Linearized subspace refinement framework to expose hidden accuracy in trained neural networks

Neural networks trained by gradient-based methods often exhibit optimization-induced accuracy plateaus in scientific machine learning tasks. We present Linearized Subspace Refinement (LSR), an architecture-agnostic post-training framework that exploits the local linearized model at a fixed trained state. By solving a reduced direct least-squares problem in a Jacobian-defined low-dimensional space, LSR computes a subspace-optimal linearized correction and yields a refined predictor with markedly improved accuracy. Across function approximation, data-driven operator learning, physics-informed operator fine-tuning, and noisy inverse problems, LSR shows that standard nonlinear training can remain far above this subspace-attainable error level. Similar accuracy plateaus persist even for the convex quadratic problem from local linearization when solved with standard iterative optimizers, identifying numerical ill-conditioning as a primary bottleneck. LSR frequently delivers order-of-magnitude error reductions, while the subspace rank provides an explicit capacity-control mechanism that balances correction strength, numerical stability, and noise sensitivity. Together, LSR exposes conditioning-limited attainable accuracy in trained-state linearized models and provides direct access to it.

Wenbo Cao, Weiwei Zhang

Institute of AI for Industries, Chinese Academy of Sciences, Nanjing 211135, China · Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China · School of Aeronautics, Northwestern Polytechnical University, Xi’an 710072, China +2

Subspace OptimizationNeural Network OptimizationNeural Network Approximation TheoryNeural Network Training Dynamics
AskPDF
Nov 26, 2025

SUPN: Shallow Universal Polynomial Networks for Efficient Function Approximation

Deep neural networks (DNNs) and Kolmogorov-Arnold networks (KANs) are flexible function approximators, but their accuracy can require substantial network depth, width, and trainable parameter counts. Beyond making the resulting networks less transparent, large parameter spaces can increase training cost and produce solutions that are sensitive to optimization and parameter initialization, leading to variability in generalization error. For these reasons, we propose Shallow Universal Polynomial Networks (SUPNs), which retain a single hidden layer while increasing the expressivity of each unit through a learnable multivariate polynomial inside each activation, which we refer to as a polynomial pre-activation. This construction trades architectural depth and width for greater expressivity within each hidden unit, reducing parameter counts and training cost while preserving a shallow computational graph. We establish two complementary universal approximation results: SUPNs are universal by increasing network width at fixed polynomial complexity, as in classical shallow-network approximation, and by increasing polynomial complexity at fixed finite width. We further prove that finite SUPNs can approach the best approximation attainable in a prescribed polynomial space to arbitrary relative tolerance, derive explicit quasi-optimal parameters, and establish corresponding convergence rates. We empirically evaluate SUPNs against DNNs, KANs, and polynomial projection on smooth, nonsmooth, and discontinuous function-approximation problems in one, two, and ten dimensions using more than 16,000 trained models. Across the target functions studied, SUPNs frequently achieve lower approximation error and variability than DNNs and KANs at comparable parameter counts while retaining lower training cost, supporting their use as efficient function approximators when the target structure is not known a priori.

Zachary Morrow, Michael Penwarden, Brian Chen +3

Polynomial Neural NetworksNeural Network Approximation TheoryShallow Neural NetworksUniversal Approximation
AskPDF
Nov 24, 2025

The Alexander-Hirschowitz theorem for neurovarieties

We study the dimension and identifiability of neurovarieties associated to polynomial neural networks. We give an independent geometric proof that the linear bounds di≥2ni−1d_i\geq 2n_i-1 on the activation degrees imply non defectiveness for any number of outputs, a dimension statement previously obtained from finite identifiability. The proof is based on a direct analysis of the differential of the parameterization. We also investigate secant and Grassmann-secant obstructions outside this range and prove global identifiability for multi-output architectures under the same degree bounds.

A. Massarenti, M. Mella

Polynomial Neural NetworksParameter IdentifiabilityNeural Network Approximation TheoryNeural Network Expressivity
AskPDF
Nov 16, 2025

On the Dimension-Free Approximation of Deep Neural Networks for Symmetric Korobov Functions

Deep neural networks have been widely used as universal approximators for functions with inherent physical structures, including permutation symmetry. In this paper, we construct symmetric deep neural networks to approximate symmetric Korobov functions and prove that both the convergence rate and the constant prefactor scale at most polynomially with respect to the ambient dimension. This represents a substantial improvement over prior approximation guarantees that suffer from the curse of dimensionality. Building on these approximation bounds, we further derive a generalization-error rate for learning symmetric Korobov functions whose leading factors likewise avoid the curse of dimensionality.

Yulong Lu, Tong Mao, Jinchao Xu +1

School of Mathematics, University of Minnesota, Minneapolis, 55455, MN, USA · Computer, Electrical and Mathematical Science and Engineering Division, King Abdullah University of Science and Technology, Thuwal 23955, Saudi Arabia · School of Mathematics, Georgia Institute of Technology, Atlanta, 30332, GA, USA

Neural Network GeneralizationNeural Network Approximation Theory
AskPDF
Nov 13, 2025

Autonomous-Flow-Based Generation

We show that using autonomous-flow-based generation, one can universally approximate orientation-preserving diffeomorphisms defined on the cube by Neural ODEs with rate O(P−1/d)\mathcal{O}(P^{-1/d}) with PP parameters. On the other hand, we show that by using only a single autonomous flow, the class of Neural ODEs is nowhere dense on the cube in dimension d≥2d \ge 2 . Under a compact-supportid_\mathrm{id} condition on (0,1)d(0,1)^d, we show that using autonomous-flow-based generation, one can universally approximate compactly supportedid_\mathrm{id} diffeomorphisms on (0,1)d(0,1)^d for any dimension with rate O((Plog⁡P)−2/d)\mathcal{O}((\frac{P}{\log P})^{-2/d}) with PP parameters and for compactly supportedid_\mathrm{id} homeomorphisms on (0,1)d(0,1)^d in dimension d≥5d \geq 5 with rate O(P−1/(d+1))\mathcal{O}(P^{-1/(d+1)}) with PP parameters and by a composition of at most IdI_d autonomous Neural ODEs with the same supportid_\mathrm{id}, where IdI_d depends only on the dimension. Moreover, we show that the class of single autonomous flows compactly supportedid_\mathrm{id} on (0,1)d(0,1)^d is meagre in the space of compactly supportedid_\mathrm{id} homeomorphisms on (0,1)d(0,1)^d for d≥2d\ge 2. By linearly lifting the domain into one higher dimension, we obtain a universal approximation result for Lipschitz functions compactly supported on (0,1)d(0,1)^d with rate O(P−1/(d+1))\mathcal{O}(P^{-1/(d+1)}) with PP parameters.

Hossein Rouhvarzi, Anastasis Kratsios

McMaster University and the Vector Institute Department of Mathematics 1280 Main Street West, Hamilton, Ontario, L8S 4K1, Canada

Universal ApproximationContinuous Normalizing FlowsNeural Network Approximation TheoryNeural ODEs
AskPDF
Oct 20, 2025

DFNN: A Deep Fréchet Neural Network Framework for Learning Metric-Space-Valued Responses

Regression with non-Euclidean responses---e.g., probability distributions, networks, symmetric positive-definite matrices, and compositions---has become increasingly important in modern applications. In this paper, we propose deep Fréchet neural networks (DFNNs), an end-to-end deep learning framework for predicting non-Euclidean responses---which are considered as random objects in a metric space---from Euclidean predictors. Our method utilizes the representation-learning power of deep neural networks (DNNs) to the task of approximating conditional Fréchet means of the response given the predictors, the metric-space analogue of conditional expectations, by minimizing a Fréchet risk. The framework is highly flexible, accommodating diverse metrics and high-dimensional predictors. We establish a universal approximation theorem for DFNNs, advancing the state-of-the-art of neural network approximation theory to general metric-space-valued responses, without making model assumptions or relying on local smoothing. We further establish rigorous generalization guarantees for DFNNs and derive corresponding risk bounds, providing, to the best of our knowledge, the first such theoretical results for deep learning regression with metric-space-valued responses. Empirical studies on synthetic distributional and network-valued responses, as well as real-world applications to predicting compositional responses in an Aitchison simplex and spherical responses, demonstrate that DFNNs consistently outperform all existing methods.

Kyum Kim, Yaqing Chen, Paromita Dubey

Rutgers University · University of Southern California

Neural Network GeneralizationNeural Network Approximation Theory
AskPDF
Oct 10, 2025

Distributionally robust approximation property of neural networks

The universal approximation property uniformly with respect to weakly compact families of measures is established for several classes of neural networks. To that end, we prove that these neural networks are dense in Orlicz spaces, thereby extending classical universal approximation theorems even beyond the traditional LpL^p-setting. The covered classes of neural networks include widely used architectures like feedforward neural networks with non-polynomial activation functions, deep narrow networks with ReLU activation functions and functional input neural networks.

Mihriban Ceylan, David J. Prömel

Neural Network Approximation TheoryUniversal Approximation
AskPDF
Oct 5, 2025

Configuration-Dependent Lower Bounds for Approximation by Shallow ReLUk^k Networks on the Sphere

We establish two related but logically distinct results for shallow ReLUk^k neural networks on the unit sphere \SSd\SS^d. First, for an arbitrary set of inner neural-network parameters, the best L2(\SSd)\mathcal{L}^2(\SS^d) approximation of a fixed target function with smoothness r>d+2k+12r>\tfrac{d+2k+1}{2} admits an asymptotic lower bound given by a constant multiple of n−1/2h‾k+1/2n^{-1/2}\underline{h}^{k+1/2}, where h‾\underline{h} denotes the antipodal separation distance of the normalized inner-parameter set. This lower bound depends explicitly on the parameter configuration through h‾\underline{h} and applies without additional assumptions on the parameters. Second, for antipodally quasi-uniform parameters, h‾≃n−1/d\underline{h}\simeq n^{-1/d}, and the lower bound establishes the exact saturation order n−d+2k+12dn^{-\frac{d+2k+1}{2d}} for such parameter families: a target function with regularity greater than d+2k+12\frac{d+2k+1}{2} and satisfying the required parity condition can be approximated at this rate, whereas approximation at any strictly faster rate forces the target function to be zero. Our results therefore place linearized neural-network approximation within the classical saturation framework and show that, although ReLUk^k network spaces can outperform finite elements of the same degree, this advantage is intrinsically limited.

Tong Mao, Jinchao Xu

Shenzhen Loop Area Institute, Shenzhen 518000, P. R. China · King Abdullah University of Science and Technology, Thuwal 23955, Saudi Arabia.

Shallow Neural NetworksNeural Network Approximation TheoryNeural Network ExpressivityFunction Approximation
AskPDF
Aug 20, 2025

Neural Operators Which Certifiably Simultaneously Solve Infinitely Many Convex Programs using Logarithmic Rank and Linear Complexity

Neural operators (NOs) are a class of deep learning models designed to simultaneously solve infinitely many related problems by casting them into an infinite-dimensional space, whereon these NOs operate. Such operator learning problems arise naturally in mathematical finance, for instance, when learning families of optimal portfolios or hedging strategies. Two significant gaps, however, remain between theory and practice: 1) most universal approximation guarantees do not describe practically implementable NOs or are pure existence statements; and 2) worst-case parameter bounds from universal approximation theorems suggest that constructed NOs may require an unrealistically large number of parameters to solve most operator learning problems, which stands in direct opposition to a slew of experimental evidence. This paper closes these gaps by exhibiting a short and simple training algorithm to construct a specific class of NOs, proximal equilibrium operators (PEOs), using (realistic) finite-dimensional deep equilibrium layers to solve families of convex optimization problems over a separable Hilbert space XX. Here, the inputs are smooth, convex loss functions on XX, and the outputs are the associated (approximate) solutions to the optimization problems defined by each input loss. Our main result is a certificate showing that, when the input losses lie in a suitable infinite-dimensional set, our PEOs trained with our algorithm do in fact uniformly approximate the corresponding solutions to arbitrary precision, with rank and width growing only logarithmically and the depth growing linearly in the reciprocal of the approximation error. We then validate both our theoretical results and the trainability of PEOs on three applications: non-linear PDEs, stochastic optimal control, and portfolio optimization and hedging problems in mathematical finance.

Anastasis Kratsios, Ariel Neufeld, Philipp Schmocker

Department of Mathematics, McMaster University and the Vector Institute, Canada · Division of Mathematical Sciences, Nanyang Technological University, Singapore

Neural OperatorsOperator LearningNeural Network Approximation TheoryProximal Optimization
AskPDF
Jul 27, 2025

Learning Latent Graph Geometry via Fixed-Point Schrödinger-Type Activation: A Theoretical Study

We study neural architectures in which each hidden layer is defined by the stationary state of a dissipative Schrödinger-type dynamics on a learned latent graph. On stable branches, the local stationary problem defines a differentiable implicit graph layer. To learn the graph itself, we optimize over the stratified moduli space of weighted graphs and equip each stratum with a non-degenerate Kähler-Hessian metric that keeps natural-gradient descent and face crossing well posed. We then show that a multilayer stationary network is equivalent to an exact global stationary problem on a supra-graph, and that it admits a penalized global relaxation whose stationary states converge to the exact one as the penalty parameter tends to infinity. Reverse-mode differentiation is recovered as the adjoint of the exact global system, and the penalized adjoint converges to it in the same limit. Finally, under finite-dimensional strong-monotonicity and admissible-lift assumptions, the corresponding represented hypothesis classes coincide among resolvent feed-forward networks, graph-stationary networks, supra-graph stationary systems, and sheaf-based architectures with unitary connection. The resulting structural identifications yield complexity bounds controlled by sparse graph or supra-graph geometry rather than dense ambient connectivity.

Dmitry Pasechnyuk-Vilensky, Martin Takáč

MBZUAI, UAE

Graph Structure LearningGraph Neural NetworksGraph Representation LearningSheaf Neural NetworksNeural Network Approximation Theory
AskPDF
Jul 11, 2025

Understanding Two-Layer Neural Networks with Smooth Activation Functions

This paper aims to understand the training solution, which is obtained by the back-propagation algorithm, of two-layer neural networks whose hidden layer is composed of the units with smooth activation functions, including the usual sigmoid type most commonly used before the advent of ReLUs. The mechanism contains four main principles: construction of Taylor series expansions, strict partial order of knots, smooth-spline implementation and smooth-continuity restriction. The universal approximation for arbitrary input dimensionality is proved and the explanation of training solutions is given. Through the principles proposed, the mystery of ``black box'' of the solution space is largely revealed. The new proofs employed also enrich approximation theory.

Changcun Huang

Shuitu Institute of Applied Mathematics, Chongqing 400700, P.R.C

Neural Network Activation FunctionsNeural Network Approximation TheoryNeural Network Training DynamicsNeural Network ExpressivityTwo-Layer Neural Networks
AskPDF
Jul 9, 2025

Discretization-independent operator learning for partial differential equations

We develop a new and general encode-approximate-reconstruct operator learning model that leverages learned neural representations of bases for input and output function distributions. We introduce the concepts of numerical operator learning and discretization independence, which clarify the relationship between theoretical formulations and practical realizations of operator learning models. Our model is discretization-independent, making it particularly effective for multiresolution learning. We establish theoretical approximation guarantees, demonstrating uniform universal approximation under strong assumptions on the input functions and statistical approximation under weaker conditions. To our knowledge, this is the first comprehensive study that investigates how discretization independence enables robust and efficient multiresolution operator learning. We validate our method through extensive numerical experiments involving both local and nonlocal PDEs, including time-independent and time-dependent problems. The results show that multiresolution training significantly improves accuracy and computational efficiency. Moreover, multiresolution training further enhances empirical discretization independence.

Jacob Hauck, Yanzhi Zhang

Department of Mathematics and Statistics, Missouri University of Science and Technology, Rolla, MO 65409

PDE Operator LearningNeural Network Approximation TheoryOperator LearningNeural Operators
AskPDF
Jun 26, 2025

Uniform Approximation of Functions with Asymmetric Growth and Decay by Deep Weighted Polynomials

Functions that grow without bound on one side of the real line and decay to zero on the other cannot be approximated uniformly by ordinary polynomials on unbounded domains. Motivated by classical weighted polynomial approximation, we introduce a class of one-sided weighted deep (composite) polynomial approximants for such asymmetric targets. The weight suppresses polynomial growth on the decaying side, while the composite polynomial remains free to capture growth on the other side. We prove that this mechanism reduces the half-line approximation problem to approximation on a compact interval whose length grows slowly with the degree, and we establish density and existence of best approximants in the appropriate closure of the model class. For computation, we first formulate the method as a trainable computational graph for deep weighted polynomial approximation. However, direct end-to-end optimization becomes increasingly ill-conditioned at high composite degree and can suffer from local minima. To address this, we introduce a fine-tuning procedure in which a fixed inner composition of monotone polynomial self-maps supplies the effective degree, while only the outer polynomial and weight parameters are trained; the outer fit reduces to a linear program. Numerical experiments on Black--Scholes option-pricing functions show that the resulting fine-tuned weighted deep polynomial achieves smaller uniform and L2L_2 errors than matched-budget polynomial baselines and resolves the decaying tail to machine precision.

Kingsley Yeon, Steven B. Damelin

Department of Statistics and CCAM, University of Chicago, Chicago, 60637, IL, USA · Department of Mathematics, ZBMATH-OPEN, Hermann-von-Helmholtz-Platz 1, Eggenstein-Leopoldshafen, 76344, Germany

Neural Network Approximation TheoryFunction Approximation
AskPDF
May 19, 2025

Parallel Layer Normalization for Universal Approximation

Layer Normalization (LN) is usually studied for its role in optimization, but it also acts as a multivariate nonlinearity between affine layers. We investigate how grouping LN units affects the approximation power of neural networks as width grows. In Parallel Layer Normalization networks (PLN-Nets), fixed-dimensional LN units are applied independently to groups between two affine layers. We prove that a shallow LN-Net with one LN unit is non-universal regardless of width, whereas PLN-Nets become universal as the number of parallel groups grows. The mechanism extends to deep networks, Root Mean Square Normalization (RMSNorm) variants, position-wise feed-forward networks, and convolutional networks under suitable grouping structures. Controlled MLP and physics-informed neural network (PINN) experiments support PLN's practical viability, while CNN and ViT experiments show that group count shapes the finite-sample fitting envelope but does not act alone: the grouped axes, replacement site, and label structure lead to distinct fitting and generalization behavior.

Yunhao Ni, Yuxin Guo, Yuhe Liu +4

State Key Laboratory of Complex and Critical Software Environment (SKLCCSE), Beihang University, Beijing, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou, China

Layer NormalizationNeural Network Approximation Theory
AskPDF
Apr 18, 2025

Causal pieces: analysing and improving spiking neural networks piece by piece

We introduce "causal pieces", a novel concept for analysing spiking neural networks (SNNs), inspired by "linear pieces" used to study expressivity and trainability in artificial neural networks (ANNs). Causal pieces partition the input and parameter space of a feedforward SNN with single-spike coding into distinct regions where the same subnetwork causes the output spikes. For networks of current-based leaky integrate-and-fire (LIF) neurons with large membrane time constants, we show that within each causal piece, output spike times are locally Lipschitz continuous with respect to inputs and network parameters. We further prove a lower bound on the approximation error that depends on the number of causal pieces. Thus, the number of causal pieces is a measure of the approximation capabilities of SNNs, which is valid despite spike-time discontinuities and applies to networks with both excitatory and inhibitory synapses. Empirically, we find that parameter initialisations yielding more causal pieces on the training set strongly correlate with SNN training success across multiple benchmarks, including Yin-Yang, Fashion-MNIST, and EuroSAT. Moreover, simulations with standard single-spike LIF neurons indicate that our findings extend beyond the theoretically analysed regime. These results establish causal pieces as a powerful and principled tool for analysing and improving the computational capabilities of SNNs.

Dominik Dold, Philipp Christian Petersen

Neural Network InitializationSpiking Neural NetworksNeural Network Approximation TheoryNeural Network Expressivity
AskPDF
Apr 3, 2025

On shallow feedforward neural networks with inputs from a topological space

We study feedforward neural networks with inputs from a topological space (TFNNs). We prove a universal approximation theorem for shallow TFNNs, which demonstrates their capacity to approximate any continuous function defined on this topological space. As an application, we obtain an approximative version of Kolmogorov's superposition theorem for compact metric spaces.

Vugar Ismailov

Institute of Mathematics and Mechanics, Baku, Azerbaijan · Center for Mathematics and its Applications, Khazar University, Baku, Azerbaijan

Feedforward Neural NetworksNeural Network Approximation TheoryShallow Neural NetworksUniversal Approximation
AskPDF
Mar 31, 2025

New universal operator approximation theorem for encoder-decoder architectures

Motivated by the rapidly growing field of mathematics for operator approximation with neural networks, we present a novel universal operator approximation theorem for broad classes of encoder-decoder architectures and a wide range of input and output spaces. In this study, we focus on the approximation of continuous operators between infinite-dimensional normed or metric spaces in the topology of uniform convergence on compact sets. Unlike standard results in the operator learning literature, we additionally investigate the case where the approximating sequence of encoder-decoder architectures can be chosen independently of the compact sets. Taking a topological perspective, we point out that compact-set-independent approximation is a strictly stronger property in most relevant operator learning frameworks. To establish our results, we introduce new approximation properties of input and output spaces tailored to encoder-decoder architectures. These properties enable us to prove a universal operator approximation theorem ensuring uniform convergence on every compact subset of the input space. Our results unify and extend existing universal operator approximation theorems for various encoder-decoder architectures, including classical DeepONets, BasisONets, MIONets, architectures based on frames and other related approaches. A notable feature of our framework is that it also applies to metric spaces beyond the normed setting. In particular, it allows the consideration of pp-Wasserstein spaces of probability measures as input or output spaces, and Skorohod spaces of càdlàg functions as input spaces. This generality also opens up potential applications in optimal transport.

Janek Gödeke, Pascal Fernsel

Center for Industrial Mathematics University of Bremen

Neural Network Approximation TheoryDeep Operator NetworksOperator Learning
AskPDF
Mar 3, 2025

The Mathematical Theory of Double Descent as Shrinkage, not Overfitting: Matching Bounds for Path-Regularized Multilayer Neural Networks

We give a nonasymptotic generalization theory for multilayer fully connected neural networks trained with path regularization, for general Lipschitz losses and without assuming the loss bounded, the architecture sparse, or any width or depth limit. Approximation error is modelled explicitly, and along the way we settle an open problem of E et al. on approximation rates in generalized Barron spaces. We treat the underparameterized and overparameterized regimes separately, which is what allows the resulting curve to be non-monotone in the width. We then prove lower bounds on the error itself, valid at every target and at every local minimizer, and combine them with the upper bounds into a theorem asserting that the error - not merely our bound on it - descends, rises and descends again as the width grows, with a peak exceeding the final plateau by a factor diverging in the sample size. Having both sides of the bound also identifies what drives the ascent, and it is not overfitting: once the regularization exceeds the dual norm of the noise, which the prescribed schedule does at every width, a slow-rate oracle inequality controls the error with no variance term at all. The rise is instead the shrinkage demanded by the regularization that the underparameterized regime requires, and we show it cannot be removed by any width-independent regularization under which the theory's guarantees hold. This mechanism is a property of the estimator analyzed here; we do not claim it accounts for the double descent reported for unregularized training.

Hao Yu

Department of Automation, Tsinghua University, Beijing, 100086, China

Double DescentNeural Network GeneralizationNeural Network Approximation TheoryGeneralization Bounds
AskPDF
Feb 26, 2025

Fourier Multi-Component and Multi-Layer Neural Networks: Unlocking High-Frequency Potential

The architecture of a neural network and the choice of its activation function are both fundamental to its performance. Equally important is ensuring that these two elements are well matched, as their alignment is key to effective representation and learning. In this paper, we introduce the Fourier Multi-Component and Multi-Layer Neural Network (FMMNN), a model that combines sine-type activations with the multi-component and multi-layer structure of MMNNs. In an FMMNN, each component is represented as a trainable linear combination of fixed random sine-type basis functions, while multi-layer composition generates more complex and adaptive high-frequency features. We establish that FMMNNs retain exponential expressive power for function approximation even under a low-rank architectural structure. We also analyze the optimization landscape of FMMNNs and find it to be substantially more favorable than that of standard fully connected neural networks, especially for high-frequency targets. In addition, we propose a scaled random initialization method for the first-layer weights in FMMNNs, which accelerates training and improves final performance when sufficient samples are available. Extensive numerical experiments support our theoretical insights, showing that FMMNNs achieve strong accuracy and favorable convergence behavior on oscillatory function-approximation benchmarks.

Shijun Zhang, Hongkai Zhao, Yimin Zhong +1

Department of Applied Mathematics Hong Kong Polytechnic University · Department of Mathematics Duke University · Department of Mathematics and Statistics Auburn University +1

Fourier Feature EmbeddingsNeural Network OptimizationNeural Network Activation FunctionsNeural Network Approximation TheoryNeural Network ExpressivityModular Neural Networks
AskPDF
Feb 4, 2025

Networks with Finite VC Dimension: Pro and Contra

Approximation and learning of classifiers of large data sets by neural networks in terms of high-dimensional geometry and statistical learning theory are investigated. The influence of the VC dimension of sets of input-output functions of networks on approximation capabilities is compared with its influence on consistency in learning from samples of data. It is shown that, whereas finite VC dimension is desirable for uniform convergence of empirical errors, it may not be desirable for approximation of functions drawn from a probability distribution modeling the likelihood that they occur in a given type of application. Based on the concentration-of-measure properties of high dimensional geometry, it is proven that both errors in approximation and empirical errors behave almost deterministically for networks implementing sets of input-output functions with finite VC dimensions in processing large data sets. Practical limitations of the universal approximation property, the trade-offs between the accuracy of approximation and consistency in learning from data, and the influence of depth of networks with ReLU units on their accuracy and consistency are discussed.

Vera Kurkova, Marcello Sanguineti

Institute of Computer Science of the Czech Academy of Sciences Pod Vodárenskou věží, 2 - 18207 Prague, Czech Republic · DIBRIS University of Genova Via Opera Pia, 13 - 16145 Genova, Italy

Neural Network GeneralizationNeural Network Approximation TheoryReLU Neural Networks
AskPDF
Nov 15, 2024

Dense ReLU Neural Networks for Temporal-spatial Model

In this paper, we focus on fully connected deep neural networks utilizing the Rectified Linear Unit (ReLU) activation function for nonparametric estimation. We derive non-asymptotic bounds that lead to convergence rates, addressing both temporal and spatial dependence in the observed measurements. By accounting for dependencies across time and space, our models better reflect the complexities of real-world data, enhancing both predictive performance and theoretical robustness. We also tackle the curse of dimensionality by modeling the data on a manifold, exploring the intrinsic dimensionality of high-dimensional data. We broaden existing theoretical findings of temporal-spatial analysis by applying them to neural networks in more general contexts and demonstrate that our proof techniques are effective for models with short-range dependence. Our empirical simulations across various synthetic response functions underscore the superior performance of our method, outperforming established approaches in the existing literature. These findings provide valuable insights into the strong capabilities of dense neural networks (Dense NN) for temporal-spatial modeling across a broad range of function classes.

Carlos Misael Madrid Padilla, Zhi Zhang, Xiaokai Luo +2

Department of Statistics and Data Science, Washington University in St. Louis. · Department of Statistics and Data Science, University of California, Los Angeles · Department of Statistics, University of Notre Dame

Nonparametric RegressionReLU Neural NetworksNeural Network Approximation Theory
AskPDF
Oct 18, 2024

Polynomial Scaling is Possible For Neural Operator Approximations of Structured Families of BSDEs

Neural operator (NO) architectures learn nonlinear maps between infinite-dimensional function spaces and are widely used to accelerate simulation and enable data-driven model discovery. While universality results ensure expressivity, they do not address complexity: for broad operator classes described only through regularity (e.g.\ uniform continuity or CrC^r-regularity), information-theoretic lower bounds imply that minimax-optimal NO approximation rates scale exponentially in the reciprocal accuracy 1/ε1/\varepsilon. This has shifted the focus of NO theory toward identifying additional problem-specific structure, beyond regularity, under which suitably tailored NO architectures can leverage to unlock polynomial scaling in 1/ε1/\varepsilon. We exhibit the first polynomial-scaling regime for NO approximations of solution operators in stochastic analysis; by identifying structured families of non-Markovian BSDEs with randomized terminal condition parameterized by the Sobolev-regular terminal condition and by Sobolev-regular additive nonlinear perturbations of the generator. We prove that their solution operator can be approximated (uniformly over the family) by a tailored NO whose number of trainable parameters grows polynomially in 1/ε1/\varepsilon. We unlock this polynomial scaling regime by informing the NO's inductive bias by factoring out the singular part of the associated semilinear elliptic PDE Green's function and by incorporating the Doléans--Dade exponential of the BSDE's common non-Markovian factor into the NO's decoding layers. As a byproduct, we extend polynomial-scaling guarantees from families of linear elliptic PDEs on regular domains to the semilinear setting.

Takashi Furuya, Anastasis Kratsios

Doshisha University, Japan · Department of Mathematics McMaster University and Vector Institute, Canada

Stochastic Differential EquationsPDE SolvingNeural Network Approximation TheoryNeural Operators
AskPDF
Oct 14, 2024

Statistical Properties of Deep Neural Networks with Dependent Data

This paper develops theory for deep neural network (DNN) estimators under dependent data. To provide theory applicable to a variety of DNN-based estimators, I first establish nonasymptotic probability bounds on the theoretical and empirical L2\mathcal{L}^{2}-errors of nonparametric sieve estimators for a general class of estimation problems under possibly nonstationary ββ-mixing data taking values in unbounded sets. I then apply the theory to fully connected and convolutional DNN estimators without bounds or sparsity restrictions on the DNN weights. For both DNN classes, I derive general results when the function to be estimated is Hölder smooth and the data are nonstationary, subgaussian, and ββ-mixing with either exponential or polynomial decay. I then specialize these to nonparametric regression, logistic regression, and quantile regression settings. Under exponential ββ-mixing, the resulting estimators attain the nonparametric minimax rate of Stone (1982) up to logarithmic factors.

Chad Brown

Department of Economics, University of Colorado Boulder, 80309, USA.

Nonparametric RegressionNeural Network Approximation TheoryStatistical Learning Theory
AskPDF