Neural Network Approximation Theory

Ask about this topic

Overview

The field asks which functions and operators neural networks can represent, how many parameters or layers they need, and how approximation error depends on target smoothness and structure. Researchers prove constructive approximation rates and lower bounds for architectures including ReLU and smooth-activation networks, Kolmogorov-Arnold networks, Transformers, and neural operators, often using tools from harmonic analysis, approximation spaces, and geometry. Applications include nonparametric regression and PDE solution maps, where Sobolev, Barron, Besov, compositional, and symmetry-aware assumptions can yield sharper rates than general worst-case guarantees. A central tension is that favorable rates often rely on strong structure or idealized parameter precision, while optimization, sampling, and finite-precision limits can determine whether those guarantees are useful in practice.

Latest papers

CardsList

242 papers29 papers in the last four weeks, up 21% on the four weeks before. 0.2% of all new papers.

Apr 16, 2026

Implicit Neural Representations: A Signal Processing Perspective

Implicit neural representations (INRs) mark a fundamental shift in signal modeling, moving from discrete sampled data to continuous functional representations. By parameterizing signals as neural networks, INRs provide a unified framework for representing images, audio, video, 3D geometry, and beyond as continuous functions of their coordinates. This functional viewpoint enables signal operations such as differentiation to be carried out analytically through automatic differentiation rather than through discrete approximations. In this article, we examine the evolution of INRs from a signal processing perspective, emphasizing spectral behavior, sampling theory, and multiscale representation. We trace the progression from standard coordinate based networks, which exhibit a spectral bias toward low frequency components, to more advanced designs that reshape the approximation space through specialized activations, including periodic, localized, and adaptive functions. We also discuss structured representations, such as hierarchical decompositions and hash grid encodings, that improve spatial adaptivity and computational efficiency. We further highlight the utility of INRs across a broad range of applications, including inverse problems in medical and radar imaging, compression, and 3D scene representation. By interpreting INRs as learned signal models whose approximation spaces adapt to the underlying data, this article clarifies the field's core conceptual developments and outlines open challenges in theoretical stability, weight space interpretability, and large scale generalization.

Dhananjaya Jayasundara, Vishal M. Patel

Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD 21218 USA

Implicit Neural RepresentationsSpectral BiasNeural Network Approximation Theory
AskPDF
Apr 8, 2026

Sparse-Aware Neural Networks for Nonlinear Functionals: Mitigating the Exponential Dependence on Dimension

Deep neural networks have emerged as powerful tools for learning operators defined over infinite-dimensional function spaces. However, existing theories frequently encounter difficulties related to dimensionality and limited interpretability. This work investigates how sparsity can help address these challenges in functional learning, a central ingredient in operator learning. We propose a framework that employs convolutional architectures to extract sparse features from a finite number of samples, together with deep fully connected networks to effectively approximate nonlinear functionals. Using universal discretization methods, we show that sparse approximators enable stable recovery from discrete samples. In addition, both the deterministic and the random sampling schemes are sufficient for our analysis. These findings lead to improved approximation rates and reduced sample sizes in various function spaces, including those with fast frequency decay and mixed smoothness. They also provide new theoretical insights into how sparsity can alleviate the curse of dimensionality in functional learning.

Jianfei Li, Shuo Huang, Han Feng +2

Department of Mathematics, Ludwig-Maximilians-Universität München (LMU Munich), Akademiestr. 7, 80799 München · Department of Mathematics, City University of Hong Kong, Hong Kong, China · School of Mathematics and Statistics, The University of Sydney, Sydney, New South Wales 2006, Australia +3

Curse of DimensionalityNeural Network Approximation TheorySparse Neural NetworksNeural Operators
AskPDF
Apr 2, 2026

Generalization Bounds and Statistical Guarantees for Multi-Task and Multiple Operator Learning with MNO Networks

Multiple operator learning concerns learning operator families {G[α]:U→V}α∈W\{G[α]:U\to V\}_{α\in W} indexed by an operator descriptor αα. Training data are collected hierarchically by sampling operator instances αα, then input functions uu per instance, and finally evaluation points xx per input, yielding noisy observations of G[α][u](x)G[α][u](x). While recent work has developed expressive multi-task and multiple operator learning architectures and approximation-theoretic scaling laws, quantitative statistical generalization guarantees remain limited. We provide a covering-number-based generalization analysis for separable models, focusing on the Multiple Neural Operator (MNO) architecture: we first derive explicit metric-entropy bounds for hypothesis classes given by linear combinations of products of deep ReLU subnetworks, and then combine these complexity bounds with approximation guarantees for MNO to obtain an explicit approximation-estimation tradeoff for the expected test error on new (unseen) triples (α,u,x)(α,u,x). The resulting bound makes the dependence on the hierarchical sampling budgets (nα,nu,nx)(n_α,n_u,n_x) transparent and yields an explicit learning-rate statement in the operator-sampling budget nαn_α, providing a sample-complexity characterization for generalization across operator instances. The structure and architecture can also be viewed as a general purpose solver or an example of a "small'' PDE foundation model, where the triples are one form of multi-modality.

Adrien Weihs, Hayden Schaeffer

Department of Mathematics, University of California Los Angeles, Los Angeles, CA 90095, USA.

Operator LearningGeneralization BoundsNeural OperatorsMulti-Task LearningNeural Network Approximation Theory
AskPDF
Apr 1, 2026

MVNN: A Measure-Valued Neural Network for Learning McKean-Vlasov Dynamics from Particle Data

Collective behaviors that emerge from interactions are fundamental to numerous biological systems. To learn such interacting forces from observations, we introduce a measure-valued neural network that infers measure-dependent interaction (drift) terms directly from particle-trajectory observations. The proposed architecture generalizes standard neural networks to operate on probability measures by learning cylindrical features, using an embedding network that produces scalable distribution-to-vector representations. On the theory side, we establish well-posedness of the resulting dynamics and prove propagation-of-chaos for the associated interacting-particle system. We further show universal approximation and quantitative approximation rates under a low-dimensional measure-dependence assumption. Numerical experiments on first and second order systems, including deterministic and stochastic Motsch-Tadmor dynamics, two-dimensional attraction-repulsion aggregation, Cucker-Smale dynamics, and a hierarchical multi-group system, demonstrate accurate prediction and strong out-of-distribution generalization.

Liyao Lyu, Xinyue Yu, Hayden Schaeffer

Department of Mathematics University of California, Los Angeles Los Angeles, CA 90024, USA

Emergent Behavior in Multi-Agent SystemsMean-Field TheoryNeural Network Approximation Theory
AskPDF
Mar 31, 2026

pp-adic Character Neural Network

We propose a new frame work of pp-adic neural network. Unlike the original pp-adic neural network by S.\ Albeverio, A.\ Khrennikov, and B.\ Tirrozi using a family of characteristic functions indexed by hyperparameters of precision as activation functions, we use a single injective pp-adic character on the topological Abelian group Zp\mathbb{Z}_p of pp-adic integers as an activation function. We prove the pp-adic universal approximation theorem for this formulation of pp-adic neural network, and reduce it to the feasibility problem of polynomial equations over the finite ring of integers modulo a power of pp.

Tomoki Mihara

Neural Network Approximation TheoryNeural Network Activation Functions
AskPDF
Mar 30, 2026

Universal Approximation Constraints of Narrow ResNets: The Tunnel Effect

We analyze the universal approximation constraints of narrow Residual Neural Networks (ResNets) both theoretically and numerically. For deep neural networks without input space augmentation, a central constraint is the inability to represent critical points of the input-output map. We prove that this has global consequences for target function approximations and show that the manifestation of this defect is typically a shift of the critical point to infinity, which we call the ``tunnel effect'' in the context of classification tasks. While ResNets offer greater expressivity than standard multilayer perceptrons (MLPs), their capability strongly depends on the signal ratio between the skip and residual channels. We establish quantitative approximation bounds for both the residual-dominant (close to MLP) and skip-dominant (close to neural ODE) regimes. These estimates depend explicitly on the channel ratio and uniform network weight bounds. Low-dimensional examples further provide a detailed analysis of the different ResNet regimes and how architecture-target incompatibility influences the approximation error.

Christian Kuehn, Sara-Viola Kuntz, Tobias Wöhrer

Technical University of Munich, School of Computation, Information and Technology, Department of Mathematics, Boltzmannstraße 3, 85748 Garching, Germany · Munich Data Science Institute (MDSI), Garching, Germany · Munich Center for Machine Learning (MCML), München, Germany +1

Universal ApproximationNeural Network Approximation TheoryFunction ApproximationNeural Network ExpressivityResidual Learning
AskPDF
Mar 30, 2026

FI-KAN: Fractal Interpolation Kolmogorov-Arnold Networks

Kolmogorov-Arnold Networks (KAN) employ B-spline bases on a fixed grid, providing no intrinsic multi-scale decomposition for non-smooth function approximation. We introduce Fractal Interpolation KAN (FI-KAN), which incorporates learnable fractal interpolation function (FIF) bases from iterated function system (IFS) theory into KAN. Two variants are presented: Pure FI-KAN (Barnsley, 1986) replaces B-splines entirely with FIF bases; Hybrid FI-KAN (Navascues, 2005) retains the B-spline path and adds a learnable fractal correction. The IFS contraction parameters give each edge a differentiable fractal dimension that adapts to target regularity during training. On a Holder regularity benchmark (α∈[0.2,2.0]α\in [0.2, 2.0]), Hybrid FI-KAN outperforms KAN at every regularity level (1.3x to 33x). On fractal targets, FI-KAN achieves up to 6.3x MSE reduction over KAN, maintaining 4.7x advantage at 5 dB SNR. On non-smooth PDE solutions (scikit-fem), Hybrid FI-KAN achieves up to 79x improvement on rough-coefficient diffusion and 3.5x on L-shaped domain corner singularities. Pure FI-KAN's complementary behavior, dominating on rough targets while underperforming on smooth ones, provides controlled evidence that basis geometry must match target regularity. A fractal dimension regularizer provides interpretable complexity control whose learned values recover the true fractal dimension of each target. These results establish regularity-matched basis design as a principled strategy for neural function approximation.

Gnankan Landry Regis N'guessan

Axiom Research Group · Dept. of Applied Mathematics and Computational Science, NM-AIST, Arusha, Tanzania · AIMS Research and Innovation Centre, Kigali, Rwanda

Kolmogorov-Arnold NetworksNeural Network Approximation Theory
AskPDF
Mar 27, 2026

Expressibility of neural quantum states: a Walsh-complexity perspective

Neural quantum states are powerful variational wavefunctions, but it remains unclear which many-body states can be represented efficiently by modern additive architectures. We introduce Walsh complexity, a basis-dependent measure of how broadly a wavefunction is spread over parity patterns. States with an almost uniform Walsh spectrum require exponentially large Walsh complexity from any good approximant. We show that shallow additive feed-forward networks cannot generate such complexity in the tame regime, e.g. polynomial activations with subexponential parameter scaling. As a concrete example, we construct a simple dimerized state prepared by a single layer of disjoint controlled-ZZ gates. Although it has only short-range entanglement and a simple tensor-network description, its Walsh complexity is maximal. Full-cube fits across system size and depth are consistent with the complexity bound: for polynomial activations, successful fitting appears only once depth reaches a logarithmic scale in NN, whereas activation saturation in tanh⁡\tanh produces a sharp threshold-like jump already at depth 33. Walsh complexity therefore provides an expressibility axis complementary to entanglement and clarifies when depth becomes an essential resource for additive neural quantum states.

Taige Wang

Department of Physics, Massachusetts Institute of Technology, Cambridge, MA 02139, USA · Department of Physics, Harvard University, Cambridge, MA 02138, USA

Neural Quantum StatesNeural Network ExpressivityShallow Neural NetworksNeural Network Approximation Theory
AskPDF
Mar 27, 2026

Kantorovich--Kernel Neural Operators: Approximation Theory, Asymptotics, and Neural Network Interpretation

This paper studies a class of multivariate Kantorovich-kernel neural network operators, including the deep Kantorovich-type neural network operators studied by Sharma and Singh. We prove density results, establish quantitative convergence estimates, derive Voronovskaya-type theorems, analyze the limits of partial differential equations for deep composite operators, prove Korovkin-type theorems, and propose inversion theorems. This paper studies a class of multivariate Kantorovich-kernel neural network operators, including the deep Kantorovich-type neural network operators studied by Sharma and Singh. We prove density results, establish quantitative convergence estimates, derive Voronovskaya-type theorems, analyze the limits of partial differential equations for deep composite operators, prove Korovkin-type theorems, and propose inversion theorems. Furthermore, this paper discusses the connection between neural network architectures and the classical positive operators proposed by Chui, Hsu, He, Lorentz, and Korovkin.

Tian-Xiao He

Illinois Wesleyan University, Bloomington, IL 61702

Neural Network Approximation TheoryNeural Operators
AskPDF
Mar 21, 2026

CFNN: Continued Fraction Neural Network

Accurately characterizing non-linear functional manifolds with singularities is a fundamental challenge in scientific computing. While Multi-Layer Perceptrons (MLPs) dominate, their spectral bias hinders resolving high-curvature features without excessive parameters. We introduce Continued Fraction Neural Networks (CFNNs), integrating continued fractions with gradient-based optimization to provide a rational inductive bias.'' This enables capturing complex asymptotics and discontinuities with extreme parameter frugality. We provide formal approximation bounds demonstrating exponential convergence and stability guarantees. To address recursive instability, we develop three implementations: CFNN-Boost, CFNN-MoE, and CFNN-Hybrid. Benchmarks show CFNNs consistently outperform MLPs in precision with one to two orders of magnitude fewer parameters, exhibiting up to a 47-fold improvement in noise robustness and physical consistency. By bridging black-box flexibility and white-box transparency, CFNNs establish a reliable grey-box'' paradigm for AI-driven scientific research.

Chao Wang, Xuancheng Zhou, Ruilin Hou +2

School of Future Technology, Shanghai University, Shanghai, 200444, China. · Institute of Artificial Intelligence, Shanghai University, Shanghai, 200444, China. · School of Computer Science and Technology, Shanghai University, Shanghai, 200444, China. +2

Function ApproximationScientific MLNeural Network Approximation Theory
AskPDF
Mar 19, 2026

Deep Hilbert--Galerkin Methods for Infinite-Dimensional PDEs and Optimal Control

We develop deep learning-based approximation methods for fully nonlinear second-order PDEs on separable Hilbert spaces, such as HJB equations for infinite-dimensional control, by parameterizing solutions via Hilbert--Galerkin Neural Operators (HGNOs). We prove the first Universal Approximation Theorems (UATs) which are sufficiently powerful to address these problems, based on novel topologies for Hessian terms and corresponding novel continuity assumptions on the fully nonlinear operator. These topologies are non-sequential and non-metrizable, making the problem delicate. In particular, we prove UATs for functions on Hilbert spaces, together with their Fréchet derivatives up to second order, and for unbounded operators applied to the first derivative, ensuring that HGNOs are able to approximate all the PDE terms. For control problems, we further prove UATs for optimal feedback controls in terms of our approximating value function HGNO. We develop numerical training methods, which we call Deep Hilbert--Galerkin and Hilbert Actor-Critic (reinforcement learning) Methods, for these problems by minimizing the Lμ2(H)L^2_μ(H)-norm of the residual of the PDE on the whole Hilbert space, not just a projected PDE to finite dimensions. This is the first paper to propose such an approach. The models considered arise in many applied sciences, such as functional differential equations in physics and Kolmogorov and HJB PDEs related to controlled PDEs, SPDEs, path-dependent systems, partially observed stochastic systems, and mean-field SDEs. We numerically solve examples of Kolmogorov and HJB PDEs related to the optimal control of deterministic and stochastic heat and Burgers' equations, demonstrating the promise of our deep learning-based approach.

Samuel N. Cohen, Filippo de Feo, Jackson Hebner +1

Mathematical Institute, University of Oxford · Institut für Mathematik, Technische Universität Berlin

HJB EquationsPDE SolvingNeural PDE SolversPDE Operator LearningOptimal ControlNeural Network Approximation Theory+1
AskPDF
Mar 19, 2026

Mathematical Foundations of Deep Learning

This draft book offers a comprehensive and rigorous treatment of the mathematical principles underlying modern deep learning. The book spans core theoretical topics, from the approximation capabilities of deep neural networks, the theory and algorithms of optimal control and reinforcement learning integrated with deep learning techniques, to contemporary generative models that drive today's advances in artificial intelligence.

Xiaojing Ye

Neural Network Approximation TheoryGenerative ModelingReinforcement LearningOptimal Control
AskPDF
Mar 16, 2026

Deep learning and the rate of approximation by flows

We investigate the dependence of the approximation capacity of deep residual networks on its depth in a continuous dynamical systems setting. This can be formulated as the general problem of quantifying the minimal time-horizon required to approximate a diffeomorphism by flows driven by a given family F\mathcal F of vector fields. We show that this minimal time can be identified as a geodesic distance on a sub-Finsler manifold of diffeomorphisms, where the local geometry is characterised by a variational principle involving F\mathcal F. This connects the learning efficiency of target relationships to their compatibility with the learning architectural choice. Further, the results suggest that the key approximation mechanism in deep learning, namely the approximation of functions by composition or dynamics, differs in a fundamental way from linear approximation theory, where linear spaces and norm-based rate estimates are replaced by manifolds and geodesic distances.

Jingpu Cheng, Qianxiao Li, Ting Lin +1

Department of Mathematics, National University of Singapore, 117543, Singapore · Department of Mathematics and Institute for Functional Intelligent Materials, National University of Singapore, 117543, Singapore · School of Mathematical Sciences, Peking University, 100871, China

Neural Network Approximation TheoryFunction ApproximationResidual Learning
AskPDF
Mar 13, 2026

Equivalence of approximation by networks of single- and multi-spike neurons

In a spiking neural network, is it enough for each neuron to spike at most once? In recent work, approximation bounds for spiking neural networks have been derived, quantifying how well they can fit target functions. However, these results are only valid for neurons that spike at most once, which is commonly thought to be a strong limitation. Here, we show that the opposite is true for a large class of spiking neuron models, including the commonly used leaky integrate-and-fire model with subtractive reset: for every approximation bound that is valid for a set of multi-spike neural networks, there is an equivalent set of single-spike neural networks with only linearly more (or less) neurons, in the maximum number of spikes, for which the bound holds. The same is true for the reverse direction too, showing that regarding their approximation capabilities in general machine learning tasks, single-spike and multi-spike neural networks are equivalent. Consequently, many approximation results in the literature for single-spike neural networks also hold for the multi-spike case.

Dominik Dold, Philipp Christian Petersen

Spiking Neural NetworksNeural Network Approximation TheoryNeural Network Expressivity
AskPDF
Mar 12, 2026

Topological DeepONets and a generalization of the Chen-Chen operator approximation theorem

Deep Operator Networks (DeepONets) provide a branch-trunk neural architecture for approximating nonlinear operators acting between function spaces. In the classical operator approximation framework, the input is a function u∈C(K1)u\in C(K_1) defined on a compact set K1K_1 (typically a compact subset of a Banach space), and the operator maps uu to an output function G(u)∈C(K2)G(u)\in C(K_2) defined on a compact Euclidean domain K2⊂RdK_2\subset\mathbb{R}^d. In this paper, we develop a topological extension in which the operator input lies in an arbitrary Hausdorff locally convex space XX. We construct topological feedforward neural networks on XX using continuous linear functionals from the dual space X∗X^* and introduce topological DeepONets whose branch component acts on XX through such linear measurements, while the trunk component acts on the Euclidean output domain. Our main theorem shows that continuous operators G:V→C(K;Rm)G:V\to C(K;\mathbb{R}^m), where V⊂XV\subset X and K⊂RdK\subset\mathbb{R}^d are compact, can be uniformly approximated by such topological DeepONets. This extends the classical Chen-Chen operator approximation theorem from spaces of continuous functions to locally convex spaces and yields a branch-trunk approximation theorem beyond the Banach-space setting.

Vugar Ismailov

Neural Network Approximation TheoryDeep Operator NetworksNeural Operators
AskPDF
Mar 11, 2026

Efficient Approximation to Analytic and LpL^p functions by Height-Augmented ReLU Networks

This work addresses two fundamental limitations in neural network approximation theory. We demonstrate that a three-dimensional network architecture enables a significantly more efficient representation of sawtooth functions, which serves as the cornerstone in the approximation of analytic and LpL^p functions. First, we establish substantially improved exponential approximation rates for several important classes of analytic functions and offer a parameter-efficient network design. Second, for the first time, we derive a quantitative and non-asymptotic approximation of high orders for general LpL^p functions. Our techniques advance the theoretical understanding of the neural network approximation in fundamental function spaces and offer a theoretically grounded pathway for designing more parameter-efficient networks.

ZeYu Li, FengLei Fan, TieYong Zeng

Chinese University of Hong Kong, Department of Mathematics, Hong Kong, 999077, NT, Hong Kong SAR, China · City University of Hong Kong, Department of Data Science, Hong Kong, 999077, Kowloon, Hong Kong SAR, China · Guangzhou Nanfang College, School of Mathematics and Statistics, Guangzhou, 510970, Guangdong, China +1

Neural Network Approximation TheoryReLU Neural Networks
AskPDF
Mar 10, 2026

Stochastic Port-Hamiltonian Neural Networks: Universal Approximation with Passivity Guarantees

Stochastic port-Hamiltonian systems represent open dynamical systems with dissipation, inputs, and stochastic forcing in an energy based form. We introduce stochastic port-Hamiltonian neural networks, SPH-NNs, which parameterize the Hamiltonian with a feedforward network and enforce skew symmetry of the interconnection matrix and positive semidefiniteness of the dissipation matrix. For Itô dynamics we establish a weak passivity inequality in expectation under an explicit generator condition, stated for a stopped process on a compact set. We also prove a universal approximation result showing that, on any compact set and finite horizon, SPH-NNs approximate the coefficients of a target stochastic port-Hamiltonian system with C2C^2 accuracy of the Hamiltonian and yield coupled solutions that remain close in mean square up to the exit time. Experiments on noisy mass spring, Duffing, and Van der Pol oscillators show improved long horizon rollouts and reduced energy error relative to a multilayer perceptron baseline.

Luca Di Persio, Matthias Ehrhardt, Youness Outaleb

Department of Computer Science - College of Mathematics University of Verona, Strada le Grazie 15 - 37134 Verona, Italy · University of Wuppertal, Chair of Applied and Computational Mathematics, Gaußstrasse 20, 42119 Wuppertal, Germany · Doctoral School in Mathematics, University of Trento Via Sommarive, 14 - 38123 Povo, Italy

Port-Hamiltonian SystemsHamiltonian Neural NetworksStochastic Differential EquationsNeural Network Approximation TheoryPassivity-Based Control
AskPDF
Mar 9, 2026

DualFlexKAN: Dual-stage Kolmogorov-Arnold Networks with Independent Function Control

Multi-Layer Perceptrons (MLPs) rely on pre-defined, fixed activation functions, imposing a static inductive bias that forces the network to approximate complex topologies solely through increased depth and width. Kolmogorov-Arnold Networks (KANs) address this limitation through edge-centric learnable functions, yet their formulation suffers from quadratic parameter scaling and architectural rigidity that hinders the effective integration of standard regularization techniques. This paper introduces the DualFlexKAN (DFKAN), a flexible architecture featuring a dual-stage mechanism that independently controls pre-linear input transformations and post-linear output activations. This decoupling enables hybrid networks that optimize the trade-off between expressiveness and computational cost. Unlike standard formulations, DFKAN supports diverse basis function families, including orthogonal polynomials, B-splines, and radial basis functions, integrated with configurable regularization strategies that stabilize training dynamics. Comprehensive evaluations across regression benchmarks, physics-informed tasks, and function approximation demonstrate that DFKAN outperforms both MLPs and conventional KANs in accuracy, convergence speed, and gradient fidelity. The proposed hybrid configurations achieve superior performance with one to two orders of magnitude fewer parameters than standard KANs, effectively mitigating the parameter explosion problem while preserving KAN-style expressiveness. DFKAN provides a principled, scalable framework for incorporating adaptive non-linearities, proving particularly advantageous for data-efficient learning and interpretable function discovery in scientific applications.

Andrés Ortiz, Nicolás J. Gallego-Molina, Carmen Jiménez-Mesa +2

Department of Communications Engineering University of Malaga Malaga, Spain · DaSCI Andalusian Institute of Data Science and Computational Intelligence University of Granada Granada, Spain · Department of signal Theory, Communications and Telematics University of Granada Granada, Spain

Function ApproximationKolmogorov-Arnold NetworksNeural Network Approximation Theory
AskPDF
Mar 4, 2026

Neural Approximation and Its Applications

Multivariate function approximation is a fundamental problem in machine learning. Classic multivariate function approximations rely on hand-crafted basis functions (e.g., polynomial basis and Fourier basis), which limits their approximation ability and data adaptation ability, resulting in unsatisfactory performance. To address these challenges, we introduce the neural basis function by leveraging an untrained neural network as the basis function. Equipped with the proposed neural basis function, we suggest the neural approximation (NeuApprox) paradigm for multivariate function approximation. Specifically, the underlying multivariate function behind the multi-dimensional data is decomposed into a sum of block terms. The clear physically-interpreted block term is the product of expressive neural basis functions and their corresponding learnable coefficients, which allows us to faithfully capture distinct components of the underlying data and also flexibly adapt to new data by readily fine-tuning the neural basis functions. Attributed to the elaborately designed block terms, the suggested NeuApprox enjoys strong approximation ability and flexible data adaptation ability over the hand-crafted basis function-based methods. We also theoretically prove that NeuApprox can approximate any multivariate continuous function to arbitrary accuracy. Extensive experiments on diverse multi-dimensional datasets (including multispectral images, light field data, videos, traffic data, and point cloud data) demonstrate the promising performance of NeuApprox in terms of both approximation capability and adaptability.

Wei-Hao Wu, Ting-Zhu Huang, Xi-Le Zhao +2

School of Mathematical Sciences, University of Electronic Science and Technology of China, Chengdu, 611731, Sichuan, China. · School of Mathematical and Statistics, Xi’an Jiaotong University, Xi’an, 710049, Shaanxi, China. · Key Laboratory of Computational Intelligence and Chinese Information Processing of Ministry of Education, Shanxi University, Taiyuan, 030006, Shanxi, China.

Neural Network Approximation Theory
AskPDF
Mar 4, 2026

Extending Neural Operators: Robust Handling of Functions Beyond the Training Set

We develop a rigorous framework for extending neural operators to handle out-of-distribution input functions. We leverage kernel approximation techniques and provide theory for characterizing the input-output function spaces in terms of Reproducing Kernel Hilbert Spaces (RKHSs). We provide theorems on the requirements for reliable extensions and their predicted approximation accuracy. We also establish formal relationships between specific kernel choices and their corresponding Sobolev Native Spaces. This connection further allows the extended neural operators to reliably capture not only function values but also their derivatives. Our methods are empirically validated through the solution of elliptic partial differential equations (PDEs) involving operators on manifolds having point-cloud representations and handling geometric contributions. We report results on key factors impacting the accuracy and computational performance of the extension approaches.

Blaine Quackenbush, Paul J. Atzberger

Department of Mathematics, University of California Santa Barbara (UCSB). · Department of Mathematics, Department of Mechanical Engineering, University of California Santa Barbara (UCSB)

Neural OperatorsOOD GeneralizationPDE Operator LearningNeural Network Approximation Theory
AskPDF
Mar 3, 2026

Coalgebras for categorical deep learning: Representability and universal approximation

Categorical deep learning (CDL) has recently emerged as a framework that leverages category theory to unify diverse neural architectures. While geometric deep learning (GDL) is grounded in the specific context of invariants of group actions, CDL aims to provide domain-independent abstractions for reasoning about models and their properties. In this paper, we contribute to this program by developing a coalgebraic foundation for equivariant representation in deep learning, as classical notions of group actions and equivariant maps are naturally generalized by the coalgebraic formalism. Our first main result demonstrates that, given an embedding of data sets formalized as a functor from SET to VECT, and given a notion of invariant behavior on data sets modeled by an endofunctor on SET, there is a corresponding endofunctor on VECT that is compatible with the embedding in the sense that this lifted functor recovers the analogous notion of invariant behavior on the embedded data. Building on this foundation, we then establish a universal approximation theorem for equivariant maps in this generalized setting. We show that continuous equivariant functions can be approximated within our coalgebraic framework for a broad class of symmetries. This work thus provides a categorical bridge between the abstract specification of invariant behavior and its concrete realization in neural architectures.

Dragan Mašulović

University of Novi Sad, Faculty of Sciences Department of Mathematics and Informatics Trg Dositeja Obradovića 3, 21000 Novi Sad, Serbia

Equivariant Representation LearningInvariant Representation LearningNeural Network Approximation TheoryCategory Theory
AskPDF
Mar 3, 2026

On the Expressive Power of Transformers for Maxout Networks and Continuous Piecewise Linear Functions

Transformer networks have achieved remarkable empirical success across a wide range of applications, yet their theoretical expressive power remains insufficiently understood. In this paper, we study the expressive capabilities of Transformer architectures. We first establish an explicit approximation of maxout networks by Transformer networks while preserving comparable model complexity. As a consequence, Transformers inherit the universal approximation capability of ReLU networks under similar complexity constraints. Building on this connection, we develop a framework to analyze the approximation of continuous piecewise linear functions by Transformers and quantitatively characterize their expressivity via the number of linear regions, which grows exponentially with depth. Our analysis establishes a theoretical bridge between approximation theory for standard feedforward neural networks and Transformer architectures. It also yields structural insights into Transformers: self-attention layers implement max-type operations, while feedforward layers realize token-wise affine transformations.

Linyan Gu, Lihua Yang, Feng Zhou

School of Mathematics, Sun Yat-sen University · School of Information Science, Guangdong University of Finance and Economics

Transformer ExpressivityNeural Network Approximation TheoryTransformer Attention
AskPDF
Mar 3, 2026

Lattice-based Deep Neural Networks: Regularity and Tailored Regularization

This survey article is concerned with the application of lattice rules to Deep Neural Networks (DNNs), lattice rules being a family of quasi-Monte Carlo methods. They have demonstrated effectiveness in various contexts for high-dimensional integration and function approximation. They are extremely easy to implement thanks to their very simple formulation -- all that is required is a good integer generating vector of length matching the dimensionality of the problem. In recent years there has been a burst of research activities on the application and theory of DNNs. We review our recent article on using lattice rules as training points for DNNs with a smooth activation function, where we obtained explicit regularity bounds of the DNNs. By imposing restrictions on the network parameters to match the regularity features of the target function, we prove that DNNs with tailored lattice training points can achieve good theoretical generalization error bounds, with implied constants independent of the input dimension. We also demonstrate numerically that DNNs trained with our tailored regularization perform significantly better than with standard ℓ2\ell_2 regularization.

Alexander Keller, Frances Y. Kuo, Dirk Nuyens +1

Neural Network Approximation TheoryGeneralization Bounds
AskPDF
Feb 24, 2026

Standard Transformers Achieve the Minimax Rate in Nonparametric Regression with Cs,λC^{s,λ} Targets

The tremendous success of Transformer models in fields such as large language models and computer vision necessitates a rigorous theoretical investigation. To the best of our knowledge, this paper is the first work proving that standard Transformers can approximate Hölder functions C^{s,λ}\left([0,1]^{d\times n}\right) $$ (s\in\mathbb{N}_{\geq0},0<λ\leq1) under the LtL^t distance (t∈[1,∞]t \in [1, \infty]) with arbitrary precision. Building upon this approximation result, we demonstrate that standard Transformers achieve the minimax optimal rate in nonparametric regression for Hölder target functions. It is worth mentioning that, by introducing two metrics: the size tuple and the dimension vector, we provide a fine-grained characterization of Transformer structures, which facilitates future research on the generalization and optimization errors of Transformers with different structures. As intermediate results, we also derive the upper bounds for the Lipschitz constant of standard Transformers and their memorization capacity, which may be of independent interest. These findings provide theoretical justification for the powerful capabilities of Transformer models.

Yanming Lai, Defeng Sun

Department of Applied Mathematics, The Hong Kong Polytechnic University, Hung Hom, Hong Kong, China

Transformer ExpressivityNeural Network Approximation TheoryNonparametric RegressionStatistical Learning Theory
AskPDF
Feb 23, 2026

Quantitative Approximation Rates for Group Equivariant Learning

The universal approximation theorem establishes that neural networks can approximate any continuous function on a compact set. Later works in approximation theory provide quantitative approximation rates for ReLU networks on the class of αα-Hölder functions f:[0,1]N→Rf: [0,1]^N \to \mathbb{R}. The goal of this paper is to provide similar quantitative approximation results in the context of group equivariant learning, where the learned αα-Hölder function is known to obey certain group symmetries. While there has been much interest in the literature in understanding the universal approximation properties of equivariant models, very few quantitative approximation results are known for equivariant models. In this paper, we bridge this gap by deriving quantitative approximation rates for several prominent group-equivariant and invariant architectures. The architectures that we consider include: the permutation-invariant Deep Sets architecture; the permutation-equivariant Sumformer and Transformer architectures; joint invariance to permutations and rigid motions using invariant networks based on frame averaging; and general bi-Lipschitz invariant models. Overall, we show that equally-sized ReLU MLPs and equivariant architectures are equally expressive over equivariant functions. Thus, hard-coding equivariance does not result in a loss of expressivity or approximation power in these models.

Jonathan W. Siegel, Snir Hordan, Hannah Lawrence +2

Department of Mathematics, Texas A&M University, College Station, TX, 77840, USA · Faculty of Mathematics, Technion- Israel Institute of Technology, Haifa, 3200003, Israel · Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, 02139 USA

Permutation-Equivariant Neural NetworksEquivariant Neural NetworksNeural Network Approximation TheoryPermutation-Invariant Learning
AskPDF
Feb 23, 2026

Smoothness Adaptivity in Constant-Depth Neural Networks: Optimal Rates via Smooth Activations

Smooth activation functions are ubiquitous in modern deep learning, yet their theoretical advantages over non-smooth counterparts remain poorly understood. In this work, we study both approximation and statistical properties of neural networks with smooth activations for learning functions in the Sobolev space Ws,∞([0,1]d)W^{s,\infty}([0,1]^d) with s>0s>0. We prove that constant-depth networks equipped with smooth activations achieve smoothness adaptivity: increasing width alone suffices to attain the minimax-optimal approximation and estimation error rates (up to logarithmic factors). In contrast, for non-smooth activations such as ReLU, smoothness adaptivity is fundamentally limited by depth: the attainable approximation order is bounded by depth, and higher-order smoothness requires proportional depth growth. These results identify activation smoothness as a fundamental mechanism, complementary to depth, for achieving optimal rates over Sobolev function classes. Technically, our analysis is based on a multi-scale approximation framework that yields explicit neural network approximators with controlled parameter norms and model size. This complexity control ensures statistical learnability under empirical risk minimization (ERM) and avoids the impractical ℓ0\ell^0-sparsity constraints commonly required in prior analyses.

Yuhao Liu, Zilin Wang, Lei Wu +1

Department of Mathematical Sciences, Tsinghua University · School of Mathematical Sciences, Peking University · Center for Machine Learning Research, Peking University +1

Neural Network Approximation TheoryNeural Network Activation FunctionsShallow Neural NetworksStatistical Learning Theory
AskPDF
Feb 20, 2026

Hardware-Friendly Input Expansion for Accelerating Function Approximation

One-dimensional function approximation is a fundamental problem in scientific computing and engineering applications. While neural networks possess powerful universal approximation capabilities, their optimization process is often hindered by flat loss landscapes induced by parameter-space symmetries, leading to slow convergence and poor generalization, particularly for high-frequency components. Inspired by the principle of symmetry breaking in physics, this paper proposes a hardware-friendly approach for function approximation through input-space expansion. The core idea involves augmenting the original one-dimensional input (e.g., xx) with constant values (e.g., ππ) to form a higher-dimensional vector (e.g., [π,π,x,π,π][π, π, x, π, π]), effectively breaking parameter symmetries without increasing the network's parameter count. We evaluate the method on ten representative one-dimensional functions, including smooth, discontinuous, high-frequency, and non-differentiable functions. Experimental results demonstrate that input-space expansion significantly accelerates training convergence (reducing LBFGS iterations by 12% on average) and enhances approximation accuracy (reducing final MSE by 66.3% for the optimal 5D expansion). Ablation studies further reveal the effects of different expansion dimensions and constant selections, with ππ consistently outperforming other constants. Our work proposes a low-cost, efficient, and hardware-friendly technique for algorithm design.

Hu Lou, Yin-Jun Gao, Dong-Xiao Zhang +3

Northwest Institute of Nuclear Technology, Xi’an, 710000, Shaanxi, China.

Neural Network Approximation TheoryNeural Network Optimization
AskPDF
Feb 19, 2026

From Approximation Rates to Loss-Landscape Barrier Decay in Shallow ReLU Networks

We study pathwise connectivity of sublevel sets for one-hidden-layer ReLU networks with constrained first-layer weights and an ℓ1\ell_1 penalty on the output layer. The data term is assumed convex and globally Lipschitz in the scalar logit. We first give a finite-width construction that connects any two points of a common sublevel through a path controlled by a loss-consistent compression functional and a first-order perturbation term. The proof replaces the quadratic perturbation estimate in the Freeman--Bruna mechanism by a direct Lipschitz bound. Positive homogeneity is then used in a direction that is compatible with the penalty: every active atom is moved monotonically from the unit ball to the unit sphere while its output coefficient is reduced. Sphere covering and cluster merging consequently give O(m−1/(n−1))O(m^{-1/(n-1)}) fixed-level thickening for n≥2n\ge2, while the one-dimensional two-ray dictionary gives exact connectivity for every m≥4m\ge4. We also prove internally that the regularized approximation values satisfy e(l)−e∞=O(l−1/2)e(l)-e_\infty=O(l^{-1/2}). More generally, a rate O(l−s)O(l^{-s}) transfers to a near-optimal barrier rate O(m−s/((n−1)s+1))O(m^{-s/((n-1)s+1)}); under the standing assumptions, this yields the explicit rate O(m−1/(n+1))O(m^{-1/(n+1)}). A theorem-aligned finite-distribution experiment complements the analysis. The primary Huber run yields a maximal best certified upper gap 1.66×10−51.66\times10^{-5} over 720 recorded pairs at widths m≥16m\ge16; a matched binary-cross-entropy rerun and a 720-endpoint dense-representation stress test probe loss robustness and the active cluster-merging mechanism.

Saveliy Baturin

Independent Researcher, Russia.

Shallow Neural NetworksReLU Neural NetworksMode ConnectivityNeural Network Approximation Theory
AskPDF
Feb 18, 2026

FEKAN: Feature-Enriched Kolmogorov-Arnold Networks

Kolmogorov-Arnold Networks (KANs) have recently emerged as a compelling alternative to multilayer perceptrons, offering enhanced interpretability via functional decomposition. However, existing KAN architectures, including spline-, wavelet-, radial-basis variants, etc., suffer from high computational cost and slow convergence, limiting scalability and practical applicability. Here, we introduce Feature-Enriched Kolmogorov-Arnold Networks (FEKAN), a simple yet effective extension that preserves all the advantages of KAN while improving computational efficiency and predictive accuracy through feature enrichment, without increasing the number of trainable parameters. By incorporating these additional features, FEKAN accelerates convergence, increases representation capacity, and substantially mitigates the computational overhead characteristic of state-of-the-art KAN architectures. We investigate FEKAN across a comprehensive set of benchmarks, including function-approximation tasks, physics-informed formulations for diverse partial differential equations (PDEs), and neural operator settings that map between input and output function spaces. For function approximation, we systematically compare FEKAN against a broad family of KAN variants, FastKAN, WavKAN, ReLUKAN, HRKAN, ChebyshevKAN, RBFKAN, and the original SplineKAN. Across all tasks, FEKAN demonstrates substantially faster convergence and consistently higher approximation accuracy than the underlying baseline architectures. We also establish the theoretical foundations for FEKAN, showing its superior representation capacity compared to KAN, which contributes to improved accuracy and efficiency.

Sidharth S. Menon, Ameya D. Jagtap

Aerospace Engineering Department, Worcester Polytechnic Institute, Worcester, MA 01609, USA.

Function ApproximationKolmogorov-Arnold NetworksNeural Network Approximation Theory
AskPDF
Feb 17, 2026

Approximation Theory for Lipschitz Continuous Transformers

Stability and robustness are critical for deploying Transformers in safety-sensitive settings. A principled way to enforce such behavior is to constrain the model's Lipschitz constant. However, approximation-theoretic guarantees for architectures that explicitly preserve Lipschitz continuity have yet to be established. In this work, we bridge this gap by introducing a class of gradient-descent-type in-context Transformers that are Lipschitz-continuous by construction. We realize both MLP and attention blocks as explicit Euler steps of negative gradient flows, ensuring inherent stability without sacrificing expressivity. We prove a universal approximation theorem for this class within a Lipschitz-constrained function space. Crucially, our analysis adopts a measure-theoretic formalism, interpreting Transformers as operators on probability measures, to yield approximation guarantees independent of token count. These results provide a rigorous theoretical foundation for the design of robust, Lipschitz continuous Transformer architectures.

Takashi Furuya, Davide Murari, Carola-Bibiane Schönlieb

Faculty of Life and Medical Sciences, Department of Biomedical Engineering, Doshisha University, RIKEN AIP · Department of Applied Mathematics and Theoretical Physics, University of Cambridge

TransformerNeural Network Approximation TheoryLipschitz Neural Networks
AskPDF