Neural Network Approximation Theory

Follow this topic Ask about this topic

Momentum

15 papers in the last four weeks, up 15% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 163

May 5, 2026math.CA

Exact ReLU realization of tensor-product refinement iterates

We study scalar dyadic refinement operators on R^2 of the form (Vf)(x,y) = sum_{(j,k) in Z^2} c_{j,k} f(2x-j, 2y-k), where only finitely many mask coefficients c_{j,k} are nonzero. Under a fixed support-window hypothesis, we prove that for every compactly supported continuous piecewise linear seed g:R^2->R, the iterates V^n g admit exact ReLU realizations of fixed width and depth O(n). This gives a first genuinely two-dimensional extension of the exact realization theory for refinement cascades. Using the one-dimensional exact loop-controller framework, the proof transports the tensor-product residual dynamics exactly on the product of two polygonal loops and reduces the remaining seam ambiguity to a final readout and selector step. The matrix cascade is then handled by a fixed-depth recursive block, and general compactly supported continuous piecewise linear seeds are reduced to a finite decomposition together with exact clamped gluing on the support window. This identifies the tensor-product dyadic case as a natural first multivariate instance of the loop-controller method for refinement iterates.

Tsogtgerel Gantumur

McGill University, Montréal, QC, Canada · National University of Mongolia, Ulaanbaatar, Mongolia · Mongolian Academy of Sciences, Institute of Mathematics and Digital Technology

ReLU Neural Networks Neural Network Approximation Theory Iterative Refinement

May 5, 2026cs.LG

Simultaneous CNN Approximation on Manifolds with Applications to Boundary Value Problems

This paper develops convolutional neural network (CNN) methods for simultaneous Sobolev approximation and elliptic boundary value problems on compact Riemannian manifolds. We prove approximation estimates for single- and multichannel CNNs, with rates governed by the intrinsic dimension and the smoothness gap. Motivated by elliptic stability, we propose a physics-informed CNN framework with a spectral boundary loss. The boundary residual is expanded in boundary Laplace--Beltrami eigenmodes and penalized by Sobolev trace weights, matching the natural

\mathcal H^{2s-1/2}(\partial\mathcal M^d)

trace norm for

2s

-order elliptic problems. This avoids smooth auxiliary constructions for exact boundary enforcement and singular Sobolev--Slobodeckij double integrals, while allowing FFT-based or precomputed spectral implementations. We also derive an error decomposition separating approximation, generalization, and spectral truncation errors, showing that the proposed loss is aligned with localized fast-rate generalization analysis. Numerical experiments on the upper hemisphere and upper half-torus demonstrate improved accuracy, convergence, and stability over standard PINNs, with one to two orders of magnitude gains for high-frequency boundary data.

Hanfei Zhou, Lei Shi

School of Mathematical Sciences, Fudan University, Shanghai, 200433, China · Shanghai Key Laboratory for Contemporary Applied Mathematics, Fudan University, Shanghai, 200433, China

PDE Solving Neural Network Approximation Theory Physics-Informed ML Convolutional Neural Networks

May 5, 2026math.OC

Parametrizing Convex Sets Using Sublinear Neural Networks

We propose a neural parameterization of convex sets by learning sublinear (positively homogeneous and convex) functions. Our networks implicitly represent both the support and gauge functions of a convex body. We prove a universal approximation theorem for convex sets under this parametrization. Empirically, we demonstrate the method on shape optimization and inverse design tasks, achieving accurate reconstruction of target shapes.

Eloi Martinet

Institute for Mathematics, University of Würzburg Germany

Neural Network Approximation Theory

May 4, 2026cs.LG

Quantitative Sobolev Approximation Bounds for Neural Operators with Empirical Validation on Burgers Equation

Neural operators have emerged as a powerful tool for learning mappings between infinite-dimensional function spaces. However, their approximation properties in Sobolev norms remain poorly quantified, even though these norms control both function values and derivatives and are the natural metrics for PDE well-posedness, stability, and generalization. We develop a functional-analytic framework for operator learning in Sobolev spaces and connect it to the numerical behavior of Fourier Neural Operators (FNOs) on a prototypical PDE. First, for a continuous nonlinear operator

\mathcal{G}: H^{s}(D)\to H^{t}(D')

with

s > d/2

and inputs restricted to a compact subset of

H^{s}(D)

, we prove that

\mathcal{G}

can be uniformly approximated in

H^{t}

-norm by a neural operator with

\mathcal{O}(\varepsilon^{-d/s})

trainable parameters. This yields an explicit complexity--error relation of the form

\|\mathcal{G}-\mathcal{G}_θ\|_{H^{t}} \lesssim C N^{-s/d}

. We then study the one-dimensional viscous Burgers solution operator

\mathcal{G}: u_{0}\mapsto u(\cdot,1)

on a bounded

H^{1}

-ball and train FNOs with an

H^{1}

-loss. Across a sweep of model sizes, we obtain test

H^{1}

-errors down to

\mathcal{O}(10^{-7})

and relative errors of order

10^{-3}

, with predictions accurately matching both solutions and spatial derivatives on held-out data. A log-log plot of Sobolev error versus parameter count exhibits an approximate power law

\|\mathcal{G}-\mathcal{G}_θ\|_{H^{1}} \approx C N^{-α}

with empirical exponent

α\approx 1.4

, and long-horizon training reveals optimization instabilities in large FNOs, providing quantitative evidence that Sobolev-space approximation theory meaningfully predicts neural-operator scaling behavior.

Nicole Hao

Department of Mathematics, Cornell University

Fourier Neural Operator PDE Solving PDE Operator Learning Neural Network Approximation Theory

May 4, 2026math.PR

Universality in Deep Neural Networks: An approach via the Lindeberg exchange principle

We consider the infinite-width limit of a fully connected deep neural network with general weights, and we prove quantitative general bounds on the

2

-Wasserstein distance between the network and its infinite-width Gaussian limit, under appropriate regularity assumptions on the activation function. Our main tool is a Lindeberg principle for Deep Neural Networks, which we use to successively replace the weights on each layer by Gaussian random variables.

Filippo Giovagnini, Sotirios Kotitsas, Marco Romito

Neural Network Approximation Theory Wasserstein Distance

May 4, 2026stat.ML

ParaRNN: An Interpretable and Parallelizable Recurrent Neural Network for Time-Dependent Data

The proliferation of large-scale and structurally complex data has spurred the integration of machine learning methods into statistical modeling. Recurrent neural networks (RNNs), a foundational class of models for time-dependent data, can be viewed as nonlinear extensions of classical autoregressive moving average models. Despite their flexibility and empirical success in machine learning, RNNs often suffer from limited interpretability and slow training, which hinders their use in statistics. This paper proposes the Parallelized RNN (ParaRNN), a novel model composed of multiple small recurrent units. ParaRNN admits an additive representation that decouples recurrent dynamics into interpretable components, whose behavior can be characterized through recurrence features. This interpretability enables its applications in nonparametric regression for time-dependent data, while the design also allows efficient parallelization. The approximation capacity and non-asymptotic prediction error bounds in a nonparametric regression setting are established for ParaRNN. Empirical results on three sequential modeling tasks further demonstrate that ParaRNN achieves performance comparable to vanilla RNNs while offering improved interpretability and efficiency.

Yuxi Cai, Lan Li, Feiqing Huang +1

Department of Statistics and Actuarial Science, University of Hong Kong

Interpretable ML Recurrent Neural Networks Nonparametric Regression Neural Network Approximation Theory

May 3, 2026cs.LG

Floating-Point Networks with Automatic Differentiation Can Represent Almost All Floating-Point Functions and Their Gradients

Theoretical studies show that for any differentiable function on a compact domain, there exists a neural network that approximates both the function values and gradients. However, such a result cannot be used in practice since it assumes real parameters and exact internal operations. In contrast, real implementations only use a finite subset of reals and machine operations with round-off errors. In this work, we investigate whether a similar result holds for neural networks under floating-point arithmetic, when the gradient with respect to the input is computed by the automatic differentiation algorithm

D^\mathtt{AD}

. We first show that given a floating-point function

φ

(e.g., a loss function), arbitrary function values and gradients can be represented by a floating-point network

f

and

D^\mathtt{AD}(φ\circ f)

, respectively. We further extend this result: given

φ_1,\dots,φ_n

,

D^\mathtt{AD}(φ_i\circ f)

can simultaneously represent arbitrary gradients while

f

represents the target values, under mild conditions. Our results hold for practical activation functions, e.g.,

\mathrm{ReLU}

,

\mathrm{ELU}

,

\mathrm{GeLU}

,

\mathrm{Swish}

,

\mathrm{Sigmoid}

, and

\mathrm{tanh}

.

Sejun Park, Yeachan Park, Geonho Hwang

Department of Artificial Intelligence, Korea University, Seoul, 02841, Republic of Korea · Department of Mathematics and Statistics, Sejong University, Seoul, 05006, Republic of Korea · Department of Mathematical Sciences, Gwangju Institute of Science and Technology, Gwangju, 61005, Republic of Korea

Automatic Differentiation Neural Network Approximation Theory

May 3, 2026math.CA

Exact Loop Controllers for ReLU Realization of Homogeneous Curve Refinements

We study homogeneous refinement operators

(Vγ)(t)=\sum_{j\in\mathbb Z}A_jγ(Mt-j)

, acting on compactly supported continuous piecewise linear curves

γ:\mathbb R\to\mathbb R^p

, where

M\ge2

and only finitely many matrices

A_j\in\mathbb R^{p\times p}

are nonzero. We prove that the iterates

V^nγ

admit exact ReLU realizations of fixed width and depth

O(n)

. The main new ingredient is an exact loop controller for the residual dynamics. Instead of propagating scalar residual surrogates, the construction transports the residual orbit by a forward-exact state on a polygonal loop. Scalar factors and digit selectors are then recovered from this loop state by complementary CPwL readouts. The loop seam is not removed, but its remaining ambiguity is confined to the final readout/selector stage, where it is harmless because the scalar atom is supported away from the seam. This gives a homogeneous

M

-ary vector-valued extension of the scalar binary refinable-function construction with a more geometric controller architecture. We also record crude exponential bounds on the network weights and biases. Affine forcing terms are handled by expanding affine iterates into finite sums of homogeneous iterates, giving exact fixed-width realizations with depth

O(n^2)

, and anchored open curves reduce to compactly supported defects with affine anchor mismatch. We also describe homogeneous polygonal generators, including dragon-type examples and a self-intersecting Hilbert-type prototype in arbitrary dimension. The extended version includes stage-dependent forcing, finite-state stacking reductions, and further geometric constructions such as Koch-, Gosper-, Morton-, and connector-based Hilbert-type variants.

Boldsaikhan Bolorkhuu, Tsogtgerel Gantumur

1McGill University · 2National University of Mongolia · Institute of Mathematics and Digital Technology, Mongolian Academy of Sciences

ReLU Neural Networks Neural Network Approximation Theory Iterative Refinement

Apr 29, 2026cs.LG

Hyper Input Convex Neural Networks for Shape Constrained Learning and Optimal Transport

We introduce Hyper Input Convex Neural Networks (HyCNNs), a novel neural network architecture designed for learning convex functions. HyCNNs combine the principles of Maxout networks with input convex neural networks (ICNNs) to create a neural network that is always convex in the input, theoretically capable of leveraging depth, and performs reliable when trained at scale compared to ICNNs. Concretely, we prove that HyCNNs require exponentially fewer parameters than ICNNs to approximate quadratic functions up to a given precision. Throughout a series of synthetic experiments, we demonstrate that HyCNNs outperform existing ICNNs and MLPs in terms of predictive performance for convex regression and interpolation tasks. We further apply HyCNNs to learn high-dimensional optimal transport maps for synthetic examples and for single-cell RNA sequencing data, where they oftentimes outperform ICNN-based neural optimal transport methods and other baselines across a wide range of settings.

Shayan Hundrieser, Insung Kong, Johannes Schmidt-Hieber

*Department of Applied Mathematics, University of Twente, Enschede, The Netherlands

Input Convex Neural Networks Neural Network Approximation Theory Optimal Transport

Apr 29, 2026cs.LG

Layer-wise Lipschitz-Product Control for Deep Kolmogorov--Arnold Network Representations of Compositionally Structured Functions

We prove that any continuous function f from [0,1]^n to R representable by a finite computation tree with N internal nodes and compositional sparsity s = O(1) admits a deep Kolmogorov-Arnold Network (KAN) representation. Each internal node is realised by a primitive KAN block with controlled block depth and Lipschitz product. The layer-wise Lipschitz product satisfies the primary domain-sensitive bound independent of the input dimension n. It simplifies to P(KAN_f) <= max(C*,1)^L_f with L_f <= c_max * N. For the standard operations {+,-,x,sin,cos} with x nodes on [0,1]-bounded inputs we obtain P(KAN) <= 1. Layer widths satisfy n_l <= n + 2 w_max * N. The uniform approximation error is bounded by N * max(C*,1)^d(f) * epsilon_Op (simplifies when C* <=1). For f in C^m we obtain optimal B-spline rates. Range bounds are also derived (B_f <= N+1 for additive trees). This addresses the gap on Lipschitz control in deep KAN stacks noted by Liu et al. (2024). Experiments confirm P(KAN)=1.0 for several compositionally structured functions.

Aleksander Tankman

Fivestar Europe OÜ, Tallinn, Estonia

Kolmogorov-Arnold Networks Neural Network Approximation Theory Neural Network Expressivity Lipschitz Neural Networks

Apr 27, 2026cs.LG

Transformer Approximations from ReLUs

We provide a systematic recipe for translating ReLU approximation results to softmax attention mechanism. This recipe covers many common approximation targets. Importantly, it yields target-specific, economic resource bounds beyond universal approximation statements. We showcase the recipe on multiplication, reciprocal computation, and min/max primitives. These results provide new analytical tools for analyzing softmax transformer models.

Jerry Yao-Chieh Hu, Mingcheng Lu, Yi-Chen Lee +1

Center for Foundation Models and Generative AI, Northwestern University, Evanston, IL 60208, USA · Department of Computer Science, Northwestern University, Evanston, IL 60208, USA · Department of Physics, National Taiwan University, Taipei 10617, Taiwan +1

Softmax Attention Neural Network Approximation Theory Attention Mechanisms

Apr 27, 2026cs.LG

Progressive Approximation in Deep Residual Networks: Theory and Validation

The Universal Approximation Theorem (UAT) guarantees universal function approximation but does not explain how residual models distribute approximation across layers. We reframe residual networks as a layer-wise approximation process that builds an approximation trajectory from input to target, and prove the existence of progressive trajectories where error decreases monotonically with depth. It reveals that residual networks can implement structured, step-by-step refinement rather than end-to-end (E2E) black-box mapping. Building on this, we propose Layer-wise Progressive Approximation (LPA), a theoretically grounded training principle that explicitly aligns each layer with its residual target to realize such trajectories. LPA is architecture-agnostic: we observe progressive behavior in residual FNNs, ResNets, and Transformers across tasks including complex surface fitting, image classification, and NLP with LLMs for generation and classification. Crucially, this enables ``train once, use

N

models": a single network yields useful predictions at every depth, supporting efficient shallow inference without retraining. Our work unifies approximation theory with practical deep learning, providing a new lens on representation learning and a flexible framework for multi-depth deployment. The source code will be released unpon acceptance at https://(open_upon_acceptance).

Wei Wang, Xiao-Yong Wei, Qing Li

Department of Computing, the Hong Kong Polytechnic University.

Efficient Neural Network Inference Residual Learning Universal Approximation Neural Network Approximation Theory Early-Exit Neural Networks

Apr 26, 2026cs.LG

Necessary and sufficient conditions for universality of Kolmogorov-Arnold networks

We analyze the universal approximation property of Kolmogorov-Arnold Networks (KANs) in terms of their edge functions. If these functions are all affine, then universality clearly fails. How many non-affine functions are needed, in addition to affine ones, to ensure universality? We show that a single one suffices. More precisely, we prove that deep KANs in which all edge functions are either affine or equal to a fixed continuous function

σ

are dense in

C(K)

for every compact set

K\subset\mathbb{R}^n

if and only if

σ

is non-affine. In contrast, for KANs with exactly two hidden layers, universality holds if and only if

σ

is nonpolynomial. We further show that the full class of affine functions is not required; it can be replaced by a finite set without affecting universality. In particular, in the nonpolynomial case, a fixed family of five affine functions suffices when the depth is arbitrary. More generally, for every continuous non-affine function

σ

, there exists a finite affine family

A_σ

such that deep KANs with edge functions in

A_σ\cup\{σ\}

remain universal. We also prove that KANs with the spline-based edge parameterization introduced by Liu et al.~\cite{Liu2024} are universal approximators in the classical sense, even when the spline degree and knot sequence are fixed in advance.

Vugar Ismailov

Kolmogorov-Arnold Networks Universal Approximation Neural Network Approximation Theory Neural Network Expressivity

Apr 25, 2026stat.ML

Explicit integral representations and quantitative bounds for two-layer ReLU networks

An approach to construct explicit integral representations for two-layer ReLU networks is presented, which provides relatively simple representations for any multivariate polynomial. Quantitative bounds are provided for a particular, sharpened ReLU integral representation, which involves a harmonic extension and a projection. The bounds demonstrate that functions can be approximated with

L^{2}(\mathcal{D})

errors that do not depend explicitly on dimension or degree, but rather the coefficients of their monomial expansions and the distribution

\mathcal{D}

. We also present a connection to the RKHS of the exponential kernel

K(x,y)=\exp\left(\left\langle x,y\right\rangle \right)

, and a very simple integral representation involving additionally multiplication via a fixed function which has better quantitative bounds.

Anthony Lee

School of Mathematics, University of Bristol

ReLU Neural Networks Reproducing Kernel Hilbert Spaces Neural Network Approximation Theory

Apr 24, 2026cs.LG

SOC-ICNN: From Polyhedral to Conic Geometry for Learning Convex Surrogate Functions

Classical ReLU-based Input Convex Neural Networks (ICNNs) are equivalent to the optimal value functions of Linear Programming (LP). This intrinsic structural equivalence restricts their representational capacity to piecewise-linear polyhedral functions. To overcome this representational bottleneck, we propose the SOC-ICNN, an architecture that generalizes the underlying optimization class from LP to Second-Order Cone Programming (SOCP). By explicitly injecting positive semi-definite curvature and Euclidean norm-based conic primitives, our formulation introduces native smooth curvature into the representation while preserving a rigorous optimization-theoretic interpretation. We formally prove that SOC-ICNNs strictly expand the representational space of ReLU-ICNNs without increasing the asymptotic order of forward-pass complexity. Extensive experiments demonstrate that SOC-ICNN substantially improves function approximation, while delivering competitive downstream decision quality. The code is available at https://anonymous.4open.science/r/SOC-ICNN-4B18/.

Kang Liu, Jianchen Hu, Wei Peng

School of Future Technology Xi’an Jiaotong University Xi’an, China · School of Future Technology School of Automation Science and Engineering Xi’an Jiaotong University Xi’an, China · School of Automation Science and Engineering Xi’an Jiaotong University Xi’an, China

Neural Surrogate Modeling Input Convex Neural Networks Neural Network Approximation Theory

Apr 23, 2026cs.CE

Scale-Parameter Selection in Gaussian Kolmogorov-Arnold Networks

Kolmogorov--Arnold Networks (KANs) have recently attracted attention as edge-based neural architectures in which learnable univariate functions replace conventional fixed activation functions. A key source of flexibility in KANs is the choice of basis functions used to parameterize the learnable edge functions. In this context, Gaussian basis functions provide a simple and efficient alternative to splines. However, their performance depends strongly on the scale (shape) parameter

ε

, whose role has not been studied systematically. In this paper, we investigate how

ε

affects Gaussian KANs through first-layer feature geometry, conditioning, and approximation behavior. Our central observation is that scale selection is governed primarily by the first layer, since it is the only layer constructed directly on the input domain and any loss of distinguishability introduced there cannot be recovered by later layers. From this viewpoint, we analyze the first-layer feature matrix and identify a practical operating interval,

ε\in \left[\frac{1}{G-1},\frac{2}{G-1}\right],

where

G

denotes the number of Gaussian centers. We interpret this interval not as a universal optimality result, but as a stable and effective design rule, and validate it through brute-force sweeps over

ε

across function-approximation problems with different collocation densities, grid resolutions, network architectures, and input dimensions, as well as physics-informed problems. We further show that this range is useful for fixed-scale selection, variable-scale constructions, constrained training of

ε

, and efficient scale search using early training MSE. In this way, the paper positions scale selection as a practical design principle for Gaussian KANs rather than as an ad hoc hyperparameter choice.

Amir Noorizadegan, Sifan Wang

Department of Mathematics, Hong Kong Baptist University, Hong Kong SAR, China · Institution for Foundations of Data Science, Yale University, New Haven, CT 06520, USA

Kolmogorov-Arnold Networks Neural Network Approximation Theory Hyperparameter Optimization

Apr 22, 2026cs.LG

Layer-wise Geometric Approximation Rates for Deep Networks

Depth is widely viewed as a central contributor to the success of deep neural networks, whereas standard neural network approximation theory typically provides guarantees only for the final output and leaves the role of intermediate layers largely unclear. We address this gap by developing a quantitative framework in which depth admits a precise scale-dependent interpretation. Specifically, we design a single shared mixed-activation architecture of fixed width

2dN+d+2

and any prescribed finite depth such that each intermediate readout

Φ_\ell

is itself an approximant to the target function

f

. For

f\in L^p([0,1]^d)

with

p\in [1,\infty)

, the approximation error of

Φ_\ell

is controlled by

(2d+1)

times the

L^p

modulus of continuity at the geometric scale

N^{-\ell}

for all

\ell

. The estimate reduces to the geometric rate

(2d+1)N^{-\ell}

if

f

is

1

-Lipschitz. Our network design is inspired by multigrade deep learning, where depth serves as a progressive refinement mechanism. For every prescribed terminal depth, the construction yields a finite nested family of prefix readouts whose earlier correction terms remain embedded in later readouts. Thus the approximation may be truncated within the prescribed depth range once the desired certified accuracy is reached.

Shijun Zhang, Zuowei Shen, Yuesheng Xu

Department of Applied Mathematics Hong Kong Polytechnic University · Department of Mathematics National University of Singapore · Department of Mathematics and Statistics Old Dominion University

Neural Network Approximation Theory Neural Network Expressivity

Apr 20, 2026cs.LG

Neural Shape Operator Surrogates -- Expression Rate Bounds

We prove error bounds for operator surrogates of solution operators for partial differential and boundary integral equations on families of domains which are diffeomorphic to one common reference (or latent) domain

D_{ref}

. The pullback of the PDE to

D_{ref}

via affine-parametric shape encoding produces a collection of holomorphic parametric PDEs on

D_{ref}

. Sufficient conditions for (uniformly with respect to the parameter) well-posedness are given, implying existence, uniqueness and stability of parametric solution families on

D_{ref}

. We illustrate the abstract hypotheses by reviewing recent holomorphy results for a suite of elliptic and parabolic PDEs. Quantified parametric holomorphy implies existence of finite-parametric, discrete approximations of the parametric solution families with convergence rates in terms of the number

N

of parameters. We obtain constructive proofs of existence of Neural and Spectral Operator surrogates for the shape-to-solution maps with error bounds and convergence rate guarantees uniform on the collection of admissible shapes. We admit principal-component shape encoders and frame decoders. Our results support in particular the (empirically reported) ability of neural operators to realize data-to-solution maps for elliptic and parabolic PDEs and BIEs that generalize across parametric families of shapes.

Helmut Harbrecht, Christoph Schwab

PDE Operator Learning Neural Network Approximation Theory Operator Learning Neural Operators

Apr 17, 2026cs.LG

Closing the Theory-Practice Gap in Spiking Transformers via Effective Dimension

Spiking transformers achieve competitive accuracy with conventional transformers while offering

38

-

57\times

energy efficiency on neuromorphic hardware, yet no theoretical framework guides their design. This paper establishes the first comprehensive expressivity theory for spiking self-attention. We prove that spiking attention with Leaky Integrate-and-Fire neurons is a universal approximator of continuous permutation-equivariant functions, providing explicit spike circuit constructions including a novel lateral inhibition network for softmax normalization with proven

O(1/\sqrt{T})

convergence. We derive tight spike-count lower bounds via rate-distortion theory:

\varepsilon

-approximation requires

Ω(L_f^2 nd/\varepsilon^2)

spikes, with rigorous information-theoretic derivation. Our key insight is input-dependent bounds using measured effective dimensions (

d_{\text{eff}}=47

--

89

for CIFAR/ImageNet), explaining why

T=4

timesteps suffice despite worst-case

T \geq 10{,}000

predictions. We provide concrete design rules with calibrated constants (

C=2.3

, 95% CI:

[1.9, 2.7]

). Experiments on Spikformer, QKFormer, and SpikingResformer across vision and language benchmarks validate predictions with

R^2=0.97

(

p<0.001

). Our framework provides the first principled foundation for neuromorphic transformer design.

Dongxin Guo, Jikun Wu, Siu Ming Yiu

*The University of Hong Kong, Hong Kong, China · †Brain Investing Limited, Hong Kong, China

Spiking Transformers Transformer Attention Spiking Neural Networks Neuromorphic Computing Neural Network Approximation Theory

Apr 16, 2026cs.CV

Implicit Neural Representations: A Signal Processing Perspective

Implicit neural representations (INRs) mark a fundamental shift in signal modeling, moving from discrete sampled data to continuous functional representations. By parameterizing signals as neural networks, INRs provide a unified framework for representing images, audio, video, 3D geometry, and beyond as continuous functions of their coordinates. This functional viewpoint enables signal operations such as differentiation to be carried out analytically through automatic differentiation rather than through discrete approximations. In this article, we examine the evolution of INRs from a signal processing perspective, emphasizing spectral behavior, sampling theory, and multiscale representation. We trace the progression from standard coordinate based networks, which exhibit a spectral bias toward low frequency components, to more advanced designs that reshape the approximation space through specialized activations, including periodic, localized, and adaptive functions. We also discuss structured representations, such as hierarchical decompositions and hash grid encodings, that improve spatial adaptivity and computational efficiency. We further highlight the utility of INRs across a broad range of applications, including inverse problems in medical and radar imaging, compression, and 3D scene representation. By interpreting INRs as learned signal models whose approximation spaces adapt to the underlying data, this article clarifies the field's core conceptual developments and outlines open challenges in theoretical stability, weight space interpretability, and large scale generalization.

Dhananjaya Jayasundara, Vishal M. Patel

Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD 21218 USA

Implicit Neural Representations Spectral Bias Neural Network Approximation Theory

Mar 13, 2026cs.NE

Equivalence of approximation by networks of single- and multi-spike neurons

In a spiking neural network, is it enough for each neuron to spike at most once? In recent work, approximation bounds for spiking neural networks have been derived, quantifying how well they can fit target functions. However, these results are only valid for neurons that spike at most once, which is commonly thought to be a strong limitation. Here, we show that the opposite is true for a large class of spiking neuron models, including the commonly used leaky integrate-and-fire model with subtractive reset: for every approximation bound that is valid for a set of multi-spike neural networks, there is an equivalent set of single-spike neural networks with only linearly more (or less) neurons, in the maximum number of spikes, for which the bound holds. The same is true for the reverse direction too, showing that regarding their approximation capabilities in general machine learning tasks, single-spike and multi-spike neural networks are equivalent. Consequently, many approximation results in the literature for single-spike neural networks also hold for the multi-spike case.

Dominik Dold, Philipp Christian Petersen

Spiking Neural Networks Neural Network Approximation Theory Neural Network Expressivity

Feb 12, 2026cs.LG

Rational Neural Networks have Expressivity Advantages

We study neural networks with trainable low-degree rational activation functions and show that they are more expressive and parameter-efficient than modern piecewise-linear and smooth activations such as ELU, LeakyReLU, LogSigmoid, PReLU, ReLU, SELU, CELU, Sigmoid, SiLU, Mish, Softplus, Tanh, Softmin, Softmax, and LogSoftmax. For an error target of

\varepsilon>0

, we establish approximation-theoretic separations: Any network built from standard fixed activations can be uniformly approximated on compact domains by a rational-activation network with only

\mathrm{poly}(\log\log(1/\varepsilon))

overhead in size, while the converse provably requires

Ω(\log(1/\varepsilon))

parameters in the worst case. This exponential gap persists at the level of full networks and extends to gated activations and transformer-style nonlinearities. In practice, rational activations integrate seamlessly into standard architectures and training pipelines, allowing rationals to match or outperform fixed activations under identical architectures and optimizers.

Maosen Tang, Alex Townsend

Center for Applied Mathematics, Cornell University, United States · Department of Mathematics, Cornell University, United States

Neural Network Activation Functions Neural Network Approximation Theory Neural Network Expressivity

Feb 5, 2026cs.LG

Clifford Kolmogorov-Arnold Networks

We introduce Clifford Kolmogorov-Arnold Network (ClKAN), a flexible and efficient architecture for function approximation in arbitrary Clifford Algebra spaces. We propose the use of Randomized Quasi-Monte Carlo grid generation as a solution to the exponential scaling associated with higher-dimensional algebras. Our ClKAN also introduces new batch normalization strategies to deal with variable domain input. ClKAN finds application in scientific discovery and engineering, and is validated in synthetic and physics-inspired tasks.

Matthias Wolff, Francesco Alesiani, Christof Duhme +1

Data Science Div. NEC Labs Europe Heidelberg, Germany

Kolmogorov-Arnold Networks Neural Network Approximation Theory Scientific ML

Feb 3, 2026cs.LG

Geometry-Preserving Neural Architectures on Manifolds with Boundary

A growing number of neural architectures have been proposed to enforce geometric constraints, including projection-based networks, exponential-map updates, constrained output layers, and manifold neural ODEs. We provide a unified framework for these geometry-preserving architectures by organizing them according to where and how constraints are enforced, either throughout the intermediate layers or only at the final output. This perspective reveals several gaps in the existing theory. To address these gaps, we prove high-level approximation theorems for projected neural ODEs, intermediate augmented architectures, and final augmented architectures on prox-regular constraint sets, including smooth manifolds with boundary. Numerical experiments on synthetic dynamics over S^2, the disk, SO(3), together with real-world protein backbone data on SE(3), demonstrate exact feasibility for analytic updates and show that the final augmentation have simpler architecture and outperform in most tasks considered. When the constraint set is unknown, we learn projections via small-time heat-kernel limits, showing diffusion/flow-matching can be used as data-based projections. Moreover, we also the demonstrate the usefulness of the architectures that enforce non-convex constraints for path planning on manifolds with boundary.

Karthik Elamvazhuthi, Shiba Biswal, Kian Rosenblum +4

Los Alamos National Laboratory · Boston College

Geometric Representation Learning Neural Network Approximation Theory Neural ODEs

Jan 30, 2026cs.LG

Agile Reinforcement Learning through Separable Neural Architecture and Applications

Deep reinforcement learning (RL) is increasingly deployed in resource-constrained environments, yet go-to function approximators - multilayer perceptrons (MLPs) - are often parameter-inefficient due to an imperfect inductive bias for the smooth structure of many value functions. This mismatch can also hinder sample efficiency and slow policy learning in this capacity-limited regime. Although model compression techniques exist, they operate post-hoc and do not improve learning efficiency. Spline-based architectures such as Kolmogorov-Arnold Networks (KANs) have been shown to offer parameter efficiency but are widely reported to exhibit significant computational overhead, especially at scale. In seeking to address these limitations, this work introduces SPAN (SPline-based Adaptive Networks) for RL. SPAN adapts the KHRONOS framework with a learnable preprocessing layer. SPAN is evaluated across discrete (PPO) and high-dimensional continuous (SAC) control tasks, offline settings (Minari/D4RL) and a real-world datacenter HVAC control application. SPAN achieves a 30-50% improvement in sample efficiency and 1.3-9 times higher success rates across benchmarks compared to MLP baselines. Despite incurring a per-step evaluation overhead of 1.2-1.8x, SPAN's superior convergence reliability yields an expected total training cost 1.3-6.3x lower than MLP baselines when accounting for convergence failures. In the HVAC application, SPAN reduces energy consumption in 9 of 12 months relative to MLP while simultaneously achieving a 1.1-3.4x reduction in thermal comfort violations across the evaluation year, demonstrating generalization to real-world engineering control. Furthermore, SPAN demonstrates superior anytime performance and robustness to hyperparameter variations, suggesting it as a viable, high-performance alternative for learning efficient policies in resource-limited settings.

Rajib Mostakim, Reza T. Batley, Sourav Saha

Department of Mechanical Engineering, Bangladesh University of Engineering and Technology, Dhaka, Bangladesh · Kevin T. Crofton Department of Aerospace and Ocean Engineering, Virginia Polytechnic Institute and State University, Blacksburg, VA, USA

RL Control Neural Network Approximation Theory Sample-Efficient RL Heating, Ventilation, and Air Conditioning Control

Jan 20, 2026cs.LG

Linearized subspace refinement framework to expose hidden accuracy in trained neural networks

Neural networks trained by gradient-based methods often exhibit optimization-induced accuracy plateaus in scientific machine learning tasks. We present Linearized Subspace Refinement (LSR), an architecture-agnostic post-training framework that exploits the local linearized model at a fixed trained state. By solving a reduced direct least-squares problem in a Jacobian-defined low-dimensional space, LSR computes a subspace-optimal linearized correction and yields a refined predictor with markedly improved accuracy. Across function approximation, data-driven operator learning, physics-informed operator fine-tuning, and noisy inverse problems, LSR shows that standard nonlinear training can remain far above this subspace-attainable error level. Similar accuracy plateaus persist even for the convex quadratic problem from local linearization when solved with standard iterative optimizers, identifying numerical ill-conditioning as a primary bottleneck. LSR frequently delivers order-of-magnitude error reductions, while the subspace rank provides an explicit capacity-control mechanism that balances correction strength, numerical stability, and noise sensitivity. Together, LSR exposes conditioning-limited attainable accuracy in trained-state linearized models and provides direct access to it.

Wenbo Cao, Weiwei Zhang

Institute of AI for Industries, Chinese Academy of Sciences, Nanjing 211135, China · Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China · School of Aeronautics, Northwestern Polytechnical University, Xi’an 710072, China +2

Neural Network Optimization Neural Network Approximation Theory Neural Network Training Dynamics

Nov 24, 2025math.AG

The Alexander-Hirschowitz theorem for neurovarieties

We study the dimension and identifiability of neurovarieties associated to polynomial neural networks. We give an independent geometric proof that the linear bounds

d_i\geq 2n_i-1

on the activation degrees imply non defectiveness for any number of outputs, a dimension statement previously obtained from finite identifiability. The proof is based on a direct analysis of the differential of the parameterization. We also investigate secant and Grassmann-secant obstructions outside this range and prove global identifiability for multi-output architectures under the same degree bounds.

A. Massarenti, M. Mella

Polynomial Neural Networks Parameter Identifiability Neural Network Approximation Theory Neural Network Expressivity

Nov 16, 2025cs.LG

On the Dimension-Free Approximation of Deep Neural Networks for Symmetric Korobov Functions

Deep neural networks have been widely used as universal approximators for functions with inherent physical structures, including permutation symmetry. In this paper, we construct symmetric deep neural networks to approximate symmetric Korobov functions and prove that both the convergence rate and the constant prefactor scale at most polynomially with respect to the ambient dimension. This represents a substantial improvement over prior approximation guarantees that suffer from the curse of dimensionality. Building on these approximation bounds, we further derive a generalization-error rate for learning symmetric Korobov functions whose leading factors likewise avoid the curse of dimensionality.

Yulong Lu, Tong Mao, Jinchao Xu +1

School of Mathematics, University of Minnesota, Minneapolis, 55455, MN, USA · Computer, Electrical and Mathematical Science and Engineering Division, King Abdullah University of Science and Technology, Thuwal 23955, Saudi Arabia · School of Mathematics, Georgia Institute of Technology, Atlanta, 30332, GA, USA

Neural Network Generalization Neural Network Approximation Theory

Oct 20, 2025stat.ML

DFNN: A Deep Fréchet Neural Network Framework for Learning Metric-Space-Valued Responses

Regression with non-Euclidean responses---e.g., probability distributions, networks, symmetric positive-definite matrices, and compositions---has become increasingly important in modern applications. In this paper, we propose deep Fréchet neural networks (DFNNs), an end-to-end deep learning framework for predicting non-Euclidean responses---which are considered as random objects in a metric space---from Euclidean predictors. Our method utilizes the representation-learning power of deep neural networks (DNNs) to the task of approximating conditional Fréchet means of the response given the predictors, the metric-space analogue of conditional expectations, by minimizing a Fréchet risk. The framework is highly flexible, accommodating diverse metrics and high-dimensional predictors. We establish a universal approximation theorem for DFNNs, advancing the state-of-the-art of neural network approximation theory to general metric-space-valued responses, without making model assumptions or relying on local smoothing. We further establish rigorous generalization guarantees for DFNNs and derive corresponding risk bounds, providing, to the best of our knowledge, the first such theoretical results for deep learning regression with metric-space-valued responses. Empirical studies on synthetic distributional and network-valued responses, as well as real-world applications to predicting compositional responses in an Aitchison simplex and spherical responses, demonstrate that DFNNs consistently outperform all existing methods.

Kyum Kim, Yaqing Chen, Paromita Dubey

Rutgers University · University of Southern California

Neural Network Generalization Neural Network Approximation Theory

Oct 5, 2025math.NA

Configuration-Dependent Lower Bounds for Approximation by Shallow ReLU $^k$ Networks on the Sphere

We establish two related but logically distinct results for shallow ReLU

^k

neural networks on the unit sphere

\SS^d

. First, for an arbitrary set of inner neural-network parameters, the best

\mathcal{L}^2(\SS^d)

approximation of a fixed target function with smoothness

r>\tfrac{d+2k+1}{2}

admits an asymptotic lower bound given by a constant multiple of

n^{-1/2}\underline{h}^{k+1/2}

, where

\underline{h}

denotes the antipodal separation distance of the normalized inner-parameter set. This lower bound depends explicitly on the parameter configuration through

\underline{h}

and applies without additional assumptions on the parameters. Second, for antipodally quasi-uniform parameters,

\underline{h}\simeq n^{-1/d}

, and the lower bound establishes the exact saturation order

n^{-\frac{d+2k+1}{2d}}

for such parameter families: a target function with regularity greater than

\frac{d+2k+1}{2}

and satisfying the required parity condition can be approximated at this rate, whereas approximation at any strictly faster rate forces the target function to be zero. Our results therefore place linearized neural-network approximation within the classical saturation framework and show that, although ReLU

^k

network spaces can outperform finite elements of the same degree, this advantage is intrinsically limited.

Tong Mao, Jinchao Xu

Shenzhen Loop Area Institute, Shenzhen 518000, P. R. China · King Abdullah University of Science and Technology, Thuwal 23955, Saudi Arabia.

Shallow Neural Networks Neural Network Approximation Theory Neural Network Expressivity