Neural Network Approximation Theory

Momentum

15 papers in the last four weeks, up 15% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 163

Oct 6, 2026math.NA

FOSLS-deRhaNN: native de Rham neural classes for H(div) and H(curl) with applications to first-order system least-squares neural network methods for partial differential equations

We construct neural approximation classes native to the graph spaces H(div) and H(curl), in two and three dimensions and, for H(div), in any dimension. Every realization lies in the space for all parameter values, and with kinked potentials, such as ReLU networks, the admissible jumps appear at finite width. The classes are images of scalar and componentwise networks under fixed operators of the de Rham complex, and do not involve a mesh or finite element emulation. For H(div) in R^n two native classes are given on an equal footing, with a skew-symmetric potential

A

\mathrm{Div}\,A+R_nq+\mathbf{h}

, with the divergence

q

as an explicit unknown, and

\mathrm{Div}\,A+\mathbf{z}

with an

H^1

field

\mathbf{z}

; for H(curl) the analogous classes are

\mathrm{grad}\,φ+Sr+\mathbf{h}

in two dimensions and

\mathrm{grad}\,φ+\mathbf{z}

in two and three dimensions. In all of them every interface jump of the field is carried by the potential term,

\mathrm{Div}\,A

\mathrm{grad}\,φ

, while the remaining part has no interface jump (it is an

H^1

field in the regular-decomposition classes); the classes with

\mathbf{z}

are the componentwise approach enriched by this term. Known or learned interface geometry enters the potential through factors with trainable amplitudes, and the remaining part if the divergence jumps. The classes lead to the FOSLS-deRhaNN method, first-order system least squares with de Rham neural networks, whose loss is the least-squares functional posed in the natural spaces of the weak formulation; for elliptic equations this includes

H^{-1}

right-hand sides and

H^{1/2}

Dirichlet data. Elliptic equations with discontinuous coefficients and curl-curl problems are treated as instances, with the functional equivalent to the error; linear transport with discontinuous solutions and conservation laws with shocks use the same flux classes.

Shun Zhang

Department of Mathematics, City University of Hong Kong, Kowloon Tong, Hong Kong, China

Neural PDE Solvers Neural Network Approximation Theory

Ask PDF

Oct 5, 2026math.NA

Singular parameters and missing limits in neural PDE solvers

Neural solvers for partial differential equations (PDEs) can approach an accurate solution while their parameters grow without bound. In such cases, the limiting solution may have no finite representation in the chosen model, leaving the best loss unattained. Our analysis connects missing limits in deep neural tanh- networks to unbounded hidden parameters or increasingly redundant neurons. For a class of models built from translated kernels, we describe the missing functions and recover them by adding kernel derivatives to the model. This completion makes the best approximation attainable under standard assumptions. Numerical studies follow the associated parameter growth and explore how completion affects PDE optimization.

Daniel Fernández

Chair for Dynamics, Control, Machine Learning, and Numerics (Alexander von Humboldt Professorship), Department of Mathematics, Friedrich-Alexander-Universität Erlangen-Nürnberg, 91058 Erlangen, Germany.

Neural PDE Solvers Neural Network Optimization Neural Network Approximation Theory

Ask PDF

Oct 4, 2026cs.LG

Beyond Overparameterization: Provable Learning of Input-Convex Multi-Layer Polynomial Networks with Active Queries

The theoretical understanding of multi-layer neural networks is largely confined to overparameterized settings, which obscure parameter identifiability and incur high sample complexity. Neural tangent kernel (NTK) provides a general theory for wide networks, but does not offer efficient sample-complexity guarantees. Recent feature-learning results go beyond kernel methods for single-neuron, multi-index, and hierarchical targets. However, the analysis is often restricted to shallow or specific architectures and to the overparameterized regime. We break this paradigm to achieve parameter-level recovery of deep target networks, albeit by using active data queries. Specifically, we study

L

-layer polynomial networks with even degree-

k

monomial activations and nonnegative higher-layer weights. This structure makes the target network input-convex, while the optimization landscape remains highly nonconvex with respect to the parameters. Leveraging input convexity and active queries, we propose \textbf{ASPIRE} (\textbf{A}ctive \textbf{S}am\textbf{P}ling for \textbf{I}terative \textbf{R}ecovery via \textbf{E}igendirections), a layerwise sampling-based diagonalization algorithm that recovers all network parameters to

δ

-accuracy with sample complexity

\widetilde O_{k,L}\left(d^{L^2+O(L)}δ^{-2e}\right)

in polynomial time. To our knowledge, this is the \emph{first} parameter-recovery guarantee for deep target networks whose exponent grows only polynomially with depth, as well as the \emph{first} justification for the effectiveness of using high-quality data in neural network training, with a remarkably \emph{exponential} separation.

Jinqi Tang, Qian Chen, Shihong Ding +1

Peking University

Polynomial Neural Networks Input Convex Neural Networks Neural Network Approximation Theory

Ask PDF

Oct 1, 2026cs.LG

Universal interpolation for deep residual self-attention networks

Universal approximation is a necessary qualitative property of learning architectures to benefit from scaling laws. While it is generically verified on a variety of neural architectures and random feature models, it typically involves infinite width limits. In this work, we focus on deep self-attention models and consider instead the `dual' regime, where approximation power is enabled entirely by depth, and featuring strong parameter sharing across layers, motivated by recent models such as the Looped Transformers. More specifically, we ask whether one can find a predefined finite set of parameters, each defining an attention block, such that the resulting finite set of transformations can map any collection of

N

sequences of

n

tokens to any other collection of

N

sequences of

n

tokens. Crucially, these transformations are \emph{fixed independently of the input and output} collections: only the order in which the blocks are applied, their signs, and their durations depend on the particular interpolation task. Our main result establishes it for residual softmax attention using only two frozen single-head blocks with Gaussian-initialized projection matrices. The result holds at both continuous and finite depth. We also characterize the restrictions imposed by causal masking and establish corresponding universal interpolation guarantees.

Sibylle Marcotte, Joan Bruna

Department of Computer Science New York University

Softmax Attention Transformer Self-Attention Neural Network Approximation Theory

Ask PDF

Sep 30, 2026stat.ML

Minimax rates for learning spectral Barron functions by deep ReLU neural networks

We study how well deep neural networks approximate and learn spectral Barron functions. Recent studies have shown that these function classes can be efficiently approximated by shallow neural networks without suffering from the curse of dimensionality. We complement these results by providing new approximation bounds for deep networks with ReLU activation and establishing the minimax rates for learning these function classes. Specifically, we show that

d

-dimensional spectral Barron functions with smoothness index

s>0

can be approximated by deep ReLU neural networks with approximation rate

\widetilde{\mathcal{O}} (S^{-\frac{1}{2}-\frac{s}{d}})

, where

S

denotes the number of nonzero parameters in the network. Using this approximation result, we further show that deep ReLU neural networks can learn spectral Barron functions in a fast rate

n^{-\frac{d+2s}{2d+2s}}

with

n

training samples. Finally, we prove that this convergence rate is minimax optimal up to logarithmic factors.

Songqiu Ma, Yunfei Yang

School of Mathematics (Zhuhai), Sun Yat-sen University, Zhuhai, China. · School of Mathematics (Zhuhai) and Guangdong Province Key Laboratory of Computational Science, Sun Yat-sen University, Zhuhai, China.

ReLU Neural Networks Minimax Estimation Neural Network Approximation Theory Statistical Learning Theory

Ask PDF

Sep 28, 2026cs.LG

Arbitrary-Accuracy Neural Approximation with Optimal Neuron Count and Near-Optimal Bit Complexity

We study the minimum number of hidden neurons required for arbitrary-accuracy approximation of multivariate Hölder-continuous functions on

[0,1]^d

and the associated encoding complexity. For

d\geq 2

, we construct a fixed, explicitly defined activation function for which a closed-form network with two hidden layers of widths

d

and

1

achieves arbitrary accuracy in the uniform norm. We prove that

d+1

is the exact minimum total number of hidden neurons among standard feedforward networks with locally integrable activations and affine outputs. We further give a simpler construction using a single elementary activation that combines the floor and exponential functions. This construction requires three hidden layers of widths

d

1

, and

2

, only two neurons above the minimum. If a skip connection is allowed, widths

d

1

, and

1

suffice. These constructions use explicit grid addressing and integer encoding of quantized function values. For a bounded

α

-Hölder class, they require

O(\varepsilon^{-d/α}\log(1/\varepsilon))

bits, matching the metric-entropy lower bound up to a logarithmic factor.

Zilan Cheng, Li-Lian Wang, Zhongjian Wang

Division of Mathematical Sciences, School of Physical and Mathematical Sciences, Nanyang Technological University, 637371, Singapore.

Neural Network Approximation Theory Feedforward Neural Networks Neural Network Expressivity

Ask PDF

Sep 27, 2026cs.LG

From Distributions to Stochastic Processes: Neural Approximation of Measure-Valued Maps

Learning mappings between probability distributions arises naturally when inputs and outputs are represented by populations of samples rather than individual observations. We develop an approximation-theoretic framework for distribution-to-distribution learning and extend it to mappings between stochastic processes. For continuous operators on

W_2

-compact families of finite-dimensional probability laws, we establish uniform neural approximation in the 2-Wasserstein metric using finite law statistics, a simplex-valued neural map, and a shared atomic output support that guarantees valid probability measures. We further extend this principle to probability laws on separable Hilbert spaces through finite-rank orthogonal projections. These results establish the representational feasibility of learning transformations between probability laws rather than deterministic vectors or functions. To demonstrate practical relevance, we study two problems naturally defined at the distribution level: prediction of first-passage-time distributions for an Ornstein--Uhlenbeck process and nonlinear response-path laws of a Duffing oscillator. Because the theory is model-agnostic and broader than any single practical architecture, the experiments use task-adapted neural models rather than reproducing the theoretical construction exactly. In both problems, the proposed models outperform a fixed-feature MLP baseline and distribution-space kernel regression. These experiments complement the theory by demonstrating the practical learnability of distribution-to-distribution transformations in random systems.

Yichen Wang, Ziyi Wang, Wenlian Lu +5

Shanghai Center for Mathematical Sciences, Shanghai, China · Department of Mathematics, Fudan University, Shanghai, China · Center for Applied Mathematics, Fudan University, Shanghai, China +2

Neural Network Approximation Theory

Ask PDF

Sep 27, 2026stat.ML

Neural Scaling Laws of Transformer Operator Network

Transformers have emerged as powerful architectures for learning solution operators of physical systems. Empirically the prediction error has been observed to decrease when the data size and model size increase, suggesting neural scaling behavior. Yet a theoretical understanding of such scaling laws for transformer-based operator learning remains limited. In this work, we develop a theoretical framework for characterizing the approximation and generalization errors of transformer-based operator learning. Our analysis builds on a local-to-global approximation principle that is naturally aligned with the softmax attention mechanism and yields discretization-invariant output functions. On approximation theory, we derive a universal approximation error of transformer-based operator learning for Hölder-regular operators. On generalization theory, we establish a power scaling law between the prediction error and the training data size. The rate of convergence represented by the scaling exponent explicitly reflects the dimensions of the input and output domains, the regularity of the underlying functions and operators, and crucially, the intrinsic dimension of the input function class. By exploiting this intrinsic low-dimensional structure, our analysis yields a power-law generalization rate for operator learning, in contrast to the logarithmic-type power-law rates appearing in existing analyses of operator learning with feedforward neural networks. Numerical experiments validate the predicted power-law scaling and confirm that the convergence rate varies systematically with the intrinsic dimension of the input function class.

Haoran Yan, Zhongjie Shi, Yuanzhe Xi +2

School of Mathematics, Georgia Institute of Technology, Atlanta, GA 30332, USA · Department of Mathematics, Emory University, Atlanta, GA 30322, USA · School of Computational Science and Engineering, Georgia Institute of Technology, Atlanta, GA 30332, USA

Neural Network Generalization Neural Network Approximation Theory Operator Learning Neural Scaling Laws

Ask PDF

Sep 24, 2026cs.NE

Activation-Flexible ANN-to-SNN Conversion with Finite-State Markov Neurons

Most ANN-to-SNN conversion methods rely on a specific correspondence between the source activation and the spiking neuron dynamics. We propose a finite-state continuous-time Markov chain (CTMC) neuron framework whose stationary spike flux can approximate every continuous nonnegative monotone activation function on a compact interval. For a generalized CTMC family with affine input-dependent transitions, we prove uniform approximation to arbitrary accuracy over this function class and derive an explicit approximation error bound. In practice, two- and three-state CTMCs fit ReLU, sigmoid, softplus, and clipped ReLU on the evaluated input ranges, and we evaluate corresponding MLP conversions for each activation with layerwise rate scaling. Moderate clipping improves the conversion cost-accuracy tradeoff on the MNIST MLP and reduces SynOps by 27% on VGG-11/MNIST at matched ANN-SNN accuracy gap criteria, whereas the trend reverses on VGG-11/CIFAR-10. Mean-field and layerwise diagnostics indicate that finite-window sampling and terminal-layer mismatch are the main residual errors. Overall, our results establish finite-state CTMC neurons as a theoretically grounded framework for activation-flexible ANN-to-SNN conversion beyond fixed activation-neuron correspondences.

Ruiyu Jia, Zhuo-Cheng Xiao

New York University Shanghai, Shanghai, China

Neural Network Activation Functions Spiking Neural Networks Neural Network Approximation Theory

Ask PDF

Sep 22, 2026cs.LG

Neural Approximation by Function Composition: Rigidity and Doubly Exponential Convergence

Deep neural networks approximate functions by composing affine maps with nonlinear activations, but how composition itself creates approximation power is not yet fully understood. We investigate a fundamental mechanism: geometrically weighted sums of iterates of a single scalar generator function. This mechanism underpins the classical tent-map construction of the function

x - x^2

and related recursive representations used by Yarotsky, W. E, et al., to analyze the approximation powers of deep neural networks. First, we establish a rigidity theorem: for continuous piecewise linear generators with a finite number of segments, any

C^3

function that can be represented in this way is at most quadratic. For non-affine quadratic functions, the geometric factor is at least

1/4

. This result both reveals limitations of the tent-map approach and complements existing methods based on hierarchical bases and recursive polynomial constructions. Second, using an exact remainder identity as guidance, we construct a smooth generator whose iterates yield doubly exponential error decay in total depth for square approximation and, through multiplication modules, for each fixed polynomial. For power series with absolutely summable coefficients on

[-1,1]^d

, distributing depth according to monomial degree yields a uniform approximation error of order

O(e^{-cL^{1/d}})

on each interior cube. These findings demonstrate how generator dynamics and remainder estimates govern depth allocation and approximation rates of deep neural networks.

Wentao Huang, Haizhang Zhang

Department of Mathematics, The University of Hong Kong, Hong Kong, P.R. China. · School of Mathematics, Sun Yat-sen University, Guangzhou, P.R. China.

Neural Network Approximation Theory Neural Network Expressivity

Ask PDF

Sep 22, 2026stat.ML

Statistical Gains from Looped Estimation under Parameter Budgets

Memory constraints in artificial intelligence motivate accurate function approximation with fewer parameters. We study looping, which repeatedly composes one update function with shared parameters; each output becomes the next input. A looped Transformer, for example, reuses one block, whereas its conventional untied counterpart uses separately parameterized blocks. We compare their parameter requirements for a given worst-case approximation accuracy, or equivalently, their approximation accuracy under a common budget limiting distinct trainable coefficients. We then ask whether this representational parsimony improves statistical accuracy. For general likelihood models, we establish an upper squared Hellinger risk bound for looped sieve maximum likelihood and a minimax lower bound for the jointly tuned untied family. Further loop iterations improve the approximation bound without adding parameters, while increasing computation and the fitted-class complexity bound. For targets of known Hölder smoothness, looped residual feedforward networks and post-layer-normalized Transformers attain the minimax polynomial rate up to logarithmic factors with a fixed number of bounded real parameters. At sufficiently large fixed budgets, looped worst-case risk vanishes while optimal worst-case untied risk remains bounded away from zero. The loop-to-untied risk ratio also tends to zero under specified growing-budget conditions. Regression, binary response, and energy-based generative models illustrate the theory.

Xinyu Tian, Xiaotong Shen

School of Statistics, University of Minnesota

Parameter Estimation Minimax Estimation Neural Network Approximation Theory Statistical Learning Theory

Ask PDF

Sep 22, 2026stat.ML

Optimal Tradeoffs Between Network Size and Parameter Magnitude in Neural Approximation and Minimax Regression

The statistical accuracy of neural networks depends on both their approximation power and the complexity of the class fitted from data. While increasing network size is a natural way to improve approximation, parameter magnitude provides another resource whose role must be quantified in both respects. We establish a sharp width--magnitude tradeoff at fixed depth using one elementary bounded

1

-Lipschitz Dyadic--Triangular Activation. For the unit

β

-Hölder ball on

[0,1]^d

with

0<β\leq1

, the optimal

L^p

approximation error for

0<p<\infty

is of order

[N^2\log(eNT)]^{-β/d}

when the network width satisfies

N\geq2d+3

and the parameter magnitudes are bounded by

T\geq1

. Matching lower bounds hold for every fixed globally Hölder activation; its Hölder exponent affects the constants but not the rate. Under bounded design densities and independent centered sub-Gaussian noise, approximate least squares over the full clipped class at depth

23

attains the classical Hölder minimax risk

\mathcal{O}(M^{-\frac{2β}{2β+d}})

without logarithmic loss whenever

N^2\log(eNT)\asymp M^{\frac{d}{2β+d}}

, where

M

is the sample size. This yields a continuum of statistically optimal choices, ranging from unit parameter radius to fixed network size. At fixed size, four hidden layers with at most

8d+7

nonzero parameters give a near-optimal radius, while six layers with at most

8d+27

attain the optimal order

\log T=\mathcal{O}(η^{-d/β})

at approximation error

η

. The same decoding method also yields fixed-size Transformer approximation.

Baicheng Li, Zuowei Shen, Haizhao Yang +1

Department of Mathematics, University of Maryland, College Park, MD 20742-4015, USA. · Department of Mathematics, National University of Singapore, Singapore 119076. · Department of Applied Mathematics, The Hong Kong Polytechnic University, Hong Kong, China.

Neural Network Optimization Nonparametric Regression Minimax Estimation Neural Network Approximation Theory

Ask PDF

Sep 17, 2026stat.ML

Error bounds in Sobolev norms for approximations with norm constrained ReLU neural networks

Recent studies have shown that smooth functions can be well approximated by ReLU neural networks with path norm constraint on the weights. We extend these results from uniform approximation to approximation in Sobolev norm. Specifically, we analyze how well Sobolev functions in

W^{n,p}

can be approximated by neural networks with width

W

, depth

L

and path norm bounded by

K

, when the approximation error is measured in the

W^{1,p}

-norm. For shallow networks with depth

L=1

, we derive the approximation error bound

\mathcal{O}(\max\{W^{-(n-1)/d}, K^{-(n-1)/(s-n)}\})

, when the smoothness index satisfies

n<s=(d+3)/2

and the input is

d

-dimensional. For deep networks, we remove the restriction on the smoothness by showing that the approximation bound

\mathcal{O}(K^{-(n-1)/(d+d/p+1)})

holds if the width

W

and depth

L

are sufficiently large.

Xianjun Li, Yunfei Yang

School of Statistics and Data Science, Capital University of Economics and Business, Beijing, China. · School of Mathematics (Zhuhai) and Guangdong Province Key Laboratory of Computational Science, Sun Yat-sen University, Zhuhai, China.

Shallow Neural Networks ReLU Neural Networks Neural Network Approximation Theory

Ask PDF

Sep 14, 2026math.NA

Physics Informed Random Feature Neural Networks for Solving PDEs

Machine learning-based partial differential equations (PDEs) solvers have attracted significant attention in recent years. Most progress in this area has been driven by deep neural networks such as physics-informed neural networks (PINNs) and kernel method (such as physics-informed Gaussian Processes). We introduce a physics-informed random feature method for countering part of the spectral bias which PINN-based solvers are facing for a certain class of PDEs. Random feature method was originally proposed to approximate large-scale kernel machines and can be viewed as a specialized randomized neural network. Compared to other state-of-the-art PINN-based solvers which require a large number of collocation points, our proposed method reduces the computational complexity. In this paper, we develop a rigorous approximation error analysis and derive high-probability error bounds on the

H^1

norm. We provide extensive numerical tests for verifying our theoretical guarantees on error decay rates, as well as several comparison tests to showcase our claimed capability for combating spectral bias in these deep learning based methods.

Chi-An Chen, Chunyang Liao, Ming Zhong

Department of Applied Mathematics, Mathematics, Illinois Institute of Technology, Chicago, IL, 60616 · Department of Mathematical Sciences, University of Arkansas, Fayetteville, AR, 72701 · Department of Mathematics, University of Houston, Houston, TX, 77204

PDE Solving Random Feature Methods Neural Network Approximation Theory Physics-Informed ML

Ask PDF

Sep 14, 2026stat.ML

Approximating Smooth Functionals with ReLU Networks

We study the uniform approximation of smooth scalar-valued functionals on an infinite-dimensional separable Hilbert space by ReLU neural networks. A key feature in deep learning for functional data is the varying importance of different coordinates/dimensions. Representing the functional input in a basis expansion, we quantify the importance of each coordinate through both the magnitude of its corresponding basis score and the directional sensitivity of the target functional. Our analysis combines coordinate truncation, anisotropic partitioning, local Taylor approximation, and ReLU network realization, while allowing unrestricted interactions among the retained coordinates. We establish a general nonasymptotic upper bound for the uniform approximation error and a complementary pseudo-dimension-based lower bound for the worst-case approximation error. Under generalized exponential coordinate decay

w_ds_d\asymp\exp(-cd^ρ)

, with

ρ>0

, the upper and lower bounds match at the leading order, which is stretched-exponential in the logarithm of the network size budget, and thus yield the nearly optimal approximation rate. This is the first work to characterize neural network approximation error for infinite-dimensional functional inputs explicitly through the joint dimensional decay of coordinate magnitudes and directional sensitivities.

Shuhao Jiao

Department of Biostatistics, City University of Hong Kong, Hong Kong SAR, China.

ReLU Neural Networks Neural Network Approximation Theory

Ask PDF

Sep 14, 2026cs.LG

Draining Fictitious Knots: Restoring Distance-Awareness Guarantees for High-Dimensional Spline Networks

Kolmogorov-Arnold Networks (KANs) with spline activations have recently shown promise for interpretable function approximation. Distance-Aware Error for Kolmogorov Networks (DAREK) introduces a computationally efficient bottom-up approach to uncertainty quantification by equipping KANs with distance-aware error bounds; yet, in high-dimensional settings, the theoretical guarantees can be weakened by the emergence of fictitious knots. Inspired by the Kolmogorov-Arnold representation theorem, DAREK adopts a componentwise formulation in which each input dimension is treated separately; as a result, induced knot locations may appear in the combined input space without corresponding to actual training data. These fictitious knots mislead the DAREK uncertainty estimator into reporting low uncertainty far from any real observation, violating the distance-awareness guarantee. We identify this failure mode precisely, characterize its geometric structure, and propose a drainage uncertainty mechanism that restores distance-awareness by constructing a monotonically decreasing uncertainty path from any fictitious knot region toward the nearest real knot. The proposed drainage method provides a practical heuristic correction that mitigates the fictitious-knot failure mode while restoring theoretical distance-awareness in high-dimensional settings. Experiments on a 2D synthetic benchmark and a 100-dimensional face dataset show that drainage raises sampled distance-awareness (SDA) from 85% to 98-99%, matching Gaussian processes at lower computational cost.

Masoud Ataei, Mohammad Javad Khojasteh, Vikas Dhiman

Electrical and Computer Engg. University of Maine Orono, ME, USA · Electrical and Microelectronic Engg. Rochester Institute of Technology Rochester, NY, USA

Uncertainty Quantification Kolmogorov-Arnold Networks Neural Network Approximation Theory

Ask PDF

Sep 14, 2026math.OC

Exact ReLU realization of binary affine refinement iterates via reflection folding and cone switching

We study vector-valued binary affine refinement operators with finitely supported matrix masks and compactly supported continuous piecewise linear input and forcing data. We prove that every finite refinement iterate admits an exact ReLU realization of fixed width and depth linear in the number of iterations. No separation of the forcing profile from the binary cell seams is required. The main mechanism is universal reflection doubling. Pairing each residual profile with its reflection replaces the two binary transition matrices by one fixed block matrix together with a fixed swap involution. The cell-seam identity makes the two branch candidates agree at the tent fold, while their swap-odd component is bounded linearly by the distance to the fold. This permits exact branch selection by a fixed continuous piecewise linear cone switch, without multiplication by a variable selector. The resulting primal recursion requires the residual orbit in reverse order. We obtain exact backward replay from the residual memory controller developed previously for affine refinement, interpreted here through the reflection quotient of circle doubling. The construction propagates the full vectorized profiles rather than decomposing the input and forcing into reference atoms. We also treat stage-dependent forcing from a fixed finite-dimensional family and show that genuine reflection equivariance reduces the doubled cascade to a single parity sector.

Boldsaikhan Bolorkhuu, Tsogtgerel Gantumur

National University of Mongolia · McGill University · Institute of Mathematics and Digital Technology, Mongolian Academy of Sciences

ReLU Neural Networks Neural Network Approximation Theory

Ask PDF

Sep 4, 2026math.NA

Shallow neural network approximation in mixed Sobolev spaces

We investigate the best

L_2

approximation of mixed Sobolev spaces by shallow neural networks with

n

neurons and general activation functions. We first establish an activation-independent Fourier-block principle: if an activation has univariate approximation order

ρ

in the sense of the Fourier-block property, then the global approximation rate has algebraic order

\min\{α,ρ\}

for target functions of mixed smoothness

α

, up to explicit logarithmic factors. To verify this property for concrete activations, we introduce a structured univariate approximation condition that implies the Fourier-block property with explicit parameters. For

\mathrm{ReLU}^k

, a matching algebraic lower bound identifies

\min\{α,k+1\}

as the optimal algebraic approximation exponent in any dimension, up to logarithmic factors in the upper bound. The framework also yields the exponent

\min\{α,k+1\}

for cardinal B-splines and soft-

\mathrm{ReLU}^k

, and the full mixed-smoothness exponent

α

for ELU and cosine activations, again up to logarithmic~factors.

Yuwen Li, Guozhi Zhang

School of Mathematical Sciences, Zhejiang University, 866 Yuhangtang Road, Hangzhou 310058, Zhejiang, China

Neural Network Approximation Theory Shallow Neural Networks Neural Network Activation Functions

Ask PDF

Sep 3, 2026math.NA

Residual neural networks overcome the curse of dimensionality for semilinear heat equations

Rigorous results show that feedforward neural networks can overcome the curse of dimensionality in the numerical approximation of high-dimensional partial differential equations (PDEs), but comparatively little is known about residual neural networks (ResNets) in the nonlinear PDE setting. We prove that ResNets overcome the curse of dimensionality in the numerical approximation of solutions of semilinear heat equations with globally Lipschitz continuous, gradient-independent nonlinearities: under polynomial growth and network approximability hypotheses on the PDE data, there exist

η\in(0,\infty)

and ResNets

Ψ_{d,\varepsilon}

d\in\mathbb{N}

\varepsilon\in(0,1]

, with at most

ηd^η\varepsilon^{-η}

parameters whose realizations approximate the solution in dimension

d

with an

L^2

-error of at most

\varepsilon

. The proof represents one deterministic realization of a multilevel Picard estimator by a ResNet whose shortcut connections transmit the spatial variable and a scalar accumulator, while the residual branches successively add the summands of the estimator. For ridge-sum initial conditions, admissible sigmoidal activations, and globally Lipschitz truncations of the nonlinearity, we obtain, for every

ξ>0

, the explicit bound

C_ξd^{4+ξ}\varepsilon^{-(3+ξ)}

on the number of parameters.

Ilkhom Mukhammadiev, Diyora Salimova

Department of Applied Mathematics, University of Freiburg, Germany

Neural PDE Solvers PDE Solving Residual Learning Neural Network Approximation Theory

Ask PDF

Sep 3, 2026math.NA

Spectral Convergence of Random Feature Method in Multiple Dimensions

We first prove spectral convergence of the random feature method (RFM) for multidimensional targets in Sobolev, Gevrey, ultra-analytic, and bandlimited classes. The analysis establishes general high-probability approximation estimates in the interpolation scale generated by a kernel integral operator. On a single event determined only by the sampled features, one random space approximates every target in a prescribed source ball; moreover, for each target, a single coefficient vector defines an approximant that attains spectral accuracy simultaneously in all admissible error norms. For both regularity-adapted frequency distributions and uniform distributions on growing frequency windows, the resulting rates range from super-exponential to algebraic, depending on the regularity of the target. Second, we establish abstract error estimates for strong- and weak-form RFM discretizations, thereby converting the preceding approximation bounds into convergence estimates for multidimensional second-order elliptic boundary value and eigenvalue problems. Finally, for random feature matrices (RFMtxs), we prove super-exponential singular-value decay with Fourier features and exponential decay with

\tanh

features, together with corresponding condition-number lower bounds. The analysis identifies a common mechanism: the same spectral approximation that yields high accuracy also drives severe ill-conditioning.

Pingbing Ming, Hao Yu

PDE Solving Random Feature Methods Neural Network Approximation Theory

Ask PDF

Sep 2, 2026stat.ML

A Closed-Form Formula for Consistent Lipschitz Regression on Metric Spaces with Sparse Neural Network Realizations

Several classical machine-learning methods, such as KRRs and SVRs, are both computationally and analytically tractable since their estimators either admit closed-form expressions or are obtained by minimizing convex training objectives; neither feature is generally available for deep neural networks. We address this by introducing a simple closed-form ``two-stage'' compositional formula

\hat{f}

for reconstructing an unknown Lipschitz function

f:\mathcal{X}\to \mathbb{R}

on a metric space

(\mathcal X,ρ)

from

N

i.i.d. noisy observations. Our main result is a high-probability uniform (

L^{\infty}

) recovery guarantee that jointly controls approximation and statistical errors while enjoying an optimization error of zero; in particular, we do not assume oracle access to an approximate ERM. Our secondary main results establish the optimality of our formula in three complementary senses. 1) Function space: On Ahlfors-regular metric spaces, the hypothesis class parameterized by our formula attains the optimal fat-shattering dimension. 2) Parameter space: Its dependence on the parameters is maximally numerically stable, in the sense that a smaller approximation error cannot be achieved with a smaller Lipschitz dependence on the model parameters. 3) Forward pass: Its dependence on the input is maximally regular, matching the Lipschitz constant of the target function

f

. When

\mathcal X=[0,1]^d

is equipped with the

\ell^\infty

norm,

\hat{f}

admits algorithmic ReLU-MLP and exact ReLU-multi-head transformer realizations of depth

\mathcal{O}(\log(N))

with

\mathcal{O}(N)

nonzero parameters.

Ruiyang Hong, Hrad Ghoukasian, Anastasis Kratsios

Neural Network Approximation Theory Statistical Learning Theory Lipschitz Neural Networks

Ask PDF

Sep 1, 2026cs.LG

RecKAN: Kolmogorov-Arnold Networks with a Learnable Recursive Polynomial Basis

Kolmogorov--Arnold Networks (KANs) replace the fixed scalar weights of a standard network with learnable univariate functions on each edge, but existing variants still fix the \emph{basis} that those functions are built from: B-splines, Chebyshev polynomials, wavelets, or Jacobi polynomials, and learn only the combination weights over it. We introduce RecKAN, which instead defines the basis itself by a second order polynomial recurrence,

R_{n+1}(x) = (ax^2+bx+c)R_n(x) + (dx+e)R_{n-1}(x)

, whose five coefficients are learned jointly with the network. We show this recurrence recovers several classical polynomial families including both kinds of Chebyshev polynomials, Fibonacci, Pell, and Jacobsthal polynomials as special cases, and prove that its degree grows linearly in

n

exactly on the sub-family containing all of them, giving a concrete sense in which the learned basis can move beyond any fixed classical choice. Across multiple benchmark datasets spanning image, text, biomedical time series classification, and time series forecasting, RecKAN outperforms three parameter-matched KAN baselines (Chebyshev, Jacobi, and spline based) on all classification tasks and achieves the lowest MSE on the ETTh1 forecasting benchmark. Additionally, when used as a classifier head with a convolutional backbone, RecKAN achieves higher accuracy than standard MLP heads on Fashion MNIST, CIFAR-10, and SVHN. On a synthetic function fitting benchmark it tracks a sharply oscillatory target that a parameter comparable MLP under fits. We further show that the learned recurrence coefficients are interpretable: on the task requiring the most local structure, training moves the basis away from the linear degree growth regime that contains every classical family we identify, consistent with our theoretical analysis of what that structural shift enables.

Amirhosein Azarpour

Department of Computer Science Shahid Beheshti University Tehran, Iran

Polynomial Neural Networks Kolmogorov-Arnold Networks Neural Network Approximation Theory

Ask PDF

Sep 1, 2026cs.LG

Why Multi-Layer Message Passing Works: Completeness Theory for Graph Neural Network Interatomic Potentials

We prove that the Hypergraph Neural Network, an invariant architecture with 3-body message passing, is a universal approximator for potential energy surfaces. Our main contribution is a multi-layer completeness theory. We show that

L

layers of message passing on sparse, cutoff-based graphs achieve the same representational power as having access to the full

L

-hop neighborhood, provided the configurations are generic, satisfy an overlap condition and a connectivity condition. This provides the first rigorous justification for the common practice of using multi-layer message passing with a per-layer cutoff smaller than the physical interaction range, the setting used by virtually all practical graph neural network based machine-learned interatomic potentials. As immediate consequences, we show that both DPA3 and CHGNet architectures inherit universal approximation.

Pingbing Ming, Han Wang

SKLMS, Institute of Computational Mathematics and Scientific/Engineering Computing, AMSS, Chinese Academy of Sciences, No. 55, East Road Zhong-Guan-Cun, Beijing 100190, China · National Key Laboratory of Computational Physics, Institute of Applied Physics and Computational Mathematics, Fenghao East Road 2, Beijing 100094, P.R. China, and HEDPS, CAPT, College of Engineering, Peking University, Beijing 100871, P.R. China

Machine Learning Interatomic Potentials GNN Expressivity Message Passing Neural Networks Hypergraph Neural Networks Neural Network Approximation Theory

Ask PDF

Aug 31, 2026stat.ML

A convolutional framework for detecting event-driven dynamics in energy price series

This paper develops a general convolutional neural network (CNN) framework for detecting heterogeneous event-driven dynamics in univariate time series windows. We show that the induced CNN class exactly represents classifiers based on range, maximum drawup, maximum drawdown and slope change, and uniformly approximates realised volatility and autoregressive explosiveness on compact domains. We further establish error bounds for representative rules in finite samples and an oracle inequality for learning across them. Simulations show that the proposed model can match or outperform classifiers based on individual statistics as the training sample grows. In an application to six daily energy price series, a hierarchical CNN distinguishes event windows and event families. Applied without retraining to observations withheld after 20 February 2026, the fitted model identifies predominantly geopolitical dynamics in several oil and refined product series around the outbreak of the 2026 Iran war, while distinguishing a contemporaneous natural gas spike associated with weather.

Caixia Xu, Piotr Fryzlewicz

1Shanghai University of Finance and Economics · 2London School of Economics

Time Series Classification Neural Network Approximation Theory Convolutional Neural Networks

Ask PDF

Aug 31, 2026cs.LG

Sharp Approximation Rates for Neural Networks with Affine Latent Parameterizations

Many parameter-efficient methods generate the parameters of a large neural network from a low-dimensional latent representation. Given an architecture

Φ

with

P_Φ

parameter slots, we write

\boldsymbolθ_f=\mathcal{G}(\boldsymbolξ_f)

, where

\mathcal{G}\colon\mathbb{R}^M\to\mathbb{R}^{P_Φ}

is a parameter generator and

\boldsymbolξ_f\in\mathbb{R}^M

is a latent representation of the target function

f

. The architecture

Φ

and the generator

\mathcal{G}

are shared across the entire target class, while each target

f

is represented by its own latent vector

\boldsymbolξ_f

, with

Φ_{\mathcal{G}(\boldsymbolξ_f)}

approximating

f

. This framework encompasses hypernetworks, low-dimensional parameterizations, parameter-efficient adaptation, and model compression. Understanding the tradeoff between the latent dimension

M

and the network budget

P

is therefore fundamental to characterizing the expressive efficiency of these methods. We study this tradeoff for affine generators and fully connected ReLU architectures. More precisely, optimizing jointly over architectures

Φ

satisfying

P_Φ\leq P

and affine generators

\mathcal{G}:\mathbb{R}^M\to \mathbb{R}^{P_Φ}

, we prove that the optimal worst-case uniform approximation error over the unit ball of

α

-Hölder functions on

[0,1]^d

, where

0<α\leq1

, has the sharp order

\bigl(P\min\{M,P\}\bigr)^{-α/d}.

In particular, our result shows that even a fixed-dimensional latent space suffices to achieve vanishing approximation error as the network budget increases.

Shijun Zhang

Department of Applied Mathematics Hong Kong Polytechnic University

Hypernetworks Neural Network Approximation Theory Neural Network Expressivity Neural Network Compression

Ask PDF

Aug 31, 2026cs.LG

Kolmogorov--Arnold against bounded translations

Historically originating from Hilbert's 13th problem, the Kolmogorov-Arnold representation theorem (KART) has recently experienced a major revitalisation through its applications to neural networks, specifically Kolmogorov-Arnold Networks (KANs). While the exact representation is well established, its stability under continuous adversarial perturbations of the hidden layer remains a critical open question. In this paper, we investigate the robustness of KART against bounded adversarial translations. We provide an explicit, self-contained, and constructive proof of an approximate representation using fixed, piecewise linear inner functions. Crucially, our construction employs a single outer function that remains invariant for all summands and is independent of the specific adversarial translation, provided its maximum bound is known a priori.

Sviatoslav V. Dzhenzher

Moscow Institute of Physics and Technology 141701, Institutskii lane, 9, Dolgoprudny, Russia.

Adversarial Robustness Kolmogorov-Arnold Networks Neural Network Approximation Theory

Ask PDF

Aug 24, 2026cs.LG

Every Layer Counts: An Exponential $L_2$ Depth Hierarchy for ReLU Networks

We prove a depth hierarchy for ReLU neural networks in which every additional ReLU layer can save exponentially many neurons. For all

k\geq2

, we construct a globally

[0,1]

-valued,

1

-Lipschitz function realized by a depth-

(k+1)

network of width

\mathcal{O}(d^4)

, whereas any depth-

k

network with unrestricted weights and width at most

\frac{2^d}{2d(k-1)}

has squared

L_2

error at least

1/24

under an absolutely continuous distribution supported at exponential distance from the origin. To the best of our knowledge, this is the first exponential hierarchy across all adjacent fixed depths, and the first exponential separation for ReLU networks between two fixed depths whose shallower network has depth at least

3

. The lower bound also immediately yields the corresponding hierarchy for exact computation. Moreover, the case

k=2

gives a compactly supported separation between depths

3

and

2

with unrestricted shallow-network weights, answering a question raised by Safran, Eldan, and Shamir (2019). The distribution used in our construction nevertheless has all its mass at exponential radius, placing the hierarchy outside the regularity regime in which such a separation would imply major threshold-circuit lower bounds. We also prove an exact separation for a more regular target, which is globally

[0,1]

-valued and

\mathcal{O}(\sqrt d)

-Lipschitz and maps the unit hypercube onto

[0,1]

. It is computed by a polynomial-width depth-

4

network, whereas any depth-

3

network agreeing with it on the unit hypercube requires exponentially many first-layer neurons, even with unrestricted weights.

Itay Safran

Stein Faculty of Computer and Information Science, Ben-Gurion University of the Negev

Shallow Neural Networks ReLU Neural Networks Neural Network Approximation Theory Neural Network Expressivity

Ask PDF

Aug 12, 2026cs.LG

HYDRA: Hyperbolic Dynamic Representation Architecture for Kolmogorov-Arnold Networks

Kolmogorov-Arnold Networks (KANs) enhance nonlinear function approximation by replacing scalar weights with learnable univariate functions. However, assigning an independent function to every connection results in substantial parameter redundancy, limiting their scalability and efficiency. To reduce this redundancy, we introduce \textbf{HY}perbolic \textbf{D}ynamic \textbf{R}epresentation \textbf{A}rchitecture (HYDRA), a parameter-efficient hyperbolic extension of KAN that combines spline-based functional learning with representations in the Poincaré ball. HYDRA maps vector-valued inputs into a bounded hyperbolic latent space, performs KAN-style updates in tangent space, and employs a low-rank prototype block to share functional transformations across hidden dimensions. The resulting hyperbolic representations provide a structured radial coordinate for interpretation, while radius control improves training stability by preventing boundary saturation. Extensive experiments across eight benchmark datasets demonstrate that HYDRA consistently achieves competitive or superior predictive performance while improving parameter efficiency and representation interpretability.

Zhao Su, Yuxin Xia, Haoran Li +4

School of Information Science and Engineering, Lanzhou University · Department of Data Science and AI, Monash University · School of Computing and Information Technology, University of Wollongong +1

Kolmogorov-Arnold Networks Neural Network Approximation Theory Neural Network Compression Hyperbolic Representation Learning

Ask PDF

Aug 11, 2026cs.LG

Accelerated Learning of High Dimensional Functions with a Tensor-Featured Training Network

In this work we present a method to accelerate the optimization of learning high dimensional functions using deep neural network (DNN). This optimization procedure introduces contextual features into the first layer of a DNN. The parameters of DNN are optimized via standard gradient descent while keeping the input-feature basis fixed. After optimization of the DNN parameters, the feature layer is provided a chance to update and change before DNN optimization resumes. The feature layer has two types of functions: those that can be evaluated quickly in a matrix-free way on the domain (i.e. rank-1 features) and more complex features that must first be decomposed using tensor network (TN) decomposition strategies (tensor features). In particular, we study the effect of adding features which distill pretrained DNN into TNs using a discretize and decompose strategy. To efficiently decompose high-dimensional functions constructed from discretized DNN, we leverage a randomized tensor decomposition strategy. Using randomization, we are able to reduce the storage cost of decomposing high dimensional functions by at least 8 orders of magnitude. Using this approach, we are able to efficiently train models between 5 and 40 dimensions.

Karl Pierce, Yuehaw Khoo, Haizhao Yang

† The University of Maryland, College Park, College Park MD 20742 USA · ∗The University of Chicago, Chicago, IL 60637 USA

Neural Network Optimization Neural Network Approximation Theory

Ask PDF

Aug 10, 2026cs.LG

Training-Free Universal Approximation by Prompting Random Transformers

How expressive is prompting a transformer? Answering this question is important for separating the roles of prompting, architecture, and pretraining in transformer models, and for determining whether task-specific behavior must be stored in model weights or can instead be induced at inference time through the prompt. We show, in an approximation-theoretic sense, that pretraining is optional: a single-layer softmax attention network with random, untrained weights can approximate any Hölder function on a compact manifold when steered by an appropriate soft prompt. Guided by the connection between softmax attention and kernel methods, we construct explicit soft prompts (a prompt per target function, independent of the query) as solutions to linear systems matching attention logits to Gaussian kernel exponents, under which the frozen transformer emulates the classical Nadaraya-Watson kernel estimator. The construction requires only a mild rank condition on the weights, which we show holds almost surely under Gaussian initialization. The prompted network inherits the theoretical guarantees of kernel regression, leading to universal approximation theorems with minimax-optimal rates that depend on the intrinsic dimension. We further quantify the cost of prompting, exposing a tradeoff between the norm of the constructed soft prompt tokens, prompt length, and hidden dimension. Numerical experiments corroborate the constructions and predicted rates.

Alexander Hsu, Rongjie Lai

Department of Mathematics, Purdue University

Transformer Expressivity Prompt Learning Transformer Neural Network Approximation Theory

Ask PDF

Momentum

Related topics

Latest papers 163