Neural Network Approximation Theory

Follow this topic Ask about this topic

Momentum

15 papers in the last four weeks, up 15% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 163

May 22, 2026cs.LG

Optimization of randomized neural networks for transfer operator approximation

RaNNDy is a randomized neural network architecture for the data-driven approximation of transfer operators associated with complex dynamical systems. The weights and biases of the hidden layers of the network are randomly initialized and kept fixed, only the output layer is trained. This has several advantages over fully optimized neural networks, notably a closed-form solution for the output layer and significantly lower training costs. Despite these advantages, RaNNDy is restricted to the initial selection of weights and biases that parametrize the basis functions required for the operator approximation. Since the basis functions are determined by the activation function, choosing an appropriate activation function for the hidden layers is crucial. In this work, we propose an algorithm that optimizes the activation function itself, while keeping the weights and biases in the randomized neural network fixed, providing a more suitable dictionary. We illustrate the efficacy of the approach using various benchmark problems, including stochastic differential equations and random walks on graphons.

Mohammad Tabish, Stefan Klus

Maxwell Institute for Mathematical Sciences, University of Edinburgh and Heriot–Watt University, Edinburgh, UK · School of Mathematical & Computer Sciences, Heriot–Watt University, Edinburgh, UK

Dynamical Systems Neural Network Optimization Neural Network Activation Functions Neural Network Approximation Theory Operator Learning

May 22, 2026cs.LG

Any-Dimensional Invariant Universality

Several machine learning models are defined for inputs of any size, such as graphs with different numbers of nodes and point clouds containing varying numbers of points. The universality properties of such any-dimensional models remain poorly understood, as universality is traditionally studied for models accepting inputs of a fixed size, defined on a compact subset of their domain. In sharp contrast, any-dimensional models can be viewed as sequences of functions defined on growing-sized inputs, and it is not clear in which sense they can be universal. We develop a systematic approach to establish any-dimensional universality, by identifying any-dimensional functions with a unique function taking inputs in a suitable infinite-dimensional limit space containing inputs of all finite sizes as well as their limits. Using the symmetries of these inputs and relations between inputs of different sizes, we show that this limit space admits a natural topology with rich families of compact sets on which any-dimensional universality can be established. We illustrate our approach by showing that several existing architectures fail to be universal, and we propose simple modifications that restore universality.

Shengtai Yao, Eitan Levin, Mateo Díaz

Department of Applied Mathematics and Statistics, Johns Hopkins University, Baltimore, MD 21218, USA. · Department of Computing and Mathematical Sciences, Caltech, Pasadena, CA 91125, USA.

Universal Approximation Neural Network Approximation Theory

May 21, 2026cs.LG

Multiple Neural Operators Achieve Near-Optimal Rates for Multi-Task Learning

We study the approximation and statistical complexity of learning collections of operators in a shared multi-task setting, with a focus on the Multiple Neural Operators (MNO) architecture. For broad classes of Lipschitz multiple operator maps, we derive near-optimal upper bounds for approximation and statistical generalization. On the lower-bound side, we establish a curse of parametric complexity and prove corresponding minimax rates. Together, these results show that shared representations across tasks do not increase the overall cost: multi-task operator learning follows the same scaling laws as single operator learning. We also compare MNO with a multi-task extension of DeepONet based on concatenated task inputs and show that, from a worst-case approximation-complexity perspective, both architectures satisfy essentially the same asymptotic rates.

Adrien Weihs, Hayden Schaeffer

Department of Mathematics, University of California Los Angeles, Los Angeles, CA 90095, USA.

Neural Network Approximation Theory Operator Learning Statistical Learning Theory

May 21, 2026math.CO

Holographic functions and neural networks

A fuzzy Boolean function is a map

f:\cube^n\to [0,1]

, where

n\in\mathbb N

. We introduce and compare three ways of saying that such a function has bounded complexity. The first is a sampling property: the value

f(x)

can be recovered, up to small error and with high probability, from the values of a bounded number of randomly chosen coordinates of

x

. We call this the holographic property. The second is a structural property:

f

is uniformly close to a bounded-degree polynomial in boundedly many bounded linear coordinate forms. The third is computational:

f

is uniformly close to the output of a neural network with a bounded number of non-input neurons, bounded Lipschitz activation functions and bounded incoming weights. We prove that these three properties are equivalent up to quantitative changes of the parameters. The implication from holography to polynomial structure uses a variant of a weak version of hypergraph regularity.

Balazs Szegedy

Neural Network Approximation Theory Neural Network Expressivity

May 21, 2026cs.LG

Neural Flow Operators can Approximate any Operator: Abstract Frameworks and Universal Approximations

We introduce an abstract neural flow framework for neural networks and neural operators. The framework contains two continuous-depth models, namely neural flows with composition and separation structures, and covers both finite-dimensional function approximation and infinite-dimensional operator approximation. We prove well-posedness and universal approximation properties for the corresponding neural flows, including, to the best of our knowledge, the first universal approximation result for flow-based models between infinite-dimensional spaces. We also obtain universal approximation results for convolutional neural flow models. Through suitable time discretizations, the composition structure recovers ResNet-type architectures, while the separation structure, via a splitting-based discretization, yields plain architectures. This gives a unified flow-based route to both residual and plain architectures for neural networks and neural operators with fully connected or convolutional linear layers.

Shuang Chen, Juncai He, Xue-Cheng Tai

Qiuzhen College, Tsinghua University, Haidian District, Beijing 100084, China · Yau Mathematical Sciences Center, Tsinghua University, Haidian District, Beijing 100084, China · Norwegian Research Center, Fantoftvegen 38, 5072 Bergen, Norway

Neural Network Approximation Theory Neural Operators

May 21, 2026cs.CR

Decision-Aware Quadratic ReLU Replacement for HE-Friendly Inference

Fully homomorphic encryption (FHE) supports only additions and multiplications, so FHE-only neural-network inference typically replaces ReLU with polynomials fitted over empirical activation intervals. Such interval fitting often requires higher-degree polynomials to control activation error, incurring homomorphic evaluation costs, while classification is determined by the final logit decision. We revisit ReLU replacement from a decision-aware perspective: given a trained single-hidden-layer ReLU MLP and a specified calibration set, can an HE-friendly low-degree polynomial replace ReLU without retraining while preserving calibration-set decisions? We focus on quadratic replacement, the lowest-degree that retains a genuine per-unit nonlinearity. For calibration sets positive-margin separable in the lifted space, we formulate quadratic replacement as a linear separation problem, yielding necessary and sufficient conditions for calibration-lossless replacement and a constructive algorithm for the coefficients. When the positive-margin condition fails -- often because a few near-boundary or misclassified calibration samples bring the lifted hulls into contact -- we extend the same geometric framework via reduced convex hulls and Lagrangian-dual soft-margin relaxations. These cap the weight any single sample can carry, converting the problem into smaller convex quadratic programs that yield approximately feasible coefficients with high empirical agreement on calibration-set decisions. In particular, at the maximal weight cap

μ=1

, the reduced-convex-hull relaxation reduces to standard convex-hull separation; the relaxation thus continuously extends the positive-margin exact theory. Under CKKS, the quadratic replacement matches plaintext top-1 accuracy on multiple benchmarks, running 3.7--4.1

\times

faster than Remez-7 in the activation module and 1.18--1.68

\times

faster end-to-end.

Rui Li, Wenyuan Wu, Weijie Miao

Chongqing Key Laboratory of Secure Computing for Biology, Chongqing Institute of Green and Intelligent Technology, Chinese Academy of Sciences, 266 Fangzheng Avenue, Beibei District, Chongqing 400714, China · Department of Industrial and Systems Engineering, The Hong Kong Polytechnic University, Hung Hom, Hong Kong, China

Efficient Neural Network Inference Homomorphic Encryption Neural Network Approximation Theory

May 20, 2026cs.LG

Approximation Theory for Neural Networks: Old and New

Universal approximation theorems provide a mathematical explanation for the expressive power of neural networks. They assert that, under mild conditions on the activation function, feedforward neural networks are dense in broad function classes, such as continuous functions on compact subsets of

\mathbb{R}^d

,

L^p

spaces, or Sobolev spaces. Over the past four decades, these qualitative universality results have evolved into a rich quantitative theory addressing approximation rates, parameter efficiency, and the role of architectural features such as depth and width. This survey presents several glimpses into this theory. We review classical density results for single-hidden-layer networks, as well as quantitative bounds that relate approximation error to network size and smoothness assumptions on target functions. Particular emphasis is placed on depth--width trade-offs and on results demonstrating that deeper architectures can achieve superior parameter efficiency for structured function classes. In addition to standard feedforward neural networks, we also review recent developments on Kolmogorov--Arnold Networks (KANs), which offer an alternative architectural paradigm and whose approximation-theoretic properties have begun to attract significant theoretical attention.

Soumendu Sundar Mukherjee, Himasish Talukdar

Statistics and Mathematics Unit, Indian Statistical Institute, Kolkata

Universal Approximation Neural Network Approximation Theory Feedforward Neural Networks Neural Network Expressivity

May 19, 2026cs.LG

Fast Reconstruction of Exact Maxwell Dynamics from Sparse Data

We introduce FLASH-MAX, a shallow, exact-by-construction neural network architecture for predicting homogeneous electromagnetic fields from sparse pointwise observations. Each hidden neuron represents a separate exact solution to Maxwell's equations, so that the network satisfies the governing equations symbolically by construction and can be trained end-to-end from sparse data within seconds. We prove a universal approximation result showing that this exact model class remains universal on arbitrary domains. FLASH-MAX reaches sub-1% relative validation error from about 1K sparse pointwise observations in seconds, all while maintaining a zero PDE residual, and keeps single-digit errors even for only 100 observations sampled from 3D space. These results suggest that moving governing structure from the loss into the hypothesis class can dramatically improve the trade-off between precision and optimization speed in scientific machine learning.

Dan DeGenaro, Xin Li, Obed Amo +4

Department of Computer Science, Georgetown University · Department of Mathematics, Georgetown University · Department of Mathematics and Statistics, Old Dominion University +2

Neural PDE Solvers Neural Network Approximation Theory Physics-Informed ML

May 18, 2026stat.ML

Dual-Channel Tensor Neural Networks: Finite-Sample Theory and Conformal Structure Selection

Tensor-valued data arise naturally in neuroimaging, genomics, climate science, and spatiotemporal networks, where multilinear dependencies across modes carry information that is destroyed under vectorization. Existing approaches either impose a single low-rank structure, which can miss localized signal, or treat the tensor as a long vector, which discards its multiway geometry. We propose a Dual-Channel Tensor Neural Network (DC-TNN) that decomposes each tensor input into a low-rank core and a sparse refinement, and processes the two components through coupled neural channels. The framework is structure-agnostic and accommodates CP, Tucker, and tensor-train cores within a single architecture. For estimation, we establish non-asymptotic risk bounds for the DC-TNN estimator that decompose into network approximation, core estimation, and refinement-selection terms, and show that the effective dimension is determined jointly by the core rank and refinement sparsity rather than by the ambient tensor size. For inference, we develop a structure-aware conformal ROC procedure that calibrates within the core-refinement latent space and produces ROC and AUC confidence bands with finite-sample, distribution-free coverage. Building on this, we propose a conformal structure selector that, to our knowledge, is the first distribution-free procedure for choosing among candidate tensor decompositions with finite-sample validity. Simulations and an analysis of a protein dataset demonstrate competitive predictive accuracy, reliable uncertainty quantification, and consistent recovery of the tensor structure.

Elynn Chen, Jiayu Li, Zheshi Zheng +1

New York University · University of Michigan · Duke University

Tensor Networks Uncertainty Quantification Tensor Decomposition Conformal Prediction Neural Network Approximation Theory

May 18, 2026stat.ML

Shallow ReLU $^s$ Networks in $L^p$ -Type and Sobolev Spaces: Approximation and Path-Norm Controlled Generalization

This paper studies approximation by shallow ReLU

^s

networks,

σ_s(t)=\max\{0,t\}^s

, together with their generalization behavior under

\ell_1

path-norm control. For the

L^p

-type integral spaces

\widetilde{\mathcal{F}}_{p,τ_d,s}

,

1\le p\le2

, spherical harmonic analysis yields approximation bounds for shallow networks. In particular, when

τ_d

is the uniform measure and

1\le p<2

, the approximation rate is

O\!\left(m^{-\frac{p(2s+2d+1)-2d}{2dp}}\right)

for

1\le p\le p^*

and

O\!\left(m^{-\frac{p(4s+3d-1)-2d+2}{4dp}}\right)

for

p^*<p<2

, where

p^*=\frac{2d+2}{d+3}

. Approximation bounds for Sobolev spaces

W^{α,p}

,

1\le p<2

, are obtained through embeddings into spectral Barron spaces. For nonparametric regression with sub-Gaussian noise, path-norm-regularized shallow ReLU

^s

networks achieve minimax-optimal rates

O\!\left(n^{-\frac{d+2s+1}{2d+2s+1}}\log n\right)

over

\mathscr{B}_s

and

O\!\left(n^{-\frac{2α}{2α+d}}\log n\right)

over

W^{α,\infty}

, with matching lower bounds up to logarithmic factors.

Weizhao Li, Fanghui Liu, Lei Shi

School of Mathematical Sciences Fudan University Shanghai, China · School of Mathematical Sciences Institute of Natural Sciences and MOE-LSC2026 Shanghai Jiao Tong University Shanghai, China · School of Mathematical Sciences Shanghai Key Laboratory for Contemporary Applied Mathematics25 Fudan University Shanghai, China

Shallow Neural Networks Nonparametric Regression Neural Network Approximation Theory Generalization Bounds

May 18, 2026cs.LG

Function graph transformers universally approximate operators between function spaces

We study the approximation of nonlinear operators between function spaces by transformers. Our approach is to lift functions to measures supported on their graphs and leverage a recently introduced measure-theoretic view of transformers. A function

h

is represented by its graph measure

γ_h

, with finite tokens

\{(x_j,h(x_j))\}_{j=1}^N

being its empirical approximations. We show that this framework elegantly models discretization refinement via convergence of measures and provides a natural setting for operator learning. Within this framework, we introduce function graph transformers, a graph-preserving subclass of measure-theoretic transformers that maps graph measures to graph measures, which is to say that outputs remain single-valued functions. Crucially, this additional structure does not reduce generality: we prove that the resulting graph-preserving maps can be approximated by finite compositions of standard softmax self-attention layers and pointwise MLPs, yielding universal approximation results for broad classes of nonlinear operators. Unlike existing theoretical approaches to operator learning with transformers, the measure-theoretic framework also accommodates regularized negative-order Sobolev inputs for which discretization invariance is particularly challenging, as well as query points on different output domains. Overall, function graph transformers provide a continuum viewpoint and mathematical toolkit for transformer-based operator learning, clarifying the roles of positional encodings, graph structure, regularization, and ensuring consistency across discretizations.

Takashi Furuya, David Mis, Ivan Dokmanić +2

Doshisha University, RIKEN AIP · Rice University · University of Basel +2

Transformer Attention Neural Network Approximation Theory Operator Learning Neural Operators

May 18, 2026cs.LG

InfoFlow: A Framework for Multi-Layer Transformer Analysis

While the approximation properties of single-layer Transformer architectures have been studied in recent works, a rigorous theoretical understanding of the multi-layer setting remains limited. In this work, we establish that multi-layer Transformers possess fundamentally different approximation capabilities from single-layer ones: for certain retrieval tasks, any single-layer Transformer requires least

Ω(\varepsilon^{-k})

parameters to achieve precision

\varepsilon

, where

k

grows linearly with sequence length

T

, whereas a two-layer Transformer with a single head per layer achieves the same approximation precision with at most

O (\varepsilon^{-1})

parameters. To understand this separation, we identify two structural mechanisms underlying multi-layer approximation. Specifically, softmax attention can only efficiently retrieve the token attaining the maximum attention score, incurring exponential-in-length parameter cost for

k

-th largest retrieval with

k \geq 2

. Moreover, the parameter cost of decoding coupled information scales with the size of the retrieved token set. Motivated by these findings, we propose InfoFlow, a framework for multi-layer Transformers. The framework tracks an information set of accessible input positions at each token and layer, assigning an explicit approximation rate to each mode of information propagation. This abstraction recovers known approximation bounds, remains consistent with experimental observations on trained networks, and yields concrete predictions in settings where direct theoretical analysis is currently intractable. Our results provide a principled framework for reasoning about the approximation efficiency of multi-layer Transformers.

Penghao Yu, Haotian Jiang, Zeyu Bao +1

Department of Mathematics National University of Singapore · Institute for Functional Intelligent Materials National University of Singapore · National University of Singapore +1

Transformer Expressivity Transformer Self-Attention Transformer Attention Neural Network Approximation Theory

May 14, 2026cs.LG

Universal Approximation of Nonlinear Operators and Their Derivatives

We show that Universal Approximation (UA) of nonlinear operators and their derivatives via Operator Learning (OL) architectures fails in

{C^k_F}

(Fréchet) compact-open topologies and in Fréchet--Sobolev norms (i.e. under operator norms). We solve this obstruction by restoring UA in natural weaker topologies:

C^k_B

(Bastiani) compact-open topologies and (novel) weighted Bastiani--Sobolev spaces for general finite input measures. In full Banach-space generality, these are the first complete generalizations of the corresponding influential classical results in [Hornik, 1991] to infinite-dimensional spaces and OL. Based on our UATs, we formulate Bastiani--Sobolev training in DIOL. These results launch Derivative-Informed Operator Learning (DIOL) (i.e. learning nonlinear operators and their derivatives) on general Banach spaces. We parameterize nonlinear operators via Encoder-Decoder Architectures, classical OL architectures available in general Banach spaces; these include DeepONets, Deep-H-ONets, and PCA-Nets, which our UATs cover. A key mathematical result is that our new weighted Bastiani--Sobolev spaces generalize classical Gaussian (Malliavin) Sobolev spaces on Banach spaces. Open frontiers where DIOL and our UATs find applications are: high-order accuracy in OL; fast constrained optimization in Banach spaces (e.g. optimal control of PDEs, inverse problems) via Learn-Then-Optimize; numerical methods for infinite-dimensional PDEs (e.g. HJB PDEs on Banach spaces from infinite-dimensional optimal control via Optimize-Then-Learn, such as optimal control of PDEs, SPDEs, path-dependent systems, partially observed systems, mean-field control).

Filippo de Feo

Institut für Mathematik, Technische Universität Berlin.

Neural Network Approximation Theory Operator Learning

May 13, 2026stat.ME

Wahkon: A Statistically Principled Deep RKHS Superposition Network

Deep learning excels at prediction but often lacks finite-sample guarantees and calibrated uncertainty; RKHS (Reproducing Kernel Hilbert Space)-based methods provide those guarantees but struggle to adapt in high dimensions. We propose Wahkon, a deep RKHS superposition network that unifies Kolmogorov's superposition principle with RKHS regularization in the smoothing-spline tradition of Wahba. This yields a finite-dimensional deep representer theorem that makes training tractable and provides explicit layerwise complexity control. We show the penalized estimator is exactly the MAP (maximum a posteriori) estimate under a hierarchical Gaussian-process prior, extending the spline/GP duality to deep compositions. Using metric-entropy arguments, we establish minimax-optimal convergence rates under mild smoothness and clarify how depth and width trade off with regularity. Empirically, Wahkon outperforms multilayer perceptrons, Neural Tangent Kernels, and Kolmogorov--Arnold Networks across simulation benchmarks and a single-cell CITE-seq study. By unifying Kolmogorov's superposition principle with RKHS regularization, Wahkon delivers accuracy, interpretability, and statistical rigor in a single framework.

Yongkai Chen, Wenxuan Zhong, Ping Ma

Department of Statistics, Harvard University · Department of Statistics, University of Georgia

Reproducing Kernel Hilbert Spaces Kernel Methods Neural Network Approximation Theory Statistical Learning Theory

May 12, 2026cs.LG

Approximation of Maximally Monotone Operators : A Graph Convergence Perspective

Operator learning has been highly successful for continuous mappings between infinite-dimensional spaces, such as PDE solution operators. However, many operators of interest-including differential operators-are discontinuous or set-valued, and lie outside classical approximation frameworks. We propose a paradigm shift by formulating approximation via graph convergence (Painlevé-Kuratowski convergence), which is well-suited for closed operators. We show that uniform and

L^p

approximation are fundamentally inadequate in this setting. Focusing on maximally monotone operators, we prove that any such operator can be approximated in the sense of local graph convergence by continuous encoder-decoder architectures, and further construct structure-preserving approximations that retain maximal monotonicity via resolvent-based parameterizations.

Takashi Furuya, Yury Korolev, Takaharu Yaguchi

Doshisha University, RIKEN AIP · Department of Mathematical Sciences, University of Bath, UK · Kyushu University, RIKEN AIP

Neural Network Approximation Theory Operator Learning

May 12, 2026cs.LG

Lower bounds for one-layer transformers that compute parity

This note shows that no self-attention layer post-processed by a rational function can sign-represent the parity function unless the product of the number of heads and the degree of the post-processing function grows linearly with the input length. Combining this lower bound with rational approximation of ReLU networks yields a margin-dependent extension for self-attention layers post-processed by ReLU networks.

Daniel Hsu

Transformer Expressivity Self-Attention Transformer Attention Neural Network Approximation Theory Multi-Head Attention Neural Network Expressivity

May 12, 2026cs.LG

Approximation Theory of Laplacian-Based Neural Operators for Reaction-Diffusion System

Neural operators provide a framework for learning solution operators of partial differential equations (PDEs), enabling efficient surrogate modeling for complex systems. While universal approximation results are now well understood, approximation analysis specific to nonlinear reaction-diffusion systems remains limited. In this paper, we study neural operators applied to the solution mapping from initial conditions to time-dependent solutions of a generalized Gierer-Meinhardt reaction-diffusion system, a prototypical model of nonlinear pattern formation. Our main results establish explicit approximation error bounds in terms of network depth, width, and spectral rank by exploiting the Laplacian spectral representation of the Green's function underlying the PDE. We show that the required parameter complexity grows at most polynomially with respect to the target accuracy, demonstrating that Laplacian eigenfunction-based neural operator architectures alleviate the curse of parametric complexity encountered in generic operator learning. Numerical experiments on the Gierer-Meinhardt system support the theoretical findings.

Takashi Furuya, Ryo Ozawa, Jenn-Nan Wang

Doshisha University · RIKEN AIP · Tohoku University +1

PDE Operator Learning Neural Network Approximation Theory Neural Operators

May 12, 2026stat.ML

Posterior Contraction Rates for Sparse Kolmogorov-Arnold Networks in Anisotropic Besov Spaces

We study posterior contraction rates for sparse Bayesian Kolmogorov-Arnold networks (KANs) over anisotropic Besov spaces, providing a statistical foundation of KANs from a Bayesian point of view. We show that sparse Bayesian KANs equipped with spike-and-slab-type sparsity priors attain the near-minimax posterior contraction. In particular, the contraction rate depends on the intrinsic anisotropic smoothness of the underlying function. Moreover, by placing a hyperprior on a single model-size parameter, the resulting posterior adapts to unknown anisotropic smoothness and still achieves the corresponding near-minimax rate. A distinctive feature of our results, compared with those for standard sparse MLP-based models, is that the KAN depth can be kept fixed: owing to the flexibility of learnable spline edge functions, the required approximation complexity is controlled through the network width, spline-grid range and size, and parameter sparsity. Our analysis develops theoretical tools tailored to sparse spline-edge architectures, including approximation and complexity bounds for Bayesian KANs. We then extend to compositional Besov spaces and show that the contraction rates depend on layerwise smoothness and effective dimension of the underlying compositional structure, thereby effectively avoiding the curse of dimensionality. Together, the developed tools and findings advance the theoretical understanding of Bayesian neural networks and provide rigorous statistical foundations for KANs.

Jeunghun Oh, Kyeongwon Lee, Jaeyong Lee +1

Department of Statistics, Seoul National University · Department of Statistics, Inha University · Department of Mathematics, University of Maryland

Bayesian Neural Networks Kolmogorov-Arnold Networks Neural Network Approximation Theory Sparse Neural Networks

May 11, 2026cond-mat.dis-nn

Exact Fixed-Point Constraints in Neural-ODEs with Provable Universality

We introduce a technique that enables Neural-ODEs to approximate arbitrary velocity fields with a priori planted fixed-points. Specifically, a recipe is given to explicitly accommodate for a finite collection of points in the reference multi-dimensional space of the Neural-ODE where the velocity field is exactly equal to zero. In this way, the gradient-based training is rigorously constrained inside the prescribed hypothesis class while leaving the expressive power of the Neural-ODE unaltered. We rigorously prove the universality of the Neural-ODE under any local constraints in the velocity field and give a computationally convenient way of imposing the fixed points. Our method is then tested on two paradigmatic physical models.

Feliciano Giuseppe Pacifico, Duccio Fanelli, Lorenzo Buffoni +3

Department of Informatics and Computer Science, University of Pisa, Italy · Department of Physics and Astronomy and INFN, University of Florence, Sesto Fiorentino, Italy

Constrained Optimization Neural Network Approximation Theory Neural ODEs

May 10, 2026cs.LG

Minimal Filling Architectures of Polynomial Neural Networks: Counterexamples, Frontier Search, and Defects

We provide counterexamples to the unimodal minimal filling architecture conjecture for polynomial neural networks (PNNs) with power activation functions. Fixing the input and output widths, the conjecture states that any minimal filling architecture has unimodal widths for the hidden layers. We found counterexamples via a frontier search, recursive dimension bounds on neurovarieties, and symbolic computation. Notably, several subarchitectures of our main example exhibit large defect, in contrast with the predominantly small-defect behavior observed in prior literature.

Kevin Dao, Jose Israel Rodriguez

Department of Mathematics, University of Wisconsin-Madison, Wisconsin, USA.

Polynomial Neural Networks Neural Network Approximation Theory Neural Network Expressivity

May 9, 2026stat.ML

Learning Theory of Transformers: Local-to-Global Approximation via Softmax Partition of Unity

This paper investigates the learning theory of Transformer networks for regression tasks on the compact Euclidean domain

[0,1]^d

and

d

-dimensional compact Riemannian manifolds. We propose a novel constructive approximation framework for Transformers that builds local approximations of the target function and aggregates them into a global approximation via softmax partition of unity. This approach leverages the attention mechanism to achieve spatial localization through affine transformations of the input. The softmax activation plays a crucial role in aggregating local approximations to a global output. From an approximation perspective, we prove that a dense Transformer equipped with only two encoder blocks and standard single-hidden-layer point-wise feed-forward networks can achieve a uniform

\varepsilon

-approximation error for

α

-Hölder continuous functions with

α\in (0,1]

using

\mathcal{O}(\varepsilon^{-d/α})

total parameters. Building upon this approximation guarantee, we establish a near minimax-optimal generalization error bound of order

\mathcal{O}\big(n^{-\frac{2α}{2α+d}} \log n\big)

for the empirical risk minimizer, where

n

is the training data size. The Transformer architecture studied in this paper is dense, shallow and wide, and employs softmax activation and sinusoidal positional encodings, closely reflecting practical implementations.

Zhongjie Shi, Wenjing Liao

School of Mathematics, Georgia Institute of Technology, Atlanta, GA 30332, United States

Neural Network Generalization Transformer Neural Network Approximation Theory Statistical Learning Theory

May 8, 2026math.FA

Structure-Preserving Reconstruction of Convex Lipschitz Functionals on Hilbert Spaces from Finite Samples

Convex functionals are ubiquitous in applied analysis, appearing as value functions, risk measures, super-hedging prices, and loss functionals in machine learning. In many applications, however, the functional is only observed through finitely many exact pointwise evaluations. We ask whether a convex functional on a separable Hilbert space

H

can be reconstructed, up to arbitrary uniform accuracy, by an explicit formula which preserves convexity and Lipschitz regularity and is finitely computable. We answer this affirmatively. For every compact convex

C\subseteq H

, every

L

-Lipschitz convex functional

ρ:C\to\mathbb{R}

, and every

\varepsilon>0

, we construct an explicit finite-sample reconstruction which is convex,

L

-Lipschitz, and uniformly

\varepsilon

-accurate on

C

. The construction uses only finitely many linear measurements

\langle b,\cdot\rangle_H

, with

b

lying in a finite-dimensional subspace of

H

, and is exactly implementable by a

\operatorname{ReLU}

-MLP. Building on this, we introduce convex neural functionals (CNFs), a structured trainable architecture class containing our reconstruction, whose every admissible parameter configuration is automatically convex and Lipschitz, providing a principled foundation for learning convex functionals from finite data.

Anastasis Kratsios

Input Convex Neural Networks Neural Network Approximation Theory

May 8, 2026cs.LG

A Deep Risk Estimator for Known Operator Learning

We describe an approach for estimating the statistical risk of deep networks that contain a mix of learned and known operators. Building on the maximal training error bounds previously established for known operator learning, we derive a deep risk estimator that connects the expected error of a layered network to the size of the training sample. The estimator decomposes the total risk into a sum over learned layers; every known operator contributes zero to this sum, while every learned layer adds an approximation term inspired by Barron's classic work and an estimation term that decreases with the number of training samples. We are able to show that the bound shrinks whenever a learned layer is replaced by a known operator and that the corresponding sample requirement scales with the number of trainable parameters of the layer that is replaced. As an application, we use computed tomography as an example and compare an operator-aware filtered backprojection network with a fully connected substitute that collapses the entire reconstruction pipeline into a single learned dense matrix. The predicted parameter ratio coincides with the structural sparsity that the analytic decomposition into a circulant filter and a sparse backprojection exposes. We confirm the predicted scaling on CPU at small image scale and on GPU at medium image scale, all on the same scaling law. Beyond CT reconstruction, the estimator applies to physics-informed neural networks that hardcode a known physical operation in its architecture, and we expect the result to be of interest for a broad community working on operator-aware deep learning. Calibrating the per-layer constants on each sweep yields a bound that tracks the empirical test MSE within a factor of two at every training-set size, so the estimator can be inverted to predict how many training samples are required to reach a target error.

Andreas Maier, Md Hasan, Paulina Conrad +1

Pattern Recognition Lab · Friedrich-Alexander-Universität Erlangen-Nürnberg · Erlangen, Germany

Neural Network Generalization Neural Network Approximation Theory Operator Learning Statistical Learning Theory

May 8, 2026cs.LG

Embedding Dimension Lower Bounds for Universality of Deep Sets and Janossy Pooling

In many practical applications it is important to build symmetries into neural network architectures. Consider the important case of permutation symmetry on point clouds consisting of

n

points in

d

dimensions. In this case the network learns a function on a set of

n

points in

\mathbb{R}^d

, and a natural paradigm for constructing invariant networks is Janossy pooling, which generalizes the popular Deep Sets architecture. We study the universality of this approach, in particular the important question of how large the embedding dimension must be to guarantee universality of this architecture. Specifically, using a novel technique, we prove new lower bounds on the required size of this embedding dimension. For Deep Sets, this gives the correct minimal dimension up to a constant factor for all

d > 1

. For

k

-ary Janossy pooling, we prove the first non-trivial lower bound on the required embedding dimension when

k > 1

.

Ali Syed, Aditya Nambiar, Jonathan W. Siegel

Department of Mathematics Texas A&M University College Station, TX 77843

Representation Learning Deep Sets Universal Approximation Neural Network Approximation Theory Invariant Representation Learning

May 8, 2026cs.LG

Approximation Error Upper and Lower Bounds for Hölder Class with Transformers

We explore the expressive power of Transformers by establishing precise approximation error upper and lower bounds for Hölder class. Specifically, a new approximation upper bound is derived for the standard Transformer architecture equipped with Softmax operators, ReLU activation functions, and residual connections. We prove that a Transformer network composed of at most

\mathcal{O}(\varepsilon^{-{d_{0}}/α})

blocks can approximate any bounded Hölder function with

d_{0}

-dimensional input and smoothness

α\in(0,1]

under any accuracy

\varepsilon>0

. In the case of approximation lower bounds, leveraging the VC-dimension upper bound, we are the first to rigorously prove that Transformers demand for at least

Ω(\varepsilon^{-{d_{0}}/({4α})})

blocks to achieve the

\varepsilon

approximation accuracy. As a final step, we extend the derived results for standard Transformers to a general regression task and establish the corresponding excess risk rates demonstrating Transformers' empirical effectiveness in real-world settings.

Xin He, Yuling Jiao, Xiliang Lu +1

School of Mathematics and Statistics, Wuhan University, Wuhan, China · School of Artificial Intelligence, Wuhan University, Wuhan, China · Hubei Key Laboratory of Computational Science, Wuhan University, Wuhan, China

Transformer Neural Network Approximation Theory

May 7, 2026cs.LG

Region Seeding via Pre-Activation Regularization: A Geometric View of Piecewise Affine Neural Networks

Deep networks with continuous piecewise affine activations induce polyhedral partitions of the input space, making the number of realized affine regions a natural measure of expressive capacity and a key determinant of how well the model can approximate nonlinear target functions. In practice, standard training realizes far fewer region refinements in data-visited neighborhoods than the architecture could in principle support, while existing region-count theory is primarily architectural and offers little guidance on how optimization shapes the realized partition near the data. Our theory provides a sufficient condition under which bringing neuron switching surfaces sufficiently close to data points ensures their intersection with local neighborhoods, which in turn implies a strict increase in the local affine-region count, yielding a principled training-time handle for seeding data-relevant partitions early in optimization. Guided by these results, we propose a plug-and-play region-seeding regularizer that encourages early partitioning while allowing task-driven refinement to dominate later in training. Experiments show that the regularizer increases the number of realized affine regions via exact enumeration and improves overall performance on toy datasets, while also improving early-stage accuracy and achieving comparable (or slightly improved) final accuracy on ImageNet-1k for classical models.

Yi Wei, Xuan Qi, Furao Shen

DITEN University of Genoa, Genoa, Italy · State Key Laboratory of Novel Software Technology School of Artificial Intelligence Nanjing University, Jiangsu, China

Neural Network Activation Functions Neural Network Approximation Theory Neural Network Training Dynamics Neural Network Expressivity

May 7, 2026stat.ML

ConquerNet: Convolution-Smoothed Quantile ReLU Neural Networks with Minimax Guarantees

Quantile regression is a fundamental tool for distributional learning but poses significant optimization challenges for deep models due to the non-smoothness of the pinball loss. We propose ConquerNet, a class of \textbf{con}volution-smoothed \textbf{qu}antil\textbf{e} \textbf{R}eLU neural \textbf{net}works, which yield smooth objectives while preserving the underlying quantile structure. We establish general nonasymptotic risk bounds for ConquerNet under mild conditions, providing minimax guarantees over Besov function classes. In numerical studies, we demonstrate that the proposed approach outperforms standard quantile neural networks at multiple quantile levels, showing improved estimation accuracy and training efficiency across the board, with particularly pronounced advantages at high and low quantiles.

Tianpai Luo, Fangwei Wu, Weichi Wu

Department of Statistics and Data Science Tsinghua University Beijing, 100084, China

Quantile Regression Minimax Estimation Neural Network Approximation Theory

May 7, 2026cs.LG

AffineLens: Capturing the Continuous Piecewise Affine Functions of Neural Networks

Piecewise affine neural networks (PANNs) provide a principled geometric perspective on neural network expressivity by characterizing the input--output map as a continuous piecewise affine (CPA) function whose complexity is governed by the number, arrangement, and shapes of its affine regions. However, existing interpretability and expressivity analyses often rely on indirect proxies (e.g., activation statistics or theoretical upper bounds) and rarely offer practical, accurate tools for enumerating and visualizing the induced region partition under realistic architectures and bounded input domains. In this work, we present AffineLens, a unified framework for computing the hyperplane arrangements and polyhedral structures underlying PANNs. Given a calibrated (bounded) input polytope, AffineLens identifies the subset of neuron-induced hyperplanes that intersect the domain, enumerates the resulting affine sub-regions in a layer-wise manner, and returns provably non-empty maximal CPA regions together with interior representatives. The framework further provides visualizations of region partitioning and decision boundaries, enabling qualitative inspection alongside quantitative region counts. By exploiting the affine restriction property of CPA networks under fixed activation patterns, AffineLens supports a broad class of modern components, including batch normalization, pooling, residual connections, multilayer perceptrons, and convolutional layers. Finally, we use AffineLens to perform a systematic empirical study of architectural expressivity, comparing networks through region complexity metrics and revealing how design choices influence the geometry of learned functions.

Yi Wei, Xuan Qi, Furao Shen +3

State Key Laboratory of Novel Software Technology School of Intelligence Science and Technology Nanjing University, Jiangsu, China · AI for Good Istituto Italiano di Tecnologia, Genoa, Italy · DITEN University of Genoa, Genoa, Italy +2

Neural Network Interpretability Neural Network Approximation Theory Neural Network Expressivity

May 7, 2026cs.LG

Structural Correspondence and Universal Approximation in Diagonal plus Low-Rank Neural Networks

The massive computational costs of scaling modern deep learning architectures have driven the widespread use of parameter-efficient low-rank structures, such as LoRA and low-rank factorization. However, theoretical guarantees for their expressive power are less explored, often relying on restrictive priors like a pretrained base matrix, ReLU activations or non-verifiable singularity conditions. We first investigate the limits of neural networks constrained strictly to low-rank manifolds without pretrained dense priors. We demonstrate a theoretical paradox: while purely rank-1 layers can exactly interpolate arbitrary scalar datasets, they collapse for function approximations. To overcome this bottleneck without surrendering parameter efficiency, we introduce a unified \textit{Structural Correspondence} framework. We prove that augmenting low-rank layers with only a minimal sparse diagonal component, say a Diagonal plus Low-Rank (DLoR) structure, is sufficient to reach Universal Approximation. We show that any full-rank transformation can be exactly reconstructed using these DLoR components by trading off network width (additive decomposition) or depth (multiplicative decomposition). By tracking asymptotic Taylor remainders, we prove that DLoR neural networks fully restore the Universal Approximation Theorem for general activation functions. Finally, we establish that multiplicative depth provides superior parameter-to-expressivity scaling compared to additive width. Our results show that dense matrices and specific activation functions are not topological prerequisites for universal expressivity.

Ying Chen, Aoxi Li, Jihun Kim +1

Department of Industrial Engineering & Operations Research University of California, Berkeley, CA 94720

Low-Rank Matrix Decomposition Low-Rank Approximation Neural Network Approximation Theory Neural Network Expressivity

May 6, 2026cs.LG

Adaptivity Under Realizability Constraints: Comparing In-Context and Agentic Learning

We compare in-context learning with fixed queries and agentic learning with adaptive queries for uniform approximation of task families. We consider two settings: an unrestricted regime, where querying and approximation are arbitrary functions, and a realizable regime, where we require these operations to be implemented by ReLU neural networks. In both settings, adaptivity never hinders approximation performance. However, this advantage can change when one passes from the unrestricted regime to the realizable regime. We identify four distinct approximation scenarios, each witnessed by an explicit task family: (a) no advantage of adaptivity; (b) an advantage in the unrestricted regime that persists under ReLU realizability; (c) an advantage that arises only under realizability; and (d) an advantage that disappears under realizability. This demonstrates that representational constraints interact profoundly with the effect of adaptivity.

Anastasis Kratsios, A. Martina Neuman, Philipp Petersen

McMaster University and the Vector Institute, Main St., Hamilton, L8S 4L8, Ontario, Canada · University of Vienna, Faculty of Mathematics, Kolingasse 14-16, 1090 Wien, Austria

ReLU Neural Networks In-Context Learning Neural Network Approximation Theory Adaptive Learning