Constructive Universal Approximation by Infinite-Dimensional Neural Networks via Gaussian-Weighted Ridgelet Transform

Sho Sonoda; Luca Galimberti; Yuanchao Xu; Isao Ishikawa; Masahiro Ikeda

cs.LGOct 16, 2026

Constructive Universal Approximation by Infinite-Dimensional Neural Networks via Gaussian-Weighted Ridgelet Transform

Authors: Sho Sonoda, Luca Galimberti, Yuanchao Xu, Isao Ishikawa, Masahiro Ikeda

Organizations: RIKEN Center for Advanced Intelligence Project (AIP), Nihonbashi 1-chome Mitsui Building, 1-4-1 Nihonbashi, Chuo-ku, Tokyo 103-0027, Japan · AI tech studio, CyberAgent, Inc., Shibuya Scramble Square, 2-24-12 Shibuya, Shibuya-ku, Tokyo 150-6121, Japan · Department of Mathematics, King’s College London, Strand Building, London, WC2R 2LS, UK · Department of Mathematics, Graduate School of Science, Kyoto University, Kyoto, 606-8502, Japan ·

Abstract

Neural networks whose inputs are functions, as in operator learning, are usually shown to be universal by existence arguments, either by reduction to finite-dimensional networks or by Stone--Weierstrass-type density; these prove that approximants exist but say little about how the parameters depend on the target. We take a constructive route. We extend the ridgelet transform, an integral transform related to the Fourier, Radon, and wavelet transforms, to continuous maps on a separable infinite-dimensional Hilbert space, replacing the Lebesgue measure that is unavailable in infinite dimensions by Gaussian weights with a homogeneous mixture on the direction space. The resulting transform has a Plancherel identity and reconstruction formulas incorporating the frame operator, the correction induced by the weights, and it assigns to each target an explicit coefficient measure, one for every admissible analysis filter. From these representations we obtain uniform approximation on compact sets for all continuous non-polynomial activations of polynomial growth, including ReLU, and, by Monte Carlo sampling of the coefficient, finite-width networks with a dimension-free $O(N^{-1/2})$ error in the width $N$ for globally Lipschitz activations and coefficients with finite variation and second moments. Explicit examples include Gaussian targets and neural-operator layers. A companion Lean~4 formalization provides machine-checked proofs, with its correspondence to the manuscript documented at https://shosonoda.github.io/lean-operator-ridgelet/ .

Figures & tables

Experiment	$d$	error at $N=4$	error at $N=4096$	slope
1: ReLU, Gaussian directions on $\ell^{2}$	$10$	$0.41$	$0.011$	$-0.53$
	$10^{2}$	$0.37$	$0.012$	$-0.50$
	$10^{3}$	$0.42$	$0.011$	$-0.51$
	$10^{4}$	$0.32$	$0.010$	$-0.50$
2: ridgelet coefficient of $f_{W}$ , target $6\,f_{g}$	$10$	$11.9$	$0.46$	$-0.47$
	$10^{2}$	$11.6$	$0.43$	$-0.48$

Table 1: Mean supremum error on

K

of the sampled networks at

N=4

and

N=4096

, and the fitted slope of the error against

N

in logarithmic coordinates, for each truncation dimension

d

Panel of Figure 1 — Figure 1: Mean supremum error on $K$ against the width $N$ for Experiments 1 (upper left), 2 (upper right), and 3(a) (bottom), for several truncation dimensions $d$ ; the dashed line is proportional to $N^{-1/2}$ .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Reference	Target class	Approximant	Norm or topology
Chen and Chen [19]	continuous functionals on a compact $K\subset C[a,b]$ or $L^{p}[a,b]$	one hidden layer, Tauber–Wiener activation applied to finitely many point values or integrals of the input	uniform on $K$
Chen and Chen [20]	continuous operators $G\colon V\to C(K_{2})$ , $V\subset C(K_{1})$ compact	branch–trunk network (the DeepONet form), Tauber–Wiener activations	uniform on $V\times K_{2}$
Stinchcombe [91]	continuous functions on compactifications; functionals through the same structure	one hidden layer, activation characterized by the compactification	uniform on the compactification
Rossi and Conan-Guez [81]	continuous functionals on a compact $K\subset L^{2}(V)$	one hidden layer of functional neurons $\rho(\int wx+b)$ , regular activation	uniform on $K$
Benth et al. [9]	continuous $f\colon X\to\mathbb{R}$ , $X$ Fréchet	one hidden layer $\sum_{i}\langle\ell_{i},\sigma(A_{i}x+b_{i})\rangle$ , operator activation $\sigma\colon X\to X$ bounded and separating	cc
Galimberti [32] (Thm. 4.2)	continuous functions on quasi-Polish spaces	same architecture composed with an injection into $\ell^{2}$ ; finite-rank projections (Thm. 4.3)	cc

Appendix

Table 2: Universal approximation with infinite-dimensional inputs: functionals, Fréchet and topological inputs, weighted spaces, and signatures.

Reference	Target class	Approximant	Norm or topology
Kovachki et al. [53]	continuous operators between Sobolev spaces on the torus, on compact input sets	Fourier neural operator	uniform on compact sets
Lanthaler et al. [65]	continuous operators between function spaces on Euclidean domains, on compact input sets	averaging neural operator (one spatial average plus pointwise nonlinearity)	uniform on compact sets
Gödeke and Fernsel [35]	continuous operators between normed or metric spaces satisfying encoder–decoder approximation properties	encoder–decoder architectures (DeepONet, BasisONet, MIONet, frame based)	cc; also compact-set-independent choice of the architecture
Yao et al. [100] (Thms. 3.2, 4.3)	$C^{1}$ operators between Sobolev spaces on the torus whose derivative is continuous into Hilbert–Schmidt operators	Fourier neural operator with smooth activation	sup over a compact set of the operator error and of the Hilbert–Schmidt derivative error; $L^{2}_{\mu}$ of both for a Borel probability $\mu$ with moment conditions
de Feo [25] (Thms. 4.3, 4.6; Cor. 4.19)	$C^{k}$ operators together with derivatives, on Banach spaces	encoder–decoder classes (DeepONet, Deep-H-ONet, Schauder nets)	not dense in Fréchet $C^{k}$ compact-open or Fréchet–Sobolev norms, already for bounded linear targets; dense in Bastiani $C^{k}$ compact-open (derivatives uniform on compacts, directions from compacts), weighted Bastiani–Sobolev, and Gaussian Sobolev $W^{k,p}_{\gamma}$ norms
This paper	continuous $f\colon\mathcal{H}\to\mathcal{Y}$ , $\mathcal{H}$ separable Hilbert, $\mathcal{Y}$ separable Hilbert	one hidden layer $\sum_{j}v_{j}\sigma(\langle a_{j},x\rangle-b_{j})$ , continuous non-polynomial $\sigma$ of polynomial growth; coefficients from the ridgelet transform of a spectral approximant	cc

Appendix

Table 3: Universal approximation with infinite-dimensional inputs: neural operators, derivatives, and this paper.

Reference	Target class	Approximant	Norm	Bound
Barron [5]	$f$ on a ball in $\mathbb{R}^{d}$ with $\int\|\xi\|\|\widehat{f}(\xi)\|\,\mathrm{d}\xi\leq C$	one hidden layer, sigmoidal, width $N$	$L^{2}(\mu)$ on the ball	$O(C/\sqrt{N})$ , no explicit $d$
Mhaskar and Hahm [72]	smooth functionals on compact subsets of $L^{p}$ or $C([-1,1]^{s})$	generalized translation networks	uniform	order bounds in the number of neurons, with matching lower bounds
Song et al. [85] (Thms. 1, 2)	functionals on $L^{p}([-1,1]^{s})$ with $\omega_{F}(r)\leq cr^{\lambda}$ , on the unit ball $K$ of $C^{\beta}$ (Thm. 2)	functional deep ReLU network: a bounded linear first layer followed by ReLU layers, $M$ nonzero weights, depth $O(L_{M}^{2})$	sup over $K$	$O(L_{M}^{-\beta\lambda/s})$ with $L_{M}=\log M/\log\log M$
Gonon et al. [36] (Thm. 1)	$f$ on a separable Hilbert space with a Fourier–Stieltjes representation, second moments, and a density w.r.t. the sampling measure	random ReLU features, width $N$ , trained readout	expected squared $L^{2}$ error over the input law	$O(N^{-1})$ for the squared error
Korolev [52] (Thm. 3.24)	variation space $F_{1}(X;Y)$ of maps between duals of separable Banach spaces with Schauder bases, Lipschitz for the weak- $*$ metric on $Y$	$\sum_{i=1}^{n}\alpha_{i}(K_{i}x)_{+}$ with finite-rank $K_{i}$ , $\\|K_{i}\\|\leq 1$	weak- $*$ metric on $Y$ , pointwise with factor $\\|x\\|$ ; Bochner $L^{p}_{\mu}$ for $\mu$ with $p$ moments	$n^{-1/2}$ times $c\\|f\\|_{F_{1}}\\|x\\|$ ; inverse theorem
Neufeld and Schmocker [78] (Thm. 4.5; Cor. 4.21)	Barron space $\mathcal{B}^{r}_{G,\theta}(X)$ of a separable Banach space of type $t$ ; in Cor. 4.21, a Barron–ridgelet class on $U\subset\mathbb{R}^{m}$ defined by an admissible filter pair	random feature model with $N$ features; random neural network with $N$ neurons	Bochner $L^{r}$ in the $X$ -norm; weighted Sobolev $W^{k,p}(U,w)$	$N^{-\kappa}$ , $\kappa=1-1/\min(r,t)$ , so $N^{-1/2}$ for type $2$ , $r\geq 2$ ; $\kappa=1-1/\min(2,p,r)$ in Cor. 4.21

Appendix

Table 4: Quantitative upper bounds for functionals and for networks with Hilbert or Banach inputs, as rates in the width

N

Reference	Target class	Approximant	Norm	Bound
Lanthaler et al. [64]	PDE solution operators; general error decomposition	DeepONet	$L^{2}(\mu)$ over the input law	encoder, approximator, reconstruction errors; algebraic rates in examples
Furuya et al. [31] (Thm. 1)	solution operator of a nonlinear parabolic equation on an $L^{\infty}$ ball, short time	weight-tied ReLU neural operator, Green-function expansion of rank $N$	sup over the ball of $L^{r}(0,T;L^{s})$	depth $O((\log\varepsilon^{-1})^{2})$ , width $O(\varepsilon^{-1}(\log\varepsilon^{-1})^{2})$ , $N$ by the Green-function error
Kratsios et al. [58]	uniformly continuous, in particular Lipschitz, operators on compact Sobolev sets	mixture of neural operators routed by a tree	uniform on the compact set	active expert of size $O(\varepsilon^{-1})$ for Lipschitz targets
Adcock et al. [3] (Thm. 4.1)	Lipschitz $F\in L^{2}_{\mu}(X;Y)$ , $\mu$ Gaussian, whose PCA latent representation lies in a Gaussian Sobolev space $H^{k}$	Hermite–PCA approximation from $N_{X},N_{Y}$ PCA samples and $M$ input–output samples	$L^{2}_{\mu}(X;Y)$	PCA tails, sampling terms, and a term improving with the Sobolev order $k$

Appendix

Table 5: Quantitative upper bounds for neural operators and surrogates of operators.

Reference	Target class	Approximant	Norm	Bound
Lanthaler and Stuart [63] (Thm. 2.11, Cor. 2.12)	$r$ -times Fréchet differentiable functionals or operators on a compact $K$ containing an infinite-dimensional hypercube $Q_{\alpha}$	functionals of neural-network type $\Phi\circ L$ , $L$ finite-rank linear, $\Phi$ a ReLU network (covers PCA-Net, DeepONet, NOMAD, FNO)	sup over $K$	complexity $\geq\exp(c\varepsilon^{-\kappa})$ , $\kappa=1/((\alpha+1+\delta)r)$
Lanthaler [62]	Lipschitz operators between Banach spaces	any architecture, size measured in bits	uniform on a compact set; expectation over an input law	size exponential in $\varepsilon^{-1}$ , for every activation
Kovachki et al. [55]	Lipschitz and Fréchet-differentiable operators	any method using $n$ input–output samples ( $n$ -widths)	as stated there	$n$ exponential in $\varepsilon^{-1}$
Adcock et al. [2]	uniformly bounded Lipschitz (or Hölder) operators between separable Hilbert spaces	any estimator from $n$ noisy samples (minimax)	$L^{2}$ risk over an input law with given covariance spectrum	no algebraic rate in $n$ ; sharp for exponentially decaying spectra

Appendix

Table 6: Lower bounds: no algebraic rate exists for the class.

Neural Network Approximation Theory

Neural network approximation theory studies how network size, depth, activations, and structure determine the accuracy of functions and operators.

See all 227 papers

Sep 14, 2026

Approximating Smooth Functionals with ReLU Networks

We study the uniform approximation of smooth scalar-valued functionals on an infinite-dimensional separable Hilbert space by ReLU neural networks. A key feature in deep learning for functional data is the varying importance of different coordinates/dimensions. Representing the functional input in a basis expansion, we quantify the importance of each coordinate through both the magnitude of its corresponding basis score and the directional sensitivity of the target functional. Our analysis combines coordinate truncation, anisotropic partitioning, local Taylor approximation, and ReLU network realization, while allowing unrestricted interactions among the retained coordinates. We establish a general nonasymptotic upper bound for the uniform approximation error and a complementary pseudo-dimension-based lower bound for the worst-case approximation error. Under generalized exponential coordinate decay

w_ds_d\asymp\exp(-cd^ρ)

, with

ρ>0

, the upper and lower bounds match at the leading order, which is stretched-exponential in the logarithm of the network size budget, and thus yield the nearly optimal approximation rate. This is the first work to characterize neural network approximation error for infinite-dimensional functional inputs explicitly through the joint dimensional decay of coordinate magnitudes and directional sensitivities.

Shuhao Jiao

Department of Biostatistics, City University of Hong Kong, Hong Kong SAR, China.

ReLU Neural Networks Neural Network Approximation Theory

Ask PDF

Aug 21, 2026

Resolution-Consistent Greedy Neural Approximation on Infinite-Dimensional Spaces

We develop constructive approximation and learning guarantees for shallow neural models with infinite-dimensional inputs observed through finitely many coordinates. The analysis is based on a parameter-normalized neural dictionary and its associated weighted variation class. Within this class, the approximation error separates into a distribution-dependent coordinate-truncation term and a greedy finite-width term. For empirical regression, a fully-corrective greedy procedure yields population guarantees whose statistical complexity is uniform in the retained input resolution. The same framework extends to Hilbert-valued responses without an explicit dependence on the output dimension. The dimension-free statements are statistical, not computational: selecting a new neuron still requires solving a nonconvex parameter-search problem. The quasi-Polish construction underlying recent infinite-dimensional universal approximation results provides a motivating example, and synthetic experiments illustrate the predicted resolution, width, and sample-size regimes.

Pablo M. Berná, Antonio Falcó, Diego Mondéjar

Universidad Tecnol´ogica Atl´antico Mediterr´aneo – UTAMED, Facultad de Empresa Digital, Tecnolog´ıa y Derecho, M´alaga, Espa˜na · Departamento de Matem´aticas, F´ısica y Ciencias Tecnol´ogicas, Universidad Cardenal Herrera-CEU, CEU Universities, San Bartolom´e 55, Alfara del Patriarca (Valencia), 46115, Spain · Departamento de Matem´aticas, CUNEF Universidad, 28040, Espa˜na

Neural Network Approximation Theory Function Approximation Shallow Neural Networks Statistical Learning Theory

Ask PDF

Mar 31, 2025

New universal operator approximation theorem for encoder-decoder architectures

Motivated by the rapidly growing field of mathematics for operator approximation with neural networks, we present a novel universal operator approximation theorem for broad classes of encoder-decoder architectures and a wide range of input and output spaces. In this study, we focus on the approximation of continuous operators between infinite-dimensional normed or metric spaces in the topology of uniform convergence on compact sets. Unlike standard results in the operator learning literature, we additionally investigate the case where the approximating sequence of encoder-decoder architectures can be chosen independently of the compact sets. Taking a topological perspective, we point out that compact-set-independent approximation is a strictly stronger property in most relevant operator learning frameworks. To establish our results, we introduce new approximation properties of input and output spaces tailored to encoder-decoder architectures. These properties enable us to prove a universal operator approximation theorem ensuring uniform convergence on every compact subset of the input space. Our results unify and extend existing universal operator approximation theorems for various encoder-decoder architectures, including classical DeepONets, BasisONets, MIONets, architectures based on frames and other related approaches. A notable feature of our framework is that it also applies to metric spaces beyond the normed setting. In particular, it allows the consideration of

p

-Wasserstein spaces of probability measures as input or output spaces, and Skorohod spaces of càdlàg functions as input spaces. This generality also opens up potential applications in optimal transport.

Janek Gödeke, Pascal Fernsel

Center for Industrial Mathematics University of Bremen

Neural Network Approximation Theory Deep Operator Networks Operator Learning

Ask PDF

Abstract

Neural Network Approximation Theory

Explore similar work

Approximating Smooth Functionals with ReLU Networks

Resolution-Consistent Greedy Neural Approximation on Infinite-Dimensional Spaces

New universal operator approximation theorem for encoder-decoder architectures