Sparse-Aware Neural Networks for Nonlinear Functionals: Mitigating the Exponential Dependence on Dimension

Jianfei Li; Shuo Huang; Han Feng; Ding-Xuan Zhou; Gitta Kutyniok

cs.LGApr 8, 2026

Sparse-Aware Neural Networks for Nonlinear Functionals: Mitigating the Exponential Dependence on Dimension

Authors: Jianfei Li, Shuo Huang, Han Feng, Ding-Xuan Zhou, Gitta Kutyniok

Organizations: Department of Mathematics, Ludwig-Maximilians-Universität München (LMU Munich), Akademiestr. 7, 80799 München · Department of Mathematics, City University of Hong Kong, Hong Kong, China · School of Mathematics and Statistics, The University of Sydney, Sydney, New South Wales 2006, Australia · Munich Center for Machine Learning (MCML), Germany · University of Tromsø, Norway · German Aerospace Center (DLR), Germany

Abstract

Deep neural networks have emerged as powerful tools for learning operators defined over infinite-dimensional function spaces. However, existing theories frequently encounter difficulties related to dimensionality and limited interpretability. This work investigates how sparsity can help address these challenges in functional learning, a central ingredient in operator learning. We propose a framework that employs convolutional architectures to extract sparse features from a finite number of samples, together with deep fully connected networks to effectively approximate nonlinear functionals. Using universal discretization methods, we show that sparse approximators enable stable recovery from discrete samples. In addition, both the deterministic and the random sampling schemes are sufficient for our analysis. These findings lead to improved approximation rates and reduced sample sizes in various function spaces, including those with fast frequency decay and mixed smoothness. They also provide new theoretical insights into how sparsity can alleviate the curse of dimensionality in functional learning.

Figures & tables

Figure 1: Functional learning pipeline proposed in Theorem 7 through functional neural networks.

Input functions	Encoder	Rate
$W_{p}^{\alpha}(\mathbb{S}^{d-1})$ ( Yang et al., 2026 )	Spherical harmonics	$\tilde{O}\left((\log K)^{-\beta\alpha/d-1}\right)$
$C^{\alpha}([-1,1]^{d})$ ( Song et al., 2023a )	Legendre polynomials	$\tilde{O}\left((\log K)^{-\beta\alpha/d}\right)$
Analytic functions ( Song et al., 2023b )		$\tilde{O}\left(\exp(-c(\log M)^{\frac{1}{d+1}})\right)$
$B^{\alpha}_{2,\infty}(\mathbb{R}^{d}),H_{0}^{\alpha}([0,1]^{d})$ ( Shi et al., 2025 )	Eigensystem induced by Mercer kernel	$\tilde{O}\left((\log K)^{-\beta\alpha/d}\right)$
Gaussian RKHS ( Shi et al., 2025 )		$\tilde{O}\left(\exp(-c(\log M)^{\frac{1}{d+1}})\right)$
$A_{1}^{\alpha}(\mathcal{D}_{N})$ ( ours )	CNN	$\tilde{O}\left((\log K)^{-\beta(\alpha-3/2)}\right)$

Table 1: Comparison of approximation rates of Hölder continuous functionals across various input function spaces and encoding schemes. We employ

\tilde{O}

notation in place of standard

O

notation to emphasize the leading term, suppressing the

\log\log

factor.

Explore similar work

Approximating Smooth Functionals with ReLU Networks
Sep 14, 2026Shuhao JiaoReLU Neural Networks Neural Network Approximation Theory
Department of Biostatistics, City University of Hong Kong, Hong Kong SAR, China.
Sparse Bayesian Deep Functional Learning with Structured Region Selection
Feb 24, 2026Xiaoxian Zhu, Yingmeng Li, Shuangge Ma +1Bayesian Neural Networks Feature Selection
School of Statistics and Data Science, Shanghai University of Finance and Economics · Department of Biostatistics, Yale School of Public Health
Learning Sparse Compositional Functions with Norm-Constrained Neural Networks
May 25, 2026Shuo Huang, Lorenzo Fiorito, Lorenzo Rosasco +1Neural Network Generalization Hierarchical Representation Learning
MaLGa, DIMA, Università degli Studi di Genova, Genoa, Italy · CBMM, Massachusetts Institute of Technology, Cambridge, MA, USA

Sep 14, 2026J/K move · Enter open · S save

Approximating Smooth Functionals with ReLU Networks

Shuhao Jiao

Department of Biostatistics, City University of Hong Kong, Hong Kong SAR, China.

ReLU Neural Networks Neural Network Approximation Theory

Open paper AskPDF arXiv

We study the uniform approximation of smooth scalar-valued functionals on an infinite-dimensional separable Hilbert space by ReLU neural networks. A key feature in deep learning for functional data is the varying importance of different coordinates/dimensions. Representing the functional input in a basis expansion, we quantify the importance of each coordinate through both the magnitude of its corresponding basis score and the directional sensitivity of the target functional. Our analysis combines coordinate truncation, anisotropic partitioning, local Taylor approximation, and ReLU network realization, while allowing unrestricted interactions among the retained coordinates. We establish a general nonasymptotic upper bound for the uniform approximation error and a complementary pseudo-dimension-based lower bound for the worst-case approximation error. Under generalized exponential coordinate decay

w_ds_d\asymp\exp(-cd^ρ)

, with

ρ>0

, the upper and lower bounds match at the leading order, which is stretched-exponential in the logarithm of the network size budget, and thus yield the nearly optimal approximation rate. This is the first work to characterize neural network approximation error for infinite-dimensional functional inputs explicitly through the joint dimensional decay of coordinate magnitudes and directional sensitivities.

Abstract

Explore similar work

Approximating Smooth Functionals with ReLU Networks

Sparse Bayesian Deep Functional Learning with Structured Region Selection

Learning Sparse Compositional Functions with Norm-Constrained Neural Networks