Optimal approximation of smooth functions by ReLU neural networks

Thomas Nagler

stat.MLOct 15, 2026

Optimal approximation of smooth functions by ReLU neural networks

Authors: Thomas Nagler

Organizations: Department of Statistics, LMU Munich Munich Center for Machine Learning (MCML)

Abstract

We study uniform approximation of the Hölder ball $\mathcal H_1^s([0,1]^D)$ by fully connected ReLU networks. Except in the one-dimensional Lipschitz case, existing constructions remain separated from the information-theoretic efficiency limit. This article closes the gap for every $s>0$ and $D\ge1$ : networks with width $O(N)$ , depth $O(L)$ , and parameter magnitude $O(1)$ attain uniform approximation error of order $(N^2L^2\log N)^{-s/D}$ . No relation between width and depth is required. The construction uses new encoding and decoding mechanisms that exploit higher-order finite-difference bounds for local Taylor coefficients of the target function. As a statistical consequence, empirical risk minimizers over the corresponding clipped network classes attain the minimax regression rate $n^{-2s/(2s+D)}$ without additional logarithmic factors. The paper also gives a detailed exposition of the key approximation techniques, covering both established and new mechanisms.

Figures & tables

Reference	Smoothness	Width	Depth	Parameter magnitude	Uniform error
Yarotsky (2018)	$0<s\leq 1$	$1$	$L$	Not tracked	$L^{-2s/D}$
Yarotsky and Zhevnerchuk (2020)	$s>0$	$1$	$L$	Not tracked	$L^{-2s/D}$
Schmidt-Hieber (2020)	$s>0$	$N^{2}$	$\log N$	$1$ (sparse)	$N^{-2s/D}$
Kohler and Langer (2021)	$s>0$	$N$	$\log N$	Not tracked	$N^{-2s/D}$
Kohler and Langer (2021)	$s>0$	$1$	$L$	Not tracked	$L^{-2s/D}$
Lu et al. (2021)	$s\in\mathds{N}_{+}$	$N\log N$	$L\log L$	Not tracked	$(N^{2}L^{2})^{-s/D}$

Table 1: Overview of quantitative approximation bounds for functions on

[0,1]^{D}

with Hölder-smoothness

s>0

. Width, depth, and parameter magnitude are stated up to constant factors.

Panel of Figure 1 — Figure 1: Structure of the approximating network. The localization module ( Section 4.6.2 ) maps $\bm{x}$ to the lower corner $\bm{\beta}/K$ of its grid cell and then to the corresponding flattened index ( Sections 4.2 and 4.5 ). For each multi-index $\bm{\alpha}$ , the network retrieves an approximation of the local Taylor coefficient $\partial^{\bm{\alpha}}f(\bm{\beta}/K)/\bm{\alpha}!$ ( Sections 4.3 and 4.4 ) and multiplies it by an approximation of the monomial $(\bm{x}-\bm{\beta}/K)^{\bm{\alpha}}$ ( Section 4.6.1 ). Summing these terms yields the local Taylor approximation, and median smoothing ( Section 4.6.3 ) controls the error near cell boundaries.

Panel of Figure 2 — Figure 2: The construction for $N=4$ with positive slope jumps. (a) The function $g_{2}^{+}$ crosses zero at $x_{i,2}$ in every group, returns to zero at $x_{i,N+1}$ , and is nonpositive until the next group. The thicker curve shows $\sigma(g_{2}^{+})$ , including its flat zero segments. (b) Each $\sigma(g_{j}^{+})$ has one knot in every group (dotted lines), and $h$ has only boundary knots (dashed lines).

Panel of Figure 3 — Figure 3: Extracting the leading digit $b_{1}$ and the remainder from a stored string of length $m=4$ ; dots mark the admissible inputs. (a) In binary, $b_{1}$ must jump between neighboring inputs separated by $2^{-m}$ . (b) In ternary with digits $\{0,1\}$ , the gap $(1/6,1/3)$ allows slope $6$ for every $m$ .

Panel of Figure 4 — Figure 4: Two sequences satisfying the same first-difference bound $|\Delta y_{i}|\leq\delta$ . In (a), oscillations at the finest admissible scale leave second differences of order $\delta$ . In (b), discrete smoothness reduces the second differences to order $\delta^{2}$ .

Panel of Figure 5 — Figure 5: Flattening a two-dimensional coefficient grid. Consecutive rows are separated by gaps filled with Hermite bridges that match the first $h_{\bm{\alpha}}-1$ derivatives. The resulting sequence preserves the required finite-difference bounds.

Panel of Figure 6 — Figure 6: Localization for $K=4$ . (a) The staircase $S_{K,\delta}$ equals $k/K$ between the strips (gray) and is continuous on them. (b) The strips form $\Omega_{K,\delta}$ for $D=2$ , and $\Lambda=\bm{\beta}/K$ on the blue part of the cell $Q_{\bm{\beta}}$ .

Panel of Figure 7 — Figure 7: Median smoothing for $D=2$ ; filled markers belong to points outside $\Omega_{K,\delta}$ . (a) The ticks mark the coordinates of the five diagonal points, and each strip contains at most one of them. (b) The three filled values lie within $\varepsilon+\omega_{f}(2\delta)$ of $f(\bm{x})$ , and so does the median $\widetilde{\phi}(\bm{x})$ of all five values.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure 8: The function $h$ in the proof of Proposition 4.2 for $\zeta=3$ and $n=2$ . Admissible inputs $\theta$ lie in the four intervals $I_{\bm{b}}$ (shaded). On each interval, the first two coordinates of $h$ are constant and the remainder coordinate is affine. Across the gaps, all coordinates are affine.

Panel of Figure 9 — Figure 9: The decoder $\phi_{\rm dec}$ of Lemma B.2 . (a) It consists of $L$ copies of one segment network acting on the state $(z,k,\bm{u},\theta)$ , with each segment covering $q$ positions. (b) Inside a segment, $\phi_{\rm stream}$ from Proposition 4.2 decodes the $q$ stored $h$ -th differences $\bm{d}$ , the networks $\phi_{Y}$ and $\phi_{U}$ evaluate the discrete Taylor formulas $Y$ and $U$ , and the indicator $\mathds{1}_{1\leq k\leq q}$ decides whether the segment writes the requested value $\overline{y}_{k}$ .

Panel of Figure 10 — Figure 10: The point-fitting network of Proposition 4.3 . (a) The indices $i=1,\dots,J$ are grouped into $M$ blocks of length $Lq$ ; the staircase $\phi_{\rm block}$ returns the block index and the sawtooth $\phi_{\rm pos}$ the position within the block. (b) The block parameters $u_{m,0},\dots,u_{m,h-1}$ , $\theta_{m}$ are retrieved from the block index and, together with $k$ , fed into the decoder $\phi_{\rm dec}$ of Lemma B.2 .

Neural Network Approximation Theory

Neural network approximation theory studies how network size, depth, activations, and structure determine the accuracy of functions and operators.

See all 227 papers

Approximating Smooth Functionals with ReLU Networks
Sep 14, 2026Shuhao JiaoReLU Neural Networks Neural Network Approximation Theory
Department of Biostatistics, City University of Hong Kong, Hong Kong SAR, China.
Shallow ReLU $^s$ Networks in $L^p$ -Type and Sobolev Spaces: Approximation and Path-Norm Controlled Generalization
May 18, 2026Weizhao Li, Fanghui Liu, Lei ShiShallow Neural Networks Nonparametric Regression
School of Mathematical Sciences Fudan University Shanghai, China · School of Mathematical Sciences Institute of Natural Sciences and MOE-LSC2026 Shanghai Jiao Tong University Shanghai, China · School of Mathematical Sciences Shanghai Key Laboratory for Contemporary Applied Mathematics25 Fudan University Shanghai, China
Error bounds in Sobolev norms for approximations with norm constrained ReLU neural networks
Sep 17, 2026Xianjun Li, Yunfei YangShallow Neural Networks ReLU Neural Networks
School of Statistics and Data Science, Capital University of Economics and Business, Beijing, China. · School of Mathematics (Zhuhai) and Guangdong Province Key Laboratory of Computational Science, Sun Yat-sen University, Zhuhai, China.

Sep 14, 2026J/K move · Enter open · S save

Approximating Smooth Functionals with ReLU Networks

Shuhao Jiao

Department of Biostatistics, City University of Hong Kong, Hong Kong SAR, China.

ReLU Neural Networks Neural Network Approximation Theory

Open paper AskPDF arXiv

We study the uniform approximation of smooth scalar-valued functionals on an infinite-dimensional separable Hilbert space by ReLU neural networks. A key feature in deep learning for functional data is the varying importance of different coordinates/dimensions. Representing the functional input in a basis expansion, we quantify the importance of each coordinate through both the magnitude of its corresponding basis score and the directional sensitivity of the target functional. Our analysis combines coordinate truncation, anisotropic partitioning, local Taylor approximation, and ReLU network realization, while allowing unrestricted interactions among the retained coordinates. We establish a general nonasymptotic upper bound for the uniform approximation error and a complementary pseudo-dimension-based lower bound for the worst-case approximation error. Under generalized exponential coordinate decay

w_ds_d\asymp\exp(-cd^ρ)

, with

ρ>0

, the upper and lower bounds match at the leading order, which is stretched-exponential in the logarithm of the network size budget, and thus yield the nearly optimal approximation rate. This is the first work to characterize neural network approximation error for infinite-dimensional functional inputs explicitly through the joint dimensional decay of coordinate magnitudes and directional sensitivities.

Abstract

Neural Network Approximation Theory

Explore similar work

Approximating Smooth Functionals with ReLU Networks

Shallow ReLUs^ss Networks in LpL^pLp-Type and Sobolev Spaces: Approximation and Path-Norm Controlled Generalization

Error bounds in Sobolev norms for approximations with norm constrained ReLU neural networks

Shallow ReLU $^s$ Networks in $L^p$ -Type and Sobolev Spaces: Approximation and Path-Norm Controlled Generalization