Efficient Approximation to Analytic and $L^p$ functions by Height-Augmented ReLU Networks

ZeYu Li; FengLei Fan; TieYong Zeng

stat.MLMar 11, 2026

Efficient Approximation to Analytic and $L^p$ functions by Height-Augmented ReLU Networks

Authors: ZeYu Li, FengLei Fan, TieYong Zeng

Organizations: Chinese University of Hong Kong, Department of Mathematics, Hong Kong, 999077, NT, Hong Kong SAR, China · City University of Hong Kong, Department of Data Science, Hong Kong, 999077, Kowloon, Hong Kong SAR, China · Guangzhou Nanfang College, School of Mathematics and Statistics, Guangzhou, 510970, Guangdong, China · Beijing Normal Hong Kong Baptist University, Institute for Advanced Study, Zhuhai, 519087, Guangdong, China

Abstract

This work addresses two fundamental limitations in neural network approximation theory. We demonstrate that a three-dimensional network architecture enables a significantly more efficient representation of sawtooth functions, which serves as the cornerstone in the approximation of analytic and $L^p$ functions. First, we establish substantially improved exponential approximation rates for several important classes of analytic functions and offer a parameter-efficient network design. Second, for the first time, we derive a quantitative and non-asymptotic approximation of high orders for general $L^p$ functions. Our techniques advance the theoretical understanding of the neural network approximation in fundamental function spaces and offer a theoretically grounded pathway for designing more parameter-efficient networks.

Figures & tables

Figure 1: Adding intra-layer links creates a new hierarchy among neurons in the same layer, which induces a new dimension referred to as height. Note that a 2D network can be regarded as a 3D network with height=1.

Target Function	Result	Width	Depth	Height	Error
Polynomial on [0,1]	Proposition 3.3	$\mathcal{O}(1)$	$\mathcal{O}(1)$	$\mathcal{O}(N)$	$\mathcal{O}\left(2^{-N}\right)$
Polynomial on [0,1]	Proposition III.5, [ 10 ]	$\mathcal{O}(1)$	$\mathcal{O}(N)$	1	$\mathcal{O}(\exp(-N))$
Analytic on $[0,1-\delta]^{d}$	Theorem 3.5	$\mathcal{O}(N^{d-1})$	$\mathcal{O}(N)$	$\mathcal{O}(N)$	$\mathcal{O}\left((1-\delta)^{N}\right)$
Analytic on $[0,1-\delta]^{d}$	Lemma III.7, [ 9 ]	$\mathcal{O}(1)$	$\mathcal{O}(N^{2d})$	1	$\mathcal{O}\left((1-\delta)^{N}\right)$
Analytic on $[0,1]^{d}$ , holomorphic in an ellipse	Theorem 3.6	$\mathcal{O}(N^{d-1})$	$\mathcal{O}(N)$	$\mathcal{O}(N)$	$\mathcal{O}\left(\rho^{-N}\right)$
Analytic on $[0,1]^{d}$ , holomorphic in an ellipse	Theorem2.2, [ 5 ]	$\mathcal{O}(N^{d+2})$	$\mathcal{O}(N^{2})$	1	$\mathcal{O}\left(2^{-N}\right)$

Table 1: Comparisons between our work and the existing results.

\gamma_{d}

is the

d

-dimensional Gaussian measure.

\omega_{d}^{r}(f,\cdot)_{p}

is the

L^{p}

modulus of smoothness of order

r

\mathbb{R}^{d}

. Note that a 3D network of width

W

, depth

K

, and height

H

is topologically equivalent to a 2D intra-linked network of width

W\times H

and depth

K

Figure 2: Implementing

f_{H}

by 3D networks in Eq. 10 .

Figure 3: Illustration of the construction in Lemma 3.3

Figure 4: Illustration of the network structure in Lemma 3.12

Figure 5: Implementation of the network approximation to analytic functions in Theorem 3.8 .

Neural Network Approximation Theory

Neural network approximation theory studies how network size, depth, activations, and structure determine the accuracy of functions and operators.

See all 227 papers

Approximation of Analytic Functions by ReLU Neural Networks with Adjustable Depth and Width
Jul 12, 2026Yanming Lai, Defeng Sun, Yang WangReLU Neural Networks Neural Network Approximation Theory
Department of Applied Mathematics, The Hong Kong Polytechnic University, Hung Hom, Hong Kong, China · Department of Mathematics, The University of Hong Kong, Pokfulam, Hong Kong, China
Approximating Smooth Functionals with ReLU Networks
Sep 14, 2026Shuhao JiaoReLU Neural Networks Neural Network Approximation Theory
Department of Biostatistics, City University of Hong Kong, Hong Kong SAR, China.
Approximation Theory for Neural Networks: Old and New
May 20, 2026Soumendu Sundar Mukherjee, Himasish TalukdarUniversal Approximation Neural Network Approximation Theory
Statistics and Mathematics Unit, Indian Statistical Institute, Kolkata

Jul 12, 2026J/K move · Enter open · S save

Approximation of Analytic Functions by ReLU Neural Networks with Adjustable Depth and Width

Yanming Lai, Defeng Sun, Yang Wang

Department of Applied Mathematics, The Hong Kong Polytechnic University, Hung Hom, Hong Kong, China · Department of Mathematics, The University of Hong Kong, Pokfulam, Hong Kong, China

ReLU Neural Networks Neural Network Approximation Theory

Open paper AskPDF arXiv

In contrast to most studies on neural network approximation theory that characterize results through a single parameter, such as the total number of network parameters, \cite{shen2020deep} pioneered the characterization of approximation rates as a joint function of the width parameter

N

and the depth parameter

L

, thereby granting greater architectural flexibility. Existing works using the

(N,L)

-characterization focus on function classes with finite smoothness

s

, establishing a typical approximation rate of

\mathcal{O}\left(N^{-2s/d}L^{-2s/d}\right)

with

d

denoting the input dimension, which indicates that network depth and width play symmetric roles for these classes. In contrast, this paper establishes upper bounds for the approximation of analytic functions, which possess infinite smoothness, via ReLU networks under the

(N,L)

-characterization. Specifically, we derive approximation rates of

\mathcal{O}\left(N^{-C L^τ}\right)

, where

C>0

is some constant and

τ>0

is a parameter influenced by the relation between

L

and

N

. In particular,

τ=1

N

scales roughly as

L^d

. Our findings reveal that depth plays a more critical role than width in the context of analytic function approximation. The main technical difficulty of obtaining such upper bounds lies in the trade-off between the smoothness parameters and the approximation accuracy. To overcome this difficulty, we employ refined constructions of several ReLU networks to approximate power functions, multivariate multiplication, and polynomials, which may be of independent interest.

Abstract

Neural Network Approximation Theory

Explore similar work

Approximation of Analytic Functions by ReLU Neural Networks with Adjustable Depth and Width

Approximating Smooth Functionals with ReLU Networks

Approximation Theory for Neural Networks: Old and New