On the growth of the parameters of approximating ReLU neural networks

Erion Morina; Martin Holler

cs.LGJun 21, 2024

On the growth of the parameters of approximating ReLU neural networks

Authors: Erion Morina, Martin Holler

Organizations: Department of Mathematics and Scientific Computing, University of Graz.

Abstract

This work focuses on the analysis of fully connected feed forward ReLU neural networks as they approximate a given, smooth function. In contrast to conventionally studied universal approximation properties under increasing architectures, e.g., in terms of width or depth of the networks, we are concerned with the asymptotic growth of the parameters of approximating networks. Such results are of interest, e.g., for error analysis or consistency results for neural network training. The main result of our work is that, for a ReLU architecture with state of the art approximation error, the realizing parameters grow at most polynomially. The obtained rate with respect to a normalized network size is compared to existing results and is shown to be superior in most cases, in particular for high dimensional input.

Figures & tables

Result	Width	Depth	Approximation	Growth of parameters	Activation
Th. 3	$\mathcal{O}(N)$	$\mathcal{O}(L)$	$\mathcal{O}(N^{\frac{-2q}{d(1+\delta)}}L^{\frac{-q}{d(1+\delta)}})$	$\mathcal{O}(N^{\frac{6q-3}{d}}L^{\frac{3q-1}{d}}\lor NL^{\frac{1}{2}}(N\lor L))$	$\relu$
[ 1 ]	$\mathcal{O}(N)$	$\mathcal{O}(1)$	$\mathcal{O}(N^{-q/d})$	$\mathcal{O}(1)$	ReQU
[ 3 ]	$\mathcal{O}(N)$	$3$	$\mathcal{O}(N^{-q/d})$	$\mathcal{O}(N^{(d+q^{2})/2})$	$\tanh$
[ 12 ]	$\mathcal{O}(N)$	$\mathcal{O}(1)$	$\mathcal{O}(N^{-2q/d})$	$\mathcal{O}(N^{(16q+2d+9)/d})$	$\frac{1}{1+\exp(-x)}$

Table 1: Comparison of state of the art results on growth of parameters realizing approximations to

f\in\mathcal{C}^{q}([0,1]^{d})

with normalized width.

Result	Nonzero weights	Approximation	Growth of parameters	Activation
Th. 3	$\mathcal{O}(W)$	$\mathcal{O}(W^{-q/d})$	$\mathcal{O}(W^{\frac{9q-4}{3d}\lor\frac{5}{6}})$	ReLU
[ 8 ]	$\mathcal{O}(W)$	$\mathcal{O}(W^{-q/d})$	$\mathcal{O}(W^{3+2q/d})$	RePU, soft+
[ 7 ]	$\mathcal{O}(W)$	$\mathcal{O}((W/\log W)^{-q/d})$	$\mathcal{O}((W/\log W)^{1/d})$	ReLU

Table 2: Comparison of state of the art result on growth of parameters realizing approximations to

f\in\mathcal{C}^{q}([0,1]^{d})

with normalized number of nonzero weights.

Panel of Figure 1 — Figure 1: Overview of structure of results in [ 13 ]

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 2 — Figure 2: The approximated function $f$ in ( 22 ) and the modification $f^{*}$ in ( 28 ).

ReLU Neural Networks

Rectified linear unit (ReLU) neural networks use piecewise-linear activations to study approximation, learning dynamics, expressivity, and verification.

See all 124 papers

Optimal approximation of smooth functions by ReLU neural networks
Oct 15, 2026Thomas NaglerNeural Network Approximation Theory ReLU Neural Networks
Department of Statistics, LMU Munich Munich Center for Machine Learning (MCML)
Approximation of Analytic Functions by ReLU Neural Networks with Adjustable Depth and Width
Jul 12, 2026Yanming Lai, Defeng Sun, Yang WangReLU Neural Networks Neural Network Approximation Theory
Department of Applied Mathematics, The Hong Kong Polytechnic University, Hung Hom, Hong Kong, China · Department of Mathematics, The University of Hong Kong, Pokfulam, Hong Kong, China
Error bounds in Sobolev norms for approximations with norm constrained ReLU neural networks
Sep 17, 2026Xianjun Li, Yunfei YangShallow Neural Networks ReLU Neural Networks
School of Statistics and Data Science, Capital University of Economics and Business, Beijing, China. · School of Mathematics (Zhuhai) and Guangdong Province Key Laboratory of Computational Science, Sun Yat-sen University, Zhuhai, China.

Oct 15, 2026J/K move · Enter open · S save

Optimal approximation of smooth functions by ReLU neural networks

Thomas Nagler

Department of Statistics, LMU Munich Munich Center for Machine Learning (MCML)

Neural Network Approximation Theory ReLU Neural Networks

Open paper AskPDF arXiv

We study uniform approximation of the Hölder ball

\mathcal H_1^s([0,1]^D)

by fully connected ReLU networks. Except in the one-dimensional Lipschitz case, existing constructions remain separated from the information-theoretic efficiency limit. This article closes the gap for every

s>0

and

D\ge1

: networks with width

O(N)

, depth

O(L)

, and parameter magnitude

O(1)

attain uniform approximation error of order

(N^2L^2\log N)^{-s/D}

. No relation between width and depth is required. The construction uses new encoding and decoding mechanisms that exploit higher-order finite-difference bounds for local Taylor coefficients of the target function. As a statistical consequence, empirical risk minimizers over the corresponding clipped network classes attain the minimax regression rate

n^{-2s/(2s+D)}

without additional logarithmic factors. The paper also gives a detailed exposition of the key approximation techniques, covering both established and new mechanisms.

Abstract

ReLU Neural Networks

Explore similar work

Optimal approximation of smooth functions by ReLU neural networks

Approximation of Analytic Functions by ReLU Neural Networks with Adjustable Depth and Width

Error bounds in Sobolev norms for approximations with norm constrained ReLU neural networks