Shallow ReLU$^s$ Networks in $L^p$-Type and Sobolev Spaces: Approximation and Path-Norm Controlled Generalization

Weizhao Li; Fanghui Liu; Lei Shi

stat.MLMay 18, 2026

Shallow ReLU $^s$ Networks in $L^p$ -Type and Sobolev Spaces: Approximation and Path-Norm Controlled GeneralizationShallow ReLU^s Networks in L^p-Type and Sobolev Spaces: Approximation and Path-Norm Controlled Generalization

Authors: Weizhao Li, Fanghui Liu, Lei Shi

Organizations: School of Mathematical Sciences Fudan University Shanghai, China · School of Mathematical Sciences Institute of Natural Sciences and MOE-LSC2026 Shanghai Jiao Tong University Shanghai, China · School of Mathematical Sciences Shanghai Key Laboratory for Contemporary Applied Mathematics25 Fudan University Shanghai, China

Abstract

This paper studies approximation by shallow ReLU $^s$ networks, $σ_s(t)=\max\{0,t\}^s$ , together with their generalization behavior under $\ell_1$ path-norm control. For the $L^p$ -type integral spaces $\widetilde{\mathcal{F}}_{p,τ_d,s}$ , $1\le p\le2$ , spherical harmonic analysis yields approximation bounds for shallow networks. In particular, when $τ_d$ is the uniform measure and $1\le p<2$ , the approximation rate is $O\!\left(m^{-\frac{p(2s+2d+1)-2d}{2dp}}\right)$ for $1\le p\le p^*$ and $O\!\left(m^{-\frac{p(4s+3d-1)-2d+2}{4dp}}\right)$ for $p^*<p<2$ , where $p^*=\frac{2d+2}{d+3}$ . Approximation bounds for Sobolev spaces $W^{α,p}$ , $1\le p<2$ , are obtained through embeddings into spectral Barron spaces. For nonparametric regression with sub-Gaussian noise, path-norm-regularized shallow ReLU $^s$ networks achieve minimax-optimal rates $O\!\left(n^{-\frac{d+2s+1}{2d+2s+1}}\log n\right)$ over $\mathscr{B}_s$ and $O\!\left(n^{-\frac{2α}{2α+d}}\log n\right)$ over $W^{α,\infty}$ , with matching lower bounds up to logarithmic factors.

Explore similar work

Sep 17, 2026stat.ML

Error bounds in Sobolev norms for approximations with norm constrained ReLU neural networks

Recent studies have shown that smooth functions can be well approximated by ReLU neural networks with path norm constraint on the weights. We extend these results from uniform approximation to approximation in Sobolev norm. Specifically, we analyze how well Sobolev functions in

W^{n,p}

can be approximated by neural networks with width

W

, depth

L

and path norm bounded by

K

, when the approximation error is measured in the

W^{1,p}

-norm. For shallow networks with depth

L=1

, we derive the approximation error bound

\mathcal{O}(\max\{W^{-(n-1)/d}, K^{-(n-1)/(s-n)}\})

, when the smoothness index satisfies

n<s=(d+3)/2

and the input is

d

-dimensional. For deep networks, we remove the restriction on the smoothness by showing that the approximation bound

\mathcal{O}(K^{-(n-1)/(d+d/p+1)})

holds if the width

W

and depth

L

are sufficiently large.

Xianjun Li, Yunfei Yang

Logarithmic Sobolev Inequalities Rectified Linear Unit Networks Neural-Network Approximation Approximation Generalization Bounds Neural Network

Ask PDF

Oct 5, 2025math.NA

Configuration-Dependent Lower Bounds for Approximation by Shallow ReLU $^k$ Networks on the SphereConfiguration-Dependent Lower Bounds for Approximation by Shallow ReLU^k Networks on the Sphere

We establish two related but logically distinct results for shallow ReLU

^k

neural networks on the unit sphere

\SS^d

. First, for an arbitrary set of inner neural-network parameters, the best

\mathcal{L}^2(\SS^d)

approximation of a fixed target function with smoothness

r>\tfrac{d+2k+1}{2}

admits an asymptotic lower bound given by a constant multiple of

n^{-1/2}\underline{h}^{k+1/2}

, where

\underline{h}

denotes the antipodal separation distance of the normalized inner-parameter set. This lower bound depends explicitly on the parameter configuration through

\underline{h}

and applies without additional assumptions on the parameters. Second, for antipodally quasi-uniform parameters,

\underline{h}\simeq n^{-1/d}

, and the lower bound establishes the exact saturation order

n^{-\frac{d+2k+1}{2d}}

for such parameter families: a target function with regularity greater than

\frac{d+2k+1}{2}

and satisfying the required parity condition can be approximated at this rate, whereas approximation at any strictly faster rate forces the target function to be zero. Our results therefore place linearized neural-network approximation within the classical saturation framework and show that, although ReLU

^k

network spaces can outperform finite elements of the same degree, this advantage is intrinsically limited.

Tong Mao, Jinchao Xu

Neural-Network Approximation Approximation Lower Bounds Saturation Sphere

Ask PDF

May 29, 2026stat.ML

Approximation and learning of anisotropic and mixed smooth functions by deep ReLU neural networks

This paper studies how efficiently deep ReLU neural networks can approximate and learn smooth functions. When the error is measured in

L^p([0,1]^d)

norm and the approximator is a network with width

W

and depth

L

, recent works have proven the supper approximation rate

\mathcal{O}((WL)^{-2s/d})

for Besov space

\mathcal{B}^s_{q,r}([0,1]^d)

under the Sobolev embedding condition

s/d>1/q-1/p

. In order to overcome the curse of dimensionality in this rate, we extent this result to anisotropic and mixed smooth function classes. We establish the approximation rate

\mathcal{O}((WL)^{-2\tilde{s}})

for anisotropic Besov space

\mathcal{B}^{\boldsymbol{s}}_{q,r}([0,1]^d)

with anisotropic smoothness

\boldsymbol{s}=(s_1,\dots,s_d)

under the embedding condition

\tilde{s} > 1/q-1/p

, where the mean smoothness

\tilde{s} = (\sum_{i=1}^d s_i^{-1})^{-1}

. For mixed smooth Besov space

\mathcal{MB}^s_{q,r}([0,1]^d)

with mixed smoothness

s>1/q-1/p

, we show that the approximation rate

\mathcal{O}((WL)^{-2s})

holds up to logarithmic factors. Using these results, we also derive approximation bounds for the composition of anisotropic Besov functions. As an application, it is shown that deep ReLU neural networks can achieve minimax optimal rates up to logarithmic factors for a wide range of smooth function classes.

Yunfei Yang, Jun Fan

Logarithmic Sobolev Inequalities Neural-Network Approximation Rectified Linear Unit Networks Approximation Neural Network Smoothing+2

Ask PDF

Abstract

Explore similar work

Error bounds in Sobolev norms for approximations with norm constrained ReLU neural networks

Configuration-Dependent Lower Bounds for Approximation by Shallow ReLUk^kk Networks on the SphereConfiguration-Dependent Lower Bounds for Approximation by Shallow ReLU^k Networks on the Sphere

Approximation and learning of anisotropic and mixed smooth functions by deep ReLU neural networks

Configuration-Dependent Lower Bounds for Approximation by Shallow ReLU $^k$ Networks on the SphereConfiguration-Dependent Lower Bounds for Approximation by Shallow ReLU^k Networks on the Sphere