Smoothness Adaptivity in Constant-Depth Neural Networks: Optimal Rates via Smooth Activations

Yuhao Liu; Zilin Wang; Lei Wu; Shaobo Zhang

stat.MLFeb 23, 2026

Smoothness Adaptivity in Constant-Depth Neural Networks: Optimal Rates via Smooth Activations

Authors: Yuhao Liu, Zilin Wang, Lei Wu, Shaobo Zhang

Organizations: Department of Mathematical Sciences, Tsinghua University · School of Mathematical Sciences, Peking University · Center for Machine Learning Research, Peking University · AI for Science Institute, Beijing

Abstract

Smooth activation functions are ubiquitous in modern deep learning, yet their theoretical advantages over non-smooth counterparts remain poorly understood. In this work, we study both approximation and statistical properties of neural networks with smooth activations for learning functions in the Sobolev space $W^{s,\infty}([0,1]^d)$ with $s>0$ . We prove that constant-depth networks equipped with smooth activations achieve smoothness adaptivity: increasing width alone suffices to attain the minimax-optimal approximation and estimation error rates (up to logarithmic factors). In contrast, for non-smooth activations such as ReLU, smoothness adaptivity is fundamentally limited by depth: the attainable approximation order is bounded by depth, and higher-order smoothness requires proportional depth growth. These results identify activation smoothness as a fundamental mechanism, complementary to depth, for achieving optimal rates over Sobolev function classes. Technically, our analysis is based on a multi-scale approximation framework that yields explicit neural network approximators with controlled parameter norms and model size. This complexity control ensures statistical learnability under empirical risk minimization (ERM) and avoids the impractical $\ell^0$ -sparsity constraints commonly required in prior analyses.

Figures & tables

Activation	Reference	Depth	Free of $\ell^{0}$ -sparsity	Norm control	Remark
ReLU	Yarotsky (2017) , Liang and Srikant (2017) , Schmidt-Hieber (2020)	$O(\log(\frac{1}{\epsilon}))$	$\times$	$\checkmark$
	Kohler and Langer (2021)	$O(\log(\frac{1}{\epsilon}))$	$\checkmark$	$\checkmark$
	Petersen and Voigtlaender (2018) , Nakada and Imaizumi (2020)	$O(s\log(s))$	$\times$	$\checkmark$
	Yang and Zhou (2024)	$2$	$\checkmark$	$\checkmark$	$s<\frac{d+3}{2}$
$\mathrm{ReLU}^{k}$	Petrushev (1998) , Pinkus (1999) , Yang and Zhou (2025)	$2$	$\checkmark$	$\checkmark$	$s<\frac{2k+d+1}{2}$
Smooth	Mhaskar (1996) , Pinkus (1999)	$2$	$\checkmark$	$\times$	No learning guarantees

Table 1: Comparison of approximation and learning results under different activation functions and architectural regimes. We compare prior work with our results along three key dimensions: depth requirement,

\ell^{0}

-sparsity constraint, and norm control. Unless otherwise stated, the listed methods achieve the optimal approximation rate

O(N^{-s/d})

and nearly optimal estimation rate

\widetilde{O}(n^{-2s/(2s+d)})

, where

N

and

n

denote the number of non-zero parameters and samples, respectively. In summary, existing results typically require either (i) depth growing with target accuracy or smoothness, (ii) sparsity constraints, or (iii) smoothness saturation. Our result is the only one that simultaneously achieves constant depth, explicit norm control, no sparsity constraints, and adaptivity to arbitrarily high smoothness orders.

Figure 1: Generalization error versus sample size for two-layer networks trained with different activation functions. Markers denote the measured generalization errors at each sample size (averaged over 5 runs), and solid lines show least-squares fits of the form

E(n)\propto n^{-\alpha}

. The fitted exponents

\alpha

, reported in the legend, indicate a faster decay of the generalization error for smooth activations as the sample size increases.

Figure 2: Illustration of the approximator construction for

f^{\star}

in Theorem B.19 with

d=1

and

K=2

. (a) Approximate

f^{\star}

by piecewise polynomials, realized as the product of global polynomials and piecewise constant functions. (b) The

4

-piece piecewise constant function on refined cells is decomposed into a summation of two

2

-piece functions defined over coarse cells, multiplied by refined-cell indicator functions. (c) The refined-cell indicator functions are realized by taking the extracted relative position information

x-a(x)

as the input for the reference indicators

\mathbbm{1}_{[0,0.25]}

and

\mathbbm{1}_{[0.25,0.5]}

, which correspond to the refined cells contained within the leftmost coarse cell

[0,0.5]

Figure 3: Illustration of the

L^{\infty}([0,1])

approximation strategy for

f^{\star}

detailed in Theorem 4.4 . Large approximation errors of

f_{i}

on the bands

\Omega_{\mathrm{band}}^{2,i}(\delta)

are nullified by the vanishing weight functions

w_{i}(x)

. Since the weights constitute a partition of unity, the global reconstruction

w_{1}f_{1}+w_{2}f_{2}

maintains the desired approximation accuracy across the entire domain

[0,1]

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Visualization of the hierarchical grid structure (

K=2,d=2

), detailing the coarse and refined cells along with their respective interior and band regions.

Figure 5: Illustration of the constructive approximation for the weight function

w_{1}

, instantiated with

K=2

. Dependencies on indices

\phi,\beta

and

\delta

are suppressed for clarity. Panels (a) and (b) visualize the approximators constructed in Lemma B.25 and Lemma B.28 , respectively. The orange shaded region denotes the domain

\Omega_{\mathrm{coarse,band}}^{K,1}(\delta)

, while the blue region corresponds to the difference set

\Omega_{\mathrm{coarse,band}}^{K,1}(2\delta)\setminus\Omega_{\mathrm{coarse,band}}^{K,1}(\delta)

Neural Network Approximation Theory

Neural network approximation theory studies how network size, depth, activations, and structure determine the accuracy of functions and operators.

See all 236 papers

Jun 4, 2026

Mitigating the Curse of Dimensionality in Uniform Convergence of Deep Neural Networks via Smooth Activations

This paper establishes a theoretical framework for the uniform convergence of smoothly activated deep neural network (DNN) estimators. While standard ReLU networks achieve minimax-optimal rates in the

L^2(P)

norm for various nonparametric regression tasks, we establish a theoretical lower bound demonstrating that least-squares ReLU estimators can suffer from the curse of dimensionality in their uniform convergence behavior. Motivated by the need for reliable uniform guarantees in downstream tasks requiring worst-case reliability, we address this limitation by analyzing smoothly activated DNNs (smooth DNNs), encompassing both feedforward and residual structures. We establish novel pseudo-dimension bounds, non-asymptotic approximation guarantees, and Hölder-norm bounds for the approximators of these models. Leveraging these results, we derive non-asymptotic uniform convergence rates for smooth DNN estimators across multiple statistical contexts, including Huber, least-squares, quantile, and logistic regression. We prove that smooth DNNs can mitigate the {curse of dimensionality} in uniform convergence by adaptively exploiting the low-dimensional hierarchical composition structure of the target function. Supported by both simulation studies and a real-world application, our results position smooth DNNs as a theoretically grounded and practically viable alternative to ReLU networks for statistical learning tasks requiring uniform guarantees.

Yizhe Ding, Runze Li, Jia Liu +1

Department of Statistics, The Pennsylvania State University

ReLU Neural Networks Curse of Dimensionality Neural Network Approximation Theory Statistical Learning Theory

Ask PDF

Jun 15, 2026

Sobolev Approximation by Fixed-Size Neural Networks with Arbitrary Accuracy

In this work, we investigate new activation functions for achieving arbitrary-accuracy Sobolev approximation by fixed-size neural networks. We first show that any function in

W^{2,\infty}((a,b)^d)

can be approximated with arbitrary accuracy, measured in the

W^{1,\infty}

-norm, by a fixed-size neural network using the Elementary Universal Activation Function (

\mathrm{EUAF}

). To extend this result to

W^{s,\infty}((a,b)^d)

for

s\in\mathbb{N}

, we introduce a smooth activation

\mathrm{DUAF}_{\infty}

from the family of Differentiable Universal Activation Functions (

\mathrm{DUAF}_n

). We prove that any function in

W^{s,\infty}((a,b)^d)

can be approximated with arbitrary accuracy in the

W^{s-1,\infty}

-norm by a fixed-size

\mathrm{DUAF}_{\infty}

-activated network. We further construct sigmoidal variants

\widetilde{\mathrm{DUAF}}_n

and show that, for every

1\leq s\leq n

, fixed-size

\widetilde{\mathrm{DUAF}}_n

-activated networks still approximate any

f\in W^{s,\infty}((a,b)^d)

with arbitrary accuracy in the

W^{s-1,\infty}

-norm. In all these results, the width and depth bounds are computed explicitly, and the proposed activations are elementary.

Baicheng Li, Haizhao Yang, Shijun Zhang

Department of Mathematics, University of Maryland, College Park, MD, USA · Department of Applied Mathematics, The Hong Kong Polytechnic University, Hong Kong, China

Neural Network Activation Functions Universal Approximation Neural Network Approximation Theory

Ask PDF

Jul 13, 2023

Deep Network Approximation: Beyond ReLU to Diverse Activation Functions

This paper explores the expressive power of deep neural networks for a diverse range of activation functions. An activation function set

\mathscr{A}

is defined to encompass the majority of commonly used activation functions, such as

\mathtt{ReLU}

\mathtt{LeakyReLU}

\mathtt{ReLU}^2

\mathtt{ELU}

\mathtt{CELU}

\mathtt{SELU}

\mathtt{Softplus}

\mathtt{GELU}

\mathtt{SiLU}

\mathtt{Swish}

\mathtt{Mish}

\mathtt{Sigmoid}

\mathtt{Tanh}

\mathtt{Arctan}

\mathtt{Softsign}

\mathtt{dSiLU}

, and

\mathtt{SRS}

. We demonstrate that for any activation function

\varrho\in \mathscr{A}

, a

\mathtt{ReLU}

network of width

N

and depth

L

can be approximated to arbitrary precision by a

\varrho

-activated network of width

3N

and depth

2L

on any bounded set. This finding enables the extension of most approximation results achieved with

\mathtt{ReLU}

networks to a wide variety of other activation functions, albeit with slightly increased constants. Significantly, we establish that the (width,

\,

depth) scaling factors can be further reduced from

(3,2)

(1,1)

\varrho

falls within a specific subset of

\mathscr{A}

. This subset includes activation functions such as

\mathtt{ELU}

\mathtt{CELU}

\mathtt{SELU}

\mathtt{Softplus}

\mathtt{GELU}

\mathtt{SiLU}

\mathtt{Swish}

, and

\mathtt{Mish}

Shijun Zhang, Jianfeng Lu, Hongkai Zhao

Department of Mathematics Duke University

ReLU Neural Networks Neural Network Activation Functions Neural Network Approximation Theory Neural Network Expressivity

Ask PDF

Abstract

Neural Network Approximation Theory

Explore similar work

Mitigating the Curse of Dimensionality in Uniform Convergence of Deep Neural Networks via Smooth Activations

Sobolev Approximation by Fixed-Size Neural Networks with Arbitrary Accuracy

Deep Network Approximation: Beyond ReLU to Diverse Activation Functions