Smoothness Adaptivity in Constant-Depth Neural Networks: Optimal Rates via Smooth Activations

Yuhao Liu; Zilin Wang; Lei Wu; Shaobo Zhang

stat.MLFeb 23, 2026

Smoothness Adaptivity in Constant-Depth Neural Networks: Optimal Rates via Smooth Activations

Authors: Yuhao Liu, Zilin Wang, Lei Wu, Shaobo Zhang

Organizations: Department of Mathematical Sciences, Tsinghua University · School of Mathematical Sciences, Peking University · Center for Machine Learning Research, Peking University · AI for Science Institute, Beijing

Abstract

Smooth activation functions are ubiquitous in modern deep learning, yet their theoretical advantages over non-smooth counterparts remain poorly understood. In this work, we study both approximation and statistical properties of neural networks with smooth activations for learning functions in the Sobolev space $W^{s,\infty}([0,1]^d)$ with $s>0$ . We prove that constant-depth networks equipped with smooth activations achieve smoothness adaptivity: increasing width alone suffices to attain the minimax-optimal approximation and estimation error rates (up to logarithmic factors). In contrast, for non-smooth activations such as ReLU, smoothness adaptivity is fundamentally limited by depth: the attainable approximation order is bounded by depth, and higher-order smoothness requires proportional depth growth. These results identify activation smoothness as a fundamental mechanism, complementary to depth, for achieving optimal rates over Sobolev function classes. Technically, our analysis is based on a multi-scale approximation framework that yields explicit neural network approximators with controlled parameter norms and model size. This complexity control ensures statistical learnability under empirical risk minimization (ERM) and avoids the impractical $\ell^0$ -sparsity constraints commonly required in prior analyses.

Figures & tables

Activation	Reference	Depth	Free of $\ell^{0}$ -sparsity	Norm control	Remark
ReLU	Yarotsky (2017) , Liang and Srikant (2017) , Schmidt-Hieber (2020)	$O(\log(\frac{1}{\epsilon}))$	$\times$	$\checkmark$
	Kohler and Langer (2021)	$O(\log(\frac{1}{\epsilon}))$	$\checkmark$	$\checkmark$
	Petersen and Voigtlaender (2018) , Nakada and Imaizumi (2020)	$O(s\log(s))$	$\times$	$\checkmark$
	Yang and Zhou (2024)	$2$	$\checkmark$	$\checkmark$	$s<\frac{d+3}{2}$
$\mathrm{ReLU}^{k}$	Petrushev (1998) , Pinkus (1999) , Yang and Zhou (2025)	$2$	$\checkmark$	$\checkmark$	$s<\frac{2k+d+1}{2}$
Smooth	Mhaskar (1996) , Pinkus (1999)	$2$	$\checkmark$	$\times$	No learning guarantees

Table 1: Comparison of approximation and learning results under different activation functions and architectural regimes. We compare prior work with our results along three key dimensions: depth requirement,

\ell^{0}

-sparsity constraint, and norm control. Unless otherwise stated, the listed methods achieve the optimal approximation rate

O(N^{-s/d})

and nearly optimal estimation rate

\widetilde{O}(n^{-2s/(2s+d)})

, where

N

and

n

denote the number of non-zero parameters and samples, respectively. In summary, existing results typically require either (i) depth growing with target accuracy or smoothness, (ii) sparsity constraints, or (iii) smoothness saturation. Our result is the only one that simultaneously achieves constant depth, explicit norm control, no sparsity constraints, and adaptivity to arbitrarily high smoothness orders.

Figure 1: Generalization error versus sample size for two-layer networks trained with different activation functions. Markers denote the measured generalization errors at each sample size (averaged over 5 runs), and solid lines show least-squares fits of the form

E(n)\propto n^{-\alpha}

. The fitted exponents

\alpha

, reported in the legend, indicate a faster decay of the generalization error for smooth activations as the sample size increases.

Figure 2: Illustration of the approximator construction for

f^{\star}

in Theorem B.19 with

d=1

and

K=2

. (a) Approximate

f^{\star}

by piecewise polynomials, realized as the product of global polynomials and piecewise constant functions. (b) The

4

-piece piecewise constant function on refined cells is decomposed into a summation of two

2

-piece functions defined over coarse cells, multiplied by refined-cell indicator functions. (c) The refined-cell indicator functions are realized by taking the extracted relative position information

x-a(x)

as the input for the reference indicators

\mathbbm{1}_{[0,0.25]}

and

\mathbbm{1}_{[0.25,0.5]}

, which correspond to the refined cells contained within the leftmost coarse cell

[0,0.5]

Figure 3: Illustration of the

L^{\infty}([0,1])

approximation strategy for

f^{\star}

detailed in Theorem 4.4 . Large approximation errors of

f_{i}

on the bands

\Omega_{\mathrm{band}}^{2,i}(\delta)

are nullified by the vanishing weight functions

w_{i}(x)

. Since the weights constitute a partition of unity, the global reconstruction

w_{1}f_{1}+w_{2}f_{2}

maintains the desired approximation accuracy across the entire domain

[0,1]

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Visualization of the hierarchical grid structure (

K=2,d=2

), detailing the coarse and refined cells along with their respective interior and band regions.

Figure 5: Illustration of the constructive approximation for the weight function

w_{1}

, instantiated with

K=2

. Dependencies on indices

\phi,\beta

and

\delta

are suppressed for clarity. Panels (a) and (b) visualize the approximators constructed in Lemma B.25 and Lemma B.28 , respectively. The orange shaded region denotes the domain

\Omega_{\mathrm{coarse,band}}^{K,1}(\delta)

, while the blue region corresponds to the difference set

\Omega_{\mathrm{coarse,band}}^{K,1}(2\delta)\setminus\Omega_{\mathrm{coarse,band}}^{K,1}(\delta)

Neural Network Approximation Theory

Neural network approximation theory studies how network size, depth, activations, and structure determine the accuracy of functions and operators.

See all 238 papers

Mitigating the Curse of Dimensionality in Uniform Convergence of Deep Neural Networks via Smooth Activations
Jun 4, 2026Yizhe Ding, Runze Li, Jia Liu +1ReLU Neural Networks Curse of Dimensionality
Department of Statistics, The Pennsylvania State University
Sobolev Approximation by Fixed-Size Neural Networks with Arbitrary Accuracy
Jun 15, 2026Baicheng Li, Haizhao Yang, Shijun ZhangNeural Network Activation Functions Universal Approximation
Department of Mathematics, University of Maryland, College Park, MD, USA · Department of Applied Mathematics, The Hong Kong Polytechnic University, Hong Kong, China
Deep Network Approximation: Beyond ReLU to Diverse Activation Functions
Jul 13, 2023Shijun Zhang, Jianfeng Lu, Hongkai ZhaoReLU Neural Networks Neural Network Activation Functions
Department of Mathematics Duke University

Jun 4, 2026J/K move · Enter open · S save

Mitigating the Curse of Dimensionality in Uniform Convergence of Deep Neural Networks via Smooth Activations

Yizhe Ding, Runze Li, Jia Liu, Lingzhou Xue

Department of Statistics, The Pennsylvania State University

ReLU Neural Networks Curse of Dimensionality Neural Network Approximation Theory Statistical Learning Theory

Open paper AskPDF arXiv

This paper establishes a theoretical framework for the uniform convergence of smoothly activated deep neural network (DNN) estimators. While standard ReLU networks achieve minimax-optimal rates in the

L^2(P)

norm for various nonparametric regression tasks, we establish a theoretical lower bound demonstrating that least-squares ReLU estimators can suffer from the curse of dimensionality in their uniform convergence behavior. Motivated by the need for reliable uniform guarantees in downstream tasks requiring worst-case reliability, we address this limitation by analyzing smoothly activated DNNs (smooth DNNs), encompassing both feedforward and residual structures. We establish novel pseudo-dimension bounds, non-asymptotic approximation guarantees, and Hölder-norm bounds for the approximators of these models. Leveraging these results, we derive non-asymptotic uniform convergence rates for smooth DNN estimators across multiple statistical contexts, including Huber, least-squares, quantile, and logistic regression. We prove that smooth DNNs can mitigate the {curse of dimensionality} in uniform convergence by adaptively exploiting the low-dimensional hierarchical composition structure of the target function. Supported by both simulation studies and a real-world application, our results position smooth DNNs as a theoretically grounded and practically viable alternative to ReLU networks for statistical learning tasks requiring uniform guarantees.

Abstract

Neural Network Approximation Theory

Explore similar work

Mitigating the Curse of Dimensionality in Uniform Convergence of Deep Neural Networks via Smooth Activations

Sobolev Approximation by Fixed-Size Neural Networks with Arbitrary Accuracy

Deep Network Approximation: Beyond ReLU to Diverse Activation Functions