Optimal Deep Nonparametric Regression with Uniformly Bounded Lipschitz Constants
Authors: Jiaxin Shi, Ziyuan Chen, Jian Huang, Hansheng Wang
Organizations: Guanghua School of Management, Peking University, Beijing, China · School of Mathematical Sciences, Peking University, Beijing, China · Department of Data Science and Artificial Intelligence, Department of Applied Mathematics, Hong Kong Polytechnic University, Hong Kong SAR, China
Stability under input perturbations is necessary for deep neural network estimators in many applications involving noisy covariates or estimated features. Weight norm constraints are widely used to control sensitivity to input perturbations by bounding the Lipschitz constant of neural networks. However, existing approximation results under fixed bounds on standard norm products are insufficient to establish optimal statistical rates. In this paper, we establish a sharp approximation theory for deep ReLU networks under a split spectral norm product constraint. This theoretical result shows that sufficiently smooth Hölder functions can be uniformly approximated to arbitrary accuracy as the network depth increases, with the norm-product bound held fixed. We then propose a penalized least squares estimator for deep nonparametric regression. The resulting estimator has a uniformly bounded Lipschitz constant while still achieving the optimal nonparametric rate up to a logarithmic factor. As applications, we establish optimal prediction rates for two-stage learning and provide robustness guarantees under adversarial input perturbations. Numerical studies show that the penalized estimator has well-controlled sensitivity to input perturbations and prediction accuracy comparable to that of classical deep network methods.
Figures & tables
MTE(SE)(10−3)
MSmax (SE)
Function
d
σε
FNN
Penalized
FNN
Penalized
True
fA,d
1
0.1
1.08 (0.02)
1.13 (0.04)
20.75 (0.60)
14.27 (0.20)
12.56
1
0.2
3.28 (0.10)
3.64 (0.17)
23.52 (0.94)
14.24 (0.32)
12.56
1
0.3
6.45 (0.18)
6.92 (0.31)
28.45 (1.37)
14.46 (0.34)
12.56
1
0.4
11.00 (0.36)
11.25 (0.62)
34.34 (1.98)
15.18 (0.53)
12.56
2
0.1
8.00 (0.23)
7.78 (0.18)
24.25 (1.09)
12.44 (0.43)
8.88
Table 1: Estimation error and input sensitivity at n=500 . Bold marks the smaller result.
Fig 1: MTE of FNN and penalized estimator. The three columns correspond to fA,d (left), fB,d (middle) and fC,d (right). The top and bottom rows correspond to d=1 and d=2 , respectively. Within each panel, blue circles represent FNN and orange squares represent penalized estimator. Different line styles indicate different noise levels.
This paper establishes a theoretical framework for the uniform convergence of smoothly activated deep neural network (DNN) estimators. While standard ReLU networks achieve minimax-optimal rates in the L2(P) norm for various nonparametric regression tasks, we establish a theoretical lower bound demonstrating that least-squares ReLU estimators can suffer from the curse of dimensionality in their uniform convergence behavior. Motivated by the need for reliable uniform guarantees in downstream tasks requiring worst-case reliability, we address this limitation by analyzing smoothly activated DNNs (smooth DNNs), encompassing both feedforward and residual structures. We establish novel pseudo-dimension bounds, non-asymptotic approximation guarantees, and Hölder-norm bounds for the approximators of these models. Leveraging these results, we derive non-asymptotic uniform convergence rates for smooth DNN estimators across multiple statistical contexts, including Huber, least-squares, quantile, and logistic regression. We prove that smooth DNNs can mitigate the {curse of dimensionality} in uniform convergence by adaptively exploiting the low-dimensional hierarchical composition structure of the target function. Supported by both simulation studies and a real-world application, our results position smooth DNNs as a theoretically grounded and practically viable alternative to ReLU networks for statistical learning tasks requiring uniform guarantees.
Yizhe Ding, Runze Li, Jia Liu +1
Department of Statistics, The Pennsylvania State University
This paper develops theory for deep neural network (DNN) estimators under dependent data. To provide theory applicable to a variety of DNN-based estimators, I first establish nonasymptotic probability bounds on the theoretical and empirical L2-errors of nonparametric sieve estimators for a general class of estimation problems under possibly nonstationary β-mixing data taking values in unbounded sets. I then apply the theory to fully connected and convolutional DNN estimators without bounds or sparsity restrictions on the DNN weights. For both DNN classes, I derive general results when the function to be estimated is Hölder smooth and the data are nonstationary, subgaussian, and β-mixing with either exponential or polynomial decay. I then specialize these to nonparametric regression, logistic regression, and quantile regression settings. Under exponential β-mixing, the resulting estimators attain the nonparametric minimax rate of Stone (1982) up to logarithmic factors.
Chad Brown
Department of Economics, University of Colorado Boulder, 80309, USA.
In this article, we study deep nonparametric regression trained via stochastic gradient Langevin dynamics (SGLD), a variant of stochastic gradient descent that injects carefully calibrated Gaussian noise during optimization. Using a PAC-Bayesian analysis tailored to deep ReLU networks, we establish an excess-risk bound that simultaneously accounts for network approximation error, statistical complexity, and the algorithmic cost accumulated along the SGLD trajectory. Under a Bernstein condition and within a stable SGLD regime, the excess-risk bound contains a stochastic term whose coefficient is inversely proportional to the variance of injected noise, while the estimator retains the optimal rate for nonparametric regression under the Hölder smoothness condition up to logarithmic factors. This result provides an approximation-error-aware, rate-optimal statistical perspective on the widely observed generalization benefits of moderate SGLD noise. For per-sample clipped SGLD, we further establish an exact f-DP guarantee and the corresponding asymptotic μ-GDP guarantee for the proposed estimator. We establish statistical convergence rates and differential privacy guarantees for DNN estimators trained using SGLD, while explicitly accounting for both network approximation error and the effect of injected noise in the excess-risk bound. Extensive simulation studies and analyses of two real datasets examine the finite-sample predictive performance and privacy-utility behavior of the proposed procedure.
Yeheng Ge, Jian Huang, Shuyuan Wu
Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University · Department of Applied Mathematics, The Hong Kong Polytechnic University · School of Statistics and Data Science, Shanghai University of Finance and Economics