stat.MLNov 18, 2026

Optimal Deep Nonparametric Regression with Uniformly Bounded Lipschitz Constants

Authors: Jiaxin Shi, Ziyuan Chen, Jian Huang, Hansheng Wang

Organizations: Guanghua School of Management, Peking University, Beijing, China · School of Mathematical Sciences, Peking University, Beijing, China · Department of Data Science and Artificial Intelligence, Department of Applied Mathematics, Hong Kong Polytechnic University, Hong Kong SAR, China

Abstract

Stability under input perturbations is necessary for deep neural network estimators in many applications involving noisy covariates or estimated features. Weight norm constraints are widely used to control sensitivity to input perturbations by bounding the Lipschitz constant of neural networks. However, existing approximation results under fixed bounds on standard norm products are insufficient to establish optimal statistical rates. In this paper, we establish a sharp approximation theory for deep ReLU networks under a split spectral norm product constraint. This theoretical result shows that sufficiently smooth Hölder functions can be uniformly approximated to arbitrary accuracy as the network depth increases, with the norm-product bound held fixed. We then propose a penalized least squares estimator for deep nonparametric regression. The resulting estimator has a uniformly bounded Lipschitz constant while still achieving the optimal nonparametric rate up to a logarithmic factor. As applications, we establish optimal prediction rates for two-stage learning and provide robustness guarantees under adversarial input perturbations. Numerical studies show that the penalized estimator has well-controlled sensitivity to input perturbations and prediction accuracy comparable to that of classical deep network methods.

Figures & tables

Explore similar work

CardsList
  1. Mitigating the Curse of Dimensionality in Uniform Convergence of Deep Neural Networks via Smooth Activations

    Jun 4, 2026Yizhe Ding, Runze Li, Jia Liu +1ReLU Neural NetworksCurse of Dimensionality

  2. Statistical Properties of Deep Neural Networks with Dependent Data

    Oct 14, 2024Chad BrownNonparametric RegressionNeural Network Approximation Theory

  3. Statistical and Privacy Guarantees for Deep M-Regression via Stochastic Gradient Langevin Dynamics

    Nov 5, 2026Yeheng Ge, Jian Huang, Shuyuan WuStochastic Gradient Langevin DynamicsNonparametric Regression