Dense ReLU Neural Networks for Temporal-spatial Model

Dense ReLU Neural Networks for Temporal-spatial Model

Abstract

Figures & tables

Nonparametric Regression

Mitigating the Curse of Dimensionality in Uniform Convergence of Deep Neural Networks via Smooth Activations

Abstract

Nonparametric Regression

Explore similar work

Mitigating the Curse of Dimensionality in Uniform Convergence of Deep Neural Networks via Smooth Activations

Generalized Deep Regression for Repeated Measurements

Deep Neural Variation Spaces: A Unifying Perspective on Depth and Complexity

Carlos Misael Madrid Padilla; Zhi Zhang; Xiaokai Luo; Daren Wang; Oscar Hernan Madrid Padilla

stat.MLNov 15, 2024

Authors: Carlos Misael Madrid Padilla, Zhi Zhang, Xiaokai Luo, Daren Wang, Oscar Hernan Madrid Padilla

Organizations: Department of Statistics and Data Science, Washington University in St. Louis. · Department of Statistics and Data Science, University of California, Los Angeles · Department of Statistics, University of Notre Dame

In this paper, we focus on fully connected deep neural networks utilizing the Rectified Linear Unit (ReLU) activation function for nonparametric estimation. We derive non-asymptotic bounds that lead to convergence rates, addressing both temporal and spatial dependence in the observed measurements. By accounting for dependencies across time and space, our models better reflect the complexities of real-world data, enhancing both predictive performance and theoretical robustness. We also tackle the curse of dimensionality by modeling the data on a manifold, exploring the intrinsic dimensionality of high-dimensional data. We broaden existing theoretical findings of temporal-spatial analysis by applying them to neural networks in more general contexts and demonstrate that our proof techniques are effective for models with short-range dependence. Our empirical simulations across various synthetic response functions underscore the superior performance of our method, outperforming established approaches in the existing literature. These findings provide valuable insights into the strong capabilities of dense neural networks (Dense NN) for temporal-spatial modeling across a broad range of function classes.

Setting / Assumptions	Rate	Dominant regime
This work
Temporal-spatial $\gamma_{i}+\epsilon_{ij}$ , $\beta$ -mixing	$\displaystyle\underbrace{\max_{(p,K)\in\mathcal{P}}(nm)^{-\frac{2p}{2p+K}}\log^{7.3}(nm)}_{\text{Term 1}}+\underbrace{\frac{\log(nm)}{n}}_{\text{Term 2}}$	$m\leq n^{K/(2p)}\,\mathrm{poly}(\log(nm))$ Term 1 $m\geq n^{K/(2p)}\,\mathrm{poly}(\log(nm))$ Term 2
i.i.d. functional $\gamma_{i}+\epsilon_{ij}$ , i.i.d.	$\displaystyle\underbrace{\max_{(p,K)\in\mathcal{P}}(nm)^{-\frac{2p}{2p+K}}\log^{6.3}(nm)}_{\text{Term 1}}+\underbrace{\frac{1}{n}}_{\text{Term 2}}$	$m\leq n^{K/(2p)}\,\mathrm{poly}(\log(nm))$ Term 1 $m\geq n^{K/(2p)}\,\mathrm{poly}(\log(nm))$ Term 2
Temporal only $\epsilon_{i,1}$ , $\gamma_{i}=0$ , $\beta$ -mixing, $m_{i}=1$	$\displaystyle\max_{(p,K)\in\mathcal{P}}n^{-\frac{2p}{2p+K}}\log^{7.3}(n)$	N/A
Related work
Kohler and Langer (2021) $\epsilon_{i,1}$ , $\gamma_{i}=0$ , $m_{i}=1$ , i.i.d., dense NN	$\displaystyle\max_{(p,K)\in\mathcal{P}}n^{-\frac{2p}{2p+K}}\log^{c^{\prime}}(n)$	N/A

Table 1: Summary of rates, assumptions, and dominant regimes. The regime boundaries are stated up to polynomial factors in

\log(nm)

. That is,

\mathrm{poly}(\log(nm))=(\log(nm))^{c}

with

c=\frac{6.3(2p+K)}{2p}

, arising from balancing the two terms in the rate.

Panel of Figure 1 — Figure 1: Comparison of methods across different scenarios, $d=7$ , $n$ , and $m_{mult}$ by box-plot. For each $(n,m_{mult})$ setting, we present the errors of the competitors in the range of within 5 times the smallest error.

Panel of Figure 2 — Figure 2: Comparison of methods across different scenarios, $d=10$ , $n$ , and $m_{mult}$ by box-plot.

Panel of Figure 3 — Figure 3: Visualization of predicted ozone levels for the 45 monitoring locations on March 3rd. Subplots (a) show the $y$ , the observed ozone levels, while subplots (b) to (f) represent predictions from Dense NN, GAM, KNN-FL, RKHS, and Trend Filtering, respectively.

Dense NN	GAM	KNN-FL	RKHS	Trend Filtering
0.0086 (0.009)	0.0574 (0.015)	0.0255 (0.010)	0.1374 (0.022)	0.0518 (0.020)

Table 2: Prediction performance (Relative Error) of different methods for ozone level prediction. The values in parentheses represent standard deviations.

d	scenario	$n$ : 500, $m_{mult}$ : 1
d	scenario	Dense NN	GAM	KNN-FL	RKHS	Trend Filtering
2	1	0.0676 (0.012)	0.3087 (0.044)	0.1249 (0.008)	1.2926 (0.135)	0.3196 (0.058)
	2	0.0133 (0.002)	0.0322 (0.002)	0.0491 (0.003)	0.1678 (0.039)	0.0370 (0.002)
	3	0.0061 (0.002)	0.3547 (0.004)	0.0616 (0.004)	0.4069 (0.004)	0.3547 (0.004)
	4	0.0008 (0.000)	0.0013 (0.000)	0.0068 (0.001)	0.2007 (0.005)	0.0014 (0.000)
	5	0.0640 (0.005)	0.4844 (0.142)	0.2419 (0.016)	0.9635 (0.196)	0.6408 (0.042)

Table 1: Results of relative error for different methods,

d

n

and

m_{mult}

: Mean (Std). We highlight the best result in bold and the second best result in bold and italic .

d	scenario	$n$ : 500, $m_{mult}$ : 2
d	scenario	Dense NN	GAM	KNN-FL	RKHS	Trend Filtering
2	1	0.0550 (0.016)	0.3043 (0.032)	0.1291 (0.021)	1.2267 (0.060)	0.3075 (0.018)
	2	0.0066 (0.000)	0.0178 (0.001)	0.0306 (0.002)	0.2019 (0.013)	0.0316 (0.002)
	3	0.0049 (0.000)	0.3598 (0.002)	0.0512 (0.003)	0.4146 (0.003)	0.3576 (0.003)
	4	0.0005 (0.000)	0.0010 (0.000)	0.0045 (0.000)	0.2320 (0.001)	0.0012 (0.000)
	5	0.0817 (0.006)	0.4716 (0.145)	0.2413 (0.016)	0.9443 (0.027)	0.4912 (0.079)

Table 2: Results of relative error for different methods,

d

n

and

m_{mult}

: Mean (Std).

d	scenario	$n$ : 1000, $m_{mult}$ : 1
d	scenario	Dense NN	GAM	KNN-FL	RKHS	Trend Filtering
2	1	0.0350 (0.008)	0.2867 (0.008)	0.0981 (0.003)	1.2588 (0.108)	0.2891 (0.004)
	2	0.0131 (0.001)	0.0260 (0.006)	0.0595 (0.004)	0.1517 (0.039)	0.0331 (0.002)
	3	0.0069 (0.001)	0.3588 (0.001)	0.0443 (0.003)	0.4139 (0.001)	0.3599 (0.002)
	4	0.0006 (0.000)	0.0011 (0.000)	0.0032 (0.001)	0.2202 (0.005)	0.0012 (0.000)
	5	0.0516 (0.004)	0.4033 (0.010)	0.2538 (0.052)	0.7392 (0.039)	0.4147 (0.014)

Table 3: Results of relative error for different methods,

d

n

and

m_{mult}

: Mean (Std)

d	scenario	$n$ : 1000, $m_{mult}$ : 2
d	scenario	Dense NN	GAM	KNN-FL	RKHS	Trend Filtering
2	1	0.0317 (0.002)	0.2918 (0.009)	0.0591 (0.004)	1.3259 (0.033)	0.2969 (0.016)
	2	0.0120 (0.001)	0.0181 (0.001)	0.0327 (0.002)	0.1602 (0.026)	0.0181 (0.001)
	3	0.0058 (0.000)	0.3578 (0.002)	0.0366 (0.002)	0.4165 (0.003)	0.3584 (0.002)
	4	0.0004 (0.000)	0.0010 (0.000)	0.0027 (0.000)	0.2328 (0.002)	0.0011 (0.000)
	5	0.0403 (0.003)	0.4226 (0.035)	0.1005 (0.007)	1.0360 (0.044)	0.4420 (0.069)

Table 4: Results of relative error for different methods,

d

n

and

m_{mult}

: Mean (Std)

d	scenario	$n$ : 2000, $m_{mult}$ : 1
d	scenario	Dense NN	GAM	KNN-FL	RKHS	Trend Filtering
2	1	0.0294 (0.005)	0.2883 (0.004)	0.0521 (0.004)	1.2793 (0.047)	0.2888 (0.006)
	2	0.0091 (0.001)	0.0323 (0.002)	0.0396 (0.003)	0.1610 (0.020)	0.0231 (0.002)
	3	0.0052 (0.000)	0.3578 (0.002)	0.0270 (0.003)	0.4162 (0.003)	0.3571 (0.002)
	4	0.0004 (0.000)	0.0010 (0.000)	0.0045 (0.000)	0.2255 (0.001)	0.0011 (0.000)
	5	0.0338 (0.008)	0.4051 (0.013)	0.0957 (0.013)	0.9558 (0.043)	0.4108 (0.018)

Table 5: Results of relative error for different methods,

d

n

and

m_{mult}

: Mean (Std)

d	scenario	$n$ : 2000, $m_{mult}$ : 2
d	scenario	Dense NN	GAM	KNN-FL	RKHS	Trend Filtering
2	1	0.0153 (0.003)	0.2794 (0.002)	0.0328 (0.002)	1.3257 (0.007)	0.2792 (0.001)
	2	0.0031 (0.000)	0.0095 (0.002)	0.0144 (0.001)	0.1489 (0.001)	0.0103 (0.002)
	3	0.0031 (0.000)	0.3587 (0.002)	0.0404 (0.011)	0.4207 (0.002)	0.3588 (0.002)
	4	0.0003 (0.000)	0.0010 (0.000)	0.0039 (0.000)	0.2338 (0.001)	0.0010 (0.000)
	5	0.0163 (0.002)	0.3773 (0.004)	0.0606 (0.002)	0.9943 (0.017)	0.3782 (0.004)

Table 6: Results of relative error for different methods,

d

n

and

m_{mult}

: Mean (Std)

Panel of Figure 4 — Figure 4: Comparison of methods across different scenarios, $d=5$ , $n$ , and $m_{mult}$ by box-plot. For each $(n,m_{mult})$ setting, we present the errors of the competitors in the range of within 5 times the smallest error.

Table 7: Hyperparameters for Dense NN Selected by

5

-fold Cross-validation

Hyperparameter	Scenario
Hyperparameter	1	2	3	4	5	6
basis $\{\text{'ps': p-spline basis, 'cp': cyclic p-spline basis}\}$ (Basis Type)	ps	ps	ps	ps	ps	ps
constraint $\{\text{None, 'convex', 'concave', \noindent\hbox{}\hfill{{\hbox{\begin{tabular}[c]{@{}c@{}}mono\\ inc\end{tabular}}}}\hfill\hbox{}, \noindent\hbox{}\hfill{{\hbox{\begin{tabular}[c]{@{}c@{}}mono\\ dec\end{tabular}}}}\hfill\hbox{}}\}$ (Constraint Type)	mono dec	mono dec	convex	None	mono dec	mono inc
$\lambda$ , with $\log_{10}(\lambda)\in[-3,3]$ (Regularization Parameter)	63.1	0.001	0.25	3.98	1000	0.015
n_splines $\in\{6,7,8,9,10\}$ (Number of Splines)	7	8	9	6	6	7
spline $\in\{1,2,3,4,5\}$ (Spline Order)	1	1	2	4	5	5

Table 8: Hyperparameters for GAM Selected by

5

-fold Cross-validation

Hyperparameter	Scenario
Hyperparameter	1	2	3	4	5	6
lambda $\in\{0.00001,0.0001,0.01,0.1,1,10,100\}$ (Regularization Parameter)	0.01	0.01	0.00001	0.0001	0.01	0.01
n_neighbors $\in\{5,10\}$ (Number of Neighbors)	10	10	5	5	5	5

Table 9: Hyperparameters for KNN-FL Selected by

5

-fold Cross-validation

Hyperparameter	Scenario
Hyperparameter	1	2	3	4	5	6
$\alpha$ , with $\log_{10}(\alpha)\in[-2,2]$ (Regularization Parameter)	100	100	100	100	100	100
kernel $\in\{\text{polynomial, laplacian, linear}\}$ (Kernel Type)	polynomial	polynomial	laplacian	polynomial	polynomial	laplacian

Table 10: Hyperparameters for RKHS Selected by

5

-fold Cross-validation

Hyperparameter	Scenario
Hyperparameter	1	2	3	4	5	6
$k\in\{0,1,2\}$ (Trend Order)	1	0	1	0	1	1
lambda, $\log_{10}(\lambda)\in[-5,2]$ (Regularization Parameter)	0.0007	0.8133	0.0001	0.1128	7.8476	0.6158

Table 11: Hyperparameters for Trend Filtering Selected by

5

-fold Cross-validation

Nonparametric regression estimates flexible relationships between predictors and outcomes without fixing a finite-dimensional model in advance.

See all 45 papers

Mitigating the Curse of Dimensionality in Uniform Convergence of Deep Neural Networks via Smooth Activations
Jun 4, 2026Yizhe Ding, Runze Li, Jia Liu +1ReLU Neural Networks Curse of Dimensionality
Department of Statistics, The Pennsylvania State University
Generalized Deep Regression for Repeated Measurements
Sep 22, 2026Kexuan LiNeural Network Generalization ReLU Neural Networks
Bristol Myers Squibb
Deep Neural Variation Spaces: A Unifying Perspective on Depth and Complexity
Jul 6, 2026Julia Nakhleh, Robert D. NowakReLU Neural Networks Neural Network Approximation Theory
Department of Computer Science · University of Wisconsin-Madison · Department of Electrical & Computer Engineering

Jun 4, 2026J/K move · Enter open · S save

Yizhe Ding, Runze Li, Jia Liu, Lingzhou Xue

Department of Statistics, The Pennsylvania State University

ReLU Neural Networks Curse of Dimensionality Neural Network Approximation Theory Statistical Learning Theory

Open paper AskPDF arXiv

This paper establishes a theoretical framework for the uniform convergence of smoothly activated deep neural network (DNN) estimators. While standard ReLU networks achieve minimax-optimal rates in the

L^2(P)

norm for various nonparametric regression tasks, we establish a theoretical lower bound demonstrating that least-squares ReLU estimators can suffer from the curse of dimensionality in their uniform convergence behavior. Motivated by the need for reliable uniform guarantees in downstream tasks requiring worst-case reliability, we address this limitation by analyzing smoothly activated DNNs (smooth DNNs), encompassing both feedforward and residual structures. We establish novel pseudo-dimension bounds, non-asymptotic approximation guarantees, and Hölder-norm bounds for the approximators of these models. Leveraging these results, we derive non-asymptotic uniform convergence rates for smooth DNN estimators across multiple statistical contexts, including Huber, least-squares, quantile, and logistic regression. We prove that smooth DNNs can mitigate the {curse of dimensionality} in uniform convergence by adaptively exploiting the low-dimensional hierarchical composition structure of the target function. Supported by both simulation studies and a real-world application, our results position smooth DNNs as a theoretically grounded and practically viable alternative to ReLU networks for statistical learning tasks requiring uniform guarantees.

Hyperparameter

Hyperparameter

$L$ $\in\{1,2,3,4,5,10\}$ (Layers)

$r$ $\in\{30,60,100,300\}$ (Neurons)