On the Generalization Behavior of Deep Residual Networks From a Dynamical System Perspective

Jinshu Huang; Mingfei Sun; Chunlin Wu

cs.LGFeb 24, 2026

On the Generalization Behavior of Deep Residual Networks From a Dynamical System Perspective

Authors: Jinshu Huang, Mingfei Sun, Chunlin Wu

Organizations: School of Mathematical Sciences, Nankai University, Tianjin, China

Abstract

Deep neural networks (DNNs) have significantly advanced machine learning, with model depth playing a central role in their successes. The dynamical system modeling approach has recently emerged as a powerful framework, offering new mathematical insights into the structure and learning behavior of DNNs. In this work, we establish generalization error bounds for both discrete- and continuous-time residual networks (ResNets) by combining Rademacher complexity, flow maps of dynamical systems, and the convergence behavior of ResNets in the deep-layer limit. The resulting bounds are of order $O(1/\sqrt{S})$ with respect to the number of training samples $S$ , and include a structure-dependent negative term, yielding depth-uniform and asymptotic generalization bounds under milder assumptions. These findings provide a unified understanding of generalization across both discrete- and continuous-time ResNets, helping to close the gap in both the order of sample complexity and assumptions between the discrete- and continuous-time settings.

Figures & tables

Symbol	Meaning	Appears in
$n_{\rm d}/n$	Dimension of input/output	Subsection 2.2
$L$	Number of layers	Eq. ( 1 )
$\tau_{L}$	Discretization step	Eq. ( 1 )
$\Theta_{L}^{\rm total}$	Discrete-time learnable parameter	Subsection 2.2
$\mathrm{x}^{l}(\cdot;{\Theta}^{\rm total}_{L})$	Output function of the $l$ -th layer in Eq.( 1 ) with ${\Theta}^{\rm total}_{L}$	Subsection 2.2
$\mathcal{F}^{l}$	Hypothesis set of discrete-time ResNet at $l$ -th	Eq. ( 3 )

Table 1: Summary of key notations and their first appearances in the paper.

Activation functions	$\qquad\psi$	$\phi_{1}$	$\phi_{2}$	$(\alpha,\beta)$
ReLU [ 14 ]	$\left\{\begin{array}[]{ll}x,\ x\geq 0\\ 0,\ x<0\end{array}\right.$	$x\boldsymbol{1}_{[0,\infty)}(x)$	$0$	$(0,\beta)$
PReLU [ 22 ]	$\left\{\begin{array}[]{ll}x,\ x\geq 0\\ a_{1}x,\ x<0\end{array}\right.$	$x\boldsymbol{1}_{[0,\infty)}(x)$	$a_{1}x\boldsymbol{1}_{[0,\infty)}(x)$	$(0,0)$
TReLU	$\left\{\begin{array}[]{ll}x-\lambda,\ x\geq\lambda\\ 0,\ x<\lambda\end{array}\right.$	$x\boldsymbol{1}_{[0,\infty)}(x)$	$0$	$(\lambda,\beta)$
ELU [ 10 ]	$\left\{\begin{array}[]{ll}x,\ x\geq 0\\ a_{2}(e^{x}-1),\ x<0\end{array}\right.$	$x\boldsymbol{1}_{[0,\infty)}(x)$	$\begin{array}[]{ll}-a_{2}(e^{-x}-1)\\ \ \ \cdot\boldsymbol{1}_{[0,\infty)}(x)\end{array}$	(0,0)
TEReLU [ 38 ]	$\left\{\begin{array}[]{ll}x-\lambda_{1},\ x\geq\lambda_{1}\\ 0,\ -\lambda_{2}<x<\lambda_{1}\\ a_{2}(e^{x+\lambda_{2}}\!-\!1),\!x\!\leq\!-\lambda_{2}\end{array}\right.$	$x\boldsymbol{1}_{[0,\infty)}(x)$	$\begin{array}[]{ll}-a_{2}(e^{-x}-1)\\ \ \ \cdot\boldsymbol{1}_{[0,\infty)}(x)\end{array}$	$(\lambda_{1},\lambda_{2})$
${\mathcal{T}}_{\lambda}$ [ 4 ]	$\left\{\begin{array}[]{ll}x-\lambda,\ x\geq\lambda\\ 0,\ -\lambda<x<\lambda\\ x+\lambda,\ x\leq-\lambda&\end{array}\right.$	$x\boldsymbol{1}_{[0,\infty)}(x)$	$x\boldsymbol{1}_{[0,\infty)}(x)$	( $\lambda,\lambda$ )

Table 2: Examples of activation functions in

\mathscr{A}(\mathbb{R})

. The parameters

a_{1}\in[0,1]

a_{2},\lambda,\lambda_{1},\lambda_{2}>0

Figure 1: Average testing-training loss gap of ResNets with $(T,L)=(6,6),(6,24)$ and $(T,L)=(8,8),(8,32)$ at last 10 epochs versus training sample size $S$ , along with least-squares fitting using $h_{\mu}(S)=\mu/\sqrt{S}$ . Plots (a1)-(a4) show results of ResNets on MNIST. Plots (b1)-(b4) show results of ResNets on CIFAR10. Plots (c1)-(c4) show results of ResNets on CIFAR100. The generalization gap decreases with increasing $S$ , closely matching the theoretical rate of $O(1/\sqrt{S})$ .

Testing accuracy
(T, L)	Samples	$10^{4}$	$2\cdot 10^{4}$	$3\cdot 10^{4}$	$4\cdot 10^{4}$	$5\cdot 10^{4}$
(6, 6)	MNIST	99.549	99.589	99.639	99.669	99.629
	CIFAR10	84.375	89.083	90.865	92.348	92.839
	CIFAR100	49.189	59.165	66.216	68.139	72.516
(6, 24)	MNIST	99.479	99.589	99.579	99.659	99.669
	CIFAR10	84.896	89.694	91.526	92.728	93.58

Table 3: The top-1 image classification testing accuracy by trained ResNets with

(T,L)=(6,6),(6,24)

and

(T,L)=(8,8),(8,32)

on MNIST, CIFAR10 and CIFAR100 datasets with different training sample sizes

S

Panel of Figure 2 — Figure 2: Training loss of ResNets with varying layer numbers in the deep-layer limit regime. The first row corresponds to $T=6$ with $L=3,6,12,24$ , and the second row corresponds to $T=8$ with $L=4,8,16,32$ . Across all datasets (MNIST, CIFAR10, CIFAR100), the training loss exhibits convergence behavior as the number of layers increases, supporting the depth-stability predicted by our theoretical analysis.

Panel of Figure 3 — Figure 3: Top-1 image classification accuracy of ResNets with varying layer numbers in the deep-layer limit regime. The first row corresponds to $T=6$ with $L=3,6,12,24$ , and the second row corresponds to $T=8$ with $L=4,8,16,32$ . Across all datasets (MNIST, CIFAR10, CIFAR100), the testing accuracy exhibits convergence behavior as the number of layers increases, supporting the depth-stability predicted by our theoretical analysis.

Figure 4: Empirical generalization gap $\hat{\mathfrak{R}}_{\rm test}-\hat{\mathfrak{R}}_{\rm train}$ versus training epoch for ResNets with different $(T,L)$ configurations. The first row corresponds to MNIST, the second row to CIFAR10, and the third row to CIFAR100. ResNets equipped with activation functions admitting a structured decomposition exhibit smaller generalization gaps on CIFAR10 and CIFAR100 in the later stage of training.

Neural Network Generalization

Neural network generalization studies why trained models perform well on unseen data and how architecture, data, and training shape that ability.

See all 299 papers

Learning Non-Vacuous Generalization Bounds from Optimization
Jun 9, 2022Chengli Tan, Jiangshe Zhang, Junmin Liu +1Neural Network Generalization Statistical Learning Theory
Northwestern Polytechnical University, Xi’an, China · Xi’an Jiaotong University, Xi’an, China · SGIT AI Lab, State Grid Corporation of China, Xi’an, China
ResNets of All Shapes and Sizes: Convergence of Training Dynamics in the Large-scale Limit
Mar 18, 2026Louis-Pierre Chaintron, Lénaïc Chizat, Javier MaassNeural Network Training Dynamics Residual Dynamics Learning
École Polytechnique Fédérale de Lausanne (EPFL), Institute of Mathematics, 1015 Lausanne, Switzerland.
Pointwise Generalization in Deep Neural Networks
May 18, 2026Shaojie Li, Yunbei XuNeural Network Generalization Representation Learning
National University of Singapore

Jun 9, 2022J/K move · Enter open · S save

Learning Non-Vacuous Generalization Bounds from Optimization

Chengli Tan, Jiangshe Zhang, Junmin Liu, Yihong Gong

Northwestern Polytechnical University, Xi’an, China · Xi’an Jiaotong University, Xi’an, China · SGIT AI Lab, State Grid Corporation of China, Xi’an, China

Neural Network Generalization Statistical Learning Theory Generalization Bounds

Open paper AskPDF arXiv

One of the fundamental challenges in the deep learning community is to theoretically understand how well a deep neural network generalizes to unseen data. However, current approaches often yield generalization bounds that are either too loose to be informative of the true generalization error or only valid to the compressed nets. In this study, we present a simple yet non-vacuous generalization bound from the optimization perspective. We achieve this goal by leveraging that the hypothesis set accessed by stochastic gradient algorithms is essentially fractal-like and thus can derive a tighter bound over the algorithm-dependent Rademacher complexity. The main argument rests on modeling the discrete-time recursion process via a continuous-time stochastic differential equation driven by fractional Brownian motion. Numerical studies demonstrate that our approach is able to yield plausible generalization guarantees for modern neural networks such as ResNet and Vision Transformer, even when they are trained on a large-scale dataset (e.g. ImageNet-1K).

Abstract

Neural Network Generalization

Explore similar work

Learning Non-Vacuous Generalization Bounds from Optimization

ResNets of All Shapes and Sizes: Convergence of Training Dynamics in the Large-scale Limit

Pointwise Generalization in Deep Neural Networks