Error Analysis of Deep Learning
Organizations: School of Artificial Intelligence, National Center for Applied Mathematics in Hubei, Hubei Key Laboratory of Computational Science, Wuhan University, Wuhan, China. · School of Mathematics and Statistics, Wuhan University, Wuhan, China. · National Center for Applied Mathematics in Hubei, Wuhan Institute for Math & AI, School of Mathematics and Statistics, Hubei Key Laboratory of Computational Science, Wuhan University, Wuhan, China. · Wuhan Institute for Math & AI, Wuhan University, Wuhan, School of Mathematical Sciences, Peking University, Beijing, China.
Abstract
This review examines deep learning theory through the lens of error decomposition, systematically addressing approximation, statistical, and optimization errors. We trace the development of error analyses from bounds on individual error components to frameworks that account for all three. Using nonparametric regression and the Deep Ritz Method as dual running examples, we demonstrate how these analyses apply across fundamentally different loss structures. We then review representative analyses in scientific machine learning, generative learning, reinforcement learning, representation learning, and large language models, showing how learning errors affect solutions, distributions, and predictions through problem-specific arguments. Throughout, we clarify which error components are controlled, how they are combined, and what assumptions are needed to obtain guarantees for trained networks.
Figures & tables
| Problem | Squared-error lower bound | |||
| Hölder regression | ||||
| Noisy-RHS elliptic recovery |
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Abbreviation | Meaning |
|---|---|
| DRM | Deep Ritz Method |
| ERM | Empirical Risk Minimizer (or Empirical Risk Minimization) |
| PNN | Parallel Neural Network |
| PGD | Projected Gradient Descent |
| PSGD | Projected Stochastic Gradient Descent |
| PDE | Partial Differential Equation |
| Symbol | Meaning | Definition / location |
|---|---|---|
| Input dimension | § 2.1 | |
| Network depth (number of affine layers, including the output layer) | § 2.1 , ( 1 ) | |
| Maximum hidden-layer width ( ) | § 2.1 | |
| Number of neurons at layer | § 2.1 | |
| Activation function | § 2.1 | |
| Weight matrix and bias at layer | § 2.1 |
| Symbol | Meaning | Definition / location |
|---|---|---|
| Point-wise loss function | § 2.2 | |
| Population (true) risk | § 2.2 | |
| Empirical risk on samples | § 2.2 | |
| Bayes risk | § 2.2 | |
| Bayes-optimal function | § 2.2 | |
| Best in-class approximator | § 2.2 |
| Symbol | Meaning | Definition / location |
|---|---|---|
| Hölder ball of smoothness , radius | § 2.7 , ( 19 ) | |
| Sobolev space, smoothness , integrability | § 2.7 | |
| Sobolev space (Hilbert case) | § 2.7 | |
| Hölder smoothness parameter | § 2.6.1 | |
| Sobolev smoothness order | § 2.6.2 | |
| Hölder ball radius (Regression smoothness bound) | § 2.6.1 |
| Symbol | Meaning | Definition / location |
|---|---|---|
| Regression | ||
| True regression function (conditional mean) | § 2.6.1 , ( 5 ), ( 122 ) | |
| Predictor and response | § 2.6.1 | |
| Noise term | § 2.6.1 | |
| Noise bound, sub-Gaussian scale, or Gaussian standard deviation | § 3.3.1 | |
| Marginal distribution of | § 2.6.1 | |
| Symbol | Meaning | Definition / location |
|---|---|---|
| Network parameters ( controls width, controls depth) | § 3.1 , Thm 3 | |
| Partition-grid resolution | § 3.1.1 | |
| Arbitrarily small parameter in the tanh approximation theorems | § 3.1.2 , Thm 6 | |
| Weight-magnitude exponent of the tanh approximation theorems | Theorem 6 | |
| Bump sharpness parameter ( ) | § 3.1.2 proof | |
| (tanh) | Tanh Taylor scaling parameter | § 3.1 |
| Symbol | Meaning | Definition / location |
|---|---|---|
| Rademacher signs ( with prob. ) | § 3.2 | |
| Empirical Rademacher complexity | § 3.2 , ( 20 ) | |
| Expected symmetric Rademacher complexity (with ) | § 3.2 , ( 21 ) | |
| Expected Rademacher complexity | § 3.2 | |
| -covering number | § 3.2 , Prop 15 | |
| Pseudo-dimension | § 3.2 |
| Symbol | Meaning | Definition / location |
|---|---|---|
| Projection radius for inner weights | § 5.1.2 | |
| Projection radius for outer coefficients | § 5.1.2 | |
| -ball constraint set on | § 5.1.2 | |
| -ball constraint set on | § 5.1.2 | |
| (PGD) | Total number of PGD iterations | § 5.1.2 |
| Step size | § 5.1.2 |
| Symbol | Meaning | Definition / location |
|---|---|---|
| Minimax risk over class , metric , experiment | § 3.4 , ( 33 ) | |
| Metric in which estimation accuracy is evaluated | § 3.4 | |
| Statistical experiment parameter ( ) | § 3.4 | |
| Target-norm order in the Fano template ( ) | § 3.4 | |
| Observation-operator order in the Fano template ( ) | § 3.4 | |
| , | Separation and distinguishability exponents in the Fano template | § 3.4 , eqs. ( 35 )–( 36 ) |
| Symbol | Meaning | Used in |
|---|---|---|
| Scientific machine learning | ||
| True state and source in the inverse problem | § 6.1 | |
| Scalar and gradient observation noises, and their componentwise amplitude bound | § 6.1 | |
| Interior and boundary sample sizes | § 6.1 | |
| Population and empirical PINN objectives | § 6.1 | |
| (inverse) | State and source candidate network classes for the inverse problem | § 6.1 , ( 74 ) |
| Label | Named result | Location |
|---|---|---|
| A1 | ReLU approximation of Hölder functions | Theorem 3 |
| A2 | Gühring–Raslan tanh Sobolev approximation | Theorem 6 |
| A3 | Norm-controlled ReLU approximation (Jiao et al.) | Theorem 32 |
| A4 | Norm-controlled tanh approximation (Ding et al.) | Theorem 33 |
| A5 | PNN tanh approximation (Jiao et al.) | Theorem 40 |
| A5r | PNN ReLU approximation (Kohler–Cho–Krzyżak) | Theorem 39 |
| Theorem | Reference | Rate | Building blocks |
|---|---|---|---|
| Classical size-controlled (§ 3 , ) | |||
| Regression slow rate | Thm 22 | A1 + S3, symmetric decomp ( 8 ) | |
| DRM slow rate | Thm 26 | A2 ( ) + S5 + Prop 2 | |
| Regression fast rate | Thm 24 | A1 + S4 (peeling), asymmetric decomp ( 9 ) | |
| DRM fast rate (external) | Thm 28 | + localized Rademacher + Dirichlet energy coercivity [ 115 ] | |
| Over-parameterized (§ 4 , ) | |||
Statistical Learning Theory
Statistical learning theory studies when data-driven algorithms generalize, how many samples they need, and what statistical or computational limits apply.
See all 635 papers