cs.LGMar 3, 2025

Path Regularization: A Near-Complete and Optimal Nonasymptotic Generalization Theory for Multilayer Neural Networks and Double Descent Phenomenon

Authors: Hao Yu

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Path regularization has shown to be a very effective regularization to train neural networks, leading to a better generalization property than common regularizations i.e. weight decay, etc. We propose a first near-complete (as will be made explicit in the main text) nonasymptotic generalization theory for multilayer neural networks with path regularizations for general learning problems. In particular, it does not require the boundedness of the loss function, as is commonly assumed in the literature. Our theory goes beyond the bias-variance tradeoff and aligns with phenomena typically encountered in deep learning. It is therefore sharply different from other existing nonasymptotic generalization error bounds. More explicitly, we propose an explicit generalization error upper bound for multilayer neural networks with σ(0)=0σ(0)=0 and sufficiently broad Lipschitz loss functions, without requiring the width, depth, or other hyperparameters of the neural network to approach infinity, a specific neural network architecture (e.g., sparsity), or boundedness of the loss function, while also taking approximation error into consideration. In particular, we solve an open problem proposed by Weinan E et. al. in 2020 regarding the approximation rates in generalized Barron spaces. Furthermore, we show the near-minimax optimality of our theory for regression problems with ReLU activations. Notably, our upper bound exhibits the famous double descent phenomenon for such networks, which is the most distinguished characteristic compared with other existing results. Our subsequent work will prove the matching lower bounds in the minimax sense, meaning that it is highly possible that our theory reveals the true underlying mechanism of the double descent phenomenon. We can also explain scaling law from this theory.

Explore similar work

CardsList
May 18, 2026stat.ML

Shallow ReLUs^s Networks in LpL^p-Type and Sobolev Spaces: Approximation and Path-Norm Controlled Generalization

This paper studies approximation by shallow ReLUs^s networks, σs(t)=max⁡{0,t}sσ_s(t)=\max\{0,t\}^s, together with their generalization behavior under ℓ1\ell_1 path-norm control. For the LpL^p-type integral spaces F~p,τd,s\widetilde{\mathcal{F}}_{p,τ_d,s}, 1≤p≤21\le p\le2, spherical harmonic analysis yields approximation bounds for shallow networks. In particular, when τdτ_d is the uniform measure and 1≤p<21\le p<2, the approximation rate is O ⁣(m−p(2s+2d+1)−2d2dp)O\!\left(m^{-\frac{p(2s+2d+1)-2d}{2dp}}\right) for 1≤p≤p∗1\le p\le p^* and O ⁣(m−p(4s+3d−1)−2d+24dp)O\!\left(m^{-\frac{p(4s+3d-1)-2d+2}{4dp}}\right) for p∗<p<2p^*<p<2, where p∗=2d+2d+3p^*=\frac{2d+2}{d+3}. Approximation bounds for Sobolev spaces Wα,pW^{α,p}, 1≤p<21\le p<2, are obtained through embeddings into spectral Barron spaces. For nonparametric regression with sub-Gaussian noise, path-norm-regularized shallow ReLUs^s networks achieve minimax-optimal rates O ⁣(n−d+2s+12d+2s+1log⁡n)O\!\left(n^{-\frac{d+2s+1}{2d+2s+1}}\log n\right) over Bs\mathscr{B}_s and O ⁣(n−2α2α+dlog⁡n)O\!\left(n^{-\frac{2α}{2α+d}}\log n\right) over Wα,∞W^{α,\infty}, with matching lower bounds up to logarithmic factors.
Weizhao Li, Fanghui Liu, Lei Shi
Logarithmic Sobolev InequalitiesRectified Linear Unit NetworksNeural-Network ApproximationApproximationGeneralization BoundsStructural Generalization
AskPDF
Sep 7, 2026cs.LG

A Theoretical Analysis of Generalization Dynamics in Neural Networks under Gradient Descent with Weight Decay

Understanding generalization remains a central challenge in machine learning because it requires jointly considering data, architecture, and training dynamics. In this paper, we develop a theoretical framework that characterizes how these factors jointly shape generalization performance throughout training. More precisely, we study a broad class of neural networks trained under the ℓ2\ell^2 loss by gradient descent (GD) with weight decay, and prove the convergence of GD to a neighbourhood of the global minimizers of the empirical loss. By partitioning the space based on the input data, we then decompose the population error into data error, optimization error, and prediction variation error, and bound them separately. In particular, for the prediction variation error, which measures the oscillations of the learned function, we propose (local) approximate homogeneity and derive explicit cellwise and layerwise bounds for its evolution along the training trajectory. These bounds yield two important implications: a necessary condition of improved generalization explains differences in layerwise generalization behavior; a sufficient condition describes delayed generalization and provides a theoretical characterization of grokking.
Yuqing Wang, Ioannis G. Kevrekidis, Mikhail Belkin
Generalization BoundsTwo-Layer Neural NetworksGradient DescentNeural Network
AskPDF
May 18, 2026cs.LG

Pointwise Generalization in Deep Neural Networks

We address the fundamental question of why deep neural networks generalize by establishing a pointwise generalization theory for fully connected networks. This framework resolves long-standing barriers to characterizing the rich nonlinear feature-learning regime and builds a new statistical foundation for representation learning. For each trained model, we characterize the hypothesis via a pointwise Riemannian Dimension, derived from the eigenvalues of the learned feature representations across layers. This establishes a principled framework for deriving hypothesis-dependent, representation-aware generalization bounds. These bounds offer a systematic upgrade over approaches based on model size, products of norms, and infinite-width linearizations, yielding guarantees that are orders of magnitude tighter in both theory and experiment. Analytically, we identify the structural properties and mathematical principles that explain the tractability of deep networks. Empirically, the pointwise Riemannian Dimension exhibits substantial feature compression, decreases with increased over-parameterization, and captures the implicit bias of optimizers. Taken together, our results indicate that deep networks are mathematically tractable in practical regimes and that their generalization is sharply explained by pointwise, feature-spectrum-aware complexity.
Shaojie Li, Yunbei Xu
Generalization BoundsDeep NetworkFeature Learning
AskPDF