Implicit Regularization and Generalization in Overparameterized Neural Networks

Zeran Johannsen

cs.LGApr 8, 2026

Implicit Regularization and Generalization in Overparameterized Neural Networks

Authors: Zeran Johannsen

Abstract

Classical statistical learning theory predicts that overparameterized models should exhibit severe overfitting, yet modern deep neural networks with far more parameters than training samples consistently generalize well. This contradiction has become a central theoretical question in machine learning. This study investigates the role of optimization dynamics and implicit regularization in enabling generalization in overparameterized neural networks through controlled experiments. We examine stochastic gradient descent (SGD) across batch sizes, the geometry of flat versus sharp minima via Hessian eigenvalue estimation and weight perturbation analysis, the Neural Tangent Kernel (NTK) regime through wide-network experiments, double descent across model scales, and the Lottery Ticket Hypothesis through iterative magnitude pruning. All experiments use PyTorch on CIFAR-10 and MNIST with multiple random seeds. Our findings demonstrate that generalization is strongly influenced by the interaction between network architecture, optimization algorithms, and loss landscape geometry. Smaller batch sizes consistently produced lower test error and flatter minima, with an 11.8x difference in top Hessian eigenvalue between small-batch and large-batch solutions corresponding to 1.61 percentage points higher test accuracy. Sparse subnetworks retaining only 10% of parameters achieved within 1.15 percentage points of full model performance when retrained from their original initialization. These results highlight the need for revised learning-theoretic frameworks capable of explaining generalization in high-dimensional model regimes.

Figures & tables

Width	Parameters	Train Acc (%)	Test Acc (%)
8	6,514	97.1	$89.7\pm 0.6$
16	13,274	99.9	$93.4\pm 0.3$
32	27,562	100.0	$95.4\pm 0.0$
64	59,210	100.0	$96.1\pm 0.1$
128	134,794	100.0	$96.7\pm 0.1$
256	335,114	100.0	$96.7\pm 0.2$

TABLE I: Double descent results for MLP on MNIST.

Channels	Parameters	Train Acc (%)	Test Acc (%)
4	72,990	39.8	$32.9\pm 1.7$
8	152,082	45.8	$27.6\pm 17.5$
16	337,050	90.7	$56.1\pm 2.3$
32	814,122	100.0	$65.8\pm 0.1$
64	2,196,810	99.9	$69.7\pm 0.5$
96	4,150,890	99.9	$72.0\pm 0.3$

TABLE II: Double descent results for CNN on CIFAR-10.

Fig. 1: Double descent curves for MLP on MNIST (left) and CNN on CIFAR-10 (right). Test error and train error plotted against number of parameters on a log scale. Vertical dashed line indicates training set size.

Config	Batch	LR	Test Acc (%)	Gen Gap (%)
SGD	32	0.005	$85.52\pm 0.26$	14.48
SGD	64	0.0075	$85.47\pm 0.33$	14.53
SGD	128	0.01	$84.72\pm 0.08$	15.28
SGD	256	0.02	$84.74\pm 0.27$	15.26
SGD	512	0.04	$84.28\pm 0.19$	15.72
SGD	1024	0.08	$84.17\pm 0.15$	15.83

TABLE III: Implicit regularization: test accuracy and generalization gap across optimizer configurations. *Full-batch GD used a 5,000-sample subset for computational feasibility.

Fig. 2: Test accuracy and generalization gap across optimizer configurations. Blue bars show test accuracy; orange bars show generalization gap.

Model	Batch Size	Test Acc (%)	Top Hessian Eigenvalue
Small-batch	32	85.78	0.19
Large-batch	1024	84.17	2.24

TABLE IV: Hessian eigenvalue comparison between small-batch and large-batch models.

Perturbation $\sigma$	Small-batch	Large-batch
0.0005	$+1.9\%$	$+3.4\%$
0.001	$+3.3\%$	$+11.8\%$
0.002	$+23.6\%$	$+52.3\%$
0.005	$+158.2\%$	$+1{,}931.9\%$
0.01	$+5{,}353.8\%$	$+46{,}205.1\%$

TABLE V: Perturbation analysis: loss increase (%) under weight perturbation.

Fig. 3: Loss landscape analysis. Left: loss increase (%) under weight perturbation at varying $\sigma$ for small-batch vs. large-batch models. Right: top Hessian eigenvalue comparison showing $11.8\times$ difference in curvature.

Width	Parameters	Test Acc (%)	$\Delta\theta_{\mathrm{rel}}$
32	27,562	91.6	0.9356
64	59,210	92.6	0.6861
128	134,794	92.9	0.4899
256	335,114	93.1	0.3485
512	932,362	93.3	0.2477
1024	2,913,290	93.5	0.1743

TABLE VI: NTK regime results: relative parameter movement and test accuracy as a function of network width.

Fig. 4: NTK regime analysis. Left: relative parameter movement decreases monotonically with network width. Right: test accuracy improves with width. Both axes use log scale for width.

Remaining Params (%)	Test Acc (%)
100 (full model)	$84.87\pm 0.16$
70	$85.27\pm 0.12$
50	$84.86\pm 0.29$
30	$84.74\pm 0.21$
20	$84.47\pm 0.32$
10	$83.72\pm 0.33$

TABLE VII: Lottery ticket pruning results: test accuracy vs. remaining parameters.

Fig. 5: Lottery ticket pruning results. Blue line shows test accuracy vs. remaining parameters (%) for subnetworks retrained from original initialization. Red marker shows random reinitialization control at 10% remaining parameters.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Optimizer	SGD with momentum (default)
Momentum	0.9
Base Learning Rate	0.01 (at batch size 128)
LR Scaling	Linear with batch size
Learning Rate Schedule	Cosine annealing
Batch Size	Varied (32–2048); 128 default

Appendix

TABLE VIII: Default experimental hyperparameters.

Parameter	Value
Optimizer	Adam
LR	0.001
$\beta_{1}$	0.9
$\beta_{2}$	0.999

Appendix

TABLE IX: Adam optimizer hyperparameters.

Overparameterized Neural Network

See all 13 papers

Jun 4, 2026

Generalization in Deep Neural Networks: Minimax Rates for Gradient Methods

Understanding the generalization performance of over-parameterized neural networks has become a central topic in deep learning theory. While recent advances, particularly works under the Neural Tangent Kernel (NTK) regime, have shed light on the behavior of shallow architectures, the statistical generalization properties of deep neural networks (DNNs), especially in regression tasks, remain far less understood. In this paper, we make significant progress toward closing this gap by providing a comprehensive generalization analysis of DNNs trained using gradient-based methods. First, we establish, for the first time, a crucial connection between the learning dynamics of a DNN with smooth activation functions trained via gradient-based methods and those of kernel methods, showing that gradient-based methods on over-parameterized DNNs can fully inherit the favorable learning dynamics of their kernel counterparts. Building on this connection and the well-established optimality of kernel methods, we derive the first known minimax-optimal rates for the excess population risk of both gradient descent (GD) and stochastic gradient descent (SGD), under the assumption that network width scales polynomially with the sample size. Our results demonstrate that, with sufficient width, DNNs trained by GD or SGD can achieve generalization performance comparable to kernel-based methods.

Junyu Zhou, Puyu Wang, Yunwen Lei +2

Mathematical Institute for Machine Learning and Data Science, Catholic University of Eichstätt-Ingolstadt · Department of Computer Science, RPTU Kaiserslautern-Landau · Department of Mathematics, The University of Hong Kong +1

Neural Network Generalization Gradient Descent Dynamics Stochastic Gradient Descent Generalization Bounds

Ask PDF

Mar 7, 2024

Branch Scaling Manifests as Implicit Architectural Regularization for Improving Generalization in Overparameterized ResNets

Scaling factors in residual branches have emerged as a prevalent method for boosting neural network performance, especially in normalization-free architectures. While prior work has primarily examined scaling effects from an optimization perspective, this paper investigates their role in residual architectures through the lens of generalization theory. Specifically, we establish that wide residual networks (ResNets) with constant scaling factors become asymptotically unlearnable as depth increases. In contrast, when the scaling factor exhibits rapid depth-wise decay combined with early stopping, over-parameterized ResNets achieve minimax-optimal generalization rates. To establish this, we demonstrate that the generalization capability of wide ResNets can be approximated by kernel regression associated with the Neural Tangent Kernel (NTK). Our theoretical findings are validated through experiments on synthetic data and real-world classification tasks, including MNIST and CIFAR-100.

Zixiong Yu, Guhan Chen, Jianfa Lai +2

Huawei Large Model Data Technology Lab, Shenzhen · Tsinghua University, Beijing · Kyoto University, Kyoto

Neural Network Generalization Residual Learning Generalization Bounds

Ask PDF

May 18, 2026

Pointwise Generalization in Deep Neural Networks

We address the fundamental question of why deep neural networks generalize by establishing a pointwise generalization theory for fully connected networks. This framework resolves long-standing barriers to characterizing the rich nonlinear feature-learning regime and builds a new statistical foundation for representation learning. For each trained model, we characterize the hypothesis via a pointwise Riemannian Dimension, derived from the eigenvalues of the learned feature representations across layers. This establishes a principled framework for deriving hypothesis-dependent, representation-aware generalization bounds. These bounds offer a systematic upgrade over approaches based on model size, products of norms, and infinite-width linearizations, yielding guarantees that are orders of magnitude tighter in both theory and experiment. Analytically, we identify the structural properties and mathematical principles that explain the tractability of deep networks. Empirically, the pointwise Riemannian Dimension exhibits substantial feature compression, decreases with increased over-parameterization, and captures the implicit bias of optimizers. Taken together, our results indicate that deep networks are mathematically tractable in practical regimes and that their generalization is sharply explained by pointwise, feature-spectrum-aware complexity.

Shaojie Li, Yunbei Xu

National University of Singapore

Neural Network Generalization Representation Learning Generalization Bounds

Ask PDF

Abstract

Overparameterized Neural Network

Explore similar work

Generalization in Deep Neural Networks: Minimax Rates for Gradient Methods

Branch Scaling Manifests as Implicit Architectural Regularization for Improving Generalization in Overparameterized ResNets

Pointwise Generalization in Deep Neural Networks