Benign Overfitting for Regression with Trained Two-Layer ReLU Networks
Organizations: *ETH Zürich, Switzerland · †Amazon · ‡Amazon
Abstract
We study the least-square regression problem with a two-layer fully-connected neural network, with ReLU activation function, trained by gradient flow. Our first result is a generalization result, that requires no assumptions on the underlying regression function or the noise other than that they are bounded. We operate in the neural tangent kernel regime, and our generalization result is developed via a decomposition of the excess risk into estimation and approximation errors, viewing gradient flow as an implicit regularizer. This decomposition in the context of neural networks is a novel perspective of gradient descent, and helps us avoid uniform convergence traps. In this work, we also establish that under the same setting, the trained network overfits to the data. Together, these results, establishes the first result on benign overfitting for finite-width ReLU networks for arbitrary regression functions.
Figures & tables
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Analytical | Sampled Weights | |
| Network | n/a | |
| Network evaluation | n/a | |
| Noise variable | n/a | |
| Noise vector | n/a |
| Population Trajectory | Empirical Trajectory | |
| Network | ||
| Network Evaluation | ||
| Noise Function | ||
| Noise vector | ||
| Error function | ||
| Error vector |
| Top eigenfunctions | Remaining eigenfunctions | |
| Network | ||
| Error function | ||
| Squared norm of | ||
| error function | ||
| Gradient function | ||
Neural Network Generalization
Neural network generalization studies why trained models perform well on unseen data and how architecture, data, and training shape that ability.
See all 323 papers