Benign Overfitting of Two-Layer ReLU Networks in Fixed Dimension through Localized Initialization
Organizations: ETH Zürich · Amazon
Abstract
Benign overfitting occurs when a model fits noisy training data exactly and still generalizes well. We establish this behavior for finite-width two-layer networks with the standard ReLU activation, trained by gradient flow in fixed dimension. Our data-independent initialization combines Gaussian weights with neurons initially active only on shrinking spherical caps. For uniform spherical inputs and any fixed bounded conditional response law, training and excess risks vanish in probability uniformly over all sufficiently late times. The parameters also converge, with probability tending to one, to an interpolating limit whose excess risk vanishes in probability. We use one schedule depending only on sample size and dimension. Our proof connects the network to a kernel flow whose limit combines ridge regression with localized spikes that fit the training residuals. We give explicit sample-size, width and training-time bounds.
Figures & tables
Benign Overfitting
Benign overfitting studies how overparameterized models can fit noisy training data yet still predict accurately on new examples.
See all 23 papers