cs.LGMay 16, 2025

Benign Overfitting of Two-Layer ReLU Networks in Fixed Dimension through Localized Initialization

Authors: Junhyung Park, Patrick Bloebaum, Shiva Prasad Kasiviswanathan

Organizations: ETH Zürich · Amazon

Abstract

Benign overfitting occurs when a model fits noisy training data exactly and still generalizes well. We establish this behavior for finite-width two-layer networks with the standard ReLU activation, trained by gradient flow in fixed dimension. Our data-independent initialization combines Gaussian weights with neurons initially active only on shrinking spherical caps. For uniform spherical inputs and any fixed bounded conditional response law, training and excess risks vanish in probability uniformly over all sufficiently late times. The parameters also converge, with probability tending to one, to an interpolating limit whose excess risk vanishes in probability. We use one schedule depending only on sample size and dimension. Our proof connects the network to a kernel flow whose limit combines ridge regression with localized spikes that fit the training residuals. We give explicit sample-size, width and training-time bounds.

Figures & tables

Explore similar work

CardsList
  1. Benign Overfitting for Regression with Trained Two-Layer ReLU Networks

    Oct 8, 2024Junhyung Park, Patrick Bloebaum, Shiva Prasad KasiviswanathanNeural Network GeneralizationBenign Overfitting

  2. Mildly Overparameterized ReLU Networks on Orthogonal Data: Incremental Learning and Implicit Bias

    May 26, 2026James Town, Etienne Boursier, Ben Lewis +2ReLU Neural NetworksNeural Network Optimization

  3. Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initialization

    May 28, 2025Etienne Boursier, Matthew Bowditch, Matthias Englert +1Loss LandscapeOverparameterized Neural Network