stat.MLNov 7, 2026

On Generalization in Bilevel Least Squares with Overparameterized Neural Networks

Authors: Fares El Khoury, Edouard Pauwels, Samuel Vaiter, Michael Arbel

Organizations: Université Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK, 38000 Grenoble, France · CNRS & Université Côte d’Azur, Laboratoire J. A. Dieudonné, 06108 Nice, France

Abstract

Bilevel optimization addresses machine learning problems in which the outer objective is a value function implicitly defined through the solution of an inner problem. While its algorithmic aspects are well developed, its generalization properties remain much less understood. In this paper, we study the generalization of inexact gradient methods when both objectives are least-squares regression problems and when the estimator of the value function's gradient, or hypergradient, is obtained by implicit differentiation in function space. This approach allows us to derive stationarity bounds for these methods in terms of the uniform generalization errors of the two regression functions needed to estimate the hypergradient. When these regression functions are defined as overparameterized two-layer ReLU networks trained by gradient flow with early stopping, we obtain explicit finite-sample bounds on these generalization errors in the Neural Tangent Kernel (NTK) regime. Combining these with our general inexact gradient analysis shows the effect of sample size on the overall convergence of the bilevel method. Numerical experiments on synthetic data illustrate our results. Beyond the NTK regime, further experiments reveal an initialization-dependent implicit bias induced by the bilevel structure, which can lead to feature learning at the inner level and overall performance improvements.

Figures & tables

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. On the Stability and Generalization of First-order Bilevel Minimax Optimization

    Apr 22, 2026Xuelin Zhang, Peipei YuanMin-Max OptimizationBilevel Optimization

  2. Newton-Based Methods for Bilevel Optimization Beyond Lower-Level Strong Convexity

    Oct 19, 2026Youran Dong, Junfeng Yang, Wei Yao +1Bilevel OptimizationSecond-Order Optimization

  3. Your Hypergradient is Skewed: Antithetic Neumann Estimation for Bilevel Optimization

    Nov 11, 2026Jason Bohne, Pawel Polak, Gary Kazantsev +1Stochastic Bilevel OptimizationBilevel Optimization