Benign Over Parameterization

Benign overparameterization explores the surprising phenomenon where neural networks with far more parameters than training data generalize well, even achieving zero training error. Current research focuses on understanding this behavior through analyses of gradient descent dynamics in various model architectures, including two-layer networks, random feature models, and deep networks, often employing techniques like random duality theory and analyzing the impact of initialization strategies. This research is significant because it challenges traditional understanding of overfitting and offers insights into the design and training of efficient and robust machine learning models across diverse applications, from web search ranking to reinforcement learning.

Papers

June 17, 2022

Tensor-on-Tensor Regression: Riemannian Optimization, Over-parameterization, Statistical-computational Gap, and Their Interplay
Yuetian Luo, Anru R. Zhang
Riemannian Gradient Riemannian Optimization Benign Over Parameterization Computational Gap Tensor on Tensor Regression

June 5, 2022

Early Stage Convergence and Global Convergence of Training Mildly Parameterized Neural Networks
Mingze Wang, Chao Ma
Neural Network Loss Function Early Stage Convergence Global Convergence Benign Over Parameterization

May 3, 2022

Smooth over-parameterized solvers for non-smooth structured optimization
Clarice Poon, Gabriel Peyré
Benign Over Parameterization Non Smooth Variational Formulation

March 20, 2022

Over-parameterization: A Necessary Condition for Models that Extrapolate
Roozbeh Yousefzadeh
Full Model Extrapolation Framework Convex Hull Benign Over Parameterization Sufficient Condition

March 19, 2022

Deep Learning Generalization, Extrapolation, and Over-parameterization
Roozbeh Yousefzadeh
Neural Network Strong Generalization Deep Network Adversarial Vulnerability Extrapolation Framework Neural Network Generalization Benign Over Parameterization Imperceptible Adversarial Perturbation

March 9, 2022

On the influence of over-parameterization in manifold based surrogates and deep neural operators
Katiana Kontolati, Somdatta Goswami, Michael D. Shields, George Em Karniadakis
External Influence Benign Over Parameterization Surrogate Assisted Version Polynomial Chaos Expansion Deep Neural Operator DeepONet Framework

February 28, 2022

Robust Training under Label Noise by Over-parameterization
Sheng Liu, Zhihui Zhu, Qing Qu, Chong You
Neural Network Label Noise Robust Training Parameterized Model Benign Over Parameterization

February 23, 2022

Benefit of Interpolation in Nearest Neighbor Algorithms
Yue Xing, Qifan Song, Guang Cheng
Neural Network Nearest Neighbor Complementary Benefit Interpolation Regime Double Descent Benign Over Parameterization

February 17, 2022

Global Convergence of Sub-gradient Method for Robust Matrix Recovery: Small Initialization, Noisy Measurements, and Over-parameterization
Jianhao Ma, Salar Fattahi
Global Convergence Low Rank Matrix Parameterized Model Benign Over Parameterization Noisy Measurement Low Rank Matrix Recovery

February 4, 2022

Demystify Optimization and Generalization of Over-parameterized PAC-Bayesian Learning
Wei Huang, Chunrui Liu, Yilan Chen, Tianyu Liu, Richard Yi Da Xu
Strong Generalization PAC Bayesian Benign Over Parameterization PAC Bayesian Bound

December 14, 2021

Training Multi-Layer Over-Parametrized Neural Network in Subquadratic Time
Zhao Song, Lichen Zhang, Ruizhe Zhang
Large Language Model Neural Network Large Pre Trained Model Parameterized Model Benign Over Parameterization