Two Layer ReLU

Two-layer ReLU networks, a foundational model in deep learning, are the subject of intense research aimed at understanding their optimization landscape and generalization capabilities. Current work focuses on analyzing the implicit biases of gradient-based optimization algorithms, exploring the role of initialization and learning rate, and developing convex relaxations to improve training efficiency and provide theoretical guarantees. These investigations are crucial for advancing our understanding of neural network training dynamics and for developing more robust and efficient machine learning models with provable properties.

Papers

November 11, 2024

On the Principles of ReLU Networks with One Hidden Layer
Changcun Huang
Neural Network Back Propagation General Principle ReLU Network Hidden Layer Two Layer ReLU Dimensional Input Deep ReLU Network

November 4, 2024

Theoretical characterisation of the Gauss-Newton conditioning in Neural Networks
Jim Zhao, Sidak Pal Singh, Aurelien Lucchi
Deep Neural Network Network Programming Neural Topic Deep Linear Two Layer ReLU Gauss Newton

October 29, 2024

Convex Formulations for Training Two-Layer ReLU Neural Networks
Karthik Prakhya, Tolga Birdal, Alp Yurtsever
Two Layer ReLU NP Complete Semidefinite Relaxation Non Convexity Convex Formulation

October 24, 2024

Initialization Matters: On the Benign Overfitting of Two-Layer ReLU CNN with Fully Trainable Layers
Shuning Shang, Xuran Meng, Yuan Cao, Difan Zou
Neural Network Model Overfitting New Initialization Two Layer ReLU Benign Overfitting Trainable Layer

October 10, 2024

Provable Privacy Attacks on Trained Shallow Neural Networks
Guy Smorodinsky, Gal Vardi, Itay Safran
Membership Inference Attack Privacy Guarantee Shallow Neural Network Two Layer ReLU Software Vulnerability Data Reconstruction Attack

October 8, 2024

Benign Overfitting for Regression with Trained Two-Layer ReLU Networks
Junhyung Park, Patrick Bloebaum, Shiva Prasad Kasiviswanathan
Gradient Descent Novel Regression Gradient Flow Neural Tangent Kernel ReLU Network Two Layer ReLU Benign Overfitting Fully Connected Neural Network

October 3, 2024

Simplicity bias and optimization threshold in two-layer ReLU networks
Etienne Boursier, Nicolas Flammarion
Neural Network Strong Generalization Interpolation Regime Overparametrization Bound Two Layer ReLU Simplicity Bias Threshold Optimization

July 5, 2024

Differentially Private Convex Approximation of Two-Layer ReLU Networks
Antti Koskela
Private Stochastic Privacy Utility Two Layer ReLU Private Gradient Private Convex Optimization Private Convex

June 10, 2024

Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizes
Dan Qiao, Kaiqi Zhang, Esha Singh, Daniel Soudry, Yu-Xiang Wang
Strong Generalization Gradient Descent Regularization Model ReLU Network Two Layer ReLU Flat Minimum Nonparametric Regression

June 5, 2024

Feature Contamination: Neural Networks Learn Uncorrelated Features and Fail to Generalize
Tianren Zhang, Chujie Zhao, Guanyu Chen, Yizhou Jiang, Feng Chen
Strong Generalization Representation Learning Distribution Generalization Two Layer ReLU Data Contamination

May 30, 2024

Complexity of Deciding Injectivity and Surjectivity of ReLU Neural Networks
Vincent Froese, Moritz Grillo, Martin Skutella
Complexity Matter ReLU Layer Two Layer ReLU NP Complete ReLU Neural Network

April 9, 2024

Exploring Neural Network Landscapes: Star-Shaped and Geodesic Connectivity
Zhanran Lin, Puheng Li, Lei Wu
Loss Landscape Two Layer ReLU Star Model Connectivity Pattern Linear Network

February 6, 2024

Convex Relaxations of ReLU Neural Networks Approximate Global Optima in Polynomial Time
Sungyoon Kim, Mert Pilanci
Polynomial Time Convex Relaxation Two Layer ReLU Local Gradient Layer ReLU Network Optimality Gap

December 28, 2023

Hidden Minima in Two-Layer ReLU Networks
Yossi Arjevani
Loss Function Two Layer ReLU

November 18, 2023

Polynomial-Time Solutions for ReLU Network Training: A Complexity Classification via Max-Cut and Zonotopes
Yifei Wang, Mert Pilanci
Polynomial Time ReLU Network ReLU Layer Two Layer ReLU Polynomial Time Approximation Polynomial Zonotopes Decay Regularization Difficulty Score

October 29, 2023

Implicit Bias of Gradient Descent for Two-layer ReLU and Leaky ReLU Networks on Nearly-orthogonal Data
Yiwen Kou, Zixiang Chen, Quanquan Gu
Gradient Descent Gradient Flow Implicit Bias Two Layer ReLU Post Hoc Orthogonalization Leaky ReLU

October 11, 2023

Why Does Sharpness-Aware Minimization Generalize Better Than SGD?
Zixiang Chen, Junkai Zhang, Yiwen Kou, Xiangning Chen, Cho-Jui Hsieh, Quanquan Gu
Neural Network Strong Generalization Stochastic Gradient Descent Sharpness Aware Minimization Two Layer ReLU

October 9, 2023

On the Convergence of Federated Averaging under Partial Participation for Over-parameterized Neural Networks
Xin Liu, Wei li, Dazhi Zhan, Yu Pan, Xin Ma, Yu Ding, Zhisong Pan
Neural Network Early Stage Convergence Federated Averaging Deep Linear Two Layer ReLU FedAvg Converges Partial Participation

July 28, 2023

Noisy Interpolation Learning with Shallow Univariate ReLU Networks
Nirmit Joshi, Gal Vardi, Nathan Srebro
Neural Network Model Overfitting Two Layer ReLU Shallow ReLU Easy to Hard Generalization Optimal Error

July 24, 2023

Early Neuron Alignment in Two-layer ReLU Networks with Small Initialization
Hancheng Min, Enrique Mallada, René Vidal
Gradient Flow Binary Classification MNIST Dataset Two Layer ReLU Small Initialization Neuron Alignment