Two Layer Neural Network

Two-layer neural networks serve as a fundamental model for understanding the behavior of deeper networks, with research focusing on their optimization dynamics, generalization capabilities, and feature learning properties. Current investigations utilize stochastic gradient descent and related algorithms, often within the context of the neural tangent kernel approximation, to analyze convergence rates and the impact of hyperparameters like learning rate and network width. These studies provide crucial insights into the theoretical foundations of deep learning, informing the design of more efficient and robust algorithms and offering a clearer understanding of phenomena like spectral bias and the emergence of skills during training.

Papers

March 22, 2022

On the (Non-)Robustness of Two-Layer Neural Networks in Different Learning Regimes
Elvis Dohmatob, Alberto Bietti
Neural Network Native Robustness Adversarial Example Adversarial Robustness Two Layer Neural Network Regime Switching Non Robust Lazy Training

February 17, 2022

The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networks
Emmanuel Abbe, Enric Boix-Adsera, Theodor Misiakiewicz
Stochastic Gradient Descent Training Dynamic Two Layer Neural Network Sufficient Condition Latent Subspace Sparse Function Low Dimension

February 16, 2022

Learning a Single Neuron for Non-monotonic Activation Functions
Lei Wu
Gradient Descent Two Layer Neural Network Single Neuron Non Monotonic Activation Function

February 11, 2022

Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Data
Spencer Frei, Niladri S. Chatterji, Peter L. Bartlett
Gradient Descent Generalization Error Noisy Data Two Layer Neural Network Benign Overfitting Neural Network Classifier

February 10, 2022

Hardness of Noise-Free Learning for Two-Hidden-Layer Neural Networks
Sitan Chen, Aravind Gollakota, Adam R. Klivans, Raghu Meka
ReLU Network Two Layer Neural Network Hardness Result Agnostic Learning Noise Robust Learning Layer ReLU Network Self Fitting Method

January 12, 2022

On neural network kernels and the storage capacity problem
Jacob A. Zavatone-Veth, Cengiz Pehlevan
Neural Network Two Layer Neural Network Wide Neural Network Neural Network Gaussian Process Neural Kernel

December 20, 2021

RvS: What is Essential for Offline RL via Supervised Learning?
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, Sergey Levine
Reinforcement Learning Supervised Learning Two Layer Neural Network Temporal Difference Learning