Two Layer Neural Network

Two-layer neural networks serve as a fundamental model for understanding the behavior of deeper networks, with research focusing on their optimization dynamics, generalization capabilities, and feature learning properties. Current investigations utilize stochastic gradient descent and related algorithms, often within the context of the neural tangent kernel approximation, to analyze convergence rates and the impact of hyperparameters like learning rate and network width. These studies provide crucial insights into the theoretical foundations of deep learning, informing the design of more efficient and robust algorithms and offering a clearer understanding of phenomena like spectral bias and the emergence of skills during training.

Papers

February 6, 2023

Stochastic Gradient Descent-Induced Drift of Representation in a Two-Layer Neural Network
Farhad Pashakhanloo, Alexei Koulakov
Stochastic Gradient Individual Representation Two Layer Neural Network Neuron Activation Representation Drift Descent Property Perceptual Representation Drift Explanation

December 7, 2022

Bi-LSTM Price Prediction based on Attention Mechanism
Jiashu Lou, Leyi Cui, Ye Li
Attention Mechanism Two Layer Neural Network Bi LSTM Bidirectional LSTM Automated Cryptocurrency Trading

November 17, 2022

On the Sample Complexity of Two-Layer Networks: Lipschitz vs. Element-Wise Lipschitz Activation
Amit Daniely, Elad Granot
Sample Complexity Activation Function Two Layer Neural Network Lipschitz Operator Norm Bounded Two Layer Network Lipschitz Activation

November 16, 2022

On the symmetries in the dynamics of wide two-layer neural networks
Karl Hajjar, Lenaic Chizat
Neural Network Gradient Flow Approximate Symmetry Two Layer Neural Network Two Layer ReLU

October 28, 2022

A Functional-Space Mean-Field Theory of Partially-Trained Three-Layer Neural Networks
Zhengdao Chen, Eric Vanden-Eijnden, Joan Bruna
Mean Field Two Layer Neural Network Three Layer Neural Network Kernel Flow

September 29, 2022

Neural Networks Efficiently Learn Low-Dimensional Representations with SGD
Alireza Mousavi-Hosseini, Sejun Park, Manuela Girotti, Ioannis Mitliagkas, Murat A. Erdogdu
Neural Network Gradient Descent Stochastic Gradient Descent Two Layer Neural Network Low Dimensional Representation

September 27, 2022

Biologically-Plausible Determinant Maximization Neural Networks for Blind Separation of Correlated Sources
Bariscan Bozkurt, Cengiz Pehlevan, Alper T. Erdogan
Source Separation Two Layer Neural Network Blind Source Separation Artificial Neuron Correlated Source Determinant Maximization

August 10, 2022

A Sublinear Adversarial Training Algorithm
Yeqi Gao, Lianke Qin, Zhao Song, Yitan Wang
Adversarial Attack Adversarial Training Adversarial Perturbation Two Layer Neural Network Adversarial Training Algorithm

June 14, 2022

Overparametrized linear dimensionality reductions: From projection pursuit to two-layer neural networks
Andrea Montanari, Kangjie Zhou
Wasserstein Distance Two Layer Neural Network Dimension Reduction Sharp Bound Projection Pursuit

June 10, 2022

Intrinsic dimensionality and generalization properties of the $\mathcal{R}$-norm inductive bias
Navid Ardeshir, Daniel Hsu, Clayton Sanford
Inductive Bias Two Layer Neural Network Generalization Property P$ Norm Optimal Generalization Intrinsic Dimensionality

June 2, 2022

Understanding the Role of Nonlinearity in Training Dynamics of Contrastive Learning
Yuandong Tian
Contrastive Learning Integral Role Nonlinear Model Training Dynamic Two Layer Neural Network Linear Activation Deep Nonlinear

May 19, 2022

Mean-Field Analysis of Two-Layer Neural Networks: Global Optimality with Linear Convergence Rates
Jingwei Zhang, Xunpeng Huang, Jincheng Yu
Mean Field Learning Dynamic Two Layer Neural Network Global Optimality Linear Convergence Rate Noisy Gradient Descent

May 17, 2022

Sharp asymptotics on the compression of two-layer neural networks
Mohammad Hossein Amani, Simone Bombari, Marco Mondelli, Rattana Pukdee, Stefano Rini
Linear Compression Two Layer Neural Network Network Compression Optimal Weight Mean Field Limit Gaussian Input

May 3, 2022

High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representation
Jimmy Ba, Murat A. Erdogdu, Taiji Suzuki, Zhichao Wang, Denny Wu, Greg Yang
High Dimensional Learning Rate Individual Representation Feature Learning Two Layer Neural Network Layer Selection Gradient Step

April 24, 2022

Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapes
Chao Ma, Daniel Kunin, Lei Wu, Lexing Ying
Loss Function Gradient Descent Loss Landscape Two Layer Neural Network Multiscale Structure

April 8, 2022

On Projectivity in Markov Logic Networks
Sagar Malhotra, Luciano Serafini
Two Layer Neural Network Projection Bias Spatial Transformer Network Projective Simulation Markov Logic

March 22, 2022

On the (Non-)Robustness of Two-Layer Neural Networks in Different Learning Regimes
Elvis Dohmatob, Alberto Bietti
Neural Network Native Robustness Adversarial Example Adversarial Robustness Two Layer Neural Network Regime Switching Non Robust Lazy Training

February 17, 2022

The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networks
Emmanuel Abbe, Enric Boix-Adsera, Theodor Misiakiewicz
Stochastic Gradient Descent Training Dynamic Two Layer Neural Network Sufficient Condition Latent Subspace Sparse Function Low Dimension

February 16, 2022

Learning a Single Neuron for Non-monotonic Activation Functions
Lei Wu
Gradient Descent Two Layer Neural Network Single Neuron Non Monotonic Activation Function

February 11, 2022

Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Data
Spencer Frei, Niladri S. Chatterji, Peter L. Bartlett
Gradient Descent Generalization Error Noisy Data Two Layer Neural Network Benign Overfitting Neural Network Classifier

Two Layer Neural Network

Papers

Stochastic Gradient Descent-Induced Drift of Representation in a Two-Layer Neural Network

Bi-LSTM Price Prediction based on Attention Mechanism

On the Sample Complexity of Two-Layer Networks: Lipschitz vs. Element-Wise Lipschitz Activation

On the symmetries in the dynamics of wide two-layer neural networks

A Functional-Space Mean-Field Theory of Partially-Trained Three-Layer Neural Networks

Neural Networks Efficiently Learn Low-Dimensional Representations with SGD

Biologically-Plausible Determinant Maximization Neural Networks for Blind Separation of Correlated Sources

A Sublinear Adversarial Training Algorithm

Overparametrized linear dimensionality reductions: From projection pursuit to two-layer neural networks

Intrinsic dimensionality and generalization properties of the $\mathcal{R}$-norm inductive bias

Understanding the Role of Nonlinearity in Training Dynamics of Contrastive Learning

Mean-Field Analysis of Two-Layer Neural Networks: Global Optimality with Linear Convergence Rates

Sharp asymptotics on the compression of two-layer neural networks

High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representation

Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapes

On Projectivity in Markov Logic Networks

On the (Non-)Robustness of Two-Layer Neural Networks in Different Learning Regimes

The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networks

Learning a Single Neuron for Non-monotonic Activation Functions

Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Data