Two Layer Neural Network

Two-layer neural networks serve as a fundamental model for understanding the behavior of deeper networks, with research focusing on their optimization dynamics, generalization capabilities, and feature learning properties. Current investigations utilize stochastic gradient descent and related algorithms, often within the context of the neural tangent kernel approximation, to analyze convergence rates and the impact of hyperparameters like learning rate and network width. These studies provide crucial insights into the theoretical foundations of deep learning, informing the design of more efficient and robust algorithms and offering a clearer understanding of phenomena like spectral bias and the emergence of skills during training.

Papers

July 13, 2023

Efficient SGD Neural Network Training via Sublinear Activated Neuron Identification
Lianke Qin, Zhao Song, Yuanyuan Yang
Neural Network Deep Learning Stochastic Gradient Descent Two Layer Neural Network Provable Convergence Neuron Identification

July 11, 2023

Fundamental limits of overparametrized shallow neural networks for supervised learning
Francesco Camilli, Daria Tieplova, Jean Barbier
Neural Network Supervised Learning Neural Network Model Information Theoretic Two Layer Neural Network Shallow Neural Network Fundamental Limit Optimal Generalization

July 3, 2023

Neural Hilbert Ladders: Multi-Layer Neural Networks in Function Space
Zhengdao Chen
Neural Network Kernel Hilbert Space Two Layer Neural Network Function Space Multi Layer Neural Network

June 29, 2023

Why Shallow Networks Struggle with Approximating and Learning High Frequency: A Numerical Study
Shijun Zhang, Hongkai Zhao, Yimin Zhong, Haomin Zhou
Average Approximation Two Layer Neural Network Low Precision Shallow Network Frequency Learning Numerical Result Computational Problem

June 28, 2023

Beyond NTK with Vanilla Gradient Descent: A Mean-Field Analysis of Neural Networks with Polynomial Width, Samples, and Time
Arvind Mahankali, Jeff Z. Haochen, Kefan Dong, Margalit Glasgow, Tengyu Ma
Neural Network Gradient Descent Gradient Flow Mean Field Two Layer Neural Network Vanilla Gradient Descent Polynomial Bound Layer NTK

May 29, 2023

May 26, 2023

Generalization Guarantees of Gradient Descent for Multi-Layer Neural Networks
Puyu Wang, Yunwen Lei, Di Wang, Yiming Ying, Ding-Xuan Zhou
Neural Network Gradient Descent Two Layer Neural Network Generalization Guarantee Multi Layer Neural Network Three Layer Neural Network

May 22, 2023

Fast Convergence in Learning Two-Layer Neural Networks with Separable Data
Hossein Taheri, Christos Thrampoulidis
Gradient Descent Generalization Bound Two Layer Neural Network Linear Classifier Separable Data Exponential Convergence Rate Tailed Loss

May 10, 2023

Phase transitions in the mini-batch size for sparse and dense two-layer neural networks
Raffaele Marino, Federico Ricci-Tersenghi
Two Layer Neural Network Phase Transition Mini Batch Two Layer

May 9, 2023

A duality framework for analyzing random feature and two-layer neural networks
Hongrui Chen, Jihao Long, Lei Wu
Kernel Hilbert Space Two Layer Neural Network Random Feature Function Space Dot Product Kernel

April 6, 2023

Training a Two Layer ReLU Network Analytically
Adrian Barbu
Gradient Descent Two Layer Neural Network Adam Optimizer ReLU Activation Two Layer ReLU

March 31, 2023

On the Effect of Initialization: The Scaling Path of 2-Layer Neural Networks
Sebastian Neumayer, Lénaïc Chizat, Michael Unser
Gradient Descent Mixed Effect Convex Set Two Layer Neural Network Root to Leaf Path New Initialization Two Layer ReLU Unbalanced Optimal Transport Regularization Path

March 29, 2023

Training Neural Networks is NP-Hard in Fixed Dimension
Vincent Froese, Christoph Hertrich
Neural Network Two Layer Neural Network Computational Complexity NP Complete Parameterized Complexity Threshold Activation

March 24, 2023

Online Learning for the Random Feature Model in the Student-Teacher Framework
Roman Worschech, Bernd Rosenow
Deep Neural Network Online Learning Two Layer Neural Network Random Feature Teacher Student Benign Over Parameterization

March 20, 2023

Convergence Guarantees of Overparametrized Wide Deep Inverse Prior
Nathan Buskulic, Yvain Quéau, Jalal Fadili
Inverse Problem Convergence Guarantee Two Layer Neural Network Overparametrization Bound Deep Prior Learned Prior

March 14, 2023

Practically Solving LPN in High Noise Regimes Faster Using Neural Networks
Haozhe Jiang, Kaiyue Wen, Yilei Chen
Neural Network Neural Network Model Two Layer Neural Network Neural Network Algorithm Parity Learning

March 12, 2023

Phase Diagram of Initial Condensation for Two-layer Neural Networks
Zhengan Chen, Yuqing Li, Tao Luo, Zhangchen Zhou, Zhi-Qin John Xu
Neural Network Two Layer Neural Network Phase Diagram Initial Condensation

March 1, 2023

Adversarial Examples Exist in Two-Layer ReLU Networks for Low Dimensional Linear Subspaces
Odelia Melamed, Gilad Yehudai, Gal Vardi
Adversarial Example Adversarial Perturbation Two Layer Neural Network Low Priority Two Layer ReLU

February 28, 2023

Learning time-scales in two-layers neural networks
Raphaël Berthier, Andrea Montanari, Kangjie Zhou
Learning Dynamic Two Layer Neural Network Time Scale Multi Layer Neural Network Gradient Flow Dynamic