Wide Neural Network

Wide neural networks, characterized by a large number of neurons in their layers, are a focus of intense research due to their unique theoretical properties and practical performance. Current research investigates their training dynamics, generalization capabilities, and connections to Gaussian processes and kernel methods, often focusing on architectures like ResNets and deep equilibrium models, and employing techniques like neural tangent kernel analysis. These studies aim to provide a deeper understanding of how and why these networks generalize well, even when overparameterized, leading to improved model design and training strategies with implications for various applications, including image classification and solving partial differential equations.

Papers

July 26, 2023

Controlling the Inductive Bias of Wide Neural Networks by Modifying the Kernel's Spectrum
Amnon Geifman, Daniel Barzilai, Ronen Basri, Meirav Galun
Gradient Descent Inductive Bias Neural Tangent Kernel Wide Neural Network Kernel Extension Time to Spectrum

June 14, 2023

Nonparametric regression using over-parameterized shallow ReLU neural networks
Yunfei Yang, Ding-Xuan Zhou
Neural Network Shallow Neural Network Wide Neural Network Nonparametric Regression Shallow ReLU Rate Maximization

May 28, 2023

Feature-Learning Networks Are Consistent Across Widths At Realistic Scales
Nikhil Vyas, Alexander Atanasov, Blake Bordelon, Depen Morwani, Sabarish Sainathan, Cengiz Pehlevan
Network Programming Wide Neural Network Large Learning Rate Ghanaian Musical Scale First Open Source Width Benchmark

May 23, 2023

Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimension
Moritz Haas, David Holzmüller, Ulrike von Luxburg, Ingo Steinwart
Neural Network Neural Tangent Kernel Small Kernel Box Dimension Wide Neural Network Spike Protein Benign Overfitting

May 17, 2023

Sparsity-depth Tradeoff in Infinitely Wide Deep Neural Networks
Chanwoo Chun, Daniel D. Lee
Sparse Network Wide Neural Network Sparse Activation Neural Network Gaussian Process Deep Bayesian

May 15, 2023

ReLU soothes the NTK condition number and accelerates optimization for wide neural networks
Chaoyue Liu, Like Hui
Optimization Purpose Activation Function Neural Tangent Kernel Wide Neural Network Linear Neural Network Rectified Linear Unit

May 4, 2023

On the Eigenvalue Decay Rates of a Class of Neural-Network Related Kernel Functions Defined on General Domains
Yicheng Li, Zixiong Yu, Guhan Chen, Qian Lin
Neural Tangent Kernel Open Domain True Class Wide Neural Network Kernel Function Eigenvalue Decay

April 6, 2023

Wide neural networks: From non-gaussian random fields at initialization to the NTK geometry of training
Luís Carvalho, João Lopes Costa, José Mourão, Gonçalo Oliveira
Training Data Gaussian Process New Initialization Wide Neural Network Random Field Width Network Layer NTK

March 28, 2023

Kernel interpolation generalizes poorly
Yicheng Li, Haobo Zhang, Qian Lin
Generalization Error Wide Neural Network Kernel Regression Benign Overfitting Kernel Interpolation

March 12, 2023

Global Optimality of Elman-type RNN in the Mean-Field Regime
Andrea Agazzi, Jianfeng Lu, Sayan Mukherjee
Mean Field Traditional RNNs \Sigma}{\Delta}$ Low Pas RNN Wide Neural Network Global Optimality Gradient Descent Dynamic

March 6, 2023

Bayesian inference with finitely wide neural networks
Chi-Ken Lu
Bayesian Inference Wide Neural Network Gaussian Kernel Non Gaussian Deep Gaussian Process Multivariate Dependence Gaussian Posterior

February 12, 2023

Generalization Ability of Wide Neural Networks on $\mathbb{R}$
Jianfa Lai, Manyun Xu, Rui Chen, Qian Lin
Neural Tangent Kernel Wide Neural Network Two Layer ReLU Benign Overfitting Neural Kernel

February 6, 2023

Toward Large Kernel Models
Amirhesam Abedsoltan, Mikhail Belkin, Parthe Pandit
Kernel Method Wide Neural Network Kernel Machine

January 26, 2023

A Simple Algorithm For Scaling Up Kernel Methods
Teng Andrea Xu, Bryan Kelly, Semyon Malamud
Neural Tangent Kernel Kernel Method Random Feature Wide Neural Network Random Feature Regression Simple Algorithm

January 18, 2023

Catapult Dynamics and Phase Transitions in Quadratic Nets
David Meltzer, Junyu Liu
Learning Rate Phase Transition Wide Neural Network Quadratic Neural Network Homogeneous Neural Network

January 1, 2023

Neural Networks with Sparse Activation Induced by Large Bias: Tighter Analysis with Bias-Generalized NTK
Hongru Yang, Ziyu Jiang, Ruizhe Zhang, Yingbin Liang, Zhangyang Wang
Strong Generalization Early Stage Convergence Neural Tangent Kernel ReLU Network Sparse Network Wide Neural Network Sparse Activation Relative Stance Bias

December 23, 2022

The Onset of Variance-Limited Behavior for Networks in the Lazy and Rich Regimes
Alexander Atanasov, Blake Bordelon, Sabarish Sainathan, Cengiz Pehlevan
Network Programming Wide Neural Network Chaotic Regime Lazy Search Width Neural Network Early Onset Width Network Regime Switching Finite Size

October 5, 2022

The Influence of Learning Rule on Representation Dynamics in Wide Neural Networks
Blake Bordelon, Cengiz Pehlevan
External Influence Learning Dynamic Wide Neural Network Dynamic Representation Dynamic Prediction Learning Rule Infinite Width Neural Network

September 23, 2022

Achieve the Minimum Width of Neural Networks for Universal Approximation
Yongqiang Cai
Neural Network Universal Approximation Wide Neural Network Universal Approximators \Ell_p$ Norm Minimum Width Approximation Power

September 16, 2022

Deep tensor networks with matrix product operators
Bojan Žunkovič
Tensor Network Wide Neural Network Matrix Product Operator