Width Neural Network

Width in neural networks is a crucial architectural parameter significantly impacting model performance and generalization. Current research focuses on understanding the relationship between network width and various aspects of learning, including feature extraction, generalization error, and the convergence of training algorithms, often employing models like residual networks and message-passing neural networks. These investigations utilize both theoretical analyses, such as examining infinite-width limits and kernel methods, and empirical studies involving gradient descent and other optimization techniques. This research aims to provide a deeper understanding of neural network behavior and to inform the design of more efficient and effective architectures for various machine learning tasks.

Papers

August 7, 2024

Bayes-optimal learning of an extensive-width neural network from quadratically many samples
Antoine Maillard, Emanuele Troiani, Simon Martin, Florent Krzakala, Lenka Zdeborová
Quadratic Neural Network Bayes Optimal Width Neural Network Randomly Initialized Optimal Denoising Bayes Error

June 10, 2024

Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning
Daniel Kunin, Allan Raventós, Clémentine Dominé, Feng Chen, David Klindt, Andrew Saxe, Surya Ganguli
Width Neural Network Initialization Bias

February 21, 2024

Asymptotics of Learning with Deep Structured (Random) Features
Dominik Schröder, Daniil Dmitriev, Hugo Cui, Bruno Loureiro
LeArning Abstract Feature Wise Random Participation Linear Neural Network Asymptotic Behavior Covariance Alignment Width Neural Network

February 9, 2024

Flexible infinite-width graph convolutional networks and the importance of representation learning
Ben Anson, Edward Milsom, Laurence Aitchison
Convolutional Neural Network Graph Neural Network Representation Learning Importance Aware Width Neural Network Neural Network Gaussian Process Flexible Graph

February 4, 2024

Defining Neural Network Architecture through Polytope Structures of Dataset
Sangmin Lee, Abbas Mammadov, Jong Chul Ye
Neural Network Data Set Neural Network Architecture Width Neural Network Complex Datasets Forward Invariant Polytopes Geometric Complexity Network Width

October 28, 2023

Efficient kernel surrogates for neural network-based regression
Saad Qadeer, Andrew Engel, Amanda Howard, Adam Tsou, Max Vargas, Panos Stinis, Tony Chiang
Deep Neural Network Neural Tangent Kernel Deep Regression DNN Accuracy Width Neural Network Kernel Based Surrogate

July 5, 2023

From NeurODEs to AutoencODEs: a mean-field control framework for width-varying Neural Networks
Cristina Cipriani, Massimo Fornasier, Alessandro Scagliotti
Supervised Autoencoder Deep Learning Architecture Mean Field Residual Neural Network Width Neural Network

May 25, 2023

Data Topology-Dependent Upper Bounds of Neural Network Widths
Sangmin Lee, Jong Chul Ye
Deep Neural Network Raw Data Universal Approximation Upper Bound Layer ReLU Network Three Layer Neural Network Width Neural Network

February 6, 2023

On Over-Squashing in Message Passing Neural Networks: The Impact of Width, Depth, and Topology
Francesco Di Giovanni, Lorenzo Giusti, Federico Barbero, Giulia Luise, Pietro Lio', Michael Bronstein
Graph Neural Network Large Depth Message Passing Neural Network Topology Problem Squashing Mitigation Approach Influential Node Minimum Width Width Neural Network

December 28, 2022

Learning Lipschitz Functions by GD-trained Shallow Overparameterized ReLU Neural Networks
Ilja Kuzborskij, Csaba Szepesvári
Gradient Descent Neural Tangent Kernel Shallow ReLU Lipschitz Operator Width Neural Network Lipschitz Learning

December 23, 2022

The Onset of Variance-Limited Behavior for Networks in the Lazy and Rich Regimes
Alexander Atanasov, Blake Bordelon, Sabarish Sainathan, Cengiz Pehlevan
Network Programming Wide Neural Network Chaotic Regime Lazy Search Early Onset Width Neural Network Width Network Regime Switching Finite Size

October 31, 2022

Globally Gated Deep Linear Networks
Qianyi Li, Haim Sompolinsky
Deep Linear Deep ReLU Network Linear Network Width Neural Network Gated Linear

October 10, 2022

Meta-Principled Family of Hyperparameter Scaling Strategies
Sho Yaida
Neural Tangent Kernel Width Neural Network Neural Tangent Large Scale Hyperparameter Optimization

October 3, 2022

On the infinite-depth limit of finite-width neural networks
Soufiane Hayou
Residual Neural Network Width Neural Network Infinite Depth Infinite Width Infinite Depth and Width Limit

May 24, 2022

Gaussian Pre-Activations in Neural Networks: Myth or Reality?
Pierre Wolinski, Julyan Arbel
Neural Network Real World Many Myth Width Neural Network Deep Narrow Progressive Propagation

March 14, 2022

Phenomenology of Double Descent in Finite-Width Neural Networks
Sidak Pal Singh, Aurelien Lucchi, Thomas Hofmann, Bernhard Schölkopf
Loss Function Neural Tangent Kernel Double Descent Width Neural Network

December 5, 2021

Joint Superposition Coding and Training for Federated Learning over Multi-Width Neural Networks
Hankyul Baek, Won Joon Yun, Yunseok Kwak, Soyi Jung, Mingyue Ji, Mehdi Bennis, Jihong Park, Joongheon Kim
Training Data Federated Learning Backward Propagation Width Neural Network Slimmable Neural Network

November 11, 2021

On the Equivalence between Neural Network and Support Vector Machine
Yilan Chen, Wei Huang, Lam M. Nguyen, Tsui-Wei Weng
Neural Network Support Vector Machine Neural Tangent Kernel Wide Neural Network Width Neural Network Infinite Width Neural Network