Width Network

Width networks, encompassing both infinite-width and large-but-finite-width neural networks, are a key area of research aiming to understand the theoretical underpinnings of deep learning's success. Current investigations focus on the relationship between finite and infinite-width models, exploring architectures like convolutional neural networks (CNNs) and fully connected networks, and analyzing training dynamics using methods such as Neural Tangent Kernels (NTKs) and stochastic differential equations (SDEs). This research clarifies the role of feature learning, the impact of optimization algorithms, and the relationship between network width, generalization error, and the capacity to learn complex functions, ultimately contributing to a more rigorous theoretical foundation for deep learning.

Papers

June 27, 2024

Infinite Width Models That Work: Why Feature Learning Doesn't Matter as Much as You Think
Luke Sernau
Neural Tangent Kernel Feature Learning Sequential Optimization Infinite Width Width Network

January 30, 2024

Rademacher Complexity of Neural ODEs via Chen-Fliess Series
Joshua Hanson, Maxim Raginsky
Neural ODE Differential Equation Ordinary Differential Equation Rademacher Complexity Width Network

September 4, 2023

Les Houches Lectures on Deep Learning at Large & Infinite Width
Yasaman Bahri, Boris Hanin, Antonin Brossollet, Vittorio Erba, Christian Keup, Rosalba Pacelli, James B. Simon
Deep Learning Deep Neural Network Statistical Mechanic Lecture Note Infinite Width Width Network Deep Random Neural Network

July 21, 2023

Local Kernel Renormalization as a mechanism for feature learning in overparametrized Convolutional Neural Networks
R. Aiudi, R. Pacelli, A. Vezzani, R. Burioni, P. Rotondo
Convolutional Neural Network Feature Learning Functional Mechanism Convolutional Architecture Fully Connected Neural Network Width Network

June 2, 2023

Network Degeneracy as an Indicator of Training Performance: Comparing Finite and Infinite Width Angle Predictions
Cameron Jakub, Mihai Nica
Neural Network Deep Neural Network Large Depth Efficient Training ReLU Layer Aggregate Level Indicator Width Network Finite Data Homogeneous Diffusive Network

April 6, 2023

Wide neural networks: From non-gaussian random fields at initialization to the NTK geometry of training
Luís Carvalho, João Lopes Costa, José Mourão, Gonçalo Oliveira
Training Data Gaussian Process New Initialization Wide Neural Network Random Field Width Network Layer NTK

December 23, 2022

The Onset of Variance-Limited Behavior for Networks in the Lazy and Rich Regimes
Alexander Atanasov, Blake Bordelon, Sabarish Sainathan, Cengiz Pehlevan
Network Programming Wide Neural Network Chaotic Regime Lazy Search Width Neural Network Early Onset Width Network Regime Switching Finite Size

September 17, 2022

Approximation results for Gradient Descent trained Shallow Neural Networks in $1d$
R. Gentile, G. Welper
Gradient Descent Non Convex Optimization Shallow Neural Network Shallow Network Approximation Property Width Network

July 19, 2022

Bounding generalization error with input compression: An empirical study with infinite-width networks
Angus Galloway, Anna Golubeva, Mahmoud Salem, Mihai Nica, Yani Ioannou, Graham W. Taylor
Deep Neural Network Empirical Study Generalization Error DNN Architecture Generalization Error Bound Infinite Width Neural Network Width Network Input Compression

June 6, 2022

December 10, 2021

Eigenspace Restructuring: a Principle of Space and Frequency in Neural Networks
Lechao Xiao
Neural Network Deep Space High Frequency General Principle Width Network

Width Network

Papers

Infinite Width Models That Work: Why Feature Learning Doesn't Matter as Much as You Think

Rademacher Complexity of Neural ODEs via Chen-Fliess Series

Les Houches Lectures on Deep Learning at Large & Infinite Width

Local Kernel Renormalization as a mechanism for feature learning in overparametrized Convolutional Neural Networks

Network Degeneracy as an Indicator of Training Performance: Comparing Finite and Infinite Width Angle Predictions

Wide neural networks: From non-gaussian random fields at initialization to the NTK geometry of training

The Onset of Variance-Limited Behavior for Networks in the Lazy and Rich Regimes

Approximation results for Gradient Descent trained Shallow Neural Networks in $1d$

Bounding generalization error with input compression: An empirical study with infinite-width networks

Spectral Bias Outside the Training Set for Deep Networks in the Kernel Regime

The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initialization

Eigenspace Restructuring: a Principle of Space and Frequency in Neural Networks