Infinite Width Neural Network

Infinite-width neural networks represent a theoretical framework for analyzing the behavior of very large neural networks by studying their properties as the number of neurons approaches infinity. Current research focuses on understanding the relationship between network depth and sample complexity, the impact of different training algorithms and activation functions on learning dynamics (including the role of the neural tangent kernel), and the equivalence between infinite-width networks and other machine learning models like Gaussian processes and support vector machines. This research provides valuable insights into the generalization ability, robustness, and efficient training of practical, finite-width networks, impacting both theoretical understanding and the development of improved algorithms and architectures.

Papers

February 13, 2024

Depth Separation in Norm-Bounded Infinite-Width Neural Networks
Suzanna Parkinson, Greg Ongie, Rebecca Willett, Ohad Shamir, Nathan Srebro
ReLU Network Norm Bounded Infinite Width Neural Network Depth Separation

October 9, 2023

Theoretical Analysis of Robust Overfitting for Wide DNNs: An NTK Approach
Shaopeng Fu, Di Wang
Neural Network Adversarial Training Neural Tangent Kernel Theoretical Analysis Wide Neural Network Robust Overfitting Infinite Width Neural Network

October 3, 2023

Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networks
Greg Yang, Dingli Yu, Chen Zhu, Soufiane Hayou
Neural Network Feature Learning Depth Distribution Tensor Program Infinite Width Neural Network Transfer Hyperparameter Optimisation

July 5, 2023

Universal Scaling Laws of Absorbing Phase Transitions in Artificial Deep Neural Networks
Keiichi Tamai, Tsuyoshi Okubo, Truong Vinh Truong Duy, Naotake Natori, Synge Todo
Phase Transition Infinite Width Neural Network Finite Size

March 9, 2023

Kernel Regression with Infinite-Width Neural Networks on Millions of Examples
Ben Adlam, Jaehoon Lee, Shreyas Padhy, Zachary Nado, Jasper Snoek
Kernel Method Fewer Example Kernel Regression Infinite Width Neural Network Neural Kernel

February 19, 2023

Guided Deep Kernel Learning
Idan Achituve, Gal Chechik, Ethan Fetaya
Deep Kernel Learning Deep Kernel Infinite Width Neural Network Neural Network Gaussian Process Kernel Optimization

February 7, 2023

On the relationship between multivariate splines and infinitely-wide neural networks
Francis Bach
Human Relationship Random Feature B Spline Random Fourier Feature Infinite Width Neural Network

October 5, 2022

The Influence of Learning Rule on Representation Dynamics in Wide Neural Networks
Blake Bordelon, Cengiz Pehlevan
External Influence Learning Dynamic Wide Neural Network Dynamic Representation Dynamic Prediction Learning Rule Infinite Width Neural Network

August 8, 2022

Deep Maxout Network Gaussian Process
Libin Liang, Ye Tian, Ge Cheng
Convolutional Neural Network Gaussian Process Deep Gaussian Process Infinite Width Neural Network Neural Kernel

July 19, 2022

Bounding generalization error with input compression: An empirical study with infinite-width networks
Angus Galloway, Anna Golubeva, Mahmoud Salem, Mihai Nica, Yani Ioannou, Graham W. Taylor
Deep Neural Network Empirical Study Generalization Error DNN Architecture Generalization Error Bound Infinite Width Neural Network Width Network Input Compression

June 3, 2022

Infinite Recommendation Networks: A Data-Centric Approach
Noveen Sachdeva, Mehak Preet Dhaliwal, Carole-Jean Wu, Julian McAuley
Neural Network Supervised Autoencoder Data Centric Representation Bottleneck Infinite Width Neural Network Different Recommendation

May 19, 2022

Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networks
Blake Bordelon, Cengiz Pehlevan
Wide Neural Network Kernel Based Infinite Width Neural Network Order Dynamic

May 17, 2022

Deep neural networks with dependent weights: Gaussian Process mixture limit, heavy tails, sparsity and compressibility
Hoil Lee, Fadhel Ayed, Paul Jung, Juho Lee, Hongseok Yang, François Caron
Deep Neural Network Sparsity Increase Non Gaussian Random Weight Infinite Width Neural Network Heavy Tail Infinite Width

January 13, 2022

An Overview of Uncertainty Quantification Methods for Infinite Neural Networks
Florian Juengermann, Maxime Laasri, Marius Merkle
Neural Network Gaussian Process Predictive Uncertainty Infinite Width Neural Network Bayesian Inference Framework

December 31, 2021

Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNs
Inbar Seroussi, Gadi Naveh, Zohar Ringel
Convolutional Neural Network Deep Neural Network Deep Convolutional Neural Network Feature Learning Separation Performance Neural Network Output Thermodynamic Integration Ghanaian Musical Scale Faster Model Infinite Width Neural Network

November 11, 2021

On the Equivalence between Neural Network and Support Vector Machine
Yilan Chen, Wei Huang, Lam M. Nguyen, Tsui-Wei Weng
Neural Network Support Vector Machine Neural Tangent Kernel Wide Neural Network Width Neural Network Infinite Width Neural Network