Infinite Width

Research on "infinite width" in neural networks investigates the theoretical properties of networks with infinitely many neurons per layer, aiming to simplify analysis and gain insights into the behavior of their finite-width counterparts. Current work focuses on understanding the dynamics of training with various optimizers (like Adam and SGD), exploring the role of feature learning and initialization strategies in different architectures (including MLPs and CNNs), and characterizing the resulting kernel functions. These analyses provide a more rigorous understanding of neural network training, potentially leading to improved optimization algorithms and hyperparameter tuning strategies for practical applications.

Papers

November 4, 2024

Local Loss Optimization in the Infinite Width: Stable Parameterization of Predictive Coding Networks and Target Propagation
Satoki Ishikawa, Rio Yokota, Ryo Karakida
Loss Function Predictive Coding Deep Linear Local Learning Infinite Width Square Root

June 27, 2024

Infinite Width Models That Work: Why Feature Learning Doesn't Matter as Much as You Think
Luke Sernau
Neural Tangent Kernel Feature Learning Sequential Optimization Infinite Width Width Network

December 19, 2023

On the Parameterization of Second-Order Optimization Effective Towards the Infinite Width
Satoki Ishikawa, Ryo Karakida
Second Order Related Hyperparameters Second Order Optimization Infinite Width

December 6, 2023

An Infinite-Width Analysis on the Jacobian-Regularised Training of a Neural Network
Taeyoung Kim, Hongseok Yang
Neural Network Deep Neural Network Multi Layer Neural Tangent Kernel Hidden Layer Infinite Width Jacobian Regularization

September 4, 2023

Les Houches Lectures on Deep Learning at Large & Infinite Width
Yasaman Bahri, Boris Hanin, Antonin Brossollet, Vittorio Erba, Christian Keup, Rosalba Pacelli, James B. Simon
Deep Learning Deep Neural Network Statistical Mechanic Lecture Note Infinite Width Width Network Deep Random Neural Network

August 3, 2023

Tensor Programs IVb: Adaptive Optimization in the Infinite-Width Limit
Greg Yang, Etai Littwin
Gradient Descent Wide Neural Network Tensor Program Adaptive Optimization Adaptive Gradient Infinite Width Neural Tangent

April 6, 2023

Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networks
Blake Bordelon, Cengiz Pehlevan
Mean Field Large Learning Rate Dynamic Prediction Infinite Width Dynamical Mean Field Theory Prediction Instability

November 29, 2022

Infinite-width limit of deep linear neural networks
Lénaïc Chizat, Maria Colombo, Xavier Fernández-Real, Alessio Figalli
Neural Network Training Dynamic Deep Linear Deterministic Neural Network Infinite Width

October 3, 2022

On the infinite-depth limit of finite-width neural networks
Soufiane Hayou
Residual Neural Network Width Neural Network Infinite Depth Infinite Width Infinite Depth and Width Limit

May 24, 2022

Empirical Phase Diagram for Three-layer Neural Networks with Infinite Width
Hanxu Zhou, Qixuan Zhou, Zhenyuan Jin, Tao Luo, Yaoyu Zhang, Zhi-Qin John Xu
Gradient Flow Deeper Network Two Layer ReLU Rectified Linear Unit Three Layer Neural Network Phase Diagram Infinite Width Dynamical Regime

May 17, 2022

Deep neural networks with dependent weights: Gaussian Process mixture limit, heavy tails, sparsity and compressibility
Hoil Lee, Fadhel Ayed, Paul Jung, Juho Lee, Hongseok Yang, François Caron
Deep Neural Network Sparsity Increase Non Gaussian Random Weight Infinite Width Neural Network Heavy Tail Infinite Width

February 24, 2022

Embedded Ensembles: Infinite Width Limit and Operating Regimes
Maksim Velikanov, Roman Kail, Ivan Anokhin, Roman Vashurin, Maxim Panov, Alexey Zaytsev, Dmitry Yarotsky
Ensemble Model Model Ensembling Infinite Width Monte Carlo Dropout Dynamical Regime Independent Model Distinct Functional Regime

February 1, 2022

Neural Tangent Kernel Beyond the Infinite-Width Limit: Effects of Depth and Initialization
Mariia Seleznova, Gitta Kutyniok
Mixed Effect Neural Tangent Kernel Large Depth ReLU Network New Initialization Infinite Width Infinite Depth and Width Limit

January 30, 2022

Stochastic Neural Networks with Infinite Width are Deterministic
Liu Ziyin, Hanlin Zhang, Xiangming Meng, Yuting Lu, Eric Xing, Masahito Ueda
Neural Network Stochastic Way Stochastic Process Stochastic Neural Network Infinite Width

Infinite Width

Papers

Local Loss Optimization in the Infinite Width: Stable Parameterization of Predictive Coding Networks and Target Propagation

Infinite Width Models That Work: Why Feature Learning Doesn't Matter as Much as You Think

On the Parameterization of Second-Order Optimization Effective Towards the Infinite Width

An Infinite-Width Analysis on the Jacobian-Regularised Training of a Neural Network

Les Houches Lectures on Deep Learning at Large & Infinite Width

Tensor Programs IVb: Adaptive Optimization in the Infinite-Width Limit

Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networks

Infinite-width limit of deep linear neural networks

On the infinite-depth limit of finite-width neural networks

Empirical Phase Diagram for Three-layer Neural Networks with Infinite Width

Deep neural networks with dependent weights: Gaussian Process mixture limit, heavy tails, sparsity and compressibility

Embedded Ensembles: Infinite Width Limit and Operating Regimes

Neural Tangent Kernel Beyond the Infinite-Width Limit: Effects of Depth and Initialization

Stochastic Neural Networks with Infinite Width are Deterministic