Infinite Depth and Width Limit

Research on the infinite depth and width limit of neural networks aims to understand the behavior of extremely large networks by analyzing their asymptotic properties. Current work focuses on characterizing these limits using differential equations and stochastic processes, particularly for architectures like ResNets, MLPs, and Transformers, often involving modifications like "shaped" activations or skip connections to ensure well-defined limits. These analyses provide insights into network trainability, covariance structure, and the relationship between depth and width, potentially leading to improved network design and a deeper understanding of deep learning's fundamental mechanisms.

Papers

October 18, 2023

Differential Equation Scaling Limits of Shaped and Unshaped Neural Networks
Mufan Bill Li, Mihai Nica
Deeper Network Multi Layer Perceptron Scaling Limit Infinite Depth and Width Limit

June 30, 2023

The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limit
Lorenzo Noci, Chuning Li, Mufan Bill Li, Bobby He, Thomas Hofmann, Chris Maddison, Daniel M. Roy
Stochastic Differential Equation Attention Model Softmax Attention Deep Learning Theory U Shaped Infinite Depth and Width Limit

March 30, 2023

Neural signature kernels as infinite-width-depth-limits of controlled ResNets
Nicola Muca Cirone, Maud Lemercier, Cristopher Salvi
Reservoir Computing Neural CDEs Infinite Depth Signature Kernel Infinite Depth and Width Limit

February 1, 2023

Width and Depth Limits Commute in Residual Networks
Soufiane Hayou, Greg Yang
Deep Neural Network Bayesian Deep Learning Gaussian Distribution Minimum Width Infinite Depth and Width Limit Depth Limit

October 3, 2022

On the infinite-depth limit of finite-width neural networks
Soufiane Hayou
Residual Neural Network Width Neural Network Infinite Depth Infinite Width Infinite Depth and Width Limit

February 1, 2022

Neural Tangent Kernel Beyond the Infinite-Width Limit: Effects of Depth and Initialization
Mariia Seleznova, Gitta Kutyniok
Mixed Effect Neural Tangent Kernel Large Depth ReLU Network New Initialization Infinite Width Infinite Depth and Width Limit