Minimum Width

Minimum width in neural networks and related models is a crucial research area focusing on determining the smallest network width needed to achieve desired approximation capabilities, such as universal approximation or effective string theory simulations. Current research investigates this minimum width across various architectures, including feed-forward networks, recurrent neural networks, and treelike committee machines, often analyzing the trade-offs between width, depth, and other architectural parameters like the use of skip connections. Understanding minimum width is vital for optimizing network efficiency, improving continual learning performance by mitigating catastrophic forgetting, and advancing theoretical understanding of network capacity and expressiveness.

Papers

November 13, 2024

New advances in universal approximation with neural networks of minimal width
Dennis Rochau, Hanno Gottschalk, Robin Chan
Neural Network Universal Approximation Intrinsic Dimension Universal Approximators Minimum Width New Finding

September 24, 2024

Numerical determination of the width and shape of the effective string using Stochastic Normalizing Flows
Michele Caselle, Elia Cellini, Alessandro Nada
Normalizing Flow Object Shape Minimum Width Effective Model Effective Theory Lattice Gauge Theory

July 6, 2024

The Role of Depth, Width, and Tree Size in Expressiveness of Deep Forest
Shen-Huan Lyu, Jin-Hui Wu, Qin-Cheng Zheng, Baoliu Ye
Decision Tree Random Forest Large Depth Expressive Speech Minimum Width Deep Forest Multi Type Galton Watson Forest

March 11, 2024

On the Diminishing Returns of Width for Continual Learning
Etash Guha, Vihan Lakshman
Neural Network Continual LEArning Catastrophic Forgetting Customer Service Minimum Width Continual Learning Paradigm

February 8, 2024

Fixed width treelike neural networks capacity analysis -- generic activations
Mihailo Stojnic
Neural Network Targeted Activation Penalty Minimum Width Capacity Characterization Random Duality Theory Replica Theory

September 19, 2023

Minimum width for universal approximation using ReLU networks on compact domain
Namjun Kim, Chanho Min, Sejun Park
ReLU Network Universal Approximation Universal Approximators Minimum Width Continuous Domain Uniform Approximation

August 30, 2023

Minimum Width for Deep, Narrow MLP: A Diffeomorphism Approach
Geonho Hwang
Universal Approximation Minimum Width Deep Narrow Time Continuous Diffeomorphisms

May 29, 2023

Minimum Width of Leaky-ReLU Neural Networks for Uniform Universal Approximation
Li'ang Li, Yifei Duan, Guanghua Ji, Yongqiang Cai
Universal Approximation Minimum Width Leaky ReLU Continuous Domain

April 5, 2023

Inapproximability of sufficient reasons for decision trees
Alexander Kozachinskiy
Decision Tree Average Approximation Hardness Result Minimum Width Inapproximability Result

February 18, 2023

Generalization and Stability of Interpolating Neural Networks with Minimal Width
Hossein Taheri, Christos Thrampoulidis
Neural Network Strong Generalization Gradient Descent Core Stability Learning Rate Generalization Bound Shallow Neural Network Minimum Width

February 6, 2023

On Over-Squashing in Message Passing Neural Networks: The Impact of Width, Depth, and Topology
Francesco Di Giovanni, Lorenzo Giusti, Federico Barbero, Giulia Luise, Pietro Lio', Michael Bronstein
Graph Neural Network Large Depth Message Passing Neural Network Topology Problem Influential Node Squashing Mitigation Approach Minimum Width Width Neural Network

February 1, 2023

Width and Depth Limits Commute in Residual Networks
Soufiane Hayou, Greg Yang
Deep Neural Network Bayesian Deep Learning Gaussian Distribution Minimum Width Infinite Depth and Width Limit Depth Limit

November 25, 2022

Minimal Width for Universal Property of Deep RNN
Chang hoon Song, Geonho Hwang, Jun ho Lee, Myungjoo Kang
Recurrent Neural Network Universal Approximation K$ Universality Minimum Width Deep Narrow

September 23, 2022

Achieve the Minimum Width of Neural Networks for Universal Approximation
Yongqiang Cai
Neural Network Universal Approximation Wide Neural Network \Ell_p$ Norm Universal Approximators Minimum Width Approximation Power

September 15, 2022

Robustness in deep learning: The good (width), the bad (depth), and the ugly (initialization)
Zhenyu Zhu, Fanghui Liu, Grigorios G Chrysos, Volkan Cevher
Deep Learning Deep Neural Network Native Robustness Large Depth New Initialization Two Layer ReLU Minimum Width Average Case Robustness

June 26, 2022

Bounding the Width of Neural Networks via Coupled Initialization -- A Worst Case Analysis
Alexander Munteanu, Simon Omlor, Zhao Song, David P. Woodruff
Neural Network Neural Tangent Kernel Worst Case General Bound Two Layer ReLU Minimum Width Robust Initialization

May 19, 2022

Neural Network Architecture Beyond Width and Depth
Zuowei Shen, Haizhao Yang, Shijun Zhang
Neural Network Large Depth Neural Network Architecture ReLU Layer Deep ReLU Network Minimum Width

May 5, 2022

Parallel Parking: Optimal Entry and Minimum Slot Dimensions
Jiri Vlasak, Michal Sojka, Zdeněk Hanzálek
Path Planning Path Planning Problem Minimum Width Parking Slot

February 8, 2022

Width is Less Important than Depth in ReLU Neural Networks
Gal Vardi, Gilad Yehudai, Ohad Shamir
Neural Network Large Depth Shallow Network Q Network ReLU Neural Network Minimum Width Depth Separation Tiny Network

December 21, 2021

On the Size and Width of the Decoder of a Boolean Threshold Autoencoder
Tatsuya Akutsu, Avraham A. Melkman
Supervised Autoencoder Size Matter MEG Decoder Minimum Width Threshold Function