Network Width

Network width, the number of neurons in a neural network's layers, significantly impacts model performance and efficiency. Current research explores the relationship between network width and generalization ability, investigating both the "wide" and "narrow" limits of network architectures like residual networks and transformers, and developing techniques like Low-Rank Adaptation to optimize training for different widths. This research aims to understand how to efficiently determine optimal network widths for various tasks and datasets, potentially leading to more efficient and effective deep learning models with reduced computational costs. Furthermore, studies are leveraging graph-based representations of network training dynamics to better understand and predict the impact of width on performance.

Papers

July 26, 2024

Robust Learning in Bayesian Parallel Branching Graph Neural Networks: The Narrow Width Limit
Zechen Zhang, Haim Sompolinsky
Graph Neural Network Robust Learning Random Neural Network Network Width

February 19, 2024

LoRA+: Efficient Low Rank Adaptation of Large Models
Soufiane Hayou, Nikhil Ghosh, Bin Yu
Fine Tuning Representation Learning Large Model Low Rank Adaptation Run LoRA Run Network Width

February 4, 2024

Defining Neural Network Architecture through Polytope Structures of Dataset
Sangmin Lee, Abbas Mammadov, Jong Chul Ye
Neural Network Data Set Neural Network Architecture Width Neural Network Complex Datasets Forward Invariant Polytopes Geometric Complexity Network Width

October 18, 2023

SHARCS: Efficient Transformers through Routing with Dynamic Width Sub-networks
Mohammadreza Salehi, Sachin Mehta, Aditya Kusupati, Ali Farhadi, Hannaneh Hajishirzi
Transformer Based Transformer Network Efficient Transformer Routing Problem Adaptive Inference Network Width

October 10, 2022

Efficient NTK using Dimensionality Reduction
Nir Ailon, Supratim Shit
Neural Network Dimensionality Reduction Neural Tangent Kernel Factorization Machine Dense Layer Network Width

May 23, 2022

From Width-Based Model Checking to Width-Based Automated Theorem Proving
Mateus de Oliveira Oliveira, Farhad Vadiee
Theorem Proving Model Checking Parameterized Complexity Graph Problem Network Width

May 2, 2022

Triangular Dropout: Variable Network Width without Retraining
Edward W. Staley, Jared Markowitz
Supervised ImageNet Supervised Autoencoder Structured Dropout Compressed Representation Network Width

March 25, 2022

Searching for Network Width with Bilaterally Coupled Network
Xiu Su, Shan You, Jiyang Xie, Fei Wang, Chen Qian, Changshui Zhang, Chang Xu
Supernet Training Bilateral Network Network Width First Open Source Width Benchmark

November 9, 2021

Leveraging the Graph Structure of Neural Network Training Dynamics
Fatemeh Vahedian, Ruiyu Li, Puja Trivedi, Di Jin, Danai Koutra
Deep Neural Network Graph Structure Training Dynamic Eigenvector Centrality Network Width