Bottleneck Structure

Bottleneck structures, characterized by a compression and expansion of information flow within a network, are a central theme in current deep learning research. Studies focus on understanding their emergence in various architectures, including convolutional neural networks (CNNs) and residual networks (ResNets), and analyzing their impact on computational efficiency and feature learning. This research aims to optimize network design by leveraging bottleneck mechanisms to reduce complexity while maintaining or improving performance, as demonstrated by improvements in models like AugShuffleNet. The insights gained are crucial for developing more efficient and effective deep learning models across diverse applications, such as robotic surgery and image classification.

Papers

June 22, 2024

SEDMamba: Enhancing Selective State Space Modelling with Bottleneck Mechanism and Fine-to-Coarse Temporal Fusion for Efficient Error Detection in Robot-Assisted Surgery
Jialang Xu, Nazir Sirajudeen, Matthew Boal, Nader Francis, Danail Stoyanov, Evangelos Mazomenos
Error Detection Robot Assisted Surgery Temporal Fusion Selective State Space Robotic Assisted Surgery Medical Error Bottleneck Structure

May 27, 2024

Hamiltonian Mechanics of Feature Learning: Bottleneck Structure in Leaky ResNets
Arthur Jacot, Alexandre Kaiser
Feature Learning Low Dimensional Representation Dimensional Input Fully Connected Neural Network Leaky ReLU Hamiltonian Function Bottleneck Structure

February 12, 2024

Which Frequencies do CNNs Need? Emergent Bottleneck Structure in Feature Learning
Yuxiao Wen, Arthur Jacot
CNN Model Feature Learning Major Challenge Bottleneck Large Learning Rate Dominant Low Frequency Variational Bottleneck Bottleneck Structure

May 30, 2023

Bottleneck Structure in Learned Features: Low-Dimension vs Regularity Tradeoff
Arthur Jacot
Low Dimensional Representation Large Learning Rate Infinite Depth L2 Regularization Learned Feature Low Dimension Bottleneck Structure

November 11, 2022

The Architectural Bottleneck Principle
Tiago Pimentel, Josef Valvoda, Niklas Stoehr, Ryan Cotterell
Contextual Representation Syntactic Information Bottleneck Structure

March 13, 2022

AugShuffleNet: Communicate More, Compute Less
Longqing Ye
Timely Communication Shuffle Model Efficient CNN Bottleneck Structure