Neighborhood Attention

Neighborhood attention is a computational technique that improves the efficiency of attention mechanisms, particularly in deep learning models processing large datasets like images or graphs. Current research focuses on integrating neighborhood attention into transformer architectures, such as Dilated Neighborhood Attention Transformers (DiNAT), to enhance feature extraction and reduce computational complexity, often achieving state-of-the-art performance in tasks like image segmentation, object detection, and speaker verification. This approach offers significant advantages in speed and memory usage compared to traditional self-attention, making it valuable for real-time applications and resource-constrained environments. The resulting improvements in accuracy and efficiency have broad implications across various fields, including medical image analysis and autonomous driving.

Papers

August 20, 2024

EdgeNAT: Transformer for Efficient Edge Detection
Jinghuai Jie, Yan Guo, Guixing Wu, Junmin Wu, Baojian Hua
Transformer Based Extreme Edge Edge Detection Edge Transformer Neighborhood Attention

July 16, 2024

Learning Multi-view Anomaly Detection
Haoyang He, Jiangning Zhang, Guanzhong Tian, Chengjie Wang, Lei Xie
Multi View Feature Learning Task Relatedness Neighborhood Attention

May 20, 2024

Neighborhood Attention Transformer with Progressive Channel Fusion for Speaker Verification
Nian Li, Jianguo Wei
Speaker Verification Neighborhood Attention Attentive Pooling

March 7, 2024

Faster Neighborhood Attention: Reducing the O(n^2) Cost of Self Attention at the Threadblock Level
Ali Hassani, Wen-Mei Hwu, Humphrey Shi
Self Attention Hidden CoST Kernel Attention Attention Pattern Window Attention Learned Kernel Neighborhood Attention Block Level

March 3, 2024

Representation Learning on Heterophilic Graph with Directional Neighborhood Attention
Qincheng Lu, Jiaqi Zhu, Sitao Luan, Xiao-Wen Chang
Representation Learning Graph Attention Network Global Attention Heterophilic Graph Neighborhood Attention Local Global Interactive Graph

December 12, 2023

NAC-TCN: Temporal Convolutional Networks with Causal Dilated Neighborhood Attention for Emotion Understanding
Alexander Mehta, William Yang
Emotion Recognition Temporal Convolutional Network Emotion Understanding Neighborhood Attention

October 19, 2023

Real-Time Motion Prediction via Heterogeneous Polyline Transformer with Relative Pose Encoding
Zhejun Zhang, Alexander Liniger, Christos Sakaridis, Fisher Yu, Luc Van Gool
Autonomous Driving Neighborhood Attention Scene Centric

May 9, 2023

CAMIL: Context-Aware Multiple Instance Learning for Cancer Detection and Subtyping in Whole Slide Images
Olga Fourkioti, Matt De Vries, Chen Jin, Daniel C. Alexander, Chris Bakal
Multiple Instance Learning Whole Slide Image Contextual Information Cancer Detection Tumor Cell Neighborhood Attention

April 22, 2023

Dilated-UNet: A Fast and Accurate Medical Image Segmentation Approach using a Dilated Transformer and U-Net Architecture
Davoud Saadati, Omid Nejati Manzari, Sattar Mirzakuchaki
Medical Image Segmentation Global Attention Hierarchical Transformer Medical Image Segmentation Datasets U Net Architecture Neighborhood Attention

November 15, 2022

Adaptive Multi-Neighborhood Attention based Transformer for Graph Representation Learning
Gaichao Li, Jinsong Chen, Kun He
Transformer Based Graph Representation Learning Graph Transformer Neighborhood Attention

September 29, 2022

Dilated Neighborhood Attention Transformer
Ali Hassani, Humphrey Shi
Hierarchical Transformer Hierarchical Vision Transformer Window Attention Neighborhood Attention

April 14, 2022

Neighborhood Attention Transformer
Ali Hassani, Steven Walton, Jiachen Li, Shen Li, Humphrey Shi
Self Attention Window Attention Neighborhood Attention

Neighborhood Attention

Papers

EdgeNAT: Transformer for Efficient Edge Detection

Learning Multi-view Anomaly Detection

Neighborhood Attention Transformer with Progressive Channel Fusion for Speaker Verification

Faster Neighborhood Attention: Reducing the O(n^2) Cost of Self Attention at the Threadblock Level

Representation Learning on Heterophilic Graph with Directional Neighborhood Attention

NAC-TCN: Temporal Convolutional Networks with Causal Dilated Neighborhood Attention for Emotion Understanding

Real-Time Motion Prediction via Heterogeneous Polyline Transformer with Relative Pose Encoding

CAMIL: Context-Aware Multiple Instance Learning for Cancer Detection and Subtyping in Whole Slide Images

Dilated-UNet: A Fast and Accurate Medical Image Segmentation Approach using a Dilated Transformer and U-Net Architecture

Adaptive Multi-Neighborhood Attention based Transformer for Graph Representation Learning

Dilated Neighborhood Attention Transformer

Neighborhood Attention Transformer