Window Attention

Window attention mechanisms in transformers aim to improve efficiency and performance by limiting the scope of attention calculations to localized "windows" within input data, addressing the quadratic complexity of standard self-attention. Current research focuses on optimizing window designs (e.g., rectangular, triangular, variable-sized, shifted windows), integrating window attention with other techniques (e.g., State Space Models, Fourier transforms), and developing novel architectures (e.g., DwinFormer, HRSAM) that leverage window attention for various tasks like image super-resolution, semantic segmentation, and language modeling. These advancements are significantly impacting computer vision and natural language processing by enabling faster and more efficient processing of high-resolution images and long sequences, leading to improved accuracy in diverse applications.

Papers

September 29, 2022

Dilated Neighborhood Attention Transformer
Ali Hassani, Humphrey Shi
Hierarchical Transformer Hierarchical Vision Transformer Window Attention Neighborhood Attention

June 23, 2022

MaskViT: Masked Visual Pre-Training for Video Prediction
Agrim Gupta, Stephen Tian, Yunzhi Zhang, Jiajun Wu, Roberto Martín-Martín, Li Fei-Fei
Video Prediction Window Attention Visual Model Visual Pre Training

May 26, 2022

Green Hierarchical Vision Transformer for Masked Image Modeling
Lang Huang, Shan You, Mingkai Zheng, Fei Wang, Chen Qian, Toshihiko Yamasaki
Convolution Layer Masked Image Modeling Hierarchical Vision Transformer Window Attention Sparse Convolution

May 15, 2022

Hero-Gang Neural Model For Named Entity Recognition
Jinpeng Hu, Yaling Shen, Yang Liu, Xiang Wan, Tsung-Hui Chang
Entity Recognition Global Context Window Attention Recurrent Layer

May 8, 2022

Transformer Tracking with Cyclic Shifting Window Attention
Zikai Song, Junqing Yu, Yi-Ping Phoebe Chen, Wei Yang
Attention Mechanism Visual Object Tracking Window Attention Pixel Attention Cross Scale Attention

April 18, 2022

VSA: Learning Varied-Size Window Attention in Vision Transformers
Qiming Zhang, Yufei Xu, Jing Zhang, Dacheng Tao
Vision Transformer Semantic Segmentation Window Attention

April 14, 2022

Neighborhood Attention Transformer
Ali Hassani, Steven Walton, Jiachen Li, Shen Li, Humphrey Shi
Self Attention Window Attention Neighborhood Attention

January 5, 2022

Lawin Transformer: Improving Semantic Segmentation Transformer with Multi-Scale Representations via Large Window Attention
Haotian Yan, Chuang Zhang, Ming Wu
Vision Transformer Semantic Segmentation Hierarchical Vision Transformer Multi Scale Representation Window Attention Transformer XL

November 25, 2021

Global Interaction Modelling in Vision Transformer via Super Tokens
Ammarah Farooq, Muhammad Awais, Sara Ahmed, Josef Kittler
Vision Transformer Swin Transformer K TOKEN Global Feature Pyramid Network Window Attention Token Transformer