Self Attention Layer

Self-attention layers are a core component of Transformer networks, enabling these models to process sequential data by weighting the importance of different elements within the sequence. Current research focuses on improving the efficiency and theoretical understanding of self-attention, including exploring its optimization dynamics, analyzing its role in generalization and hallucination in large language models, and developing alternative attention mechanisms like Locality Sensitive Hashing or polynomial-based approaches to reduce computational cost. These advancements are driving improvements in various applications, from image segmentation and super-resolution to natural language processing and visual place recognition, by enhancing model performance and scalability.

Papers

March 3, 2022

Multi-Tailed Vision Transformer for Efficient Inference
Yunke Wang, Bo Du, Wenyuan Wang, Chang Xu
Vision Transformer Efficient Inference Self Attention Layer Transformer Encoder Transformer Encoders

February 24, 2022

TrimBERT: Tailoring BERT for Trade-offs
Sharath Nittur Sridhar, Anthony Sarah, Sairam Sundaresan
Natural Language Processing Ticket BERT Self Attention Layer BERT Base

December 2, 2021

Stacked Temporal Attention: Improving First-person Action Recognition by Emphasizing Discriminative Clips
Lijin Yang, Yifei Huang, Yusuke Sugano, Yoichi Sato
Action Recognition Temporal Attention Self Attention Layer Temporal Attention Module CLIP Enhanced Blockwise Classification

November 30, 2021

Shunted Self-Attention via Multi-Scale Token Aggregation
Sucheng Ren, Daquan Zhou, Shengfeng He, Jiashi Feng, Xinchao Wang
Vision Transformer Self Attention Self Attention Layer Spatial Token Focal Transformer

Self Attention Layer

Papers

Multi-Tailed Vision Transformer for Efficient Inference

TrimBERT: Tailoring BERT for Trade-offs

Stacked Temporal Attention: Improving First-person Action Recognition by Emphasizing Discriminative Clips

Shunted Self-Attention via Multi-Scale Token Aggregation