Self Attention Layer

Self-attention layers are a core component of Transformer networks, enabling these models to process sequential data by weighting the importance of different elements within the sequence. Current research focuses on improving the efficiency and theoretical understanding of self-attention, including exploring its optimization dynamics, analyzing its role in generalization and hallucination in large language models, and developing alternative attention mechanisms like Locality Sensitive Hashing or polynomial-based approaches to reduce computational cost. These advancements are driving improvements in various applications, from image segmentation and super-resolution to natural language processing and visual place recognition, by enhancing model performance and scalability.

Papers

March 7, 2023

Event Voxel Set Transformer for Spatiotemporal Representation Learning on Event Streams
Bochen Xie, Yongjian Deng, Zhanpeng Shao, Qingsong Xu, Youfu Li
Self Attention Layer Spatiotemporal Representation Event Stream Event Based Object Recognition Voxel Transformer

February 12, 2023

A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexity
Hongkang Li, Meng Wang, Sijia Liu, Pin-yu Chen
LeArning Abstract Vision Transformer Strong Generalization Theoretical Understanding Sample Complexity Attention Map Self Attention Layer Self Attention Module Shallow Transformer

January 27, 2023

On the Connection Between MPNN and Graph Transformer
Chen Cai, Truong Son Hy, Rose Yu, Yusu Wang
Graph Transformer Graph Learning Self Attention Layer New Connection Message Passing Neural Network Graph Benchmark

January 22, 2023

Interpretability in Activation Space Analysis of Transformers: A Focused Survey
Soniya Vijayakumar
Transformer Megatron Decepticons Latent Space Inherent Interpretability Comprehensive Survey Attention Layer Self Attention Layer Activation Space Feed Forward Layer

January 4, 2023

Infomaxformer: Maximum Entropy Transformer for Long Time-Series Forecasting Problem
Peiwang Tang, Xianchao Zhang
Transformer Architecture Temporal Dependency Self Attention Layer Long Term Time Series Forecasting Encoder Decoder Architecture Long Range Dependency Transformer Based Entropy Model

October 13, 2022

Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer
Yanjing Li, Sheng Xu, Baochang Zhang, Xianbin Cao, Peng Gao, Guodong Guo
Vision Transformer Multiplier Free Quantization Self Attention Layer Pre Trained Vision Transformer Bit Vision Transformer

October 9, 2022

KSAT: Knowledge-infused Self Attention Transformer -- Integrating Multiple Domain-Specific Contexts
Kaushik Roy, Yuxin Zi, Vignesh Narayanan, Manas Gaur, Amit Sheth
Self Attention Self Attention Layer Domain Specific Task

September 30, 2022

Verifiable and Energy Efficient Medical Image Analysis with Quantised Self-attentive Deep Neural Networks
Rakshith Sathish, Swanand Khare, Debdoot Sheet
Convolutional Neural Network Medical Image Analysis Convolutional Layer Self Attention Layer Traditional CNNs

September 20, 2022

Relaxed Attention for Transformer Models
Timo Lohrenz, Björn Möller, Zhengyang Li, Tim Fingscheidt
Transformer Model Self Attention Layer Confidence Relaxation Autoregressive Decoder

September 19, 2022

Integrative Feature and Cost Aggregation with Transformers for Dense Correspondence
Sunghwan Hong, Seokju Cho, Seungryong Kim, Stephen Lin
Transformer Megatron Decepticons Self Attention Layer Dense Correspondence Transformer Based Neural Network Dense Matching Cost Aggregation Cost Volume

August 18, 2022

Learning Spatial-Frequency Transformer for Visual Object Tracking
Chuanming Tang, Xiao Wang, Yuanchao Bai, Zhe Wu, Jianlin Zhang, Yongmei Huang
LeArning Abstract Self Attention Layer Visual Object Tracking Siamese Tracker Spectral Spatial Transformer

August 4, 2022

DropKey
Bonan Li, Yinhan Hu, Xuecheng Nie, Congying Han, Xiangjian Jiang, Tiande Guo, Luoqi Liu
Vision Transformer Softmax Function Structured Dropout Self Attention Layer

July 5, 2022

Weakly Supervised Grounding for VQA in Vision-Language Transformers
Aisha Urooj Khan, Hilde Kuehne, Chuang Gan, Niels Da Vitoria Lobo, Mubarak Shah
Visual Question Answering Visual Grounding Self Attention Layer Visual Encoder Vision Language Transformer

June 28, 2022

ZoDIAC: Zoneout Dropout Injection Attention Calculation
Zanyar Zohourianshahzadi, Jugal Kalita
Self Attention Self Attention Layer Self Attention Module Patch Attention

June 24, 2022

Excavating RoI Attention for Underwater Object Detection
Xutao Liang, Pinhao Song
Attention Module Self Attention Layer Underwater Object Detection Direct Roi Prediction

June 7, 2022

Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse
Lorenzo Noci, Sotiris Anagnostidis, Luca Biggio, Antonio Orvieto, Sidak Pal Singh, Aurelien Lucchi
Transformer Megatron Decepticons Integral Role Natural Gradient Gradient Norm Self Attention Layer Propagation Environment Rank Collapse Theoretical Consideration Effective Depth Up Scaling

April 1, 2022

March 15, 2022

SATS: Self-Attention Transfer for Continual Semantic Segmentation
Yiqiao Qiu, Yixing Shen, Zhuohao Sun, Yanchong Zheng, Xiaobin Chang, Weishi Zheng, Ruixuan Wang
Convolutional Neural Network Transformer Based Self Attention Self Attention Layer Continual Semantic Segmentation

March 14, 2022

Simplicial Attention Neural Networks
L. Giusti, C. Battiloro, P. Di Lorenzo, S. Sardellitti, S. Barbarossa
Self Attention Neural Network Architecture Self Attention Layer Topological Signal Simplicial Attention