Dot Product Self Attention

Dot product self-attention is a core mechanism in Transformer networks, enabling them to process sequential data by weighting the importance of different input elements. Current research focuses on addressing limitations of standard dot-product attention, such as quadratic computational complexity, susceptibility to representation collapse, and overconfidence in predictions, through methods like elliptical attention, optimal transport-based alternatives (e.g., SeTformer), and Lipschitz regularization. These advancements aim to improve the efficiency, robustness, and calibration of Transformer models across diverse applications, including image recognition, natural language processing, and sequential recommendation.

Papers

June 19, 2024

Elliptical Attention
Stefan K. Nielsen, Laziz U. Abdullaev, Rachel S.Y. Teo, Tan M. Nguyen
Attention Mechanism Dot Product Attention Axial Attention Dot Product Self Attention

February 1, 2024

Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling
Mingze Wang, Weinan E
Transformer Based Sequence Modeling Functional Mechanism Expressive Power Feed Forward Layer Approximation Rate Dot Product Self Attention

January 7, 2024

SeTformer is What You Need for Vision and Language
Pourya Shamsolmoali, Masoumeh Zareapoor, Eric Granger, Michael Felsberg
Human Language Vision Paper Softmax Attention Dot Product Self Attention

June 12, 2023

Mitigating Transformer Overconfidence via Lipschitz Regularization
Wenqian Ye, Yunsheng Ma, Xu Cao, Kun Tang
Vision Task Vision Benchmark Lipschitz Bound Lipschitz Regularization Dot Product Self Attention Regularized Transformer

September 11, 2022

On The Computational Complexity of Self-Attention
Feyza Duman Keles, Pruthuvi Mahesakya Wijewardena, Chinmay Hegde
Self Attention Transformer Architecture Computational Complexity Attention Computation Dot Product Self Attention

May 17, 2022

Unraveling Attention via Convex Duality: Analysis and Interpretations of Vision Transformers
Arda Sahiner, Tolga Ergen, Batu Ozturkler, John Pauly, Morteza Mardani, Mert Pilanci
Vision Transformer General Analysis Attention Mechanism Fourier Neural Operator Convex Function Self Attention Network Dot Product Self Attention

April 22, 2022

Paramixer: Parameterizing Mixing Links in Sparse Factors Works Better than Dot-Product Self-Attention
Tong Yu, Ruslan Khalitov, Lei Cheng, Zhirong Yang
Neural Model Internet Link Low Rank Bottleneck Dot Product Self Attention Attention Coefficient Factorized Attention

January 28, 2022

O-ViT: Orthogonal Vision Transformer
Yanhong Fei, Yingjie Liu, Xian Wei, Mingsong Chen
Vision Transformer Orthogonal Transforms Self Attention Block Dot Product Self Attention

January 16, 2022

Sequential Recommendation via Stochastic Self-Attention
Ziwei Fan, Zhiwei Liu, Alice Wang, Zahra Nazari, Lei Zheng, Hao Peng, Philip S. Yu
Sequential Recommendation Strong Transitivity Relation Discrete Behavior Dot Product Self Attention Item Item