Self Attention Computation

Self-attention computation, a core component of transformer-based models, is being actively researched to improve the efficiency and scalability of large language and vision models. Current efforts focus on reducing computational complexity through techniques like sparse attention (e.g., using low-rank key vectors or sliding windows), optimized memory management (e.g., employing product quantization for key-value caching), and architectural innovations (e.g., incorporating convolutional layers or wavelet transforms). These advancements are crucial for enabling the deployment of increasingly powerful models on resource-constrained devices and for handling longer sequences, impacting various applications from natural language processing to computer vision.

Papers

July 1, 2024

PQCache: Product Quantization-based KVCache for Long Context LLM Inference
Hailin Zhang, Xiaodong Ji, Yilin Chen, Fangcheng Fu, Xupeng Miao, Xiaonan Nie, Weipeng Chen, Bin Cui
LLM Inference Key Value Cache Product Attribute KV Cache Long Context LLM Self Attention Computation Memory Bottleneck

June 4, 2024

Loki: Low-rank Keys for Efficient Sparse Attention
Prajwal Singhania, Siddharth Singh, Shwai He, Soheil Feizi, Abhinav Bhatele
Large Language Model Sparse Attention Attention Computation Secret Key Self Attention Computation

May 27, 2024

SWAT: Scalable and Efficient Window Attention-based Transformers Acceleration on FPGAs
Zhenyu Bai, Pranav Dangi, Huize Li, Tulika Mitra
Potential Scalability Field Programmable Gate Array Sparse Attention Dataflow Architecture Attention Based Transformer Police Public Interaction Self Attention Computation Window Selection

February 23, 2024

ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition
Lu Ye, Ze Tao, Yong Huang, Yang Li
Large Language Model Self Attention Self Attention Module Chunk Wise Kernel Attention KV Cache Self Attention Computation Phase Dynamic

October 9, 2023

Plug n' Play: Channel Shuffle Module for Enhancing Tiny Vision Transformers
Xuwei Xu, Sen Wang, Yudong Chen, Jiajun Liu
Vision Transformer Self Attention Convolutional Layer Play Style Self Attention Computation Channel Mixing

July 18, 2023

Attention over pre-trained Sentence Embeddings for Long Document Classification
Amine Abdaoui, Sourav Dutta
Human Attention NLP Task Sentence Transformer Long Document Classification Pre Trained Sentence Self Attention Computation

May 11, 2023

Patch-wise Mixed-Precision Quantization of Vision Transformer
Junrui Xiao, Zhikai Li, Lianwei Yang, Qingyi Gu
Vision Transformer Low Precision Mixed Precision Quantization Self Attention Computation

February 23, 2023

MossFormer: Pushing the Performance Limit of Monaural Speech Separation using Gated Single-Head Transformer with Convolution-Augmented Joint Self-Attentions
Shengkui Zhao, Bin Ma
Self Attention Head Transformer Attention Based Sequence Dual Path Transformer Self Attention Computation Monaural Speech Separation Performance Limit

January 5, 2023

Skip-Attention: Improving Vision Transformers by Paying Less Attention
Shashanka Venkataramanan, Amir Ghodrati, Yuki M. Asano, Fatih Porikli, Amirhossein Habibian
Vision Transformer Semantic Segmentation Self Attention Self Attention Block Self Attention Computation Skip Attention

October 28, 2022

Contextual Learning in Fourier Complex Field for VHR Remote Sensing Images
Yan Zhang, Xiyuan Gao, Qingyan Duan, Jiaxu Leng, Xiao Pu, Xinbo Gao
Context Learning Image Classification Remote Sensing High Resolution Remote Sensing Self Attention Computation Naive Adaptation

July 11, 2022

Wave-ViT: Unifying Wavelet and Transformers for Visual Representation Learning
Ting Yao, Yingwei Pan, Yehao Li, Chong-Wah Ngo, Tao Mei
Vision Transformer Transformer Megatron Decepticons Visual Representation Learning Wavelet Transform Self Attention Computation Multi Scale Vision Transformer ImageNet 64 Model