Skip Attention

Skip attention techniques aim to improve the efficiency and performance of attention mechanisms in various deep learning models, particularly large language models and vision transformers, by selectively reducing redundant computations. Current research focuses on developing methods to identify and skip less important attention operations, often employing novel architectures like Siamese Self-Attention Blocks or propagation-of-information adapters, to achieve faster inference speeds without significant accuracy loss. These advancements are significant because they address the high computational cost of attention, enabling the deployment of more powerful models in resource-constrained environments and accelerating training and inference across diverse applications like image processing, natural language understanding, and multimodal learning.

Papers

March 22, 2024

Not All Attention is Needed: Parameter and Computation Efficient Transfer Learning for Multi-modal Large Language Models
Qiong Wu, Weihao Ye, Yiyi Zhou, Xiaoshuai Sun, Rongrong Ji
Human Attention Many Parameter Multi Modal Large Language Model Information Redundancy Multi Head Attention Inference Speed Efficient Transfer Learning Skip Attention

December 28, 2023

DarkShot: Lighting Dark Images with Low-Compute and High-Quality
Jiazhang Zheng, Lei Li, Qiuping Liao, Cheng Li, Li Li, Yangxing Liu
High Quality Low Light Low Light Condition Low Light Image Enhancement Task Low Compute Night Photography Skip Attention

July 8, 2023

Bidirectional Attention as a Mixture of Continuous Word Experts
Kevin Christian Wibisono, Yixin Wang
Language Model Mixture Component Bidirectional Attention Skip Attention

January 5, 2023

Skip-Attention: Improving Vision Transformers by Paying Less Attention
Shashanka Venkataramanan, Amir Ghodrati, Yuki M. Asano, Fatih Porikli, Amirhossein Habibian
Vision Transformer Semantic Segmentation Self Attention Self Attention Block Self Attention Computation Skip Attention

October 17, 2022

Attention Attention Everywhere: Monocular Depth Prediction with Skip Attention
Ashutosh Agarwal, Chetan Arora
Deep Convolutional Neural Network Monocular Depth Estimation Attention Based Attention Head Pixel Wise Depth Skip Attention

Skip Attention

Papers

Not All Attention is Needed: Parameter and Computation Efficient Transfer Learning for Multi-modal Large Language Models

DarkShot: Lighting Dark Images with Low-Compute and High-Quality

Bidirectional Attention as a Mixture of Continuous Word Experts

Skip-Attention: Improving Vision Transformers by Paying Less Attention

Attention Attention Everywhere: Monocular Depth Prediction with Skip Attention