Multi Head Attention

Multi-head attention is a mechanism within transformer networks that allows the model to attend to different aspects of input data simultaneously, improving performance on various tasks. Current research focuses on optimizing multi-head attention for efficiency, including exploring alternative architectures like grouped-query attention and methods to reduce computational complexity without sacrificing accuracy, such as pruning or low-precision approximations. These advancements are significant because they enable the application of transformer models to larger datasets and more complex problems across diverse fields, including image processing, audio classification, and natural language processing.

Papers

May 30, 2024

Why Larger Language Models Do In-context Learning Differently?
Zhenmei Shi, Junyi Wei, Zhuoyan Xu, Yingyu Liang
Large Language Model Context Learning Multi Head Attention Larger Language Model Layer Transformer

May 26, 2024

Disentangling and Integrating Relational and Sensory Information in Transformer Architectures
Awni Altabaa, John Lafferty
Attention Mechanism Transformer Architecture Message Passing Multi Head Attention Sensory Input Disentangle Content Relational Concept

May 14, 2024

Improving Transformers with Dynamically Composable Multi-Head Attention
Da Xiao, Qingye Meng, Shengping Li, Xingyuan Yuan
Transformer Megatron Decepticons Human Attention Transformer Architecture Multi Head Attention Composable Framework

April 18, 2024

Transformer tricks: Removing weights for skipless transformers
Nils Graef
Balancing Weight Multi Head Attention Skip Transformer

April 11, 2024

LATTE: Low-Precision Approximate Attention with Head-wise Trainable Threshold for Efficient Transformer
Jiing-Ping Wang, Ming-Guang Lin, An-Yeu, Wu
Efficient Transformer Multi Head Attention Approximate Attention

March 22, 2024

Not All Attention is Needed: Parameter and Computation Efficient Transfer Learning for Multi-modal Large Language Models
Qiong Wu, Weihao Ye, Yiyi Zhou, Xiaoshuai Sun, Rongrong Ji
Human Attention Many Parameter Multi Modal Large Language Model Information Redundancy Multi Head Attention Inference Speed Efficient Transfer Learning Skip Attention

March 16, 2024

EfficientMorph: Parameter-Efficient Transformer-Based Architecture for 3D Image Registration
Abu Zahid Bin Aziz, Mokshagna Sai Teja Karanam, Tushar Kataria, Shireen Y. Elhabian
Attention Mechanism Image Registration Medical Image Registration Efficient Transformer Multi Head Attention

February 16, 2024

LongHeads: Multi-Head Attention is Secretly a Long Context Processor
Yi Lu, Xin Zhou, Wei He, Jun Zhao, Tao Ji, Tao Gui, Qi Zhang, Xuanjing Huang
Large Language Model Long Context Multi Head Attention Length Generalization

February 6, 2024

Provably learning a multi-head attention layer
Sitan Chen, Yuanzhi Li
Large Language Model Transformer Architecture Multi Head Attention Attention Matrix

January 30, 2024

Superiority of Multi-Head Attention in In-Context Linear Regression
Yingqian Cui, Jie Ren, Pengfei He, Jiliang Tang, Yue Xing
Human Attention Transformer Architecture Multi Head Attention Attention Head Softmax Attention

January 11, 2024

TAnet: A New Temporal Attention Network for EEG-based Auditory Spatial Attention Decoding with a Short Decision Window
Yuting Ding, Fei Chen
Temporal Attention Brain Computer Interface Multi Head Attention Auditory Attention Audio Recognition Electroencephalography Based Auditory Attention Faster Decision

October 28, 2023

TLM: Token-Level Masking for Transformers
Yangjun Wu, Kebin Fang, Dongxiang Zhang, Han Wang, Hao Zhang, Gang Chen
Transformer Megatron Decepticons Structured Dropout Multi Head Attention Multi Head Attention Mechanism Token Masking

October 26, 2023

Sliceformer: Make Multi-head Attention as Simple as Sorting in Discriminative Tasks
Shen Yuan, Hongteng Xu
Transformer Based Attention Layer Multi Head Attention Discriminative Task Model Slicing

October 23, 2023

Sentiment analysis with adaptive multi-head attention in Transformer
Fanfei Meng, Chen-Ao Wang
Transformer Based Sentiment Analysis Attention Mechanism Multi Head Attention Multi Head Review Dataset

October 16, 2023

October 11, 2023

Pit One Against Many: Leveraging Attention-head Embeddings for Parameter-efficient Multi-head Attention
Huiyin Xue, Nikolaos Aletras
Pre Trained Language Model Human Head Multi Head Attention Attention Head Multi Head Graph PIT

September 5, 2023

MA-VAE: Multi-head Attention-based Variational Autoencoder Approach for Anomaly Detection in Multivariate Time-series Applied to Automotive Endurance Powertrain Testing
Lucas Correia, Jan-Christoph Goos, Philipp Klein, Thomas Bäck, Anna V. Kononova
Anomaly Detection Variational Autoencoder Multivariate Time Series Multi Head Attention Industrial Datasets

August 7, 2023

RCMHA: Relative Convolutional Multi-Head Attention for Natural Language Modelling
Herman Sugiharto, Aradea, Husni Mubarok
Attention Module Multi Head Attention Multi Head Relative Positional Encoding Natural Language Model

July 11, 2023

Mao-Zedong At SemEval-2023 Task 4: Label Represention Multi-Head Attention Model With Contrastive Learning-Enhanced Nearest Neighbor Mechanism For Multi-Label Text Classification
Che Zhang, Ping'an Liu, Zhenyang Xiao, Haojun Fei
SemEval 2022 Task Computational Linguistics Multi Head Attention Multi Label Text Classification Human Value Multi Head Attention Mechanism Nearest Neighbor Contrastive Learning

Multi Head Attention

Papers

Why Larger Language Models Do In-context Learning Differently?

Disentangling and Integrating Relational and Sensory Information in Transformer Architectures

Improving Transformers with Dynamically Composable Multi-Head Attention

Transformer tricks: Removing weights for skipless transformers

LATTE: Low-Precision Approximate Attention with Head-wise Trainable Threshold for Efficient Transformer

Not All Attention is Needed: Parameter and Computation Efficient Transfer Learning for Multi-modal Large Language Models

EfficientMorph: Parameter-Efficient Transformer-Based Architecture for 3D Image Registration

LongHeads: Multi-Head Attention is Secretly a Long Context Processor

Provably learning a multi-head attention layer

Superiority of Multi-Head Attention in In-Context Linear Regression

TAnet: A New Temporal Attention Network for EEG-based Auditory Spatial Attention Decoding with a Short Decision Window

TLM: Token-Level Masking for Transformers

Sliceformer: Make Multi-head Attention as Simple as Sorting in Discriminative Tasks

Sentiment analysis with adaptive multi-head attention in Transformer

On the Relevance of Temporal Features for Medical Ultrasound Video Recognition

Interpreting and Exploiting Functional Specialization in Multi-Head Attention under Multi-task Learning

Pit One Against Many: Leveraging Attention-head Embeddings for Parameter-efficient Multi-head Attention

MA-VAE: Multi-head Attention-based Variational Autoencoder Approach for Anomaly Detection in Multivariate Time-series Applied to Automotive Endurance Powertrain Testing

RCMHA: Relative Convolutional Multi-Head Attention for Natural Language Modelling

Mao-Zedong At SemEval-2023 Task 4: Label Represention Multi-Head Attention Model With Contrastive Learning-Enhanced Nearest Neighbor Mechanism For Multi-Label Text Classification