Multi-Head Attention
Overview
Multi-head attention studies how Transformers use parallel query-key-value computations to select and combine information across tokens or features. Researchers probe head specialization with activation analysis, ablation, and circuit discovery, while theoretical work examines expressivity, learning dynamics, and the number of heads needed for particular computations. Architectural approaches include sparse and grouped attention, cross-head communication, and compressed projections, evaluated in settings such as long-context retrieval, language modeling, vision, and reasoning benchmarks. A central tension is that specialized heads can support useful capabilities, but their roles vary by task and model, and reducing attention cost does not always preserve accuracy.
Latest papers
77 papers6 papers in the last four weeks, down 25% on the four weeks before. 0.0% of all new papers.
Hard Sample-Conditioned Decorrelated Attention for Robust Face Recognition
Head Dimension Matters in Language Models for Long-Context Retrieval
Formally Verified Autoresearch for Analyzing the Expressivity of Multi-Head Attention
Can Capability Enhancement be Localized?
Choosing the Number of Attention Heads from Intrinsic Dimension: A Small-Scale Study
Asymptotic Analysis of Discrete-Time Self-Attention Dynamics in Transformers
Data-Adaptive Mahalanobis Metric Learning for Cross-Head Attention in Transformers
Gaussian Equivalence for Multi-Head Self-Attention
Let the Heads Talk: Beyond Diagonal Graph Attention
Learned Queries and Keys Are All You Need: Replacing the Value Projection with Structured Transforms
MorphAtt: A Neuromorphic Accelerator for Efficient Multi-Head Attention Processing in Spiking Vision Transformers
CoWindow Attention: Full Causal Coverage Is a Collective Property
Transformer Heads Looking for Order
The Head Complexity of Boolean Functions in Single-Layer Attention
Multi-Head Self Attention is a Parameter Identification Mechanism
meaningful'' parameters to all parameters. Simple numerical examples demonstrate that training can indeed involve updates that overlap model-invariant subspaces that arise from a lack of identification. As part of our experiments we use a rebalancing'' approach that can ``fix'' updates that overlap unindentified subspaces but do not try to present evidence this should actually be adopted. Instead we simply view our numerical results as exploring and confirming the theoretical results. As a whole we discuss a purely mathematical/statistical explanation, identification, for why specific architectural choices in transformers may have improved performance.