Attention Head Specialization

Follow this topic Ask about this topic

Attention head specialization asks how individual heads divide work, such as retrieving context, tracking positions, handling modalities, or supporting particular computations. Researchers study these roles through mechanistic probes, activation patching and ablation, and analyses of how specialization emerges during training; some methods also encourage roles explicitly. Head-aware pruning, KV-cache policies, and hybrid attention use these differences to reduce computation in settings such as long-context retrieval, multimodal reasoning, and video generation. A central challenge is distinguishing a head that appears specialized from one that is causally necessary, since roles can depend on the task, architecture, and intervention used.

Latest papers

40 papers5 papers in the last four weeks, up 67% on the four weeks before. 0.0% of all new papers.

HeadRouter: Dynamic Head-Weight Routing for Task-Adaptive Audio Token Pruning in Large Audio Language Models
Apr 26, 2026Peize He, Yaodi Luo, Xiaoqian Liu +7Audio Token Compression Self-Attention
1EPIC Lab, Shanghai Jiao Tong University · 2DAIL Tech · 3Northeastern University +2
What Is the Minimum Architecture for Prolepsis? Early Irrevocable Commitment Across Tasks in Small Transformers
Apr 16, 2026Éric JacopinTransformer Interpretability Attention Head Analysis
Cosmic AI
Decoupling and Regulating Modality Heads for Fake News Detection under Missing Modality
Apr 8, 2026Kai Qian, Dongjie Xu, Weijie Shi +7Fake News Detection Multimodal Robustness
Soochow University · Hong Kong University of Science and Technology · Computer Science and Engineering, The Chinese University of Hong Kong +2
NDT: Non-Differential Transformer and Its Application to Sentiment Analysis
Mar 21, 2026Soudeep Ghoshal, Himanshu Buckchash, Sarita Paudel +1Transformer Attention Attention Mechanisms
Kalinga Institute of Industrial Technology (KIIT), Bhubaneswar, Odisha 751024, India. · IMC University of Applied Sciences Krems, Piaristengasse 1, 3500 Austria.
Specialization of softmax attention heads: insights from the high-dimensional single-location model
Mar 4, 2026M. Sagitova, O. Duranthon, L. ZdeborováSoftmax Attention Multi-Head Attention
Statistical physics of computation laboratory, École Polytechnique Fédérale de Lausanne, Switzerland
DeAR: Fine-Grained VLM Adaptation by Decomposing Attention Head Roles
Mar 1, 2026Yiming Ma, Hongkun Yang, Lionel Z. Wang +3VLM Adaptation Prompt Tuning
Chongqing Research Institute of Harbin Institute of Technology, China · Ocean University of China, China · The Hong Kong Polytechnic University, Hong Kong SAR +1
Identifying Crucial Attention Heads for Multilingual Language Models: Retrieval and Retrieval-Transition Heads
Feb 25, 2026Shaswat Patel, Vishvesh Trivedi, Yue Han +2Multilingual Language Models Cross-Lingual Reasoning in Language Models
New York University
Incremental Learning of Sparse Attention Patterns in Transformers
Feb 22, 2026Oğuz Kaan Yüksel, Rodrigo Alvarez Lucendo, Nicolas FlammarionSelf-Attention Transformer Training Dynamics
TML Lab, EPFL
The Anxiety of Influence: Bloom Filters in Transformer Attention Heads
Feb 19, 2026Peter BaloghAttention Head Analysis Attention Head Specialization
A Mechanistic Study of Transformers Training Dynamics
Oct 31, 2024Ambroise Odonnat, Wassim Bouaziz, Vivien CabannesTransformer Interpretability Attention Mechanisms
Inria, Univ. Rennes 2 · Mistral AI; work done while at Meta. · FAIR at Meta.

Apr 26, 2026J/K move · Enter open · S save

HeadRouter: Dynamic Head-Weight Routing for Task-Adaptive Audio Token Pruning in Large Audio Language Models

Peize He, Yaodi Luo, Xiaoqian Liu, Xuyang Liu+6

1EPIC Lab, Shanghai Jiao Tong University · 2DAIL Tech · 3Northeastern University+2

Audio Token Compression Self-Attention Audio Understanding Audio-Language Models Token Pruning Multimodal Token Compression+3

Open paper AskPDF arXiv

Recent large audio language models (LALMs) demonstrate remarkable capabilities in processing extended multi-modal sequences, yet incur high inference costs. Token compression is an effective method that directly reduces redundant tokens in the sequence. Existing compression methods usually assume that all attention heads in LALMs contribute equally to various audio tasks and calculate token importance by averaging scores across all heads. However, our analysis demonstrates that attention heads exhibit distinct behaviors across diverse audio domains. We further reveal that only a sparse subset of attention heads actively responds to audio, with completely different performance when handling semantic and acoustic tasks. In light of this observation, we propose HeadRouter, a head-importance-aware token pruning method that perceives the varying importance of attention heads in different audio tasks to maximize the retention of crucial tokens. HeadRouter is training-free and can be applied to various LALMs. Extensive experiments on the AudioMarathon and MMAU-Pro benchmarks demonstrate that HeadRouter achieves state-of-the-art compression performance, exceeding the baseline model even when retaining 70% of the audio tokens and achieving 101.8% and 103.0% of the vanilla average on Qwen2.5-Omni-3B and Qwen2.5-Omni-7B, respectively.