Attention Head Specialization

Follow this topic Ask about this topic

Attention head specialization asks how individual heads divide work, such as retrieving context, tracking positions, handling modalities, or supporting particular computations. Researchers study these roles through mechanistic probes, activation patching and ablation, and analyses of how specialization emerges during training; some methods also encourage roles explicitly. Head-aware pruning, KV-cache policies, and hybrid attention use these differences to reduce computation in settings such as long-context retrieval, multimodal reasoning, and video generation. A central challenge is distinguishing a head that appears specialized from one that is causally necessary, since roles can depend on the task, architecture, and intervention used.

Latest papers

40 papers5 papers in the last four weeks, up 67% on the four weeks before. 0.0% of all new papers.

Do Syntax Heads Influence Model Behavior?
Oct 13, 2026William Timkey, Tal LinzenTransformer Interpretability Attention Head Specialization
New York University
Understanding Unified VLMs: From Attention Probing to Guidance Tuning for Multimodal Generation
Oct 10, 2026Zhengyang Luo, Xiaoyu Qiu, Zhiheng Tan +3Unified Multimodal Models Multimodal Generation
University of Michigan, Ann Arbor · The Ohio State University · Indiana University Indianapolis
Which Attention Heads are like the Human Head? Not the Ones that Compute
Sep 29, 2026Christopher Pinier, Gustaw Opiełka, Hannes Rosenbusch +3Attention Head Analysis Cross-Modal Alignment
Psychological Methods, University of Amsterdam, Amsterdam, The Netherlands · Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA
How Language Models Differ in Redistributing Attention-Head Activity Under Serial Demand
Sep 28, 2026Johnny Jingze Li, Abdulla Kuleib, Kalyan Basu +1Attention Head Analysis LLM Interpretability
Carnegie Mellon University · University of California, San Diego · Qualtrics LLC
When Can Attention Heads Be Statically Defined?
Sep 28, 2026Weixian Waylon Li, Yintao Tai, Marcio Fonseca +1Attention Head Analysis Efficient Attention
University of Edinburgh, United Kingdom · Chamber of Deputies, Brazil
Rethinking Contextualization by Reinterpreting Attention Head Channels
Sep 27, 2026Hakaze Cho, Haolin Yang, Zhun Sun +3Attention Head Analysis Representation Geometry in Language Models
RIKEN · Tohoku University · New York University +2
Temporal-Attention Head Specialization During Video Diffusion Training
Sep 14, 2026Taewoo Ha, Shafayat Mowla Anik, Dae Yeol Lee +2Video Diffusion Models Attention Head Analysis
Independent Researcher · University of Colorado - Colorado Springs · Dolby Laboratories +1
Modern Transformers Are Implicit Hybrids: From Functional Differentiation to Principled Hybrid Architecture Design
Sep 2, 2026Runlin Shi, Bojian Yin, Guoqi LiRotary Positional Embeddings Linear Attention
Institute of Automation, Chinese Academy of Sciences, Beijing, China
Semantic Head Specialization Guides Hybrid ViT Attention for Multimodal LLMs
Aug 28, 2026Chenhong He, Lei Li, Shicheng Li +5Vision Transformer Hybrid Attention
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Xiaomi Corporation, LLM-Core Team · Department of Computer Science, The University of Hong Kong
Not All Attention Heads Contribute to Critical Visual Token Selection: Head-Aware Pruning Matters More
Aug 26, 2026Chaofang Ma, Lin Jiang, Carol Jingyi Li +4Efficient VLM Inference Visual Token Pruning
The Hong Kong University of Science and Technology · Northeastern University · The Hong Kong University of Science and Technology (GZ)
Diffract: Spectral View of LLM Domain Adaptation
Aug 11, 2026Nikita Borodin, Maria Krylova, Artem Zabolotnyi +6Domain Adaptation Domain-Adaptive Pretraining
Risk AI Research Lab · Applied AI Institute
Autonomy-of-Heads: Data-Free Sparse Attention from Frozen Query-Key Geometry
Aug 7, 2026Yehan Yang, Junyuan Shang, Yang Li +3Attention Head Analysis LLM Inference Acceleration
Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Baidu Inc. +1
Interpretability-Guided Soft Pruning of Attention Heads in Vision Transformers
Jul 31, 2026Kamil Książek, Piotr Suszyński, Michał Jan Włodarczyk +2Transformer Interpretability Attention Head Analysis
Centre for Credible Artificial Intelligence Warsaw University of Technology Warsaw, Poland · Faculty of Mathematics and Computer Science Jagiellonian University Krakow, Poland · University of Warsaw Warsaw, Poland
HeadCast: Casting Attention Heads for Efficient Autoregressive Video Generation
Jul 22, 2026Jinliang Shen, Lianghao Su, Zheming Li +4KV-Cache Management Autoregressive Video Generation
1Peking University · 2KlingAI Research
HydraHead: From Head-Level Functional Heterogeneity to Specialized Attention Hybridization
Jun 18, 2026Zhentao Tan, Wei Chen, Jingyi Shen +4Softmax Attention Long-Context Modeling
Alibaba Group
Where Computation Lives Inside TabPFN: Causal Localisation of Attention Head Function
Jun 11, 2026Atharva Gupta, Dhruv Kumar, Murari Mandal +1Attention Head Specialization Attention Head Analysis
Department of Computer Science, Birla Institute of Technology and Science, Pilani, India · School of Computer Engineering, Kalinga Institute of Industrial Technology, Bhubaneswar, India · Birla AI Labs, Office of Ananya Birla, Aditya Birla Group, India
Ablation-Reversible Heads Don't Transfer: A Stress Test for Mechanistic Role Claims in Transformers
Jun 6, 2026Philip QuirkeAttention Head Analysis Mechanistic Interpretability
Martian
A Unifying View of Attention Sinks: From Mechanisms to Architectural Interventions
Jun 6, 2026Lukas Fesser*, Mozes Jacobs*, Thomas Fel* +2Transformer Interpretability Vision Transformer
Kempner Institute Harvard University Cambridge, MA
AdaRoPE: Not All Attention Heads Should Rotate and Scale Equally
Jun 5, 2026Shaowen Wang, Yuke Zheng, Tansheng Zhu +4Rotary Positional Embeddings Self-Attention
Tsinghua University · Hunyuan Team, Tencent · Xiongan AI Institute +1
Mechanistic Insights into Functional Sparsity in Multimodal LLMs via CoRe Heads
Jun 4, 2026Ruoxi Sun, Quantong Qiu, Juntao Li +3Multimodal Large Language Models VLM Interpretability
Soochow University · Peking University
Detection vs. Execution: Single-Bucket Probes Miss Half the Mamba-2 State Sink
May 30, 2026Yuhang JiangMamba Attention Head Analysis
Independent Researcher
Positional versus Symbolic Attention Heads: Learning Dynamics, RoPE Geometry, and Length Generalization
May 29, 2026Felipe Urrutia, Juan José Alegría, Cinthia Sanchez Macias +3Rotary Positional Embeddings Attention Head Analysis
CENIA & Faculty of Mathematics UC Santiago, Chile · CENIA Santiago, Chile · IMC UC & CENIA Santiago, Chile
Function-Vector Heads Are Two Populations: Writers and Cancellers in In-Context Learning
May 25, 2026Han-yu WangTransformer Interpretability Task Vectors
The University of Hong Kong
Spectral Probe-Circuits: A Three-Step Recipe for Identifying Attention-Head Circuits in Pretrained Transformers
May 22, 2026Yongzhong XuTransformer Interpretability Circuit Discovery
From Parameters to Data: A Task-Parameter-Guided Fine-Tuning Pipeline for Efficient LLM Alignment
May 20, 2026Hao Chen, Qi Zhang, Liyao Li +7LLM Alignment LLM Fine-Tuning
Zhejiang University · Ant Group · Eastern Institute of Technology
Multi-Head Attention as Ensemble Nadaraya-Watson Estimation: Variance Reduction, Decorrelation, and Optimal Head Diversity
May 18, 2026Ernest FokouéSelf-Attention Ensemble Learning
School of Mathematics and Statistics, College of Science Rochester Institute of Technology Rochester, New York 14623, USA
Head Forcing: Long Autoregressive Video Generation via Head Heterogeneity
May 14, 2026Jiahao Tian, Yiwei Wang, Gang Yu +1Diffusion Transformer Autoregressive Diffusion
AGI Lab, Westlake University · University of California at Merced · StepFun
Pyramid Forcing: Head-Aware Pyramid KV Cache Policy for High-Quality Long Video Generation
May 13, 2026Jiayu Chen, Junbei Tang, Wenbiao Zhao +6Self-Attention Long-Horizon Video Generation
Peking University · South China University of Technology · Xinjiang University +2
GuidedVLA: Specifying Task-Relevant Factors via Plug-and-Play Action Attention Specialization
May 12, 2026Xiaosong Jia, Bowen Yang, Zuhao Ge +17VLM Robustness Robotic Manipulation
Institute of Trustworthy Embodied AI (TEAI), Fudan University · 2Shanghai Key Laboratory of Multimodal Embodied AI · Shanghai Jiao Tong University +1
Learning Less Is More: Premature Upper-Layer Attention Specialization Hurts Language Model Pretraining
May 11, 2026Jinchang Zhu, Jindong Li, Yuwen Hao +3Language Model Pretraining Self-Attention
The Hong Kong University of Science and Technology (Guangzhou)

Oct 13, 2026J/K move · Enter open · S save

Do Syntax Heads Influence Model Behavior?

William Timkey, Tal Linzen

New York University

Transformer Interpretability Attention Head Specialization Language Modeling Causal Interventions in Language Models

Open paper AskPDF arXiv

Transformer language models develop attention heads that specialize in syntactic relations, for example, attending from verbs to their subjects. Do these heads play a causal role in the model's syntactic predictions? Evidence for such a role is mixed to negative: even when we force these heads to attend to a distractor noun other than the subject, the model's syntactic predictions are still more consistent with the subject than with the distractor. Interventions on specialized syntax heads had a significantly stronger effect on syntactic predictions than interventions on random heads, but only in one of the two measures we propose.