Attention Head Specialization

Follow this topic Ask about this topic

Momentum

4 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 30

All topics

Which Attention Heads are like the Human Head? Not the Ones that Compute
Sep 29, 2026Christopher Pinier, Gustaw Opiełka, Hannes Rosenbusch +3Attention Head Analysis Cross-Modal Alignment
Psychological Methods, University of Amsterdam, Amsterdam, The Netherlands · Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA
When Can Attention Heads Be Statically Defined?
Sep 28, 2026Weixian Waylon Li, Yintao Tai, Marcio Fonseca +1Attention Head Analysis Efficient Attention
University of Edinburgh, United Kingdom · Chamber of Deputies, Brazil
Rethinking Contextualization by Reinterpreting Attention Head Channels
Sep 27, 2026Hakaze Cho, Haolin Yang, Zhun Sun +3Attention Head Analysis Representation Geometry in Language Models
RIKEN · Tohoku University · New York University +2
Temporal-Attention Head Specialization During Video Diffusion Training
Sep 14, 2026Taewoo Ha, Shafayat Mowla Anik, Dae Yeol Lee +2Video Diffusion Models Attention Head Analysis
Independent Researcher · University of Colorado - Colorado Springs · Dolby Laboratories +1
Modern Transformers Are Implicit Hybrids: From Functional Differentiation to Principled Hybrid Architecture Design
Sep 2, 2026Runlin Shi, Bojian Yin, Guoqi LiRotary Positional Embeddings Linear Attention
Institute of Automation, Chinese Academy of Sciences, Beijing, China
Autonomy-of-Heads: Data-Free Sparse Attention from Frozen Query-Key Geometry
Aug 7, 2026Yehan Yang, Junyuan Shang, Yang Li +3Attention Head Analysis LLM Inference Acceleration
Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Baidu Inc. +1
Interpretability-Guided Soft Pruning of Attention Heads in Vision Transformers
Jul 31, 2026Kamil Książek, Piotr Suszyński, Michał Jan Włodarczyk +2Transformer Interpretability Attention Head Analysis
Centre for Credible Artificial Intelligence Warsaw University of Technology Warsaw, Poland · Faculty of Mathematics and Computer Science Jagiellonian University Krakow, Poland · University of Warsaw Warsaw, Poland
HeadCast: Casting Attention Heads for Efficient Autoregressive Video Generation
Jul 22, 2026Jinliang Shen, Lianghao Su, Zheming Li +4KV-Cache Management Autoregressive Video Generation
1Peking University · 2KlingAI Research
HydraHead: From Head-Level Functional Heterogeneity to Specialized Attention Hybridization
Jun 18, 2026Zhentao Tan, Wei Chen, Jingyi Shen +4Softmax Attention Long-Context Modeling
Alibaba Group
Where Computation Lives Inside TabPFN: Causal Localisation of Attention Head Function
Jun 11, 2026Atharva Gupta, Dhruv Kumar, Murari Mandal +1Tabular Foundation Models Attention Head Analysis
Department of Computer Science, Birla Institute of Technology and Science, Pilani, India · School of Computer Engineering, Kalinga Institute of Industrial Technology, Bhubaneswar, India · Birla AI Labs, Office of Ananya Birla, Aditya Birla Group, India
Ablation-Reversible Heads Don't Transfer: A Stress Test for Mechanistic Role Claims in Transformers
Jun 6, 2026Philip QuirkeAttention Head Analysis Mechanistic Interpretability
Martian
A Unifying View of Attention Sinks: From Mechanisms to Architectural Interventions
Jun 6, 2026Lukas Fesser*, Mozes Jacobs*, Thomas Fel* +2Transformer Interpretability Vision Transformer
Kempner Institute Harvard University Cambridge, MA
Mechanistic Insights into Functional Sparsity in Multimodal LLMs via CoRe Heads
Jun 4, 2026Ruoxi Sun, Quantong Qiu, Juntao Li +3Multimodal Large Language Models VLM Interpretability
Soochow University · Peking University
Detection vs. Execution: Single-Bucket Probes Miss Half the Mamba-2 State Sink
May 30, 2026Yuhang JiangMamba Attention Head Analysis
Independent Researcher
Positional versus Symbolic Attention Heads: Learning Dynamics, RoPE Geometry, and Length Generalization
May 29, 2026Felipe Urrutia, Juan José Alegría, Cinthia Sanchez Macias +3Rotary Positional Embeddings Attention Head Analysis
CENIA & Faculty of Mathematics UC Santiago, Chile · CENIA Santiago, Chile · IMC UC & CENIA Santiago, Chile
Function-Vector Heads Are Two Populations: Writers and Cancellers in In-Context Learning
May 25, 2026Han-yu WangTransformer Interpretability Task Vectors
The University of Hong Kong
Spectral Probe-Circuits: A Three-Step Recipe for Identifying Attention-Head Circuits in Pretrained Transformers
May 22, 2026Yongzhong XuTransformer Interpretability Circuit Discovery
From Parameters to Data: A Task-Parameter-Guided Fine-Tuning Pipeline for Efficient LLM Alignment
May 20, 2026Hao Chen, Qi Zhang, Liyao Li +7LLM Alignment LLM Fine-Tuning
Zhejiang University · Ant Group · Eastern Institute of Technology
Multi-Head Attention as Ensemble Nadaraya-Watson Estimation: Variance Reduction, Decorrelation, and Optimal Head Diversity
May 18, 2026Ernest FokouéSelf-Attention Ensemble Learning
School of Mathematics and Statistics, College of Science Rochester Institute of Technology Rochester, New York 14623, USA
Head Forcing: Long Autoregressive Video Generation via Head Heterogeneity
May 14, 2026Jiahao Tian, Yiwei Wang, Gang Yu +1Diffusion Transformer Autoregressive Diffusion
AGI Lab, Westlake University · University of California at Merced · StepFun
Pyramid Forcing: Head-Aware Pyramid KV Cache Policy for High-Quality Long Video Generation
May 13, 2026Jiayu Chen, Junbei Tang, Wenbiao Zhao +6Self-Attention Long-Horizon Video Generation
Peking University · South China University of Technology · Xinjiang University +2
GuidedVLA: Specifying Task-Relevant Factors via Plug-and-Play Action Attention Specialization
May 12, 2026Xiaosong Jia, Bowen Yang, Zuhao Ge +17VLM Robustness Robotic Manipulation
Institute of Trustworthy Embodied AI (TEAI), Fudan University · 2Shanghai Key Laboratory of Multimodal Embodied AI · Shanghai Jiao Tong University +1
Learning Less Is More: Premature Upper-Layer Attention Specialization Hurts Language Model Pretraining
May 11, 2026Jinchang Zhu, Jindong Li, Yuwen Hao +3Language Model Pretraining Self-Attention
The Hong Kong University of Science and Technology (Guangzhou)
HeadRouter: Dynamic Head-Weight Routing for Task-Adaptive Audio Token Pruning in Large Audio Language Models
Apr 26, 2026Peize He, Yaodi Luo, Xiaoqian Liu +7Audio Token Compression Self-Attention
1EPIC Lab, Shanghai Jiao Tong University · 2DAIL Tech · 3Northeastern University +2
What Is the Minimum Architecture for Prolepsis? Early Irrevocable Commitment Across Tasks in Small Transformers
Apr 16, 2026Éric JacopinTransformer Interpretability Attention Head Analysis
Cosmic AI
Bridging Latent Reasoning and Target-Language Generation via Retrieval-Transition Heads
Feb 25, 2026Shaswat Patel, Vishvesh Trivedi, Yue Han +2Multilingual Language Models Cross-Lingual Reasoning in Language Models
New York University
Incremental Learning of Sparse Attention Patterns in Transformers
Feb 22, 2026Oğuz Kaan Yüksel, Rodrigo Alvarez Lucendo, Nicolas FlammarionSelf-Attention Transformer Attention
TML Lab, EPFL
A Mechanistic Study of Transformers Training Dynamics
Oct 31, 2024Ambroise Odonnat, Wassim Bouaziz, Vivien CabannesTransformer Interpretability Attention Mechanisms
Inria, Univ. Rennes 2 · Mistral AI; work done while at Meta. · FAIR at Meta.
AdaRoPE: Not All Attention Heads Should Rotate and Scale Equally
Date pendingShaowen Wang, Yuke Zheng, Tansheng Zhu +4Rotary Positional Embeddings Self-Attention
Tsinghua University · Hunyuan Team, Tencent · Xiongan AI Institute +1
Diffract: Spectral View of LLM Domain Adaptation
Date pendingNikita Borodin, Maria Krylova, Artem Zabolotnyi +6Domain Adaptation Domain-Adaptive Pretraining
Risk AI Research Lab · Applied AI Institute

Sep 29, 2026, cs.AIJ/K move · Enter open · S save

Which Attention Heads are like the Human Head? Not the Ones that Compute

Christopher Pinier, Gustaw Opiełka, Hannes Rosenbusch, Taylor Webb+2

Psychological Methods, University of Amsterdam, Amsterdam, The Netherlands · Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA

Attention Head Analysis Cross-Modal Alignment Attention Head Specialization

Open paper AskPDF arXiv

Brain-AI alignment is often interpreted as a sign that model and brain perform similar computations. Whether the aligned units are causally involved in model computation is rarely checked. On an abstract pattern-completion task (AAABAAA

\rightarrow

B), we compare LLM attention-head representations with human EEG and test how ablating those heads affects task performance. Alignment and causation dissociate: brain-aligned heads contribute to performance, but their removal is substantially less disruptive than removal of heads selected via attribution patching. We compare two head sets that prior interpretability work defines without reference to the brain: concept vectors (CVs), which represent abstract patterns across formats, and function vectors (FVs), selected for their contribution to correct-answer prediction. Brain alignment shows little association with FV scores, while its association with CV scores varies across models. Among brain-aligned heads, we find recurring attention profiles: one emphasizes distinctive elements (novelty heads), the other repeating elements (repetition heads). The novelty family tracks salience and attends to the same elements that humans look at, yet its removal is less damaging than random ablation on average. Repetition heads contribute modestly to performance and are associated with abstract-pattern representation (CVs). Across 17 models spanning 3B-72B parameters, FV-ranked removal is substantially more disruptive than brain-ranked removal. Brain alignment thus captures how the model reads the stimulus, and only faintly captures how it represents the pattern and solves the task.

Momentum

Related topics

Latest papers 30