Attention Head Specialization

Overview

Attention head specialization asks how individual heads divide work, such as retrieving context, tracking positions, handling modalities, or supporting particular computations. Researchers study these roles through mechanistic probes, activation patching and ablation, and analyses of how specialization emerges during training; some methods also encourage roles explicitly. Head-aware pruning, KV-cache policies, and hybrid attention use these differences to reduce computation in settings such as long-context retrieval, multimodal reasoning, and video generation. A central challenge is distinguishing a head that appears specialized from one that is causally necessary, since roles can depend on the task, architecture, and intervention used.

Latest papers

CardsList

43 papers3 papers in the last four weeks, down 50% on the four weeks before. 0.0% of all new papers.

  1. Axis-LoRA: Probing and Adapting Embedding Models for Personal-Memory Retrieval

    Nov 5, 2026Dohoon Kim, Junho Park, Hyunwoong Bae +2Text Embedding ModelsKnowledge Retrieval in Language Models

  2. Understanding Unified VLMs: From Attention Probing to Guidance Tuning for Multimodal Generation

    Oct 10, 2026Zhengyang Luo, Xiaoyu Qiu, Zhiheng Tan +3Unified Multimodal ModelsMultimodal Generation

  3. Which Attention Heads are like the Human Head? Not the Ones that Compute

    Sep 29, 2026Christopher Pinier, Gustaw Opiełka, Hannes Rosenbusch +3Attention Head AnalysisCross-Modal Alignment

  4. How Language Models Differ in Redistributing Attention-Head Activity Under Serial Demand

    Sep 28, 2026Johnny Jingze Li, Abdulla Kuleib, Kalyan Basu +1Attention Head AnalysisLLM Interpretability

  5. When Can Attention Heads Be Statically Defined?

    Sep 28, 2026Weixian Waylon Li, Yintao Tai, Marcio Fonseca +1Attention Head AnalysisEfficient Attention

  6. Rethinking Contextualization by Reinterpreting Attention Head Channels

    Sep 27, 2026Hakaze Cho, Haolin Yang, Zhun Sun +3Attention Head AnalysisRepresentation Geometry in Language Models

  7. Temporal-Attention Head Specialization During Video Diffusion Training

    Sep 14, 2026Taewoo Ha, Shafayat Mowla Anik, Dae Yeol Lee +2Video Diffusion ModelsAttention Head Analysis

  8. Modern Transformers Are Implicit Hybrids: From Functional Differentiation to Principled Hybrid Architecture Design

    Sep 2, 2026Runlin Shi, Bojian Yin, Guoqi LiRotary Positional EmbeddingsLinear Attention

  9. Semantic Head Specialization Guides Hybrid ViT Attention for Multimodal LLMs

    Aug 28, 2026Chenhong He, Lei Li, Shicheng Li +5Vision TransformerHybrid Attention

  10. Not All Attention Heads Contribute to Critical Visual Token Selection: Head-Aware Pruning Matters More

    Aug 26, 2026Chaofang Ma, Lin Jiang, Carol Jingyi Li +4Efficient VLM InferenceVisual Token Pruning

  11. Diffract: Spectral View of LLM Domain Adaptation

    Aug 11, 2026Nikita Borodin, Maria Krylova, Artem Zabolotnyi +6Domain AdaptationDomain-Adaptive Pretraining

  12. Autonomy-of-Heads: Data-Free Sparse Attention from Frozen Query-Key Geometry

    Aug 7, 2026Yehan Yang, Junyuan Shang, Yang Li +3Attention Head AnalysisLLM Inference Acceleration

  13. Interpretability-Guided Soft Pruning of Attention Heads in Vision Transformers

    Jul 31, 2026Kamil Książek, Piotr Suszyński, Michał Jan Włodarczyk +2Transformer InterpretabilityAttention Head Analysis

  14. HeadCast: Casting Attention Heads for Efficient Autoregressive Video Generation

    Jul 22, 2026Jinliang Shen, Lianghao Su, Zheming Li +4KV-Cache ManagementAutoregressive Video Generation

  15. Where Computation Lives Inside TabPFN: Causal Localisation of Attention Head Function

    Jun 11, 2026Atharva Gupta, Dhruv Kumar, Murari Mandal +1Attention Head SpecializationAttention Head Analysis

  16. A Unifying View of Attention Sinks: From Mechanisms to Architectural Interventions

    Jun 6, 2026Lukas Fesser*, Mozes Jacobs*, Thomas Fel* +2Transformer InterpretabilityVision Transformer

  17. AdaRoPE: Not All Attention Heads Should Rotate and Scale Equally

    Jun 5, 2026Shaowen Wang, Yuke Zheng, Tansheng Zhu +4Rotary Positional EmbeddingsSelf-Attention

  18. Mechanistic Insights into Functional Sparsity in Multimodal LLMs via CoRe Heads

    Jun 4, 2026Ruoxi Sun, Quantong Qiu, Juntao Li +3Multimodal Large Language ModelsVLM Interpretability

  19. Positional versus Symbolic Attention Heads: Learning Dynamics, RoPE Geometry, and Length Generalization

    May 29, 2026Felipe Urrutia, Juan José Alegría, Cinthia Sanchez Macias +3Rotary Positional EmbeddingsAttention Head Analysis

  20. From Parameters to Data: A Task-Parameter-Guided Fine-Tuning Pipeline for Efficient LLM Alignment

    May 20, 2026Hao Chen, Qi Zhang, Liyao Li +7LLM AlignmentLLM Fine-Tuning

  21. Multi-Head Attention as Ensemble Nadaraya-Watson Estimation: Variance Reduction, Decorrelation, and Optimal Head Diversity

    May 18, 2026Ernest FokouéSelf-AttentionEnsemble Learning

  22. Head Forcing: Long Autoregressive Video Generation via Head Heterogeneity

    May 14, 2026Jiahao Tian, Yiwei Wang, Gang Yu +1Diffusion TransformerAutoregressive Diffusion