Attention Sinks

Follow this topic Ask about this topic

Momentum

6 papers in the last four weeks, up 50% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 42

All topics

A Single Layer to Explain Them All:Understanding Massive Activations in Large Language Models
May 8, 2026Zeru Shi, Zhenting Wang, Fan Yang +2LLM Interpretability Attention Sinks
Rutgers University · Wake Forest University · Meta AI
Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing prevention
May 8, 2026Peter Súkeník, Cristina López Amado, Christoph H. Lampert +1Self-Attention Transformer Attention
Institute of Science and Technology (ISTA) Austria
The Structural Origin of Attention Sink: Variance Discrepancy, Super Neurons, and Dimension Disparity
May 7, 2026Siquan Li, Kaiqi Jiang, Jiacheng Sun +1Self-Attention Attention Sinks
1The Chinese University of Hong Kong, Shenzhen · 2Huawei
Attention Sinks in Massively Multilingual Neural Machine Translation:Discovery, Analysis, and Mitigation
May 2, 2026Hillary Mutisya, John MuganeMultilingual Language Models Cross-Attention
Thiomi NLP · Harvard University
LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models
Apr 27, 2026Rinyoichi Takezoe, Yaqian Li, Zihao Bo +3Efficient VLM Inference Attention Sinks
Li Auto Inc.
Sink-Token-Aware Pruning for Fine-Grained Video Understanding in Efficient Video LLMs
Apr 22, 2026Kibum Kim, Jiwan Kim, Kyle Min +4Efficient VLM Inference Attention Sinks
Korea Advanced Institute of Science and Technology (KAIST) · Oracle · University of California, San Diego +1
SinkRouter: Sink-Aware Routing for Efficient Long-Context Decoding in Large Language and Multimodal Models
Apr 18, 2026Junnan Liu, Xinyan Liu, Peifeng Gao +4GPU Acceleration Self-Attention
Harbin Institute of Technology (Weihai) · University of Hong Kong, Hong Kong, China · City University of Hong Kong, Hong Kong, China
A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigation
Apr 16, 2026Yuval Ran-Milo, Hila Ofek, Shahar MendelSelf-Attention Decoder-Only Language Models
Tel Aviv University
When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models
Apr 1, 2026Jiho Choi, Jaemin Kim, Sanghwan Kim +2Large Vision-Language Models VLM Interpretability
KAIST, South Korea · Chung-Ang University, South Korea · Technical University of Munich, Germany
Threshold Differential Attention for Sink-Free, Ultra-Sparse, and Non-Dispersive Language Modeling
Jan 17, 2026Xingyue Huang, Xueying Ding, Mingxuan Ju +3Self-Attention Long-Context Language Modeling
University of Oxford · Carnegie Mellon University · Snap Inc.
Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruning
Jan 11, 2026Jaewon Sok, Jewon Yeom, Seonghyeon Park +2LLM Pruning Attention Head Analysis
Department of Rural Systems Engineering, Seoul National University · Graduate School of Data Science, Seoul National University · Department of Aerospace Engineering, Seoul National University
RefAM: Attention Magnets for Zero-Shot Referral Segmentation
Sep 26, 2025Anna Kukleva, Enis Simsar, Alessio Tonioni +4Zero-Shot Learning Image Segmentation
Max Planck Institute for Informatics, SIC · ETH Zürich · Google +1

May 8, 2026, cs.CLJ/K move · Enter open · S save

A Single Layer to Explain Them All:Understanding Massive Activations in Large Language Models

Zeru Shi, Zhenting Wang, Fan Yang, Qifan Wang+1

Rutgers University · Wake Forest University · Meta AI

LLM Interpretability Attention Sinks Attention Mechanisms

Open paper AskPDF arXiv

We investigate the origins of massive activations in large language models (LLMs) and identify a specific layer named the \textbf{Massive Emergence Layer (ME Layer)}, that is consistently observed across model families, where massive activations first emerge and subsequently propagate to deeper layers through residual connections. We show that, within the ME Layer both the RMSNorm and the FFN parameters jointly contribute to the emergence of massive activations. Once formed, the massive activation token representation remains largely invariant across layers, reducing the diversity of hidden representations passed to the attention module. Motivated by this limitation, we propose a simple and effective method to reduce the rigidity of the massive activation token. Our approach consistently improves LLM performance across multiple tasks, including instruction following and math reasoning, in both training free and fine tuning settings. Moreover, we show that our method mitigates attention sinks by selectively weakening their influence, elucidating their origin at the hidden state level and shedding new light on principled mitigation strategies.

Momentum

Related topics

Latest papers 42