Attention Sinks

Follow this topic Ask about this topic

Attention sinks are tokens—often the beginning-of-sequence token—that receive large attention weights even when they carry little task-relevant content. Researchers study how softmax normalization, causal masking, model architecture, and training create these patterns, using mechanistic analysis and causal interventions to distinguish their roles. Methods either suppress or redistribute sink attention for grounding, pruning, and long-context tasks, or use sinks as anchors for efficient computation; evaluations include language and retrieval benchmarks as well as vision-language and video tasks. The main tension is that sinks can dilute useful evidence and encourage shortcuts, yet may also stabilize representations or support default behavior, so their value depends on the model, layer, and task.

Latest papers

55 papers5 papers in the last four weeks, down 29% on the four weeks before. 0.0% of all new papers.

Why don't Graph Transformers attend to the same token?
Oct 14, 2026Alexandre BismuthAttention Sinks Attention Mechanisms
BOLD Lab University of Oxford
Before Attention Sinks Form: Hidden-State Geometry Reveals and Modulates Their Formation
Oct 12, 2026Fan Yang, Yue SongAttention Sinks
College of AI, Tsinghua University
Exploring Attention Sinks in Text-to-Image Transformers from Diffusion and Autoregressive Models
Oct 9, 2026Binglei Li, Mengping Yang, Zhiyu Tan +3Attention Sinks Transformer Interpretability
Fudan University · Shanghai Innovation Institute · Shanghai Academy of AI for Science
Spectral Geometry of Attention: From Information Routing to Uncertainty
Oct 5, 2026Giulio Viganò, Simone Melzi, Maks OvsjanikovAttention Head Analysis Attention Sinks
University of Milano-Bicocca Milan, Italy · Ecole Polytechnique · LIX, Ecole Polytechnique IP Paris
The Life Cycle of a Massive Activation: Stochastic Birth, Weight-Decay-Driven Growth, and Competitive Consolidation
Sep 30, 2026S. Aaron McClendon, Jorge Gallego-Feliciano, Antonios SaravanosWeight Decay Attention Sinks
Aimpoint Digital Labs, Atlanta, GA, USA · New York University, New York, NY, USA
Global Ranks Survive, Selected Heads Shift: BOS-Sink Topology under 4-bit Weight-Only Quantization
Sep 20, 2026Kuanlin Chen, Chen-Wei Kuo, Cheng-En OuAttention Head Analysis 4-Bit Quantization
Independent Researcher Taoyuan, Taiwan · National Tsing Hua University Hsinchu, Taiwan
ValueDiff: Value-Geometric KV Cache Eviction for Sink-Suppressed LLMs
Sep 20, 2026Junyoung Park, Jungwook Choi, Mingu LeeLLM Inference Efficiency KV-Cache Eviction
Qualcomm AI Research · Hanyang University
It's Not RoPE that Creates Sinks: The Role of Self-Concentration and Value-Non-Mixing in Attention
Sep 8, 2026Raito Kiya, Satoki Ohashi, Kosuke Sato +6Self-Attention Attention Sinks
1Tohoku University · 2SOKENDAI · 3NINJAL +3
Do New Attention Mechanisms Actually Fix Attention Sinks at Million-Token Context?
Sep 8, 2026Sara Rizwan, Samaanah Abdus Salam, Mohammed Mudassir UddinLong-Context Language Model Evaluation Long-Context Language Modeling
Shadan Women’s College of Engineering and Technology Hyderabad, Telangana, India
Separating Stream Stability from Long-Term Recall in Language Models
Sep 7, 2026Peipei Cao, Xin Zhang, Jie Tang +3Memory-Augmented Language Models Long-Context Language Model Evaluation
School of Computer and Electronic Information, Guangxi University · School of Information Science and Engineering, Chongqing Jiaotong University · School of Computer Science and Technology, Guangdong University of Technology
SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Models
Sep 1, 2026Shiyu Li, Zi-Yuan Hu, Shijia Huang +3Efficient VLM Inference Multimodal Large Language Models
1The Chinese University of Hong Kong · 2Weitu AI · 3Peking University
Text Capability Loss in Vision-Language Adaptation: An Attention-Sink Diagnosis
Sep 1, 2026Minsik Choi, Geewook Kim, Young Geun KimVision-Language Models VLM Adaptation
Korea University · NAVER Cloud AI
Sliding-window beats linear attention
Aug 28, 2026Alexia Jolicoeur-Martineau, Rhea Sanjay Sukthanker, Pashmina Cameron +1Linear Attention KV Caching
Microsoft Applied Sciences Group (ASG) · Independent
Hybrid Gated Attention
Aug 12, 2026Zekun Zhou, Ruobing Xie, Lanrui Wang +1Low-Rank Matrix Decomposition Gated Attention
Tencent Hunyuan · Peking University
Which Question Is Your Attention Metric Answering? Attention Rows as Compositional Data
Aug 11, 2026Marios Papamichalis, Regina RuaneAttention Head Analysis Transformer Interpretability
Human Nature Lab, Yale University, New Haven, CT 06511 · Department of Statistics and Data Science, The Wharton School, University of Pennsylvania, 3733 Spruce Street, Philadelphia, PA 19104-6340
Disentangling Semantic Attention from Structural Bias in the Attention Manifold
Jul 27, 2026Pengkun Jiao, Bin Zhu, Jingjing Chen +1Object Hallucination in VLMs Multimodal Large Language Models
Fudan University · Singapore Management University
Text Template Tokens Are Implicit Semantic Registers in Diffusion Transformers
Jul 21, 2026Maohua Li, Qirui Li, Yanke Zhou +10Transformer Interpretability Diffusion Transformer
Nanjing University · Alibaba Group · Zhejiang University
VisReflect: Latent Visual Reflection for Fine-Grained Perception in Long Visual Context
Jun 29, 2026Xiaoqian Shen, Mohamed ElhoseinyEfficient VLM Inference Attention Sinks
King Abdullah University of Science and Technology, Saudi Arabia
All Routes Lead to Collapse
Jun 21, 2026K. R. BalasubramanianRepresentation Collapse Attention Sinks
Independent Research
A Unifying View of Attention Sinks: From Mechanisms to Architectural Interventions
Jun 6, 2026Lukas Fesser*, Mozes Jacobs*, Thomas Fel* +2Transformer Interpretability Vision Transformer
Kempner Institute Harvard University Cambridge, MA
Where does Absolute Position come from in decoder-only Transformers?
Jun 4, 2026Valeria Ruscio, Umberto Nanni, Fabrizio SilvestriRotary Positional Embeddings Transformer Interpretability
Intuition Machines · Sapienza University of Rome
P-Cast Precision in FP8 Attention: Sink-Induced Collapse and the Optimality of S=2^8
Jun 2, 2026Reed LauSoftmax Attention Attention Sinks
Tencent
When Graph Tokens Sink: A Mechanistic Analysis of Graph Language Models
Jun 2, 2026Ding Zhang, Runtao Zhou, Wenqing Zheng +3LLM Interpretability Mechanistic Interpretability
University of Virginia · Capital One
When Do Attention Circuits Form? Developmental Trajectories of Capability and Attention-Sink Emergence Across Three 1B-ClassArchitectures
Jun 1, 2026Yongzhong XuLanguage Model Pretraining Transformer Interpretability
Massive Spikes in LLMs are Bias Vectors: Mechanistic Uncovering and Spike-Free Quantization
Jun 1, 2026Yung-Chin Chen, Chung Peng Lee, Ze-Wei Liou +1LLM Quantization Activation Outliers
Princeton University, NJ, USA · EnCharge AI, CA, USA
Detection vs. Execution: Single-Bucket Probes Miss Half the Mamba-2 State Sink
May 30, 2026Yuhang JiangMamba Attention Head Analysis
Independent Researcher
Contribution Weights: A Geometrical Analysis of Self-Attention Transformers
May 29, 2026Harry Jake Cunningham, Nicola Muca CironeTransformer Interpretability Feature Attribution
Department of Computer Science, University College London, London, UK. · Cartesia.AI, San Francisco, US.
Addressing Exacerbated Attention Sink for Source-Free Cross-Domain Few-Shot Learning
May 25, 2026Shuai Yi, Yixiong Zou, Yuhua Li +1Source-Free Cross-Domain Few-Shot Learning Source-Free Domain Adaptation
Huazhong University of Science and Technology
ASAP: Attention Sink Anchored Pruning
May 21, 2026Jaehyuk Lee, Hanyoung Kim, Yanggee Kim +1Self-Attention Efficient ViTs
Department of Mathematics Korea University Seoul, Republic of Korea · Korea University Seoul, Republic of Korea
DySink: Dynamic Frame Sinks for Autoregressive Long Video Generation
May 20, 2026Bo Ye, Xinyu Cui, Jian Zhao +2Long-Horizon Video Generation Memory-Augmented Video Generation
School of Computer Science and Engineering, Southeast University · Key Lab. of Computer Network and Information Integration, Southeast University · Southeast University +3

Oct 14, 2026J/K move · Enter open · S save

Why don't Graph Transformers attend to the same token?

Alexandre Bismuth

BOLD Lab University of Oxford

Attention Sinks Attention Mechanisms Graph Transformer Gated Attention

Open paper AskPDF arXiv

In large language models (LLMs), attention concentrates heavily on the first token. This 'attention sink' is believed to protect deep models against over-mixing. Even though graph transformers face a related pressure, we find that a SoTA graph transformer, GRIT, rarely forms sinks on molecules, and that it sinks less when trained on larger graphs, the opposite of the trend in LLMs. In principle, a graph can prevent sinks: we prove and show empirically that its symmetry and locality limit the attention any node can receive. On molecules, however, with distinguishable atoms and global attention, neither limit prevents sinks; what stops them lies in the model. We find that the attention scoring function is the main cause. Replacing GRIT's gated logit with a dot product raises its sink rate from 0.09 to 0.68 on identical graphs. In Graph Transformers using dot-product attention, removing the sink gives about the same error as removing a random node, while GRIT's most-attended nodes matter to its predictions. Attention sinks are therefore a property of dot-product attention rather than of Transformers. In our experiments, they form when dot-product scoring has a distinguishable node to concentrate on.