Attention Sinks

Follow this topic Ask about this topic

Momentum

6 papers in the last four weeks, up 50% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 42

All topics

Spectral Geometry of Attention: From Information Routing to Uncertainty
Oct 5, 2026Giulio Viganò, Simone Melzi, Maks OvsjanikovAttention Head Analysis Attention Sinks
University of Milano-Bicocca Milan, Italy · Ecole Polytechnique · LIX, Ecole Polytechnique IP Paris
The Life Cycle of a Massive Activation: Stochastic Birth, Weight-Decay-Driven Growth, and Competitive Consolidation
Sep 30, 2026S. Aaron McClendon, Jorge Gallego-Feliciano, Antonios SaravanosWeight Decay Attention Sinks
Aimpoint Digital Labs, Atlanta, GA, USA · New York University, New York, NY, USA
Global Ranks Survive, Selected Heads Shift: BOS-Sink Topology under 4-bit Weight-Only Quantization
Sep 20, 2026Kuanlin Chen, Chen-Wei Kuo, Cheng-En OuAttention Head Analysis 4-Bit Quantization
Independent Researcher Taoyuan, Taiwan · National Tsing Hua University Hsinchu, Taiwan
ValueDiff: Value-Geometric KV Cache Eviction for Sink-Suppressed LLMs
Sep 20, 2026Junyoung Park, Jungwook Choi, Mingu LeeLLM Inference Efficiency KV-Cache Eviction
Qualcomm AI Research · Hanyang University
It's Not RoPE that Creates Sinks: The Role of Self-Concentration and Value-Non-Mixing in Attention
Sep 8, 2026Raito Kiya, Satoki Ohashi, Kosuke Sato +6Self-Attention Attention Sinks
1Tohoku University · 2SOKENDAI · 3NINJAL +3
Do New Attention Mechanisms Actually Fix Attention Sinks at Million-Token Context?
Sep 8, 2026Sara Rizwan, Samaanah Abdus SalamLong-Context Language Model Evaluation Long-Context Language Modeling
Shadan Women’s College of Engineering and Technology Hyderabad, Telangana, India
Separating Stream Stability from Long-Term Recall in Language Models
Sep 7, 2026Peipei Cao, Xin Zhang, Jie Tang +3Memory-Augmented Language Models Long-Context Language Model Evaluation
School of Computer and Electronic Information, Guangxi University · School of Information Science and Engineering, Chongqing Jiaotong University · School of Computer Science and Technology, Guangdong University of Technology
SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Models
Sep 1, 2026Shiyu Li, Zi-Yuan Hu, Shijia Huang +3Efficient VLM Inference Multimodal Large Language Models
1The Chinese University of Hong Kong · 2Weitu AI · 3Peking University
Text Capability Loss in Vision-Language Adaptation: An Attention-Sink Diagnosis
Sep 1, 2026Minsik Choi, Geewook Kim, Young Geun KimVision-Language Models VLM Adaptation
Korea University · NAVER Cloud AI
Sliding-window beats linear attention
Aug 28, 2026Alexia Jolicoeur-Martineau, Rhea Sanjay Sukthanker, Pashmina Cameron +1KV Caching LLM Inference Acceleration
Microsoft Applied Sciences Group (ASG) · Independent
Hybrid Gated Attention
Aug 12, 2026Zekun Zhou, Ruobing Xie, Lanrui Wang +1Low-Rank Matrix Decomposition Gated Attention
Tencent Hunyuan · Peking University
Disentangling Semantic Attention from Structural Bias in the Attention Manifold
Jul 27, 2026Pengkun Jiao, Bin Zhu, Jingjing Chen +1Object Hallucination in VLMs Multimodal Large Language Models
Fudan University · Singapore Management University
Text Template Tokens Are Implicit Semantic Registers in Diffusion Transformers
Jul 21, 2026Maohua Li, Qirui Li, Yanke Zhou +10Transformer Interpretability Diffusion Transformer
Nanjing University · Alibaba Group · Zhejiang University
VisReflect: Latent Visual Reflection for Fine-Grained Perception in Long Visual Context
Jun 29, 2026Xiaoqian Shen, Mohamed ElhoseinyEfficient VLM Inference Attention Sinks
King Abdullah University of Science and Technology, Saudi Arabia
All Routes Lead to Collapse
Jun 21, 2026K. R. BalasubramanianRepresentation Collapse Attention Sinks
Independent Research
A Unifying View of Attention Sinks: From Mechanisms to Architectural Interventions
Jun 6, 2026Lukas Fesser*, Mozes Jacobs*, Thomas Fel* +2Transformer Interpretability Vision Transformer
Kempner Institute Harvard University Cambridge, MA
Where does Absolute Position come from in decoder-only Transformers?
Jun 4, 2026Valeria Ruscio, Umberto Nanni, Fabrizio SilvestriRotary Positional Embeddings Transformer Interpretability
Intuition Machines · Sapienza University of Rome
P-Cast Precision in FP8 Attention: Sink-Induced Collapse and the Optimality of S=2^8
Jun 2, 2026Reed LauSoftmax Attention Attention Sinks
Tencent
When Graph Tokens Sink: A Mechanistic Analysis of Graph Language Models
Jun 2, 2026Ding Zhang, Runtao Zhou, Wenqing Zheng +3LLM Interpretability Mechanistic Interpretability
University of Virginia · Capital One
When Do Attention Circuits Form? Developmental Trajectories of Capability and Attention-Sink Emergence Across Three 1B-ClassArchitectures
Jun 1, 2026Yongzhong XuLanguage Model Pretraining Transformer Interpretability
Massive Spikes in LLMs are Bias Vectors: Mechanistic Uncovering and Spike-Free Quantization
Jun 1, 2026Yung-Chin Chen, Chung Peng Lee, Ze-Wei Liou +1LLM Quantization Activation Outliers
Princeton University, NJ, USA · EnCharge AI, CA, USA
Detection vs. Execution: Single-Bucket Probes Miss Half the Mamba-2 State Sink
May 30, 2026Yuhang JiangMamba Attention Head Analysis
Independent Researcher
Contribution Weights: A Geometrical Analysis of Self-Attention Transformers
May 29, 2026Harry Jake Cunningham, Nicola Muca CironeTransformer Interpretability Feature Attribution
Department of Computer Science, University College London, London, UK. · Cartesia.AI, San Francisco, US.
Addressing Exacerbated Attention Sink for Source-Free Cross-Domain Few-Shot Learning
May 25, 2026Shuai Yi, Yixiong Zou, Yuhua Li +1Source-Free Domain Adaptation Few-Shot Domain Adaptation
Huazhong University of Science and Technology
ASAP: Attention Sink Anchored Pruning
May 21, 2026Jaehyuk Lee, Hanyoung Kim, Yanggee Kim +1Self-Attention Efficient ViTs
Department of Mathematics Korea University Seoul, Republic of Korea · Korea University Seoul, Republic of Korea
DySink: Dynamic Frame Sinks for Autoregressive Long Video Generation
May 20, 2026Bo Ye, Xinyu Cui, Jian Zhao +2Long-Horizon Video Generation Memory-Augmented Video Generation
School of Computer Science and Engineering, Southeast University · Key Lab. of Computer Network and Information Integration, Southeast University · Southeast University +3
Attention Sinks and Outliers in Attention Residuals
May 18, 2026Haozheng Luo, Haoran Dai, Shaoyang Zhang +10LLM Quantization Attention Residual Connections
Department of Computer Science, Northwestern University · Department of Computer Science, Illinois Institute of Technolog · Department of Computer Science and Engineering, University of Michigan +5
SLASH the Sink: Sharpening Structural Attention Inside LLMs
May 11, 2026Yiming Liu, Bin Lu, Xinbing Wang +2LLM Reasoning with Graphs LLM Interpretability
Shanghai Jiao Tong University, Shanghai, China · Institute of Geographical Science and Natural Resources Research, Chinese Academy of Sciences, Beijing, China
FocuSFT: Bilevel Optimization for Dilution-Aware Long-Context Fine-Tuning
May 11, 2026Zehua Pei, Hui-Ling Zhen, Xianzhi Yu +3Supervised Fine-Tuning Self-Attention
The Chinese University of Hong Kong · Huawei Technologies Co., Ltd
Attention Sinks in Diffusion Transformers: A Causal Analysis
May 10, 2026Fangzheng Wu, Brian SummaTransformer Interpretability Diffusion Transformer
Department of Computer Science, Tulane University, New Orleans, LA, USA.
A Single Layer to Explain Them All:Understanding Massive Activations in Large Language Models
May 8, 2026Zeru Shi, Zhenting Wang, Fan Yang +2LLM Interpretability Attention Sinks
Rutgers University · Wake Forest University · Meta AI
Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing prevention
May 8, 2026Peter Súkeník, Cristina López Amado, Christoph H. Lampert +1Self-Attention Transformer Attention
Institute of Science and Technology (ISTA) Austria
The Structural Origin of Attention Sink: Variance Discrepancy, Super Neurons, and Dimension Disparity
May 7, 2026Siquan Li, Kaiqi Jiang, Jiacheng Sun +1Self-Attention Attention Sinks
1The Chinese University of Hong Kong, Shenzhen · 2Huawei
Attention Sinks in Massively Multilingual Neural Machine Translation:Discovery, Analysis, and Mitigation
May 2, 2026Hillary Mutisya, John MuganeMultilingual Language Models Cross-Attention
Thiomi NLP · Harvard University
LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models
Apr 27, 2026Rinyoichi Takezoe, Yaqian Li, Zihao Bo +3Efficient VLM Inference Attention Sinks
Li Auto Inc.
Sink-Token-Aware Pruning for Fine-Grained Video Understanding in Efficient Video LLMs
Apr 22, 2026Kibum Kim, Jiwan Kim, Kyle Min +4Efficient VLM Inference Attention Sinks
Korea Advanced Institute of Science and Technology (KAIST) · Oracle · University of California, San Diego +1
SinkRouter: Sink-Aware Routing for Efficient Long-Context Decoding in Large Language and Multimodal Models
Apr 18, 2026Junnan Liu, Xinyan Liu, Peifeng Gao +4GPU Acceleration Self-Attention
Harbin Institute of Technology (Weihai) · University of Hong Kong, Hong Kong, China · City University of Hong Kong, Hong Kong, China
A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigation
Apr 16, 2026Yuval Ran-Milo, Hila Ofek, Shahar MendelSelf-Attention Decoder-Only Language Models
Tel Aviv University
When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models
Apr 1, 2026Jiho Choi, Jaemin Kim, Sanghwan Kim +2Large Vision-Language Models VLM Interpretability
KAIST, South Korea · Chung-Ang University, South Korea · Technical University of Munich, Germany
Threshold Differential Attention for Sink-Free, Ultra-Sparse, and Non-Dispersive Language Modeling
Jan 17, 2026Xingyue Huang, Xueying Ding, Mingxuan Ju +3Self-Attention Long-Context Language Modeling
University of Oxford · Carnegie Mellon University · Snap Inc.
Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruning
Jan 11, 2026Jaewon Sok, Jewon Yeom, Seonghyeon Park +2LLM Pruning Attention Head Analysis
Department of Rural Systems Engineering, Seoul National University · Graduate School of Data Science, Seoul National University · Department of Aerospace Engineering, Seoul National University
RefAM: Attention Magnets for Zero-Shot Referral Segmentation
Sep 26, 2025Anna Kukleva, Enis Simsar, Alessio Tonioni +4Zero-Shot Learning Image Segmentation
Max Planck Institute for Informatics, SIC · ETH Zürich · Google +1

Oct 5, 2026, cs.LGJ/K move · Enter open · S save

Spectral Geometry of Attention: From Information Routing to Uncertainty

Giulio Viganò, Simone Melzi, Maks Ovsjanikov

University of Milano-Bicocca Milan, Italy · Ecole Polytechnique · LIX, Ecole Polytechnique IP Paris

Attention Head Analysis Attention Sinks

Open paper AskPDF arXiv

In this work, we study transformer attention through the lens of spectral geometry and operator theory. We view each attention head as a functional map between Hilbert spaces of functions on the token sequence and derive a Token Difference Operator, whose spectral structure controls how token-space information is routed to the output. We show that standard Euclidean spectra are structurally biased by sinks, conflating mass concentration with genuine routing capacity. By recasting token space in the intrinsic probability geometry induced by attention, the token difference spectrum disentangles sink effects from routing capacity and provides a spectral description of the dimensionality of the head output. This yields a unified framework for analyzing attention maps, explaining sinks, routing collapse, and output dimensionality within a single operator-theoretic framework. In practice, by grounding attention heuristics in spectral geometry, we develop a novel attention-based uncertainty estimator that complements probability-based scores, with the largest gains on long-context inputs.

Momentum

Related topics

Latest papers 42