Attention Mechanisms

Follow this topic Ask about this topic

Momentum

42 papers in the last four weeks, up 100% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 378

All topics

Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing prevention
May 8, 2026Peter Súkeník, Cristina López Amado, Christoph H. Lampert +1Self-Attention Transformer Attention
Institute of Science and Technology (ISTA) Austria
GazeVLM: Active Vision via Internal Attention Control for Multimodal Reasoning
May 8, 2026Brown Ebouky, Gabriele Carrino, Niccolo Avogaro +3Active Perception VLM Reasoning
IBM Research · ETH Zurich · TU Wien
Revisiting Transformer Layer Parameterization Through Causal Energy Minimization
May 8, 2026Jin Xu, Camille Couturier, Victor Rühle +2Transformer Energy-Based Models
Microsoft · Microsoft Research Cambridge
Attention Transfer Is Not Universally Effective for Vision Transformers
May 8, 2026Huaiyuan Qin, Muli Yang, Gabriel James Goenawan +4Self-Attention Vision Transformer
Institute for Infocomm Research (I2R), A*STAR, Singapore · 3Shanghai Jiao Tong University · 2Sichuan University
The Structural Origin of Attention Sink: Variance Discrepancy, Super Neurons, and Dimension Disparity
May 7, 2026Siquan Li, Kaiqi Jiang, Jiacheng Sun +1Self-Attention Attention Sinks
1The Chinese University of Hong Kong, Shenzhen · 2Huawei
Long Context Pre-Training with Lighthouse Attention
May 7, 2026Bowen Peng, Subho Ghosh, Jeffrey QuesnelleLanguage Model Pretraining Self-Attention
Nous Research
Look Beyond Saliency: Low-Attention Guided Dual Encoding for Video Semantic Search
May 7, 2026Faisal Aljehrai, Mohammed A. Alkhrashi, Alreem Almuhrij +6Visual Representation Learning Information Retrieval
Saudi Data And Artificial Intelligence Authority (SDAIA)
Neuromorphic visual attention for Sign-language recognition on SpiNNaker
May 7, 2026Sarka Liskova, Olha Vedmedenko, Mazdak Fatahi +3Event-Based Vision Visual Attention
Dept. of Cybernetics, Faculty of Electrical Engineering, Czech Technical University in Prague, Czech Republic · Faculty of Information Technology, Czech Technical University in Prague, Czech Republic
Retrieval from Within: An Intrinsic Capability of Attention-Based Models
May 7, 2026Elad Hoffer, Yochai Blau, Edan Kinderman +3Retrieval-Augmented Generation Question Answering
NVIDIA · Department of Electrical Engineering, Technion, Haifa, Israel
Large Vision-Language Models Get Lost in Attention
May 7, 2026Gongli Xi, Ye Tian, Mengyu Yang +5Transformer FFNs Vision-Language Models
School of Cyberspace Security, Beijing University of Posts and Telecommunications, Beijing, China · State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China · School of Computer Science (National Pilot Software Engineering School), Beijing University of Posts and Telecommunications, Beijing, China +3
Ensuring Reliability in Programming Knowledge Tracing: A Re-evaluation of Attention-augmented Models and Experimental Protocols
May 6, 2026Jaewook Kim, Hyeoncheol KimProgramming Education Temporal Data Leakage
Korea University, Seoul, Republic of Korea
Average Attention Transformers and Arithmetic Circuits
May 6, 2026Lena Ehrmuth, Laura StriekerTransformer Expressivity Transformer
Institute for Theoretical Computer Science, Leibniz University Hanover, Hanover, Germany
Angle-I2P: Angle-Consistent-Aware Hierarchical Attention for Cross-Modality Outlier Rejection
May 6, 2026Muyao Peng, Shun Zou, Pei An +2Point Cloud Registration Cross-Modal Feature Matching
School of Electronic Information and Communications, Huazhong University of Science and Technology
FLUID: Continuous-Time Hyperconnected Sparse Transformer for Sink-Free Learning
May 6, 2026Waleed Razzaq, Yun-Bo ZhaoTransformer Transformer Attention
School of Automation University of Science and Technology China Hefei, Anhui
Where Reliability Lives in Vision-Language Models: A Mechanistic Study of Attention, Hidden States, and Causal Circuits
May 5, 2026Logan Mann, Ajit Saravanan, Ishan Dave +4VLM Evaluation VLM Interpretability
UC Santa Barbara · UC Berkeley · NVIDIA +2
Gradient Flow Structure and Quantitative Dynamics of Multi-Head Self-Attention
May 5, 2026Ayan PendharkarSoftmax Attention Attention Head Analysis
How Language Models Process Negation
May 4, 2026Zhejian Zhou, Tianyi Zhou, Robin Jia +1LLM Interpretability Language Modeling
Information Sciences Institute, University of Southern California · Thomas Lord Department of Computer Science, University of Southern California
When Attention Collapses: Residual Evidence Modeling for Compositional Inference
May 4, 2026Niklas HoubaGravitational-Wave Astronomy Self-Attention
Institute for Particle Physics and Astrophysics ETH Zurich
Projection-Free Transformers via Gaussian Kernel Attention
May 4, 2026Debarshi Kundu, Archisman Ghosh, Swaroop Ghosh +1Transformer Self-Attention
The Pennsylvania State University, University Park, USA
Characterizing the Expressivity of Local Attention in Transformers
May 1, 2026Jiaoda Li, Ryan CotterellTransformer Expressivity Transformer
How Language Models Process Out-of-Distribution Inputs: A Two-Pathway Framework
Apr 30, 2026Hamidreza SaghirLLM Interpretability Attention Mechanisms
Independent Researcher
Attention Is Where You Attack
Apr 30, 2026Aviral Srivastava, Sourav PandaSelf-Attention Adversarial Attacks
Amazon · The Pennsylvania State University
Better Models, Faster Training: Sigmoid Attention for single-cell Foundation Models
Apr 29, 2026Vijay Sadashivaiah, Georgios Dasoulas, Judith Mueller +1Softmax Attention Self-Attention
Merck & Co., Inc., Cambridge, MA, USA
From Architecture to Output: Structural Origins of Hallucination in Large Language Models and the Amplifying Role of Data
Apr 29, 2026Md. Rejaul Korim Sadi, Toufiqur Rahman Tasin, Golam Mostofa NaeemHallucination in Language Models LLM Reliability
*Dept. of Computer Science and Engineering, Metropolitan University, Sylhet, Bangladesh
GateMOT: Q-Gated Attention for Dense Object Tracking
Apr 29, 2026Mingjin Lv, Zelin Liu, Feifei Shao +4Visual Object Tracking Gated Attention
Huazhong University of Science and Technology, Wuhan, China · Zhejiang University, Hangzhou, China · La Trobe University, Melbourne, Australia
Comparative Analysis of AutoML and BiLSTM Models for Cyberbullying Detection on Indonesian Instagram Comments
Apr 29, 2026Raihana Adelia Putri, Aisyah Musfirah, Anggi Puspita Ningrum +3Long Short-Term Memory Networks Attention Mechanisms
Institut Teknologi Sumatera
AMMA: A Multi-Chiplet Memory-Centric Architecture for Low-Latency 1M Context Attention Serving
Apr 28, 2026Zhongkai Yu, Haotian Ye, Chenyang Zhou +9Compute-in-Memory LLM Inference Acceleration
University of California, San Diego · Columbia University · Yonsei University +2
Block-Wise Differentiable Sinkhorn Attention: Tail-Refinement Gradients with a Gap-Aware Dustbin Bridge
Apr 28, 2026Dylan FordeAutomatic Differentiation Transformer Attention
Independent Researcher
QAROO: AI-Driven Online Task Offloading for Energy-Efficient and Sustainable MEC Networks
Apr 28, 2026Yongtao Yao, Yao Yang, Haorui Shi +3Edge Computing Wireless Resource Allocation
Emergent Self-Attention from Astrocyte-Gated Associative Memory Dynamics
Apr 28, 2026Arnau Vivet, Alex ArenasSoftmax Attention Dynamical Systems
1Departament d’Enginyeria Inform`atica i Matem`atiques, Universitat Rovira i Virgili, 43007 Tarragona, Spain · 2Complexity Science Hub Vienna, Metternichgasse 8, 1030 Vienna, Austria

May 8, 2026, cs.LGJ/K move · Enter open · S save

Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing prevention

Peter Súkeník, Cristina López Amado, Christoph H. Lampert, Marco Mondelli

Institute of Science and Technology (ISTA) Austria

Self-Attention Transformer Attention Attention Sinks Attention Mechanisms

Open paper AskPDF arXiv

This paper studies the role of sinks and diagonal patterns as attention switch and anti-oversmoothing mechanisms. We analyze geometric conditions under which sinks can be represented, showing a necessary alignment between the embedding of the sink and all other embeddings. Next, we refine the current understanding of the role of sinks in oversmoothing prevention: we specify the conditions under which dense attention provably smooths more than sparse attention, and empirically verify that such conditions are often satisfied in practice. We further prove an equivalence between sinks and hard attention switch, in which the output of the attention is identically 0. Finally, we relax the hard attention switch by allowing token self-communication: we provide a quantitative comparison of the costs of representing sinks vs.\ diagonal patterns, showing why sinks are favored in pretrained transformers. The introduction and analysis of diagonal patterns and the generalization of the attention switch close the gap between what oversmoothing prevention requires and what sinks provide, while also establishing when and why attention layers act like MLPs if token communication is not necessary.

Momentum

Related topics

Latest papers 378