Attention Mechanisms

Follow this topic Ask about this topic

Attention mechanisms let a model select and combine information from other tokens, image regions, or data points, supporting tasks such as language understanding, visual recognition, and retrieval. The standard approach uses query-key-value comparisons with softmax, while sparse, linear or kernel-based, local, and hybrid methods aim to reduce the cost of processing long sequences; positional encodings and cross-attention adapt interactions to order, geometry, or another input. Researchers also study attention heads and circuits to understand model behavior, alongside mathematical analyses of expressivity, training dynamics, and representation collapse. A central challenge is balancing accuracy and generalization against memory and computation, while determining when attention weights provide reliable explanations rather than merely useful internal signals.

Latest papers

517 papers61 papers in the last four weeks, up 30% on the four weeks before. 0.4% of all new papers.

RayRoPE: Projective Ray Positional Encoding for Multi-view Attention
Jan 21, 2026Yu Wu, Minsik Jeon, Jen-Hao Rick Chang +2Multi-View 3D Reconstruction Novel View Synthesis
Carnegie Mellon University · Apple
Threshold Differential Attention for Sink-Free, Ultra-Sparse, and Non-Dispersive Language Modeling
Jan 17, 2026Xingyue Huang, Xueying Ding, Mingxuan Ju +3Self-Attention Long-Context Language Modeling
University of Oxford · Carnegie Mellon University · Snap Inc.
Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruning
Jan 11, 2026Jaewon Sok, Jewon Yeom, Seonghyeon Park +2LLM Pruning Attention Head Analysis
Department of Rural Systems Engineering, Seoul National University · Graduate School of Data Science, Seoul National University · Department of Aerospace Engineering, Seoul National University
NoiseFormer -- Noise Diffused Symmetric Attention Transformer
Jan 10, 2026Phani Kumar, Nyshadham, Jyothendra Varma +1Transformer Attention Attention Mechanisms
Intel Corporation
Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference
Dec 18, 2025Dhruv Deshmukh, Saurabh Goyal, Nipun Kwatra +1Self-Attention Long-Context Language Model Inference
Microsoft Research India.
Exact Flow Linear Attention: Exact Solution from Continuous-Time Dynamics
Dec 14, 2025Jingdi Lei, Di Zhang, Soujanya PoriaDynamical Systems Linear Attention
Nanyang Technological University · Fudan University
Block Sparse Flash Attention
Dec 7, 2025Daniel Ohayon, Itay Lamprecht, Itay Hubara +3GPU Acceleration Self-Attention
Technion – Israel Institute of Technology, Haifa, Israel · Intel - Habana Labs · Intel – Habana Labs, Tel Aviv, Israel
Data-Free Pruning of Self-Attention Layers in LLMs
Dec 3, 2025Dhananjay Saikumar, Blesson VargheseLLM Inference Acceleration LLM Compression
School of Computer Science University of St Andrews United Kingdom
Selective Rotary Position Embedding
Nov 21, 2025Sajad Movahedi, Timur Carstensen, Arshia Afzal +3Softmax Attention Rotary Positional Embeddings
ELLIS Institute Tübingen1, LIONS, EPFL2, University of Freiburg3, Max-Planck-Institute for Intelligent Systems4, Prior Labs5
Predicting the Emergence of Induction Heads in Language Model Pretraining
Nov 21, 2025Tatsuya Aoyama, Ethan Gotlieb Wilcox, Nathan SchneiderLanguage Model Pretraining In-Context Learning
Department of Linguistics, Georgetown University.
Unifying Convolution and Attention via Convolutional Nearest Neighbors
Nov 18, 2025Mingi Kang, Jeová Farias Sales Rocha NetoHybrid CNN-Transformer Architectures Self-Attention
Bowdoin College
Controllably Efficient Language Models
Nov 7, 2025Jatin Prakash, Aahlad Puli, Rajesh RanganathEfficient Transformer Inference Long-Context Language Modeling
New York University
Emergence of Minimal Circuits for Indirect Object Identification in Attention-Only Transformers
Oct 28, 2025Rabin AdhikariAttention Head Analysis Transformer
Saarland University 66123 Saarbrücken
NOSA: Native and Offloadable Sparse Attention
Oct 15, 2025Yuxiang Huang, Pengjie Wang, Jicheng Han +9KV-Cache Offloading GPU Acceleration
NLP Group, DCST, IAI, BNRIST, Tsinghua University, Beijing, China. · OpenBMB, China. · BUPT, Beijing, China. +1
Performance-Efficiency Tradeoffs in Transformers: An Approximation Theory Perspective
Oct 4, 2025Ruoxi Yu, Haotian Jiang, Jingpu Cheng +3Softmax Attention Attention Head Analysis
Center for Data Science, Peking University · Institute for Functional Intelligent Materials, National University of Singapore · Department of Mathematics, National University of Singapore +1
Accelerating Attention with Basis Decomposition
Oct 2, 2025Jialin ZhaoSelf-Attention Transformer Attention
Department of Computer Science Tsinghua University
Eyes-on-Me: Scalable RAG Poisoning through Transferable Attention-Steering Attractors
Oct 1, 2025Yen-Shan Chen, Sian-Yao Huang, Cheng-Lin Yang +1Language Model Steering RAG Poisoning Attacks
CyCraft AI Lab, Taiwan · National Taiwan University
RefAM: Attention Magnets for Zero-Shot Referral Segmentation
Sep 26, 2025Anna Kukleva, Enis Simsar, Alessio Tonioni +4Attention Mechanisms Image Segmentation
Max Planck Institute for Informatics, SIC · ETH Zürich · Google +1
Rotary Position Encodings for Graphs
Sep 26, 2025Isaac Reid, Arijit Sehanobish, Cederik Höfs +7Graph Positional Encoding Rotary Positional Embeddings
University of Cambridge · Google DeepMind · Independent Researcher +3
Cross-Attention is Half Explanation in Speech-to-Text Models
Sep 22, 2025Sara Papi, Dennis Fucci, Marco Gaido +2Transformer Interpretability Cross-Attention
Fondazione Bruno Kessler, Italy
Dynamic Relational Priming Improves Transformer in Multivariate Time Series
Sep 15, 2025Hunjae Lee, Corey ClarkTransformer Attention Attention Mechanisms
Department of Computer Science, Southern Methodist University, Dallas TX USA · SMU Guildhall
Multipole Semantic Attention: A Fast Approximation of Softmax Attention for Pretraining
Sep 12, 2025Rupert Mitchell, Kristian KerstingLanguage Model Pretraining Softmax Attention
Department of Computer Science, TU Darmstadt, Darmstadt, Germany · Hessian Center for AI (hessian.AI), Darmstadt, Germany · German Research Center for Artificial Intelligence (DFKI), Darmstadt, Germany +1
Robust Filter Attention: Self-Attention as Precision-Weighted State Estimation
Sep 4, 2025Peter RacioppoSelf-Attention Long-Context Language Modeling
Independent Researcher, Los Angeles, CA, USA.
FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernel
Aug 25, 2025Ran Yan, Youhe Jiang, Zhuoming Chen +3GPU Kernel Optimization GPU Acceleration
1. The Hong Kong University of Science and Technology · 2. Carnegie Mellon University
RCStat: A Statistical Framework for using Relative Contextualization in Transformers
Jun 24, 2025Debabrata Mahapatra, Shubham Agarwal, Apoorv Saxena +1Attention Mechanisms Transformer Interpretability
Adobe Research, India
MGDFIS: Multi-scale Global-detail Feature Integration Strategy for Small Object Detection
Jun 15, 2025Yuxiang Wang, Xuecheng Bai, Chuanzhi Xu +2Remote Sensing Small Object Detection Small Object Detection
School of Computer Science, The University of Sydney, Sydney, Australia · 2Aviation Traffic Control Technology (SHENZHEN) Co., Ltd., Shenzhen, China
SEMA: a Scalable and Efficient Mamba like Attention via Token Localization and Averaging
Jun 10, 2025Nhat Thanh Tran, Fanghui Xue, Shuai Zhang +4Self-Attention Vision Transformer
Department of Mathematics; University of California, Irvine; Irvine, CA 92697, USA · Qualcomm AI Research; San Diego, CA 92121, USA
ISAC: Training-Free Instance-to-Semantic Attention Control for Multi-Instance Generation
May 27, 2025Sanghyun Jo, Wooyeol Lee, Ziseok Lee +3Controllable Image Generation T2I Generation
OGQ, Seoul, Korea · Seoul National University, Seoul, Korea
Recurrent Self-Attention Dynamics: An Energy-Agnostic Perspective from Jacobians
May 26, 2025Akiyoshi Tomihari, Ryo KarakidaSelf-Attention Attention Mechanisms
Artificial Intelligence Research Center, AIST, Japan · Department of Computer Science, The University of Tokyo, Japan · RIKEN Center for Advanced Intelligence Project
The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs
Apr 24, 2025Piotr Nawrot, Robert Li, Renjie Huang +3Transformer Inference LLM Evaluation
University of Edinburgh · Cohere · Meta

Jan 21, 2026J/K move · Enter open · S save

RayRoPE: Projective Ray Positional Encoding for Multi-view Attention

Yu Wu, Minsik Jeon, Jen-Hao Rick Chang, Oncel Tuzel+1

Carnegie Mellon University · Apple

Multi-View 3D Reconstruction Novel View Synthesis Positional Encoding Attention Mechanisms Stereo Depth Estimation

Open paper AskPDF arXiv

We study positional encodings for multi-view transformers that process tokens from a set of posed input images, and seek a mechanism that encodes patches uniquely, allows SE(3)-invariant attention with multi-frequency similarity, and can adapt to the geometry of the underlying 3D scene. We find that prior (absolute or relative) encoding schemes for multi-view attention do not meet these desiderata, and present RayRoPE to address this gap. RayRoPE represents patch positions based on associated rays and computes query-frame projective coordinates to ensure SE(3) invariance. To adapt to scene geometry, RayRoPE predicts (without direct supervision) a per-token depth to obtain its position along the corresponding ray, while also modeling uncertainty and analytically computing the expected positional encoding. We validate our method on the tasks of novel-view synthesis, stereo depth estimation, and feed-forward 3DGS reconstruction. While remaining efficient, RayRoPE consistently improves over alternate position encoding schemes (e.g., 24% relative improvement on LPIPS in RE10K).