Attention Mechanisms

Follow this topic Ask about this topic

Attention mechanisms let a model select and combine information from other tokens, image regions, or data points, supporting tasks such as language understanding, visual recognition, and retrieval. The standard approach uses query-key-value comparisons with softmax, while sparse, linear or kernel-based, local, and hybrid methods aim to reduce the cost of processing long sequences; positional encodings and cross-attention adapt interactions to order, geometry, or another input. Researchers also study attention heads and circuits to understand model behavior, alongside mathematical analyses of expressivity, training dynamics, and representation collapse. A central challenge is balancing accuracy and generalization against memory and computation, while determining when attention weights provide reliable explanations rather than merely useful internal signals.

Latest papers

517 papers61 papers in the last four weeks, up 30% on the four weeks before. 0.4% of all new papers.

MICA: Multivariate Infini Compressive Attention for Time Series Forecasting
Apr 7, 2026Willa Potosnak, Nina Żukowska, Michał Wiliński +4Multivariate Time Series Forecasting Time Series Transformers
Carnegie Mellon University, Pittsburgh, PA, USA · Amazon, Seattle, WA, USA
TriAttention: Efficient Long Reasoning with Trigonometric KV Compression
Apr 6, 2026Weian Mao, Xi Lin, Wei Huang +5Long-Context Reasoning in Language Models KV-Cache Compression
MIT · ZJU · NVIDIA
Automated Attention Pattern Discovery at Scale in Large Language Models
Apr 4, 2026Jonathan Katzy, Razvan-Mihai Popescu, Erik Mekkes +2Transformer Interpretability LLM Interpretability
Delft University of Technology
Gradient Boosting within a Single Attention Layer
Apr 3, 2026Saleh SargolzaeiAttention Mechanisms Softmax Attention
University of Windsor
Stochastic KV Routing: Enabling Adaptive Depth-Wise Cache Sharing
Apr 3, 2026Anastasiia Filippova, David Grangier, Marco Cuturi +1KV-Cache Management Memory-Efficient Inference
Apple
Enhancing Medical Visual Grounding via Knowledge-guided Spatial Prompts
Apr 2, 2026Yifan Gao, Tao Zhou, Yi Zhou +3Medical Image Grounding Medical Image Localization
School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China · School of Computer Science and Engineering, Southeast University, Nanjing 211189, China · Yong Yoo Lin School of Medicine, National University of Singapore, Singapore +1
Cosine-Normalized Attention for Hyperspectral Image Classification
Apr 2, 2026Muhammad Ahmad, Manuel MazzaraHyperspectral Image Classification Transformer Attention
SDAIA-KFUPM, Joint Research Center for Artificial Intelligence (JRCAI), King Fahd University of Petroleum and Minerals, Dhahran, 31261, Saudi Arabia · Institute of Software Development and Engineering, Innopolis University, Innopolis, 420500, Russia
Invertible Query-Key Coupling Composes with Attention Mechanisms
Apr 2, 2026Barak Gahtan, Alex M. BronsteinInvertible Neural Networks Self-Attention
Technion – Israel Institute of Technology · ISTA – Institute of Science and Technology Austria
Regularizing Attention Scores with Bootstrapping
Apr 1, 2026Neo Christopher Chung, Maxim LaletinTransformer Interpretability Attention Mechanisms
University of Warsaw · Samsung AI Center, Warsaw
Screening Is Enough
Apr 1, 2026Ken M. NakanishiSelf-Attention Long-Context Language Modeling
Center for Emergent Matter Science (CEMS), RIKEN · Graduate School of Science, The University of Tokyo
IWP: Token Pruning as Implicit Weight Pruning in Large Vision Language Models
Apr 1, 2026Dong-Jae Lee, Sunghyun Baek, Junmo KimEfficient VLM Inference Large Vision-Language Models
KAIST, South Korea
When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models
Apr 1, 2026Jiho Choi, Jaemin Kim, Sanghwan Kim +2Large Vision-Language Models VLM Interpretability
KAIST, South Korea · Chung-Ang University, South Korea · Technical University of Munich, Germany
From Dispersion to Attraction: Spectral Dynamics of Hallucination Across Whisper Model Scales
Mar 31, 2026Ivan Viakhirev, Kirill Borodin, Grach MkrtchianTransformer Interpretability Automatic Speech Recognition
Information Technologies, Mechanics and Optics University, Russia · lab260, Moscow Technical University of Communications and Informatics, Russia · BitmanagerAI, UAE
QUEST: A robust attention formulation using query-modulated spherical attention
Mar 31, 2026Hariprasath Govindarajan, Per Sidén, Jacob Roll +1Transformer Attention Attention Mechanisms
Linköping University, Sweden · Qualcomm Auto Ltd Sweden Filial
Tucker Attention: A generalization of approximate attention mechanisms
Mar 31, 2026Timon Klein, Jonas Kusch, Sebastian Sager +2Tucker Decomposition Attention Mechanisms
Department of Mathematics; Otto von Guericke University Magdeburg; 39106 Magdeburg; Germany. · Scientific Computing, Norwegian University of Life Sciences, ˚As, Norway · Max Planck Institute for Dynamics of Complex Technical Systems; 39106 Magdeburg; Germany. +1
HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention
Mar 30, 2026Yufei Xu, Fanxu Meng, Fan Jiang +11Self-Attention LLM Inference Acceleration
Xiaohongshu
CSAttention: Centroid-Scoring Attention for Accelerating LLM Inference
Mar 30, 2026Chuxu Song, Zhencan Peng, Jiuqi Wei +1LLM Inference Acceleration Long-Context Language Model Inference
Department of Computer Science, Rutgers University · Oceanbase, Ant Group.
Preconditioned Attention: Enhancing Efficiency in Transformers
Mar 28, 2026Hemanth SaratchandranAttention Mechanisms Neural Network Optimization
Australian Institute for Machine Learning (AIML), Adelaide University
On the Expressive Power of Transformers for Contextual Relations
Mar 26, 2026Demián FraimanTransformer Expressivity Transformer Attention
Instituto de Cálculo Universidad de Buenos Aires
Pure and physics-guided deep learning approaches for spatio-temporal groundwater level prediction
Mar 26, 2026Matteo Salis, Gabriele Sartor, Rosa Meo +2Physics-Informed ML Attention Mechanisms
Computer Science Department, University of Turin, Turin, Italy · LISTIC Laboratory, Université Savoie Mont Blanc, Annecy-le-vieux, France · Interuniversity Department of Regional and Urban Studies and Planning, Politecnico di Torino and University of Turin, Turin, Italy
Mitigating Object Hallucinations in LVLMs via Attention Imbalance Rectification
Mar 25, 2026Han Sun, Qin Li, Peixin Wang +1Object Hallucination in VLMs VLM Hallucination Mitigation
Shanghai Key Laboratory of Trustworthy Computing, East China Normal University
Learning Cross-Joint Attention for Generalizable Video-Based Seizure Detection
Mar 24, 2026Omar Zamzam, Takfarinas Medani, Chinmay Chinara +1Attention Mechanisms Video Anomaly Detection
Ming Hsieh Department of Electrical and Computer Engineering University of Southern California Los Angeles, CA 90089
Looking Beyond the Window: Global-Local Aligned CLIP for Training-free Open-Vocabulary Semantic Segmentation
Mar 24, 2026ByeongCheol Lee, Hyun Seok Seong, Sangeek Hyun +3Open-Vocabulary Semantic Segmentation Sliding-Window Attention
Sungkyunkwan University
Stochastic approximation in non-markovian environments revisited
Mar 22, 2026Vivek Shripad BorkarStochastic Approximation Continual Learning
Department of Electrical Engineering, Indian Institute of Technology Bombay, Powai, Mumbai 400076, INDIA.
HiCI: Hierarchical Construction-Integration for Long-Context Attention
Mar 21, 2026Xiangyu Zeng, Qi Xu, Yunke Wang +1Long-Context Language Modeling Long-Context Modeling
School of Computer Science, University of Sydney, Sydney, Australia · 2Westlake University, Hangzhou, China.
NDT: Non-Differential Transformer and Its Application to Sentiment Analysis
Mar 21, 2026Soudeep Ghoshal, Himanshu Buckchash, Sarita Paudel +1Transformer Attention Attention Mechanisms
Kalinga Institute of Industrial Technology (KIIT), Bhubaneswar, Odisha 751024, India. · IMC University of Applied Sciences Krems, Piaristengasse 1, 3500 Austria.
MKA: Memory-Keyed Attention for Efficient Long-Context Reasoning
Mar 21, 2026Dong Liu, Yanxuan Yu, Ben Lengerich +1Attention Mechanisms Memory-Efficient Attention
University of California, Los Angeles, Los Angeles, California, USA · Columbia University, New York, New York, USA · University of Wisconsin-Madison, Madison, Wisconsin, USA
GDEGAN: Gaussian Dynamic Equivariant Graph Attention Network for Ligand Binding Site Prediction
Mar 20, 2026Animesh, Plaban Kumar Bhowmick, Pralay MitraGraph Attention Networks Equivariant GNNs
Department of Artificial Intelligence, Indian Institute of Technology, Kharagpur, India · Department of Computer Science, Indian Institute of Technology, Kharagpur, India
NeuroGame Transformer: Gibbs-Inspired Attention Driven by Game Theory and Statistical Physics
Mar 19, 2026Djamel Bouchaffra, Faycal Ykhlef, Hanene Azzag +2Transformer Attention Attention Mechanisms
Paris-Saclay University, UVSQ, DAVID Lab, 78035, Versailles, France · TSAM, ASM, Centre for Development of Advanced Technologies, Algiers, Algeria · LIPN, UMR CNRS 7030, Sorbonne Paris Nord University, Villetaneuse, France
MANAR: Memory-augmented Attention with Navigational Abstract Conceptual Representation
Mar 19, 2026Zuher Jahshan, Ben Ben Ishay, Leonid YavitsAttention Mechanisms Multi-Head Attention
Bar Ilan University, Ramat Gan, Israel

Apr 7, 2026J/K move · Enter open · S save

MICA: Multivariate Infini Compressive Attention for Time Series Forecasting

Willa Potosnak, Nina Żukowska, Michał Wiliński, Dan Howarth+3

Carnegie Mellon University, Pittsburgh, PA, USA · Amazon, Seattle, WA, USA

Multivariate Time Series Forecasting Time Series Transformers Attention Mechanisms

Open paper AskPDF arXiv

Multivariate forecasting with Transformers faces a core scalability challenge: modeling cross-channel dependencies via attention compounds attention's quadratic sequence complexity with quadratic channel scaling, making full cross-channel attention impractical for high-dimensional time series. We propose Multivariate Infini Compressive Attention (MICA), an architectural design to extend channel-independent Transformers to channel-dependent forecasting. By adapting efficient attention techniques from the sequence dimension to the channel dimension, MICA adds a cross-channel attention mechanism to channel-independent backbones that scales linearly with channel count and context length. We evaluate channel-independent Transformer architectures with and without MICA across multiple forecasting benchmarks. MICA reduces forecast error over its channel-independent counterparts by 5.4% on average and up to 25.4% on individual datasets, highlighting the importance of explicit cross-channel modeling. Moreover, models with MICA rank first among deep multivariate Transformer and MLP baselines. MICA models also scale more efficiently with respect to both channel count and context length than Transformer baselines that compute attention across both the temporal and channel dimensions, establishing compressive attention as a practical solution for scalable multivariate forecasting.