Attention Mechanisms

Ask about this topic

Overview

Attention mechanisms let a model select and combine information from other tokens, image regions, or data points, supporting tasks such as language understanding, visual recognition, and retrieval. The standard approach uses query-key-value comparisons with softmax, while sparse, linear or kernel-based, local, and hybrid methods aim to reduce the cost of processing long sequences; positional encodings and cross-attention adapt interactions to order, geometry, or another input. Researchers also study attention heads and circuits to understand model behavior, alongside mathematical analyses of expressivity, training dynamics, and representation collapse. A central challenge is balancing accuracy and generalization against memory and computation, while determining when attention weights provide reliable explanations rather than merely useful internal signals.

Latest papers

CardsList

517 papers61 papers in the last four weeks, up 30% on the four weeks before. 0.4% of all new papers.

  1. Understanding axial attention in TSFMs through single location regression

    Oct 29, 2026Andrei-Tiberiu Pantea, Aymeric DieuleveutAttention MechanismsTime-Series Foundation Models

    CMAP, École polytechnique, IP Paris; Electricité de France R&D; Palaiseau, France · CMAP, École polytechnique, IP Paris; Palaiseau, France

  2. A learning-based method of automatic music transcription

    Oct 29, 2026Phu-Anh-Tai Do, Huu-Hoang Nguyen, Duc-Hoang Nguyen +1Automatic Music TranscriptionAttention Mechanisms

    Department of Information Technology, FPT University, Ho Chi Minh City, Vietnam · Faculty of Information Technology, Ho Chi Minh City Open University, Ho Chi Minh City, Vietnam

  3. Merge and Branch: Efficient Tree-Structured Supervised Fine-Tuning of LLMs for Recommendation

    Oct 27, 2026Yunqin Zhu, Yao XieLarge Language Model-Based RecommendationLLM Fine-Tuning

    Georgia Institute of Technology, Atlanta, GA, USA

  4. HIDRA: Hierarchical Dual-Routing Attention for Replay-Free Lifelong Imitation Learning

    Oct 26, 2026Fanqi Yu, Matteo Tiezzi, Cigdem Beyan +2Robot Imitation LearningAttention Mechanisms

    AI for Good (AIGO) Research Unit, Istituto Italiano di Tecnologia, Genoa, Italy · DITEN, University of Genoa, Genoa, Italy · Pattern Analysis and Computer Vision (PAVIS), Istituto Italiano di Tecnologia, Genoa, Italy +1

  5. Beyond Multiplicity Correction: Sharp Retrieval Thresholds for Attention with Correlated Distractors

    Oct 26, 2026Heehyun ParkAttention Mechanisms

  6. Cross-Loop Query Handoff: How Later Loops Redirect Two-Hop Retrieval in the Ouro Looped Language Models

    Oct 25, 2026Ziyu Mu, Zehua WangMulti-Hop RetrievalMulti-Hop QA

    Faculty of Science The University of British Columbia · Department of Electrical and Computer Engineering The University of British Columbia

  7. ARENA: Active Region attENtion Aggregation for interpretable solar flare prediction

    Oct 23, 2026Luis Bauer, Annalena Kofler, Andrés Asensio Ramos +3Attention Mechanisms

    MPI for Intelligent Systems, Tübingen · University of Tübingen · MPI for Gravitational Physics, Potsdam +4

  8. GANet: A Lightweight Group Attention Network for Plant Disease Identification on Embedded Systems

    Oct 21, 2026Anubhab Tripathi, Bibek Das, Asfak Ali +1Plant Disease ClassificationLightweight CNNs

    Department of Electronics and Telecommunication Engineering, Jadavpur University, Kolkata, India

  9. Geometric knowledge spaces, Attention pairing and Architectures

    Oct 21, 2026Shang XuAttention MechanismsTransformer

  10. Delay-Guided Attention for Quality Prediction in Multistage Manufacturing

    Oct 21, 2026Jiarui Zhang, Zhaoyang Guan, Zhixuan Xia +1ManufacturingAttention Mechanisms

    Department of Engineering Sciences and Applied Mathematics, Northwestern University, Evanston, IL, USA · Department of Industrial Engineering and Management Sciences, Northwestern University, Evanston, IL, USA

  11. Speedrunning Relational Foundation Model Pretraining

    Oct 20, 2026Gabriel Wedson Mendonça de Souza Veloso, Luan Borges Teodoro Reis Sena, Denis Oliveira Correa +1Relational Foundation ModelsRelational Deep Learning

    Kunumi Institute, Belo Horizonte, Minas Gerais, Brazil · Universidade Federal de Minas Gerais, Belo Horizonte, Minas Gerais, Brazil

  12. DocLayout-YOLO: Axis-Attentive Document Layout Detector with Mesh-Guided Synthetic Pretraining

    Oct 20, 2026Zhiyuan Zhao, Hengrui Kang, Guang Liang +2Document Layout AnalysisEfficient Object Detection

    Shanghai AI Laboratory

  13. Structured-Token Attention for Fine-Grained mmWave Liquid Identification

    Oct 20, 2026Junseo Min, Joongheon KimClassificationAttention Mechanisms

    Department of Defense Convergence Technology, Korea University, Seoul, Republic of Korea · Department of Electrical & Computer Engineering, Korea University, Seoul, Republic of Korea

  14. Kronecker Delta Rule for Long Context Linear Attention

    Oct 19, 2026Anamaria-Roberta Hartl, Levente Zólyomi, Michael List +3Delta-Rule Linear AttentionAttention Mechanisms

    Institute for Machine Learning, Johannes Kepler University, Linz · NXAI GmbH, Linz

  15. Dynamical Kernel Attention: Estimating the Present from a Record of the Past

    Oct 16, 2026Peter RacioppoAttention MechanismsSelf-Attention

    Independent Researcher Los Angeles, CA

  16. DK-VLM: Geometry as Position for Spatial Reasoning in Vision-Language Models

    Oct 15, 2026Han Su, Meng Tian, Tianyu Huang +7Visual Spatial ReasoningSpatial Reasoning

    Harbin Institute of Technology · Yinwang Intelligent Technology · Huawei

  17. ABNAT: Attention-Based Noise-Aware Training for Robust Transformers on Analog Compute-in-Memory Systems

    Oct 15, 2026Wenshuai Yao, Wenyong Zhou, Hanyong Shao +5Noise RobustnessTransformer

    School of Integrated Circuits, Peking University, Beijing, China · Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong · School of Integrated Circuit Science and Engineering, Beihang University, Beijing, China +1

  18. FlashAttention with Statistics: Low-Overhead Attention Features for Compact Hallucination Detectors

    Oct 15, 2026Andrei Volodichev, Maksim Savichev, Nikita Sviridov +3LLM Hallucination DetectionLanguage Model Probing

    Applied AI Institute · Yandex · Risk AI Research Lab

  19. GeoPano: Towards Geometrically Accurate 3D Reconstruction from a Single Panorama

    Oct 15, 2026Jing OU, Zidong Cao, Liaoyuan Fan +7Panoramic 3D ReconstructionSingle-View 3D Reconstruction

    The Hong Kong University of Science and Technology (Guangzhou), China · The Hong Kong University, Hong Kong · Sun Yat-sen University, Guangzhou, China

  20. Language Models Share a Readout Bottleneck in Working Memory

    Oct 15, 2026Dongyu GongWorking Memory in Language ModelsAttention Mechanisms

    Yale University

  21. Beyond Static Positional Encoding: Bayesian Time-aware Attention for Non-Stationary Long-term Time Series Forecasting

    Oct 14, 2026Xinjie Lan, Yunbo Mei, Leheng Zhou +1Long-Term Time Series ForecastingNon-Stationary Time Series Forecasting

    Rice University · University of Hong Kong · University of North Carolina at Charlotte

  22. Why don't Graph Transformers attend to the same token?

    Oct 14, 2026Alexandre BismuthAttention SinksAttention Mechanisms

    BOLD Lab University of Oxford

  23. FlashMask-3: Efficient and Expressive Mask-Aware Distributed Attention

    Oct 13, 2026Guoxia Wang, Qianyue He, Siming Wu +6Context ParallelismAttention Mechanisms

    Baidu Inc.

  24. On the Statistical Mechanics of Attention

    Oct 12, 2026Jerry Yao-Chieh Hu, Yu-An Chen, Maojiang Su +1Transformer AttentionAttention Mechanisms

    Center for Foundation Models and Generative AI, Northwestern University Department of Computer Science, Northwestern University · Department of Computer Science, College of Connected Computing, Vanderbilt University · Department of Physics, National Taiwan University +1

  25. AReA: Adaptive Receptive-Field Attention for Efficient High-Resolution Diffusion Transformers

    Oct 12, 2026Javad Rajabi, Koorosh Roohi, David B. Lindell +1Diffusion TransformerDiffusion-Based Image Generation

    University of Toronto · Vector Institute

  26. Softmax-Free Attention in Spiking Transformers as the Zero-Temperature Limit of Two-Level Systems

    Oct 11, 2026Roberto C. Sotero, Pouya Taghipour, Jose M. Sanchez-BornotSpiking TransformersAttention Mechanisms

    Department of Radiology, University of Calgary, Calgary, AB, Canada · Department of Biomedical Engineering, University of Calgary, Calgary, AB, Canada · Hotchkiss Brain Institute, University of Calgary, Calgary, AB, Canada +1

  27. Align-VAE: Aligning Attention and Geometry in VecSet VAEs for 3D Reconstruction

    Oct 10, 2026Huaizhi Qu, Mingxin Yang, Xintong Han +2Variational Autoencoders3D Reconstruction

    The Hong Kong University of Science and Technology · Tencent Hunyuan

  28. Orthros: Phase-Aware Heterogeneous Attention for Efficient Transformers

    Oct 9, 2026Yuhong Chou, Zehao Liu, Yuqi Pan +4Transformer AttentionAttention Mechanisms

    The Hong Kong Polytechnic University · University of Chinese Academy of Sciences · Independent Researcher +1

  29. PANDA: Prior-guided Attentional Dual-path Architecture

    Oct 9, 2026Rishabh Jain, Federico Belotti, Stefano Coniglio +2Hypergraph Neural NetworksHigher-Order GNNs

    University of Cambridge, UK · University of Bergamo, Italy · Independent Researcher

  30. Asymptotic Analysis of Discrete-Time Self-Attention Dynamics in Transformers

    Oct 9, 2026Claudio AltafiniSelf-AttentionAttention Mechanisms

    Division of Automatic Control, Dept. of Electrical Engineering, Linköping University, SE-58183, Linköping, Sweden