Attention Mechanisms

Ask about this topic

Overview

Attention mechanisms let a model select and combine information from other tokens, image regions, or data points, supporting tasks such as language understanding, visual recognition, and retrieval. The standard approach uses query-key-value comparisons with softmax, while sparse, linear or kernel-based, local, and hybrid methods aim to reduce the cost of processing long sequences; positional encodings and cross-attention adapt interactions to order, geometry, or another input. Researchers also study attention heads and circuits to understand model behavior, alongside mathematical analyses of expressivity, training dynamics, and representation collapse. A central challenge is balancing accuracy and generalization against memory and computation, while determining when attention weights provide reliable explanations rather than merely useful internal signals.

Latest papers

CardsList

517 papers61 papers in the last four weeks, up 30% on the four weeks before. 0.4% of all new papers.

  1. HAViT: Historical Attention Vision Transformer

    Mar 19, 2026Swarnendu Banik, Manish Das, Shiv Ram Dubey +1Cross-Layer AttentionVision Transformer

    Computer Vision and Biometrics Lab, Indian Institute of Information Technology, Allahabad

  2. Learning When to Attend: Conditional Memory Access for Long-Context LLMs

    Mar 18, 2026Sakshi Choudhary, Aditya Chattopadhyay, Luca Zancato +4Self-AttentionLong-Context Language Modeling

    Department of Electrical and Computer Engineering, Purdue University, USA · AWS Agentic AI, USA

  3. Revisiting Cross-Attention Mechanisms: Leveraging Beneficial Noise for Domain-Adaptive Learning

    Mar 18, 2026Zelin Zang, Yehui Yang, Fei Wang +2Unsupervised Domain AdaptationAttention Mechanisms

    School of Engineering, Westlake University, Hangzhou, China · Centre for Artificial Intelligence and Robotics (CAIR), Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences (HKISI-CAS), Hong Kong, China · State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences (CASIA), Beijing, China +2

  4. SympFormer: Accelerated attention blocks via Inertial Dynamics on Density Manifolds

    Mar 17, 2026Viktor Stein, Wuchen Li, Gabriele SteidlTransformer AttentionAttention Mechanisms

    Institute of Mathematics, Technische Universität Berlin, Straße des 17. Juni 136, 10623 Berlin, Germany · Department of Mathematics, University of South Carolina, Columbia. 1523 Greene St, Columbia, SC 29208, USA

  5. ReFORM: Review-aggregated Profile Generation via LLM with Multi-Factor Attention for Restaurant Recommendation

    Mar 17, 2026Moonsoo Park, Seulbeen Je, Donghyeon ParkRecommender SystemsLarge Language Model-Based Recommendation

    University of Southern California, USA · Sejong University, South Korea

  6. WAND: Windowed Attention and Knowledge Distillation for Efficient Autoregressive Text-to-Speech Models

    Mar 17, 2026Hanna Lee, Tan Dat Nguyen, Jaehoon Kang +1TTS SynthesisSelf-Attention

    Korea Advanced Institute of Science and Technology, Republic of Korea · Sungkyunkwan University, Republic of Korea

  7. Residual Stream Duality in Modern Transformer Architectures

    Mar 17, 2026Yifan ZhangTransformerAttention Mechanisms

  8. Not All Retrievals are Useful: Cross-Attention for Input-Aware RAG in Time Series Forecasting

    Mar 16, 2026Seunghan Lee, Jaehoon Lee, Jun Seo +7Zero-Shot Time Series ForecastingCross-Attention

    LG AI Research Seoul, South Korea

  9. PA-Net: Precipitation-Adaptive Mixture-of-Experts for Long-Tail Rainfall Nowcasting

    Mar 14, 2026Xinyu Xiao, Sen Lei, Eryun Liu +5Precipitation NowcastingMixture-of-Experts Models

    Fudan University · the School of Information Science and Technology, Southwest Jiaotong University · Zhejiang University +2

  10. IGASA: Integrated Geometry-Aware and Skip-Attention Modules for Enhanced Point Cloud Registration

    Mar 13, 2026Dongxu Zhang, Jihua Zhu, Shiqi Li +4Point Cloud RegistrationPoint Cloud Learning

  11. 98×\times Faster LLM Routing Without a Dedicated GPU: Flash Attention, Prompt Compression, and Near-Streaming for the vLLM Semantic Router

    Mar 13, 2026Xunzhuo Liu, Bowei He, Xue Liu +3LLM RoutingLLM Serving

    vLLM Semantic Router Project · MBZUAI · McGill University +2

  12. Why Attend to Everything? Focus is the Key

    Mar 12, 2026Hengshuai Yao, Xing Chen, Ahmed Murtadha +8Attention MechanismsEfficient Attention

    Sapient Intelligence · University of Alberta · Tsinghua University

  13. COTONET: A custom cotton detection algorithm based on YOLO11 for stage of growth cotton boll detection

    Mar 12, 2026Guillem González, Guillem Alenyà, Sergi FoixObject DetectionEfficient Object Detection

    Institut de Robòtica i Informàtica Industrial (CSIC-UPC), Llorens i Artigas 4-6, Barcelona, 08028, Spain

  14. Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Tasks

    Mar 12, 2026Yuval Ran-MiloAttention SinksAttention Mechanisms

    Tel Aviv University

  15. On the Computational Hardness of Transformers

    Mar 11, 2026Barna Saha, Yinzhan Xu, Christopher Ye +1TransformerAttention Mechanisms

    University of California San Diego · Columbia University

  16. Beyond Pairwise Attention: Higher-Order Modular Attention for Efficient Sequence Learning

    Mar 11, 2026Shirin Amiraslani, Xin GaoSelf-AttentionTransformer Attention

    Department of Mathematics and Statistics York University Toronto, ON M3J 1P3, Canada

  17. LAtte: Hyperbolic Lorentz Attention for Cross-Subject EEG Classification

    Mar 11, 2026Ahmad Bdeir, Johannes Burchert, Tom Hanika +2Cross-Subject EEG DecodingElectroencephalography

    Data Science Group, Universität Hildesheim · ISMLL, Universität Hildesheim

  18. QV May Be Enough: Toward the Essence of Attention in LLMs

    Mar 11, 2026Zhang EdwardAttention MechanismsTransformer Attention

  19. BinaryAttention: One-Bit QK-Attention for Vision and Diffusion Transformers

    Mar 10, 2026Chaodong Xiao, Zhengqiang Zhang, Lei ZhangAttention QuantizationAttention Mechanisms

    The Hong Kong Polytechnic University · OPPO Research Institute

  20. Telogenesis: Goal Is All U Need

    Mar 10, 2026Zhuoran Deng, Yizhi Zhang, Ziyi Zhang +1Uncertainty-Aware ExplorationAttention Mechanisms

    Independent Research

  21. Efficient Reasoning at Fixed Test-Time Cost via Length-Aware Attention Priors and Gain-Aware Training

    Mar 10, 2026Rian AtriTransformer AttentionEfficient Transformer Inference

    Serval Systems

  22. Emotion is Not Just a Label: Latent Emotional Factors in LLM Processing

    Mar 10, 2026Benjamin Reichman, Adar Avsian, Samuel Webster +1Emotion Representation in Language ModelsLLM Interpretability

    Georgia Institute of Technology

  23. FVG-PT: Adaptive Foreground View-Guided Prompt Tuning for Vision-Language Models

    Mar 9, 2026Haoyang Li, Liang Wang, Siyu Zhou +5Attention-Guided DistillationVLM Adaptation

    University of Technology Sydney, NSW 2007, Australia · Shanghai University, Shanghai 200444, China

  24. Cluster-Aware Attention-Based Deep Reinforcement Learning for Pickup and Delivery Problems

    Mar 9, 2026Wentao Wang, Lifeng Han, Guangyu ZouVehicle RoutingRL for Combinatorial Optimization

    Leicester International Institute, Dalian University of Technology, Panjin, 124221, Liaoning, China. · Department of Electronic and Information Technology, Dalian University of Technology, Panjin, 124221, Liaoning, China.

  25. Bayesian Transformer for Probabilistic Load Forecasting in Smart Grids

    Mar 9, 2026Sajib Debnath, Md. Uzzal MiaElectricity Load ForecastingProbabilistic Time Series Forecasting

    O&M Analytics, AES Clean Energ; The AES Corporation; Louisville, CO 80027, USA · Department of Information and Communication Engineering; Pabna University of Science and Technology; Rajapur, Pabna, Bangladesh

  26. StructSAM: Structure- and Spectrum-Preserving Token Merging for Segment Anything Models

    Mar 7, 2026Duy M. H. Nguyen, Tuan A. Tran, Duong Nguyen +17Token MergingImage Segmentation

    University of Stuttgart · DFKI · Max Planck Research School +10

  27. Rank-Factorized Implicit Neural Bias: Scaling Super-Resolution Transformer with FlashAttention

    Mar 6, 2026Dongheon Lee, Seokju Yun, Jaegyun Im +1Image Super-ResolutionAttention Mechanisms

    University of Seoul · KAIST AI

  28. Vessel-Aware Deep Learning for OCTA-Based Detection of AMD

    Mar 6, 2026Margalit G. Mitzner, Moinak Bhattacharya, Zhilin Zou +2Medical Image AnalysisExplainable Medical Image Analysis

    Stony Brook University, Stony Brook, NY, USA

  29. Data-Aware Random Feature Kernel for Transformers

    Mar 4, 2026Amirhossein Farzam, Hossein Mobahi, Nolan Andrew Miller +1Transformer AttentionAttention Mechanisms

    Google DeepMind

  30. Adaptive Enhancement and Dual-Pooling Sequential Attention for Lightweight Underwater Object Detection with YOLOv10

    Mar 4, 2026Md. Mushibur Rahman, Umme Fawzia Rahim, Enam Ahmed TaufikObject DetectionUnderwater Object Detection

    Department of Computer Science and Engineering, Dhaka University of Engineering & Technology Gazipur, 1707, Bangladesh · Department of Computer Science and Engineering, BRAC University Dhaka, Bangladesh