Overview

Attention mechanisms let a model select and combine information from other tokens, image regions, or data points, supporting tasks such as language understanding, visual recognition, and retrieval. The standard approach uses query-key-value comparisons with softmax, while sparse, linear or kernel-based, local, and hybrid methods aim to reduce the cost of processing long sequences; positional encodings and cross-attention adapt interactions to order, geometry, or another input. Researchers also study attention heads and circuits to understand model behavior, alongside mathematical analyses of expressivity, training dynamics, and representation collapse. A central challenge is balancing accuracy and generalization against memory and computation, while determining when attention weights provide reliable explanations rather than merely useful internal signals.

Latest papers

CardsList

541 papers30 papers in the last four weeks, down 57% on the four weeks before. 0.2% of all new papers.

  1. Kernel Ridge Attention: Nonlinear Memory with Selective Forgetting

    Nov 18, 2026Haowei Chen, Yichen Wang, Congfu XuAttention MechanismsMemory-Augmented Language Models

  2. Runtime Verification of Multiple Natural Language Criteria for Agent Governance

    Nov 17, 2026Silviu Pitis, Parand A. Alamdari, Jessica Tang +2Runtime Verification for AI AgentsAI Agent Governance

  3. GaugeRoPE: RoPE-Compatible Executable Latent Attention

    Nov 16, 2026Yuanhang Gao, Xiangrui Yang, Lingyuan Meng +2KV-Cache CompressionLLM Inference Acceleration

  4. Two-Clustering Regime of Token Dynamics in Causal Attention

    Nov 13, 2026Trinh T. Nguyen, Duy-Tung Pham, Hoang-Son Do +2Causal AttentionAttention Mechanisms

  5. Sanity Checks for Attention Surgery: Null Models and Matched Interventions for Representation-Motivated Design

    Nov 10, 2026Basavaraj R M, Aryan Bharat KumarAttention MechanismsAttention Head Analysis

  6. Everything in its right place: attention is a radio head

    Nov 9, 2026Lluís Arola-Fernández, Aissam Khadraoui, Alex ArenasAttention Mechanisms

  7. NearFormer: Linear-Size Sparse Attention via Randomized Near Neighbors

    Nov 9, 2026Ali Parviz, Alex Cloninger, Gal MishneGraph TransformerLong-Range Graph Learning

  8. Understanding the Effect of Visual Attention in CNNs: A Decision-Space Alignment Perspective

    Nov 7, 2026Niladri B. Puhan, K. Kamala PriyaVisual AttentionAttention Mechanisms

  9. Beyond Quadratic Attention: Separable Gaussian Operators for ODE Vision Transformers

    Nov 5, 2026Carlos Boned Riera, David Romero Sanchez, Oriol Ramos TerradesTransformer AttentionEfficient ViTs

  10. STAG-Net: Structure-Aware Attention over Landmark Groups for Lightweight Head Pose Estimation

    Nov 5, 2026Asiri Gawesha Lindamulage, Sanka Mohottala, Dharshana Kasthurirathna +1Object Pose EstimationAttention Mechanisms

  11. The Intrusion Pathway: How Non-Evidential Context Degrades Language Model Performance

    Nov 5, 2026Dikshant Kukreja, Gautam Gupta, Avni Mittal +8Language Model RobustnessMechanistic Interpretability

  12. ZYFA: Speed and Accuracy of FP8 FlashAttention on AMD CDNA4

    Nov 5, 2026Quentin Anthony, Ganesh Bikshandi, Praneeth Medepalli +1Efficient AttentionGPU Acceleration

  13. Attending to Memory Writes Reduces Seed Variability in Recurrent Length Generalization

    Nov 3, 2026Sungwoo Goo, Hwi-yeol Yun, Sangkeun JungRecurrent Neural NetworksLength Generalization

  14. Exploiting Biases in Attention for Transformers

    Nov 3, 2026Charles Herr, Gabriele Spadaro, Enzo TartaglioneTransformer AttentionAttention Mechanisms

  15. Length Is Cheap, Competition Is Not: Attention Predicts Context Rot in Retrieval

    Nov 3, 2026Yinuo LiuKnowledge Retrieval in Language ModelsLong-Context Language Model Evaluation

  16. Nonreciprocal Spherical Attention and Retrieval: Sharp Coarsening Clocks and Certified Score Correction

    Nov 1, 2026Faruk Alpay, Taylan AlpayApproximate Nearest Neighbor SearchAttention Mechanisms

  17. Kernel-based Interpolatory Methods for Context Compression of Attention Mechanisms

    Oct 29, 2026Michel Fabrice Serret, Laura GrigoriLLM Context CompressionKV-Cache Compression

  18. Understanding axial attention in TSFMs through single location regression

    Oct 29, 2026Andrei-Tiberiu Pantea, Aymeric DieuleveutAxial AttentionAttention Mechanisms

  19. A learning-based method of automatic music transcription

    Oct 29, 2026Phu-Anh-Tai Do, Huu-Hoang Nguyen, Duc-Hoang Nguyen +1Automatic Music TranscriptionAttention Mechanisms

  20. Merge and Branch: Efficient Tree-Structured Supervised Fine-Tuning of LLMs for Recommendation

    Oct 27, 2026Yunqin Zhu, Liyang Shu, Yusi Zhang +4Large Language Model-Based RecommendationLLM Fine-Tuning

  21. HIDRA: Hierarchical Dual-Routing Attention for Replay-Free Lifelong Imitation Learning

    Oct 26, 2026Fanqi Yu, Matteo Tiezzi, Cigdem Beyan +2Hierarchical AttentionRobot Imitation Learning

  22. Cross-Loop Query Handoff: How Later Loops Redirect Two-Hop Retrieval in the Ouro Looped Language Models

    Oct 25, 2026Ziyu Mu, Zehua WangMulti-Hop RetrievalMulti-Hop QA

  23. ARENA: Active Region attENtion Aggregation for interpretable solar flare prediction

    Oct 23, 2026Luis Bauer, Annalena Kofler, Andrés Asensio Ramos +3Space Weather ForecastingAttention Mechanisms

  24. GANet: A Lightweight Group Attention Network for Plant Disease Identification on Embedded Systems

    Oct 21, 2026Anubhab Tripathi, Bibek Das, Asfak Ali +1Plant Disease ClassificationLightweight CNNs