Overview

Self-attention lets each token build a context-dependent representation by weighting information from other tokens, and is central to language, vision, audio, and multimodal Transformers. Common approaches include multi-head softmax attention with positional encodings such as RoPE, sparse and block-sparse attention, and linear or recurrent variants that reduce the cost of processing long sequences. Researchers evaluate these methods on language-model perplexity, long-context retrieval and reasoning, and vision tasks, while also studying attention heads, information flow, and phenomena such as attention sinks. A central tension is how to cut computation and memory for long contexts without losing relevant information or the accuracy and flexibility of full attention.

Latest papers

CardsList

231 papers4 papers in the last four weeks, down 71% on the four weeks before. 0.0% of all new papers.

  1. Self-attention is a Dyson equation for many-body machine learning

    Nov 16, 2026Jarvist Moore FrostSelf-AttentionNeural Quantum States

  2. Sanity Checks for Attention Surgery: Null Models and Matched Interventions for Representation-Motivated Design

    Nov 10, 2026Basavaraj R M, Aryan Bharat KumarAttention MechanismsAttention Head Analysis

  3. Spectral Degeneration of Self-Attention under Isotropic Score Geometry

    Nov 2, 2026Mengda Li, Jianfeng YaoSelf-AttentionSoftmax Attention

  4. Radial Kernel Attention for Global Feature Exchange in Polymer Machine Learning Force Fields

    Oct 30, 2026Wenbo Fu, Chenghui Mao, Haifeng Lv +2Machine Learning Interatomic PotentialsSelf-Attention

  5. ACSAN: All-Convolutional Self-Attention Network Integrating Convolutional Operations with Vision Transformer

    Oct 26, 2026Zimian Wei, Tianqi Zhang, Hengyue PanHybrid CNN-Transformer ArchitecturesEfficient ViTs

  6. A Mixture-Gated Theory of Anisotropic Representation Dynamics in Transformers

    Oct 16, 2026Nguyen Do, My T. ThaiTransformerSelf-Attention

  7. Measure Neural Mappings

    Oct 12, 2026Ricardo Baptista, Edoardo Calvello, Bohan Chen +2Operator LearningNeural Operators

  8. Counting with Attention and LayerNorm: Norm Scaling and Representation Geometry

    Oct 12, 2026Ayyub Shaffy, Manoj Dhakal, Safal Shrestha +1Self-AttentionLayer Normalization

  9. Asymptotic Analysis of Discrete-Time Self-Attention Dynamics in Transformers

    Oct 9, 2026Claudio AltafiniSelf-AttentionAttention Mechanisms

  10. Tucker Bottleneck Attention for Multi-Dimensional Sequence Modeling

    Oct 6, 2026Ryan Solgi, Parsa Madinei, Zheng ZhangAttention MechanismsLow-Rank Attention

  11. WNet: Discrete Wavelets Transform for Efficient Token Mixing

    Oct 3, 2026Rana Aref Salama, Abdou Youssef, Mona DiabTransformerSelf-Attention

  12. Universal interpolation for deep residual self-attention networks

    Oct 1, 2026Sibylle Marcotte, Joan BrunaSoftmax AttentionTransformer

  13. Contrastive Attention Mitigates Spectral Bias in Spiking Transformers

    Oct 1, 2026Xiaoli Liu, Malu Zhang, Yang YangSpiking TransformersSelf-Attention

  14. Retrieval Capacity of Self-Attention Under Competition

    Sep 29, 2026Timur Mudarisov, Mikhail Burtsev, Radu StateSelf-AttentionLong-Context Language Modeling

  15. PQ-HSA: Reusing Product-Quantized Scores for Hybrid Sparse-Approximate Attention

    Sep 27, 2026Kunming Shao, Jierun Chen, Yanli Wang +4Self-AttentionLLM Inference Acceleration

  16. Nonequilibrium Phases of Repulsive Self-Attention: Chaos, Attention Condensation, and Emergent Locality

    Sep 23, 2026Qucheng Gao, Zuyi Yang, Xiao ChenSelf-AttentionChaotic Dynamical Systems

  17. Prescriptive SVD-Inspired Attention via Spectral Energy Retention

    Sep 21, 2026Vasileios Arampatzakis, Vasileios Sevetlidis, George PavlidisTransformer InterpretabilityLow-Rank Attention

  18. It's Not RoPE that Creates Sinks: The Role of Self-Concentration and Value-Non-Mixing in Attention

    Sep 8, 2026Raito Kiya, Satoki Ohashi, Kosuke Sato +6Self-AttentionAttention Sinks

  19. Disentangling Attention in Deep Operator Learning: A Controlled Study of Data-Driven and Physics-Informed Architectures

    Sep 3, 2026Amar Alem Koric, Qibang Liu, Seid KoricPDE Operator LearningDeep Operator Networks

  20. The Head Complexity of Boolean Functions in Single-Layer Attention

    Sep 3, 2026Rajmohan Rajaraman, Ravi Sundaram, Amanuel TesfayeSelf-AttentionAttention Mechanisms

  21. All You Need Is Non-Commutative Words

    Aug 29, 2026Carla M. Quispe Flores, Stanley Salvatierra, Renan CabreraCompositional Representation LearningSelf-Attention

  22. TANGO: Treating Tokens as Operators

    Aug 22, 2026Joshua NunleyTransformer FFNsTransformer