Self-Attention

Follow this topic Ask about this topic

Self-attention lets each token build a context-dependent representation by weighting information from other tokens, and is central to language, vision, audio, and multimodal Transformers. Common approaches include multi-head softmax attention with positional encodings such as RoPE, sparse and block-sparse attention, and linear or recurrent variants that reduce the cost of processing long sequences. Researchers evaluate these methods on language-model perplexity, long-context retrieval and reasoning, and vision tasks, while also studying attention heads, information flow, and phenomena such as attention sinks. A central tension is how to cut computation and memory for long contexts without losing relevant information or the accuracy and flexibility of full attention.

Latest papers

225 papers11 papers in the last four weeks, up 22% on the four weeks before. 0.1% of all new papers.

ACSAN: All-Convolutional Self-Attention Network Integrating Convolutional Operations with Vision Transformer
Oct 26, 2026Zimian Wei, Tianqi Zhang, Hengyue PanHybrid CNN-Transformer Architectures Efficient ViTs
College of Computer Science and Technology, National University of Defense Technology, 109 Deya Road, Changsha, 410073, Hunan, China
A Mixture-Gated Theory of Anisotropic Representation Dynamics in Transformers
Oct 16, 2026Nguyen Do, My T. ThaiTransformer Self-Attention
Department of Computer & Information Science & Engineering, University of Florida, Gainesville, FL, USA
Dynamical Kernel Attention: Estimating the Present from a Record of the Past
Oct 16, 2026Peter RacioppoAttention Mechanisms Self-Attention
Independent Researcher Los Angeles, CA
Measure Neural Mappings
Oct 12, 2026Ricardo Baptista, Edoardo Calvello, Bohan Chen +2Operator Learning Neural Operators
University of Toronto · Lawrence Berkeley National Laboratory · California Institute of Technology +1
Counting with Attention and LayerNorm: Norm Scaling and Representation Geometry
Oct 12, 2026Ayyub Shaffy, Manoj Dhakal, Safal Shrestha +1Self-Attention Layer Normalization
New York University Abu Dhabi
Asymptotic Analysis of Discrete-Time Self-Attention Dynamics in Transformers
Oct 9, 2026Claudio AltafiniSelf-Attention Attention Mechanisms
Division of Automatic Control, Dept. of Electrical Engineering, Linköping University, SE-58183, Linköping, Sweden
Tucker Bottleneck Attention for Multi-Dimensional Sequence Modeling
Oct 6, 2026Ryan Solgi, Parsa Madinei, Zheng ZhangAttention Mechanisms Low-Rank Attention
University of California-Santa Barbara Santa Barbara, CA, USA
An equality condition for the Dobrushin bound on attention rollout and how often it holds in trained transformers
Oct 4, 2026Przemysław RolaTransformer Interpretability Self-Attention
Department of Mathematics, Kraków University of Economics
WNet: Discrete Wavelets Transform for Efficient Token Mixing
Oct 3, 2026Rana Aref Salama, Abdou Youssef, Mona DiabTransformer Self-Attention
George Washington University, Washington DC, USA · Faculty of Computers and Artificial Intelligence, Cairo University, Egypt · Carnegie Mellon University, Pittsburgh, Pennsylvania, USA
Universal interpolation for deep residual self-attention networks
Oct 1, 2026Sibylle Marcotte, Joan BrunaSoftmax Attention Transformer
Department of Computer Science New York University
Contrastive Attention Mitigates Spectral Bias in Spiking Transformers
Oct 1, 2026Xiaoli Liu, Malu Zhang, Yang YangSpiking Transformers Self-Attention
School of Computer Science and Engineering, University of Electronic Science and Technology of China
Retrieval Capacity of Self-Attention Under Competition
Sep 29, 2026Timur Mudarisov, Mikhail Burtsev, Radu StateSelf-Attention Long-Context Language Modeling
University of Luxembourg, Luxembourg
PQ-HSA: Reusing Product-Quantized Scores for Hybrid Sparse-Approximate Attention
Sep 27, 2026Kunming Shao, Jierun Chen, Yanli Wang +4Self-Attention LLM Inference Acceleration
The Hong Kong University of Science and Technology · Huawei Technologies Ltd. · Sun Yat-sen University +1
Nonequilibrium Phases of Repulsive Self-Attention: Chaos, Attention Condensation, and Emergent Locality
Sep 23, 2026Qucheng Gao, Zuyi Yang, Xiao ChenSelf-Attention Chaotic Dynamical Systems
Department of Physics, Boston College, Chestnut Hill, Massachusetts 02467, USA
Prescriptive SVD-Inspired Attention via Spectral Energy Retention
Sep 21, 2026Vasileios Arampatzakis, Vasileios Sevetlidis, George PavlidisTransformer Interpretability Low-Rank Attention
Athena Research Center, Greece
Increasing Skill Level Recruits Deeper Attention Layers in a Frozen Chess Transformer
Sep 20, 2026David LitmanTransformer Interpretability Attention Head Analysis
Computational Neurobiology Laboratory, Salk Institute
It's Not RoPE that Creates Sinks: The Role of Self-Concentration and Value-Non-Mixing in Attention
Sep 8, 2026Raito Kiya, Satoki Ohashi, Kosuke Sato +6Self-Attention Attention Sinks
1Tohoku University · 2SOKENDAI · 3NINJAL +3
Linear Algebra Foundations of Efficient Attention: A Phase Reversal in Rank Collapse Under SVD Compression
Sep 6, 2026Anjaneya Teja Sarma KalvakolanuLow-Rank Compression Self-Attention
Disentangling Attention in Deep Operator Learning: A Controlled Study of Data-Driven and Physics-Informed Architectures
Sep 3, 2026Amar Alem Koric, Qibang Liu, Seid KoricPDE Operator Learning Deep Operator Networks
Department of Mechanical Engineering, Stanford University, Stanford, CA, 94305, USA · National Center for Supercomputing Applications, University of Illinois at Urbana-Champaign, 1205 W. Clark St., Urbana, IL, 61801, USA · The Grainger College of Engineering, Mechanical Science and Engineering, University of Illinois at Urbana-Champaign, 1206 W. Green St., Urbana, IL, 61801, USA
The Head Complexity of Boolean Functions in Single-Layer Attention
Sep 3, 2026Rajmohan Rajaraman, Ravi Sundaram, Amanuel TesfayeSelf-Attention Attention Mechanisms
Northeastern University.
Multi-Head Self Attention is a Parameter Identification Mechanism
Sep 1, 2026W. Ross MorrowSoftmax Attention Parameter Identifiability
All You Need Is Non-Commutative Words
Aug 29, 2026Carla M. Quispe Flores, Stanley Salvatierra, Renan CabreraCompositional Representation Learning Self-Attention
Department of Physics, Colorado School of Mines, Golden, Colorado 80401, USA · Lookia MX, OR 0401, BO · Yardley, PA 19067, USA
Mahalanobis-Based Multi-Head Attention for Complex State Propagation
Aug 25, 2026Xiaohe LiAttention Mechanisms Multi-Head Attention
GuangDong Police College
TANGO: Treating Tokens as Operators
Aug 22, 2026Joshua NunleyTransformer FFNs Transformer
Department of Informatics Luddy School of Informatics, Computing, and Engineering Cognitive Science Program Indiana University Bloomington
From Attention Masks to Inert Zero-Vector Tokens: OAttention and O-Closure for Token Dynamics
Aug 21, 2026Heyang GongAttention Mechanisms Self-Attention
Ask Self, Ask Others: Relation Is All You Need
Aug 20, 2026Yuting Ge, Pengju Yang, Mingkai NieSelf-Attention Decoder-Only Language Models
Department of Electrical Engineering City University of Hong Kong · Jilin University · National University of Singapore
Residual Dominance as a Structural Account of Last-Item Reliance in Causal Self-Attention Recommenders
Aug 14, 2026Keito Kozaki, Keigo Sakurai, Ren Togo +2Sequential Recommendation Self-Attention
Hokkaido University Sapporo, Hokkaido, Japan
SparSTAR: Sparse Attention for SpaceTime AutoRegressive Video Synthesis
Aug 11, 2026Jongbeom Lee, Hyunwoo Yu, Jincheol Yang +2Image-to-Video Generation Self-Attention
Sogang University Seoul, Republic of Korea
Power law graph attention: exact generalization of scaled dot-product attention, empirical collapse at inference
Aug 10, 2026Burc GokdenGraph Attention Networks LLM Inference
Fromthesky Research Labs LLC, Oregon, USA
Why Post-Norm Transformers Collapse: Attention Amplification and Gradient Repair Failure
Aug 10, 2026Xingjian Wang, Qingyu Han, Xiaodong Luo +1Transformer Self-Attention
The Chinese University of Hong Kong, Shenzhen · Shenzhen Research Institute of Big Data

Oct 26, 2026J/K move · Enter open · S save

ACSAN: All-Convolutional Self-Attention Network Integrating Convolutional Operations with Vision Transformer

Zimian Wei, Tianqi Zhang, Hengyue Pan

College of Computer Science and Technology, National University of Defense Technology, 109 Deya Road, Changsha, 410073, Hunan, China

Hybrid CNN-Transformer Architectures Efficient ViTs Self-Attention Image Classification Multi-Scale Attention

Open paper AskPDF arXiv

In the past few years, Vision Transformer (ViT) has achieved excellent performance in the field of computer vision. However, there still some problems remain. For instance, ViTs heavily rely on large-scale training data since they simply contain a large number of trainable parameters, which slows down their learning efficiency and limits their applicability in resource-constrained scenarios. In this paper, we propose a novel deep learning model, namely All-Convolutional Self-Attention Network (ACSAN), which applies convolution operations to replace fully-connected matrices multiplications in the ViT framework. To implement ACSAN, we firstly design Dynamic Convolutional Self-Attention (DCSA) as the basic module, which employs multi-scale dilated convolutions to generate Query, Key, and Value. We defined spatial self-attention and channel self-attention to generate attention features respectively. The two attention features are finally merged as the output of the DCSA module. Based on DCSA, we then construct the basic ACSAN block. Comparing with ViT, we use DCSA to replace the regular self-attention operations, and use convolution layers to replace MLPs. To evaluate the performance of the proposed method, we train ACSAN on three widely-used image classification databases, i.e. CIFAR-10, CIFAR-100 and ImageNet-1k, from scratch. Experimental results show that ACSAN achieves excellent classification accuracy comparing with standard ViT with comparable or fewer parameters, as well as other state-of-the-art methods. Ablation studies verify that each component contributes positively to the final performance.