Self-Attention

Follow this topic Ask about this topic

Momentum

9 papers in the last four weeks, up 50% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 195

All topics

Multi-Head Attention as Ensemble Nadaraya-Watson Estimation: Variance Reduction, Decorrelation, and Optimal Head Diversity
May 18, 2026Ernest FokouéSelf-Attention Ensemble Learning
School of Mathematics and Statistics, College of Science Rochester Institute of Technology Rochester, New York 14623, USA
DashAttention: Differentiable and Adaptive Sparse Hierarchical Attention
May 18, 2026Yuxiang Huang, Nuno M. T. Gonçalves, Federico Alvetreti +5Self-Attention Long-Context Modeling
1Tsinghua University · 2Instituto Superior Técnico, Universidade de Lisboa. · 3Instituto de Telecomunicações. +6
RAVE: Re-Allocating Visual Attention in Large Multimodal Models
May 18, 2026Xi Leng, Xinhong Ma, Ziqiang Dong +4Visual Attention Self-Attention
Qwen Business Unit of Alibaba · The Chinese University of Hong Kong, Shenzhen · Beijing Institute of Technology
InfoFlow: A Framework for Multi-Layer Transformer Analysis
May 18, 2026Penghao Yu, Haotian Jiang, Zeyu Bao +1Transformer Expressivity Transformer
Department of Mathematics National University of Singapore · Institute for Functional Intelligent Materials National University of Singapore · National University of Singapore +1
Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps
May 16, 2026Yanke Zhou, Yiduo Li, Hanlin Tang +6Self-Attention Long-Context Language Model Inference
Nanjing University · Alibaba Group
CompactAttention: Accelerating Chunked Prefill with Block-Union KV Selection
May 16, 2026Jiwon Song, Dongwon Jo, Beomseok Kang +1Self-Attention KV-Cache Management
Seoul National University
Attend Locally, Remember Linearly: Linear Attention as Cross-Frame Memory for Autoregressive Video Diffusion
May 15, 2026Kunyang Li, Mubarak Shah, Yuzhang ShangMemory-Augmented Neural Networks Video Diffusion Models
Institute of Artificial Intelligence, University of Central Florida
Grokking as Structural Inference: Transformers Need Bayesian Lottery Tickets
May 15, 2026Kai Hidajat, Solden Stoll, Joseph AnNeural Network Generalization Transformer
Department of Computer Science University of Washington Seattle, WA 98195
From Sparsity to Simplicity: Enabling Simpler Sequential Replacements via Sparse Attention Distillation
May 15, 2026Yuxin Ren, Maxwell D Collins, Miao Hu +1Self-Attention Vision Transformer
University of Arizona, Tucson AZ, USA · TetraMem, Inc., San Jose CA, USA
RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably
May 15, 2026Yufeng Du, Phillip Harris, Minyang Tian +5Rotary Positional Embeddings Self-Attention
University of Illinois at Urbana-Champaign · USA · University of Bonn +3
STS: Efficient Sparse Attention with Speculative Token Sparsity
May 15, 2026Ceyu Xu, Jiangnan Yu, Yongji Wu +1Self-Attention Long-Context Language Model Inference
The Hong Kong University of Science and Technology · UC Berkeley
DualKV: Shared-Prompt Flash Attention for Efficient RL Training with Large Rollouts and Long Contexts
May 14, 2026Jiading Gai, Shuai Zhang, Xiang Song +2GPU Acceleration Self-Attention
1Amazon Web Services · 2Google · University of Minnesota
GQLA: Group-Query Latent Attention for Hardware-Adaptive Large Language Model Decoding
May 14, 2026Fanxu MengGrouped-Query Attention GPU Acceleration
Institute for Artificial Intelligence, Peking University
MambaRain: Multi-Scale Mamba-Attention Framework for 0-3 Hour Precipitation Nowcasting
May 14, 2026Chunlei Shi, Cui Wu, Xiang Xu +10Mamba Precipitation Forecasting
School of Automation, Southeast University · Nanjing XinDa Institute of Meteorological Science and Technology · Beijing Leninainfo Technology Co., Ltd. +5
Head Forcing: Long Autoregressive Video Generation via Head Heterogeneity
May 14, 2026Jiahao Tian, Yiwei Wang, Gang Yu +1Diffusion Transformer Autoregressive Diffusion
AGI Lab, Westlake University · University of California at Merced · StepFun
Self-Pruned Key-Value Attention: Learning When to Write by Predicting Future Utility
May 13, 2026Gergely Szilvasy, Manuel Faysse, Maria Lomeli +5Self-Attention KV Caching
Meta FAIR · MICS, CentraleSupélec
Rethinking Graph Convolution for 2D-to-3D Hand Pose Lifting
May 13, 2026Chanyoung Kim, Donghyun Kim, Dong-Hyun Sim +2Graph Attention Networks Graph Representation Learning
Emory University · Yonsei University · WHATs Lab
Exact Linear Attention
May 13, 2026Weinuo OuEfficient Transformer Inference Self-Attention
Wuyi University
Pyramid Forcing: Head-Aware Pyramid KV Cache Policy for High-Quality Long Video Generation
May 13, 2026Jiayu Chen, Junbei Tang, Wenbiao Zhao +6Self-Attention Long-Horizon Video Generation
Peking University · South China University of Technology · Xinjiang University +2
When Attention Closes: How LLMs Lose the Thread in Multi-Turn Interaction
May 13, 2026Vardhan Dongre, Joseph Hsieh, Viet Dac Lai +3Self-Attention LLM Interpretability
University of Illinois Urbana-Champaign · Adobe Research
ASAP: Amortized Doubly-Stochastic Attention via Sliced Dual Projection
May 13, 2026Huy Tran, Max Milkert, David HydeTransformer Inference Self-Attention
Department of Computer Science Vanderbilt University
A Unified Framework for Critical Scaling of Inverse Temperature in Self-Attention
May 12, 2026Tomohiro Hayase, Ryo KarakidaSoftmax Attention Self-Attention
AIST
The Routing and Filtering Structure of Attention
May 12, 2026Shafayeth Jamil, Rehan KapadiaAttention Head Analysis Self-Attention
Ming Hsieh Department of Electrical and Computer Engineering University of Southern California Los Angeles, CA 90089
Lower bounds for one-layer transformers that compute parity
May 12, 2026Daniel HsuTransformer Expressivity Self-Attention
Training-Inference Consistent Segmented Execution for Long-Context LLMs
May 12, 2026Xianpeng Shang, Jiang Li, Zehua Duo +2Self-Attention Long-Context Language Modeling
College of Computer Science, Inner Mongolia University, Hohhot 010021, China · National & Local Joint Engineering Research Center of Intelligent Information Processing Technology for Mongolian, Hohhot 010021, China · Inner Mongolia Key Laboratory of Multilingual Artificial Intelligence Technology, Hohhot 010021, China +1
USEMA: a Scalable Efficient Mamba Like Attention for Medical Image Segmentation
May 11, 2026Elisha Dayag, Nhat Thanh Tran, Jack XinImage Segmentation Hybrid CNN-Transformer Architectures
University of California Irvine, Irvine CA 92697
Uniform Scaling Limits in AdamW-Trained Transformers
May 11, 2026William Gibson, Christoph ReisingerTransformer Self-Attention
Mathematical Institute · University of Oxford · Oxford, OX2 6GG, UK
RelFlexformer: Efficient Attention 3D-Transformers for Integrable Relative Positional Encodings
May 11, 2026Byeongchan Kim, Arijit Sehanobish, Avinava Dubey +23D ViTs Self-Attention
1Seoul National University · 2Independent · 3Google Research +2
Prompt-Activation Duality: Improving Activation Steering via Attention-Level Interventions
May 11, 2026Diancheng Kang, Zheyuan Liu, Ningshan Ma +3Language Model Steering Self-Attention
1Southern University of Science and Technology · University of Notre Dame · 3Massachusetts Institute of Technology
Learning Less Is More: Premature Upper-Layer Attention Specialization Hurts Language Model Pretraining
May 11, 2026Jinchang Zhu, Jindong Li, Yuwen Hao +3Language Model Pretraining Self-Attention
The Hong Kong University of Science and Technology (Guangzhou)

May 18, 2026, stat.MLJ/K move · Enter open · S save

Multi-Head Attention as Ensemble Nadaraya-Watson Estimation: Variance Reduction, Decorrelation, and Optimal Head Diversity

Ernest Fokoué

School of Mathematics and Statistics, College of Science Rochester Institute of Technology Rochester, New York 14623, USA

Self-Attention Ensemble Learning Transformer Attention Statistical Learning Theory Multi-Head Attention Variance Reduction+1

Open paper AskPDF arXiv

We develop a rigorous statistical theory of multi-head attention (MHA) as an ensemble of Nadaraya-Watson (NW) kernel regression estimators. Building on the algebraic identity between single-head softmax attention and the NW estimator, we prove that MHA is a structured ensemble of H NW estimators, each operating in a distinct learned projection subspace of the key space. We derive an explicit Bias-Variance-Covariance decomposition of the MHA mean squared error, showing that variance reduction depends not merely on the number of heads H but fundamentally on the decorrelation of head outputs. Decorrelation is governed by the principal angles between learned projection subspaces: orthogonal projections yield maximum variance reduction; aligned projections yield none. We introduce the Head Diversity Index (HDI), a computable spectral measure of inter-head decorrelation, and prove that MHA mean squared error is monotonically decreasing in HDI. This provides the first rigorous theoretical explanation for the empirically observed specialization of attention heads. Under a fixed total-dimension budget D = H * d_k, we solve the optimal head-dimension allocation problem, deriving the MSE-minimizing pair (H*, d_k*) from data distribution and regression smoothness. The solution yields a new architectural scaling law: the optimal per-head dimension grows logarithmically with training set size, while the optimal number of heads grows nearly linearly with the total budget D. Our framework unifies three strands of prior work: the NW theory of single-head attention, the general weighting theory for ensemble learning, and the decorrelation-variance-reduction isomorphism between biological and computational ensembles. Multi-head attention is the Transformer's instantiation of a universal principle: identical agents plus diversity-enforcing mechanisms yields emergent optimality.

Momentum

Related topics

Latest papers 195