Self-Attention

Follow this topic Ask about this topic

Momentum

9 papers in the last four weeks, up 50% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 195

All topics

Self-Attention as a Covariance Readout: A Unified View of In-Context Learning and Repetition
May 11, 2026Haoren Xu, Guanhua FangSoftmax Attention Self-Attention
Fudan University
Remember to Forget: Gated Adaptive Positional Encoding
May 11, 2026Riccardo Ali, Alessio Borgi, Mario Severino +2Rotary Positional Embeddings Self-Attention
Department of Computer Science and Technology, University of Cambridge, United Kingdom · Department of Computer, Control and Management Engineering, Sapienza University, Italy · Department of Information Engineering, University of Padova, Italy
FocuSFT: Bilevel Optimization for Dilution-Aware Long-Context Fine-Tuning
May 11, 2026Zehua Pei, Hui-Ling Zhen, Xianzhi Yu +3Supervised Fine-Tuning Self-Attention
The Chinese University of Hong Kong · Huawei Technologies Co., Ltd
Key-Value Means: Transformers with Expandable Block-Recurrent Compressed Memory
May 11, 2026Daniel Goldstein, Eugene CheahTransformer Inference Self-Attention
Featherless AI · Eleuther AI
Nectar: Neural Estimation of Cached-Token Attention via Regression
May 10, 2026João Monteiro, Michal Klein, Pierre Ablin +1Softmax Attention Self-Attention
Apple
CALYREX: Cross-Attention LaYeR EXtended Transformers for System Prompt Anchoring
May 10, 2026Li LixingSelf-Attention Cross-Attention
Cornell University Ithaca, NY 14853
Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction
May 10, 2026Ngoc Bui, Hieu Trung Nguyen, Arman Cohan +1Self-Attention KV Caching
Department of Computer Science, Yale University · The Chinese University of Hong Kong
A Game Theoretic Free Energy Analysis of Higher Order Synergy in Attention Heads of Large Language Models
May 10, 2026Djamel BouchaffraLLM Pruning Attention Head Analysis
DAVID Lab, University of Paris-Saclay, UVSQ Campus, 78035 Versailles, France
Positional LSH: Binary Block Matrix Approximation for Attention with Linear Biases
May 10, 2026Daniel Wolfson, Tal WagnerSelf-Attention Long-Context Language Modeling
Blavatnik School of Computer Science and AI, Tel Aviv University
How LLMs Are Persuaded: A Few Attention Heads, Rerouted
May 10, 2026Xiangkun Sun, Lingkai Kong, Aoqi Zhang +2Self-Attention LLM Interpretability
Northeastern University · Harvard University · Tsinghua University +1
Attention Sinks in Diffusion Transformers: A Causal Analysis
May 10, 2026Fangzheng Wu, Brian SummaTransformer Interpretability Diffusion Transformer
Department of Computer Science, Tulane University, New Orleans, LA, USA.
Kinetic theory for Transformers and the lost-in-the-middle phenomenon
May 9, 2026Mitia Duerinckx, Borjan Geshkovski, Stefano RossiSelf-Attention Transformer Attention
Scaling Limits of Long-Context Transformers
May 8, 2026Giuseppe Bruno, Shi Chen, Zhengjiang Lin +2Softmax Attention Self-Attention
Transformers Can Implement Preconditioned Richardson Iteration for In-Context Gaussian Kernel Regression
May 8, 2026Mingsong Yan, Dongyang Li, Charles Kulick +1Softmax Attention Transformer Interpretability
Department of Mathematics, University of California, Santa Barbara, CA
Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing prevention
May 8, 2026Peter Súkeník, Cristina López Amado, Christoph H. Lampert +1Self-Attention Transformer Attention
Institute of Science and Technology (ISTA) Austria
An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inference
May 8, 2026Feiyu Yao, Zhixiong Niu, Xiaqing Li +3GPU Acceleration Self-Attention
Beijing University of Technology Beijing, China · Microsoft Research Beijing, China · Beijing Jiaotong University Beijing, China
A Marine Debris Detection Framework for Ocean Robots via Self-Attention Enhancement and Feature Interaction Optimization
May 8, 2026Yuyang Li, Jiashu Han, Yinyi Lai +2Self-Attention Small Object Detection
School of Department of Mechanical Engineering, Hohai University, Changzhou, 213200, China · Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei, 230031, China · Department of Mechanical Engineering, City University of Hong Kong, Hong Kong SAR, 999077, China
MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inference
May 8, 2026Ruijie Zhou, Fanxu Meng, Yufei Xu +4Self-Attention Long-Context Language Model Inference
Attention Transfer Is Not Universally Effective for Vision Transformers
May 8, 2026Huaiyuan Qin, Muli Yang, Gabriel James Goenawan +4Self-Attention Vision Transformer
Institute for Infocomm Research (I2R), A*STAR, Singapore · 3Shanghai Jiao Tong University · 2Sichuan University
Sparse Attention as a Range Searching Problem: Towards an Inference-Efficient Index for KV Cache
May 7, 2026Mohsen Dehghankar, Abolfazl AsudehSelf-Attention KV Caching
Department of Computer Science University of Illinois Chicago
The Structural Origin of Attention Sink: Variance Discrepancy, Super Neurons, and Dimension Disparity
May 7, 2026Siquan Li, Kaiqi Jiang, Jiacheng Sun +1Self-Attention Attention Sinks
1The Chinese University of Hong Kong, Shenzhen · 2Huawei
Transformers Efficiently Perform In-Context Logistic Regression via Normalized Gradient Descent
May 7, 2026Chenyang Zhang, Yuan CaoLogistic Regression Self-Attention
School of Computing & Data Science, The University of Hong Kong
Long Context Pre-Training with Lighthouse Attention
May 7, 2026Bowen Peng, Subho Ghosh, Jeffrey QuesnelleLanguage Model Pretraining Self-Attention
Nous Research
Don't Lose Focus: Activation Steering via Key-Orthogonal Projections
May 7, 2026Haoyan Luo, Mateo Espinosa Zarlenga, Mateja JamnikLanguage Model Steering Self-Attention
University of Cambridge · University of Oxford
Shallow Prefill, Deep Decoding: Efficient Long-Context Inference via Layer-Asymmetric KV Visibility
May 7, 2026Jungsuk Oh, Hyeseo Jeon, Hyunjune Ji +2Self-Attention Long-Context Language Model Inference
Graduate School of Data Science Seoul National University Seoul, Republic of Korea
Budgeted Attention Allocation: Cost-Conditioned Compute Control for Efficient Transformers
May 7, 2026Amrit NidhiEfficient Transformer Inference Self-Attention
Independent Researcher
Large Vision-Language Models Get Lost in Attention
May 7, 2026Gongli Xi, Ye Tian, Mengyu Yang +5Transformer FFNs Vision-Language Models
School of Cyberspace Security, Beijing University of Posts and Telecommunications, Beijing, China · State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China · School of Computer Science (National Pilot Software Engineering School), Beijing University of Posts and Telecommunications, Beijing, China +3
CuBridge: An LLM-Based Framework for Understanding and Reconstructing High-Performance Attention Kernels
May 6, 2026Xing Ma, Yangjie Zhou, Wu Sun +6GPU Kernel Optimization GPU Acceleration
Shanghai Jiao Tong University · National University of Singapore · Shanghai Qi Zhi Institute
Self-Attention as Transport: Limits of Symmetric Spectral Diagnostics
May 6, 2026Dominik Dahlem, Diego Maniloff, Mac MisiuraTransformer Interpretability Attention Head Analysis
Red Hat AI
Gradient Flow Structure and Quantitative Dynamics of Multi-Head Self-Attention
May 5, 2026Ayan PendharkarSoftmax Attention Attention Head Analysis

May 11, 2026, cs.LGJ/K move · Enter open · S save

Self-Attention as a Covariance Readout: A Unified View of In-Context Learning and Repetition

Haoren Xu, Guanhua Fang

Fudan University

Softmax Attention Self-Attention Autoregressive Language Modeling In-Context Learning Mode Collapse Text Generation

Open paper AskPDF arXiv

Large language models (LLMs) exhibit two striking and ostensibly unrelated behaviours: in-context learning (ICL) and repetitive generation. In both, the model behaves as though it had summarised the context into a population-level statistic and discarded token-level detail. We ask whether this ``summarisation and forgetting'' can be derived from the attention mechanism itself, and answer in the affirmative. Under stationary, ergodic and elliptical inputs, the softmax attention output converges almost surely to

Θ_VΣΘ_K^{\top}Θ_Q x_t

, where

Σ

is the input covariance; the long-context limit is therefore a linear readout of the input's second-order statistics. Two consequences follow. (i) For in-context linear regression, a single softmax head can implement one step of population gradient descent. Stacking such heads with residual connections iterates this update and implements multiple gradient descent steps. (ii) Propagated across an

L

-layer transformer, this readout drives the terminal hidden state at the parametric

1/t

rate to a deterministic function of the current token alone, so that autoregressive generation collapses asymptotically to a first-order Markov chain whose attracting orbits furnish a structural account of repetition and mode collapse. The two phenomena thus emerge as facets of a single covariance-readout principle.

Momentum

Related topics

Latest papers 195