Transformer Attention

Latest papers 306

41 papers in the last four weeks, up 86% on the four weeks before. 0.2% of all new papers.

Learning Source Structure: Bayesian Prediction with Transformers
Oct 13, 2026Meir Feder, Ruediger UrbankeTransformer Sequence Modeling
School of Electrical and Computer Engineering, Tel Aviv University, Tel Aviv, Israel · School of Computer and Communication Sciences, EPFL, Lausanne, Switzerland
The Effect of Layer Ordering in Hybrid Architectures
Oct 13, 2026John Cooper, Mingchen Ma, Jiaqi Mo +4Hybrid Attention Transformer Attention
Department of Computer Sciences, University of Wisconsin-Madison · Department of Computer Sciences, John Hopkins University
Same Function, Different Trajectory: No Standard Optimizer Is Equivariant to Transformer Reparameterization
Oct 13, 2026Bum Jun KimDeep Learning Optimization Transformer
The University of Tokyo
On the Statistical Mechanics of Attention
Oct 12, 2026Jerry Yao-Chieh Hu, Yu-An Chen, Maojiang Su +1Transformer Attention Attention Mechanisms
Center for Foundation Models and Generative AI, Northwestern University Department of Computer Science, Northwestern University · Department of Computer Science, College of Connected Computing, Vanderbilt University · Department of Physics, National Taiwan University +1
CacheLink: Functional KV Transport for Zero-Prefill Cross-Model Context Handoff
Oct 11, 2026Haomiao Qiu, Yining Tang, Zijia Song +3LLM Serving KV-Cache Reuse
Harbin Institute of Technology, Shenzhen · Great Bay University · Zhejiang University
HodgePhaseNet: Discrete Hodge Decomposition of Semantic Flows in Transformer Attention
Oct 10, 2026Kiyotaka Kasubuchi, Kazuo FukiyaTransformer Attention Hodge Decomposition
Learnable Low-Rank Polynomial Sketch for Effective Linear Attention
Oct 9, 2026Zongyue Qin, Haoran Deng, Jason Cong +1Transformer Attention Linear Attention
Department of Computer Science, University of California Los Angeles, CA, USA
REFLECT: Role-Aware Detection of Sentiment, Emotion, Politeness, and Formality
Oct 9, 2026Sangpil Youm, Annabel Stocks Natalias, Hayden Heninger +4Emotion Recognition in Conversations Sentiment Analysis
University of Florida
Orthros: Phase-Aware Heterogeneous Attention for Efficient Transformers
Oct 9, 2026Yuhong Chou, Zehao Liu, Yuqi Pan +4Transformer Attention Attention Mechanisms
The Hong Kong Polytechnic University · University of Chinese Academy of Sciences · Independent Researcher +1
Neural expectation-maximization and meta-models
Oct 8, 2026Zhuolin FuTransformer Transformer Attention
Kronecker Product Attention: Memory-Efficient KV Compression via Structured Factorization
Oct 8, 2026Bahareh Nikpour, Hossein Rajabzadeh, Maryam Dialameh +2Transformer Attention KV-Cache Compression
Huawei Technologies, Toronto, Canada · University of Waterloo, Waterloo, Canada · Huawei Noah’s Ark Lab
An all-transistor physical-learning transformer can be trained to perform in-context learning
Oct 8, 2026Michael C. Abbott, Benjamin B. MachtaTransformer Attention Equilibrium Propagation
Yale Physics & QBio
4-Tensor Attention Model for Semantic Physical Reality
Oct 8, 2026Jongwook Kim, Sangheon YunAttention Mechanisms Transformer Attention
IndigoWave, Center for Quantum Spacetime, Sogang University, 35 Baekbeom-ro, Mapo-gu, Seoul 04107, Republic of Korea
Emergent Inverse-Depth Scaling From Nonlinearity In Attention
Oct 8, 2026Zirui Peng, Yizhou Liu, Ziming Liu +1Neural Scaling Laws Scaling Laws
Department of Physics, Peking University · Department of Physics, Massachusetts Institute of Technology, Cambridge, MA · College of AI, Tsinghua University
Force without transmission: a depth-induced rank collapse that no loss on the representation reopens
Oct 7, 2026Martin Hofmann, Patrick MäderTransformer Attention
Data-intensive Systems and Visualization Group (dAI.SY), Technische Universität Ilmenau, Max-Planck-Ring 14, 98693 Ilmenau, Thuringia, Germany · German Centre for Integrative Biodiversity Research (iDiv) Halle–Jena–Leipzig, Deutscher Platz 5e, 04103 Leipzig, Saxony, Germany · Faculty of Biological Sciences, Friedrich Schiller University, Fürstengraben 1, 07745 Jena, Thuringia, Germany
Fully Interpretable Minimal Transformers: From Geometry to Algorithm
Oct 7, 2026Raneem Mahajne, Toviah MoldwinTransformer Interpretability Mechanistic Interpretability
Edmond and Lily Safra Center for Brain Sciences, The Hebrew University of Jerusalem
Spatial Induction Heads: In-Context Learning of Multidimensional Cellular Automata
Oct 6, 2026Kimia Kazemian, Menghan Xu, John Thickstun +1In-Context Learning Induction Heads
Cornell University
A Self-Pruning Transformer: Extreme KV-Cache Compression with Universal Attention
Oct 6, 2026Davis Wertheimer, Haochen Shen, Ahan Gupta +6KV-Cache Compression Transformer Attention
IBM Research, USA · SSAIL Lab, University of Illinois Urbana-Champaign
Random Feature Gaussian Process Attention: Linear-Time Probabilistic Attention with Calibrated Uncertainty
Oct 6, 2026Amir Mohammad Mahfoozi, Zi Yang, Ying Li +1Transformer Attention Attention Mechanisms
Department of Computer Engineering, Sharif University of Technology · School of Computing and Data Science, The University of Hong Kong
AttSVD:Prompt-Adaptive Low-Rank KV Cache Compression via Attention-Guided SVD
Oct 3, 2026Sara Abdali, Jongwoo Ko, Pashmina CameronKV-Cache Compression Low-Rank Compression
Microsoft Applied Sciences Group (ASG)
Shared Weights, Selected Computations: How Looped Transformers Route What Each Loop Does
Sep 30, 2026Jiaju Wu, Yi Hu, Muhan ZhangTransformer Interpretability Transformer Attention
Institute for Artificial Intelligence, Peking University. · School of Mathematical Sciences, Peking University.
Pattern Formation in Transformers
Sep 29, 2026Erkan Turan, Gaspard Abel, Maks OvsjanikovTransformer Transformer Attention
LIX, École Polytechnique, IP Paris, France · Centre Borelli, ENS Paris-Saclay, Université Paris-Saclay, France · Centre d’Analyse et de Mathématique Sociales, EHESS, CNRS, 75006 Paris, France
Pixel-Level Transformers in Remote Sensing: A Canopy Height Case Study
Sep 29, 2026Sven Ligensa, Jan Pauls, Karsten Schrödter +2Remote Sensing Image Understanding Remote Sensing
University of Münster Münster, Germany · Laboratoire des Sciences du Climat et de l’Environnement Paris, France
Learned Queries and Keys Are All You Need: Replacing the Value Projection with Structured Transforms
Sep 29, 2026Ene Meco, Emadeldeen Hamdan, A. Enis CetinTransformer Attention Efficient ViTs
University of Illinois Chicago, Chicago, IL 60607
Theory on Attention Dynamics for Out-of-Distribution In-Context Learning
Sep 29, 2026Junze Deng, Daouda Sow, Sen Lin +1In-Context Learning OOD Generalization
Ohio State University Ohio, USA · University of Houston Houston, USA
How to Loop MoE: Flatten the Experts, Untie the Attention
Sep 28, 2026Shouren Wang, Chuang Ma, Mohsen Hariri +6Mixture-of-Experts Language Models Sparse Mixture-of-Experts
Case Western Reserve University · Kyoto University · NII LLMC
Which the Eye Fears: Writing with Read-Blindness Explains Massive Activations in Transformers
Sep 28, 2026Swagatam Mukhopadhyay, Vishal Vivek Saley, Vraj Parikh +1Transformer Interpretability Transformer FFNs
PsiDagger · IIT Delhi
Attention Graphons: A Graph Limit Perspective on Graph Transformers
Sep 28, 2026Caio F. Deberaldini Netto, Moshe Eliasof, Luana RuizTransformer Attention Graph Transformer
Johns Hopkins University Baltimore, MD · University of Cambridge Cambridge, UK
Dual-Stream Simultaneous Translation via 2D Grid Attention
Sep 28, 2026Yu Pu, Wei-Qiang ZhangTransformer Attention Attention Mechanisms
Department of Electronic Engineering, Tsinghua University, Beijing 100084, China · Institute for Embodied Intelligence and Robotics, Tsinghua University, Beijing 100084, China
Query Expansion and Key Specialization in Transformer Attention Geometry
Sep 28, 2026Vidit Gupta, Siddhesh Nadkarni, Mihik Chaudhari +2Neural Representation Geometry Transformer Attention
Dwarkadas J. Sanghvi College of Engineering Mumbai, India

Oct 13, 2026J/K move · Enter open · S save

Learning Source Structure: Bayesian Prediction with Transformers

Meir Feder, Ruediger Urbanke

School of Electrical and Computer Engineering, Tel Aviv University, Tel Aviv, Israel · School of Computer and Communication Sciences, EPFL, Lausanne, Switzerland

Transformer Sequence Modeling Transformer Attention Transformer FFNs Transformer Interpretability

Open paper AskPDF arXiv

A transformer trained on sequences whose source parameters are drawn afresh each time can learn only what the sequences share: the source family and its prior. The Bayesian predictor that knows both, which we call informed Bayes, is the unique minimizer of the training loss and is computable from the family alone; we use it as a quantitative, falsifiable reference. Where earlier work compared transformers with Bayes on sources whose contexts can be learned independently, we build Markov sources with up to 64 symbols in which observations in one context inform prediction in another: product sources with known or hidden encodings, transition rows sharing an unknown base law and depth, and a mixture of three families. Our main result is that informed Bayes predicts, from the family and prior alone, how a trained transformer behaves in four tests that go beyond the learning curve: the loss that remains under a source outside the training family (predicted 2.237 bits, measured 2.309), the doubling of the extra loss when two independent components change rather than one (0.345 predicted, 0.350 measured), the benefit of other contexts when a token appears for the first time, and the depth at which copying an earlier continuation starts to pay. On the learning curves themselves, small transformers match informed Bayes when the shared structure is visible in the tokens, and hidden structure needs more layers and training, with gaps below 0.01 bits per token for the largest models. We explain why the imitation is possible and where it must fail: informed Bayes decomposes into sufficient statistics that attention selects and accumulates and an estimator, fixed by the prior, that the feed-forward network learns; training pays the complexity of informed Bayes under the architecture's prior, and finite-memory universal prediction bounds how closely a bounded network can reach it.