Attention Routing

Follow this topic Ask about this topic

Latest papers

19 papers6 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Anisotropic Representation Dynamics in Transformers
Oct 16, 2026Nguyen Do, My T. ThaiTransformer Self-Attention
Department of Computer & Information Science & Engineering, University of Florida, Gainesville, FL, USA
Velox: Spatial Memory Makes World Models Efficient
Oct 15, 2026Qiuhong Shen, Shizun Wang, Xinchao WangAttention Routing Video World Models
National University of Singapore
Evaluating Trajectory Features for Routing Final-Layer Attention
Oct 7, 2026Yupeng YaoAttention Routing LLM Routing
SMat-Attention: Structured Long-Context Sequence Modeling
Sep 28, 2026Emile Anand, Abdullah Ateyeh, Archer Wang +1Long-Context Modeling Linear Attention
Massachusetts Institute of Technology
StarWM: Self-Supervised Trained Attention Routing for Robust World Models
Sep 25, 2026Zeqiang Zhang, Fabian Wurzberger, Maximilian Otte +4Attention Routing World Model Learning
Ulm University · German Aerospace Center (DLR)
Attention as a Routing Graph: Live Circuit Extraction from a Single Forward Pass
Sep 21, 2026Ash Manvi, Samreena TajreenCircuit Discovery Attention Mechanisms
Aquin Labs
Block-Sparse Attention with Semantic-Geometric Decoupled Routing
Sep 19, 2026Xinwei Long, Weigao Sun, Weibo Gao +6Attention Routing Sparse Attention
Department of Electronic Engineering, Tsinghua University, Beijing, China · Intern at Tongyi Lab, Alibaba Group · Tongyi Lab, Alibaba Group, Shanghai, China +2
Attention-Aware Routing: Coupling Routing and Attention in MoEs
Sep 17, 2026Despoina Kosmopoulou, Anastasios Tsetsilas, Efthymios Georgiou +3LLM Routing Mixture-of-Experts Language Models
National Technical University of Athens · University of Bern · Amazon AGI +1
SpotAttention: Plug-In Block-Sparse Routing for Pretrained Long-Context Transformers
Jun 22, 2026Huzama Ahmad, Se-Young YunSparse Attention Block-Sparse Attention
KAIST
All Routes Lead to Collapse
Jun 21, 2026K. R. BalasubramanianRepresentation Collapse Attention Sinks
Independent Research
Low-Rank Attention Residuals
Jun 19, 2026Jonathan SuLow-Rank Attention Decoder-Only Language Models
Independent Researcher
When Does Routing Become Interpretable? Causal Probes on Block Attention Residuals
Jun 11, 2026Aydin JavadovTransformer Interpretability Attention Residual Connections
ETH Zurich
Meta-Attention: Bayesian Per-Token Routing for Efficient Transformer Inference
May 27, 2026Alan FerrariEfficient Transformer Inference Self-Attention
Knowledge Lab AG Zürich, Switzerland
Delta Attention Residuals
May 13, 2026Cheng Luo, Zefan Cai, Junjie HuCross-Layer Attention Attention Residual Connections
Independent Researcher · University of Wisconsin–Madison
The Routing and Filtering Structure of Attention
May 12, 2026Shafayeth Jamil, Rehan KapadiaMulti-Head Attention Self-Attention
Ming Hsieh Department of Electrical and Computer Engineering University of Southern California Los Angeles, CA 90089
When Does Content-Based Routing Work? Representation Requirements for Selective Attention in Hybrid Sequence Models
Mar 22, 2026Abhinaba BasuAttention Routing LLM Routing
National Institute of Electronics and Information Technology (NIELIT)
Directional Routing in Transformers
Mar 16, 2026Kevin TaylorAttention Routing Transformer
Understanding the Physics of Key-Value Cache Compression for LLMs through Attention Dynamics
Mar 2, 2026Samhruth Ananthanarayanan, Ayan Sengupta, Tanmoy ChakrabortyAttention Routing KV-Cache Compression
IIT Delhi, India
Routing Absorption in Sparse Attention: Why Random Gates Are Hard to Beat
Feb 11, 2026Keston Aquino-MichaelsTransformer Self-Attention
No Way Labs

Oct 16, 2026J/K move · Enter open · S save

Anisotropic Representation Dynamics in Transformers

Nguyen Do, My T. Thai

Department of Computer & Information Science & Engineering, University of Florida, Gainesville, FL, USA

Transformer Self-Attention Attention Routing Gated Attention

Open paper AskPDF arXiv

Small differences between internal representations can shrink or grow as they pass through a Transformer layer, affecting how information is preserved and how sensitive the model is to perturbations. Yet it remains unclear when changes in attention weights can turn a representation difference that would otherwise shrink into one that grows. We develop a Mixture-Gated Theory of Anisotropic Representation Dynamics that separates self-attention into value transport, query gating, and key gating. Our analysis reveals how the geometry of attended representations constrains gating and how interactions between these mechanisms can reverse contraction induced by value transport. We formalize this phenomenon as gating-induced regime switching, where a representation difference contracts under fixed attention routing but expands under full attention. We derive a necessary and sufficient spectral criterion that certifies the existence or absence of switching and identifies switching directions without exhaustive search. Beyond existence, our theory characterizes maximum relative switching strength, certifies the severity of contraction that gating can overcome, and determines the largest dimension of a subspace in which every nonzero direction switches. Experiments across four Transformer architectures validate these predictions through analytical and finite-forward comparisons, revealing distinct and joint query--key gating mechanisms and nontrivial switching subspaces. Our results provide a mechanistic and spectral understanding of how attention routing can alter local representation dynamics.