Gated Attention

Follow this topic Ask about this topic

Momentum

7 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 20

All topics

How Local Mixing Encodes Relative Position in Global NoPE Attention
Sep 29, 2026Cutter Dawes, Nick Alonso, Tom Figliolia +1Gated Attention Positional Encoding
Zyphra Research
MS-GLA: Multi-Scale Gated Linear Attention for Addressing Representational Bottlenecks via Multi-Temporal Resolution
Sep 28, 2026Prasoon Dev, Anirudh Sankar, Vasudeva VarmaLong-Context Language Modeling Gated Attention
Language Technologies Research Center International Institute of Information Technology Hyderabad Hyderabad, Telangana, India
TSGate: Timestep-Aware Gated Attention for Diffusion Transformers
Sep 28, 2026Boyu Zhang, Yifan Liu, Shuxia Lin +3Diffusion Transformer Gated Attention
Alibaba Token Hub, Alibaba Group · Southeast University
GTR: Gated Token Recurrence for Efficient Dense Prediction
Sep 22, 2026Zhe Feng, Longfei Liu, Wei Liu +7Efficient Neural Network Inference Gated Attention
Didi International Business Group · Intellindust AI Lab · Institute of Automation, Chinese Academy of Sciences +2
Rethinking Vision Architectures with Gated Linear Attention and KAN
Sep 18, 2026Ali Mehizel, Oussama KhaldiVisual Representation Learning Vision Transformer
Department of Applied Statistics, ENSSEA, Kolea, Tipaza, Algeria
Abstention and Noise Filtering: Two Missing Primitives of Softmax Attention
Sep 18, 2026Richard Zhe WangSoftmax Attention Gated Attention
St. John Fisher University, Rochester, New York, USA.
Do New Attention Mechanisms Actually Fix Attention Sinks at Million-Token Context?
Sep 8, 2026Sara Rizwan, Samaanah Abdus SalamLong-Context Language Model Evaluation Long-Context Language Modeling
Shadan Women’s College of Engineering and Technology Hyderabad, Telangana, India
Liquid Gated Attention
Aug 31, 2026Yiheng Jiang, Yuanbo Xu, Yongjian YangGated Attention Linear Attention
Mobile Intelligent Computing Laboratory, College of Computer Science and Technology, Jilin University, China
Hybrid Gated Attention
Aug 12, 2026Zekun Zhou, Ruobing Xie, Lanrui Wang +1Low-Rank Matrix Decomposition Gated Attention
Tencent Hunyuan · Peking University
WQ-Fusion: Dynamic Gated Attention for Cross-Domain Audio Representation
Jun 25, 2026Mingda Lin, Lei Ding, Xinyue Zhou +6Audio Representation Learning Gated Attention
School of Electronic Information, Wuhan University, Wuhan, Hubei, China · Tencent AI Lab Seattle, Seattle, USA · CIAIC, Northwestern Polytechnical University, Xi’an, Shaanxi, China +1
Gated Graph Attention Networks with Learnable Temperature
May 28, 2026Zhongtian Ma, Hao Wu, Yexin Zhang +2Graph Attention Networks Graph Neural Networks
School of Cybersecurity, Northwestern Polytechnical University, Xi’an 710072, China · Shanghai Artificial Intelligence Laboratory, Shanghai 200232, China
Energy-Gated Attention and Wavelet Positional Encoding: Complementary Inductive Biases for Transformer Attention
May 25, 2026Athanasios ZerisSelf-Attention Gated Attention
Independent Researcher, Athens, Greece.
Energy-Gated Attention: Spectral Salience as an Inductive Bias for Transformer Attention
May 21, 2026Athanasios ZerisSelf-Attention Gated Attention
Independent Researcher, Athens, Greece.
Provably Shorter Scratchpads in Hybrid DeltaNet-Attention Decoders
May 15, 2026Tomasz SteiferGated Attention Recurrent Transformers
Centre for Credible AI Warsaw University of Technology Rektorska 4, Warszawa 00-614, Poland
Budgeted Attention Allocation: Cost-Conditioned Compute Control for Efficient Transformers
May 7, 2026Amrit NidhiEfficient Transformer Inference Self-Attention
Independent Researcher
GateMOT: Q-Gated Attention for Dense Object Tracking
Apr 29, 2026Mingjin Lv, Zelin Liu, Feifei Shao +4Visual Object Tracking Gated Attention
Huazhong University of Science and Technology, Wuhan, China · Zhejiang University, Hangzhou, China · La Trobe University, Melbourne, Australia
Tool Attention Is All You Need: Dynamic Tool Gating and Lazy Schema Loading for Eliminating the MCP/Tools Tax in Scalable Agentic Workflows
Apr 23, 2026Anuj Sadani, Deepak KumarGated Attention LLM Tool Use
Capacity-Controlled Global Attention for Graph Transformers
Apr 19, 2026Yang Liu, Dongxin Guo, Tom Zheng +3Gated Attention Transformer Attention
Brain Investing Limited Hong Kong, China · The University of Hong Kong Hong Kong, China · Stellaris AI Limited Hong Kong, China
Gating Enables Curvature: A Geometric Expressivity Gap in Attention
Apr 16, 2026Satwik Bathula, Anand A. JoshiInformation Geometry Neural Representation Geometry
Department of Electrical and Computer Engineering University of Southern California
Routing Absorption in Sparse Attention: Why Random Gates Are Hard to Beat
Feb 11, 2026Keston Aquino-MichaelsTransformer Self-Attention
No Way Labs

Sep 29, 2026, cs.CLJ/K move · Enter open · S save

How Local Mixing Encodes Relative Position in Global NoPE Attention

Cutter Dawes, Nick Alonso, Tom Figliolia, Beren Millidge

Zyphra Research

Gated Attention Positional Encoding Sliding-Window Attention Hybrid Attention

Open paper AskPDF arXiv

The attention operation is naively position invariant. However, positional information is fundamental to natural language, and therefore a variety of explicit position encodings have been developed in transformer-based models, such as rotary position encoding (RoPE). Although explicit position encodings have long been assumed to be required, recent methods that interleave local mixing layers, such as sliding window attention (SWA) and gated linear attention, while not encoding position (NoPE) in global attention layers has recently been shown to be successful at scale. How and why this approach works is not well-understood. In this paper, we develop an explanation of how hybrid models of this sort can implicitly encode position at global NoPE layers. Supported by both theoretical and empirical evidence, our central argument is that SWA and gated linear attention induce a recency bias in the residual stream that propagates to, and is selected by, the global attention logits. Moreover, in contrast to the implicit position encodings found in models with only global NoPE attention, in which positional information arises solely from the causal mask, the recency bias in hybrid models can be maintained across long sequences. In addition to deepening our understanding of how hybrid models encode position, these findings may provide insights for how to encode position in a way that can extrapolate to longer sequence lengths indefinitely.

Momentum

Related topics

Latest papers 20