Gated Attention

Ask about this topic

Overview

Gated attention studies how learned controls can regulate information flow in Transformers and related models. Common approaches use multiplicative gates on attention outputs, values, heads, or tokens, while gated linear and recurrent methods use input-dependent updates or decay to handle long sequences efficiently. These methods aim to improve selectivity, robustness, and computational cost, with applications spanning language modeling, long-context retrieval, vision, graphs, and multimodal tasks. Results vary by task and scale, and an open challenge is separating gains from gating itself from model adaptation, while checking whether soft learned gates deliver reliable benefits when converted to hard pruning or sparse inference.

Latest papers

CardsList

30 papers8 papers in the last four weeks, up 60% on the four weeks before. 0.0% of all new papers.

  1. A Mixture-Gated Theory of Anisotropic Representation Dynamics in Transformers

    Oct 16, 2026Nguyen Do, My T. ThaiTransformerSelf-Attention

    Department of Computer & Information Science & Engineering, University of Florida, Gainesville, FL, USA

  2. Why don't Graph Transformers attend to the same token?

    Oct 14, 2026Alexandre BismuthAttention SinksAttention Mechanisms

    BOLD Lab University of Oxford

  3. Harnessing SPAD Noise: Test-Time Gated Attention for Indoor Depth Estimation

    Oct 11, 2026Yiquan GaoDepth EstimationMetric Depth Estimation

    Heriot-Watt University

  4. Beyond Gate Weights: Decision-Level Interpretation of Radiomics-Image Feature Fusion for RCC Classification

    Oct 9, 2026Yuan Liang, Sourav Bhattacharjee, Abraham CampbellExplainable Medical Image AnalysisRadiomics

    Research Ireland Centre for Research Training in Machine Learning · School of Computer Science, University College Dublin, Dublin, Ireland · School of Veterinary Medicine, University College Dublin, Dublin, Ireland

  5. Diagnosis-Conditioned Spatial Gating and Decoder-Level Supervised Contrastive Learning for Radiology Report Generation

    Oct 3, 2026Md Mustafizur Rahman, Mylene C. Q. FariasRadiology Report GenerationGated Attention

    Department of Computer Science, Texas State University, San Marcos, TX, USA

  6. How Local Mixing Encodes Relative Position in Global NoPE Attention

    Sep 29, 2026Cutter Dawes, Nick Alonso, Tom Figliolia +1Gated AttentionPositional Encoding

    Zyphra Research

  7. MS-GLA: Multi-Scale Gated Linear Attention for Addressing Representational Bottlenecks via Multi-Temporal Resolution

    Sep 28, 2026Prasoon Dev, Anirudh Sankar, Vasudeva VarmaLong-Context Language ModelingGated Attention

    Language Technologies Research Center International Institute of Information Technology Hyderabad Hyderabad, Telangana, India

  8. TSGate: Timestep-Aware Gated Attention for Diffusion Transformers

    Sep 28, 2026Boyu Zhang, Yifan Liu, Shuxia Lin +3Diffusion TransformerGated Attention

    Alibaba Token Hub, Alibaba Group · Southeast University

  9. GTR: Gated Token Recurrence for Efficient Dense Prediction

    Sep 22, 2026Zhe Feng, Longfei Liu, Wei Liu +7Efficient Neural Network InferenceGated Attention

    Didi International Business Group · Intellindust AI Lab · Institute of Automation, Chinese Academy of Sciences +2

  10. Rethinking Vision Architectures with Gated Linear Attention and KAN

    Sep 18, 2026Ali Mehizel, Oussama KhaldiVisual Representation LearningVision Transformer

    Department of Applied Statistics, ENSSEA, Kolea, Tipaza, Algeria

  11. Abstention and Noise Filtering: Two Missing Primitives of Softmax Attention

    Sep 18, 2026Richard Zhe WangSoftmax AttentionGated Attention

    St. John Fisher University, Rochester, New York, USA.

  12. Do New Attention Mechanisms Actually Fix Attention Sinks at Million-Token Context?

    Sep 8, 2026Sara Rizwan, Samaanah Abdus Salam, Mohammed Mudassir UddinLong-Context Language Model EvaluationLong-Context Language Modeling

    Shadan Women’s College of Engineering and Technology Hyderabad, Telangana, India

  13. Liquid Gated Attention

    Aug 31, 2026Yiheng Jiang, Yuanbo Xu, Yongjian YangGated AttentionTime Series Modeling

    Mobile Intelligent Computing Laboratory, College of Computer Science and Technology, Jilin University, China

  14. Sigmoid Attention as a Better Substrate for Learned KV Cache Eviction

    Aug 24, 2026Isaac, LiKV-Cache EvictionGated Attention

    Rucheng · University of Pittsburgh, Pittsburgh, PA, USA.

  15. Hybrid Gated Attention

    Aug 12, 2026Zekun Zhou, Ruobing Xie, Lanrui Wang +1Low-Rank Matrix DecompositionGated Attention

    Tencent Hunyuan · Peking University

  16. WQ-Fusion: Dynamic Gated Attention for Cross-Domain Audio Representation

    Jun 25, 2026Mingda Lin, Lei Ding, Xinyue Zhou +6Audio Representation LearningGated Attention

    School of Electronic Information, Wuhan University, Wuhan, Hubei, China · Tencent AI Lab Seattle, Seattle, USA · CIAIC, Northwestern Polytechnical University, Xi’an, Shaanxi, China +1

  17. Gated Graph Attention Networks with Learnable Temperature

    May 28, 2026Zhongtian Ma, Hao Wu, Yexin Zhang +2Graph Attention NetworksGraph Neural Networks

    School of Cybersecurity, Northwestern Polytechnical University, Xi’an 710072, China · Shanghai Artificial Intelligence Laboratory, Shanghai 200232, China

  18. Energy-Gated Attention and Wavelet Positional Encoding: Complementary Inductive Biases for Transformer Attention

    May 25, 2026Athanasios ZerisSelf-AttentionGated Attention

    Independent Researcher, Athens, Greece.

  19. Energy-Gated Attention: Spectral Salience as an Inductive Bias for Transformer Attention

    May 21, 2026Athanasios ZerisSelf-AttentionGated Attention

    Independent Researcher, Athens, Greece.

  20. Provably Shorter Scratchpads in Hybrid DeltaNet-Attention Decoders

    May 15, 2026Tomasz SteiferGated AttentionRecurrent Transformers

    Centre for Credible AI Warsaw University of Technology Rektorska 4, Warszawa 00-614, Poland

  21. Budgeted Attention Allocation: Cost-Conditioned Compute Control for Efficient Transformers

    May 7, 2026Amrit NidhiEfficient Transformer InferenceSelf-Attention

    Independent Researcher

  22. GateMOT: Q-Gated Attention for Dense Object Tracking

    Apr 29, 2026Mingjin Lv, Zelin Liu, Feifei Shao +4Visual Object TrackingGated Attention

    Huazhong University of Science and Technology, Wuhan, China · Zhejiang University, Hangzhou, China · La Trobe University, Melbourne, Australia

  23. Tool Attention Is All You Need: Dynamic Tool Gating and Lazy Schema Loading for Eliminating the MCP/Tools Tax in Scalable Agentic Workflows

    Apr 23, 2026Anuj Sadani, Deepak KumarGated AttentionLLM Tool Use

  24. Capacity-Controlled Global Attention for Graph Transformers

    Apr 19, 2026Yang Liu, Dongxin Guo, Tom Zheng +3Gated AttentionTransformer Attention

    Brain Investing Limited Hong Kong, China · The University of Hong Kong Hong Kong, China · Stellaris AI Limited Hong Kong, China

  25. Gating Enables Curvature: A Geometric Expressivity Gap in Attention

    Apr 16, 2026Satwik Bathula, Anand A. JoshiInformation GeometryNeural Representation Geometry

    Department of Electrical and Computer Engineering University of Southern California

  26. Learning to Adapt: In-Context Learning Beyond Stationarity

    Apr 13, 2026Zhen Qin, Jiachen Jiang, Zhihui ZhuIn-Context LearningGated Attention

    Michigan Institute for Computational Discovery and Engineering, Department of Electrical Engineering and Computer Science, Department of Statistics, University of Michigan · Department of Computer Science and Engineering, The Ohio State University

  27. Gated-SwinRMT: Unifying Swin Windowed Attention with Retentive Manhattan Decay via Input-Dependent Gating

    Apr 7, 2026Dipan Maity, Suman Mondal, Arindam RoyVision TransformerGated Attention

    Student Kolkata, West Bengal, India · Department of Computer Science Yogoda Satsanga Palpara Mahavidyalaya West Bengal, India · Department of Computer Science & Application Prabhat Kumar College Contai West Bengal, India

  28. The Model Knows Which Tokens Matter: Automatic Token Selection via Noise Gating

    Mar 7, 2026Landi He, Xiaoyu Yang, Lijian XuEfficient VLM InferenceVisual Token Pruning

    Shenzhen University of Advanced Technology

  29. Gated Differential Linear Attention: A Linear-Time Decoder for High-Fidelity Medical Segmentation

    Mar 3, 2026Hongbo Zheng, Afshin Bozorgpour, Dorit Merhof +1Medical Image SegmentationLinear Attention

    University of Illinois Urbana-Champaign · University of Regensburg

  30. Routing Absorption in Sparse Attention: Why Random Gates Are Hard to Beat

    Feb 11, 2026Keston Aquino-MichaelsTransformerSelf-Attention

    No Way Labs