Overview

Gated attention studies how learned controls can regulate information flow in Transformers and related models. Common approaches use multiplicative gates on attention outputs, values, heads, or tokens, while gated linear and recurrent methods use input-dependent updates or decay to handle long sequences efficiently. These methods aim to improve selectivity, robustness, and computational cost, with applications spanning language modeling, long-context retrieval, vision, graphs, and multimodal tasks. Results vary by task and scale, and an open challenge is separating gains from gating itself from model adaptation, while checking whether soft learned gates deliver reliable benefits when converted to hard pruning or sparse inference.

Latest papers

CardsList

37 papers4 papers in the last four weeks, down 56% on the four weeks before. 0.0% of all new papers.

  1. Muir: Linear Attention with Sensitivity-Allocated Gating and Matched-Filter Readout

    Nov 5, 2026Tushaar Gangavarapu, Gil Tabak, Clemens JS SchaeferDelta-Rule Linear AttentionSequence Modeling

  2. The Gating Mechanism of KV Cache Eviction

    Nov 5, 2026Hao Gu, Xintong Yang, Zhiwei Li +10KV-Cache EvictionKV-Cache Compression

  3. GatedV: Self-Gated Values for Transformer Language Models

    Nov 1, 2026Li-Ming Yu, Hongyu ZhouTransformer AttentionGated Attention

  4. Zero-Gated Cross-Attention

    Oct 31, 2026Basavaraj R M, Manav VermaCross-AttentionGated Attention

  5. A Mixture-Gated Theory of Anisotropic Representation Dynamics in Transformers

    Oct 16, 2026Nguyen Do, My T. ThaiTransformerSelf-Attention

  6. Why don't Graph Transformers attend to the same token?

    Oct 14, 2026Alexandre BismuthAttention SinksAttention Mechanisms

  7. Beyond Gate Weights: Decision-Level Interpretation of Radiomics-Image Feature Fusion for RCC Classification

    Oct 9, 2026Yuan Liang, Sourav Bhattacharjee, Abraham CampbellExplainable Medical Image AnalysisRadiomics

  8. Diagnosis-Conditioned Spatial Gating and Decoder-Level Supervised Contrastive Learning for Radiology Report Generation

    Oct 3, 2026Md Mustafizur Rahman, Mylene C. Q. FariasRadiology Report GenerationGated Attention

  9. How Local Mixing Encodes Relative Position in Global NoPE Attention

    Sep 29, 2026Cutter Dawes, Nick Alonso, Tom Figliolia +1Gated AttentionPositional Encoding

  10. MS-GLA: Multi-Scale Gated Linear Attention for Addressing Representational Bottlenecks via Multi-Temporal Resolution

    Sep 28, 2026Prasoon Dev, Anirudh Sankar, Vasudeva VarmaLong-Context Language ModelingGated Attention

  11. TSGate: Timestep-Aware Gated Attention for Diffusion Transformers

    Sep 28, 2026Boyu Zhang, Yifan Liu, Shuxia Lin +3Diffusion TransformerGated Attention

  12. GTR: Gated Token Recurrence for Efficient Dense Prediction

    Sep 22, 2026Zhe Feng, Longfei Liu, Wei Liu +7Efficient Neural Network InferenceGated Attention

  13. Rethinking Vision Architectures with Gated Linear Attention and KAN

    Sep 18, 2026Ali Mehizel, Oussama KhaldiVisual Representation LearningVision Transformer

  14. Abstention and Noise Filtering: Two Missing Primitives of Softmax Attention

    Sep 18, 2026Richard Zhe WangSoftmax AttentionGated Attention

  15. Do New Attention Mechanisms Actually Fix Attention Sinks at Million-Token Context?

    Sep 8, 2026Sara Rizwan, Samaanah Abdus Salam, Mohammed Mudassir UddinLong-Context Language Model EvaluationLong-Context Language Modeling

  16. Liquid Gated Attention

    Aug 31, 2026Yiheng Jiang, Yuanbo Xu, Yongjian YangGated AttentionTime Series Modeling

  17. Sigmoid Attention as a Better Substrate for Learned KV Cache Eviction

    Aug 24, 2026Isaac, LiKV-Cache EvictionGated Attention

  18. Hybrid Gated Attention

    Aug 12, 2026Zekun Zhou, Ruobing Xie, Lanrui Wang +1Low-Rank Matrix DecompositionGated Attention

  19. WQ-Fusion: Dynamic Gated Attention for Cross-Domain Audio Representation

    Jun 25, 2026Mingda Lin, Lei Ding, Xinyue Zhou +6Audio Representation LearningGated Attention

  20. Gating and Learnable Temperature for Graph Attention Mechanisms

    May 28, 2026Zhongtian Ma, Hao Wu, Yexin Zhang +2Graph Attention NetworksGraph Neural Networks

  21. Provably Shorter Scratchpads in Hybrid DeltaNet-Attention Decoders

    May 15, 2026Tomasz SteiferGated AttentionRecurrent Transformers

  22. GateMOT: Q-Gated Attention for Dense Object Tracking

    Apr 29, 2026Mingjin Lv, Zelin Liu, Feifei Shao +4Visual Object TrackingGated Attention

  23. Capacity-Controlled Global Attention for Graph Transformers

    Apr 19, 2026Yang Liu, Dongxin Guo, Tom Zheng +3Gated AttentionTransformer Attention

  24. Gating Enables Curvature: A Geometric Expressivity Gap in Attention

    Apr 16, 2026Satwik Bathula, Anand A. JoshiInformation GeometryNeural Representation Geometry