Overview

Gated attention studies how learned controls can regulate information flow in Transformers and related models. Common approaches use multiplicative gates on attention outputs, values, heads, or tokens, while gated linear and recurrent methods use input-dependent updates or decay to handle long sequences efficiently. These methods aim to improve selectivity, robustness, and computational cost, with applications spanning language modeling, long-context retrieval, vision, graphs, and multimodal tasks. Results vary by task and scale, and an open challenge is separating gains from gating itself from model adaptation, while checking whether soft learned gates deliver reliable benefits when converted to hard pruning or sparse inference.

Latest papers

CardsList

37 papers4 papers in the last four weeks, down 56% on the four weeks before. 0.0% of all new papers.

  1. Learning to Adapt: In-Context Learning Beyond Stationarity

    Apr 13, 2026Zhen Qin, Jiachen Jiang, Zhihui ZhuIn-Context LearningGated Attention

  2. Gated-SwinRMT: Unifying Swin Windowed Attention with Retentive Manhattan Decay via Input-Dependent Gating

    Apr 7, 2026Dipan Maity, Suman Mondal, Arindam RoyVision TransformerGated Attention

  3. The Model Knows Which Tokens Matter: Automatic Token Selection via Noise Gating

    Mar 7, 2026Landi He, Xiaoyu Yang, Lijian XuEfficient VLM InferenceVisual Token Pruning

  4. Gated Differential Linear Attention: A Linear-Time Decoder for High-Fidelity Medical Segmentation

    Mar 3, 2026Hongbo Zheng, Afshin Bozorgpour, Dorit Merhof +1Medical Image SegmentationLinear Attention

  5. A Statistical Theory of Gated Attention through the Lens of Hierarchical Mixture of Experts

    Feb 1, 2026Viet Nguyen, Tuan Minh Pham, Thinh Cao +4Gated AttentionSample Complexity

  6. GateTS: Versatile and Efficient Forecasting via Attention-Inspired routed Mixture-of-Experts

    Aug 24, 2025Kyrylo Yemets, Mykola Lukashchuk, Ivan IzoninTime Series ForecastingMixture-of-Experts Models