Efficient Attention

Ask about this topic

Overview

Efficient attention studies how to reduce the cost of comparing queries with keys, especially for long sequences and constrained hardware. Approaches include IO-aware fused kernels, sparse or hierarchical retrieval, linear and recurrent attention, and KV-cache compression; many combine algorithm design with GPU or accelerator-specific kernels. Evaluation spans language modeling and long-context retrieval as well as vision, video, and other sequence tasks, where quality, latency, memory, and energy can trade off differently. A central tension is that practical methods can work well on structured inputs, while preserving full-attention quality across arbitrary inputs—and providing strong guarantees—remains difficult.

Latest papers

CardsList

138 papers17 papers in the last four weeks, down 6% on the four weeks before. 0.1% of all new papers.

  1. S-VGGT: Structure-Aware Subscene Decomposition for Scalable 3D Foundation Models

    Mar 18, 2026Xinze Li, Pengxu Chen, Yiyuan Wang +23D Foundation ModelsEfficient Attention

    Beijing Normal-Hong Kong Baptist University · Jilin University · Hong Kong Baptist University +1

  2. Scaling Attention via Feature Sparsity

    Mar 17, 2026Yan Xie, Tiansheng Wen, Tangda Huang +4Transformer AttentionEfficient Attention

    School of Electronic Engineering, Xidian University · Stony Brook University · TUM +2

  3. MobileLLM-Flash: Latency-Guided On-Device LLM Design for Industry Scale Deployment

    Mar 16, 2026Hanxian Huang, Igor Fedorov, Andrey Gromov +14On-Device Language Model InferenceLLM Inference Acceleration

    Meta AI

  4. Why Attend to Everything? Focus is the Key

    Mar 12, 2026Hengshuai Yao, Xing Chen, Ahmed Murtadha +8Attention MechanismsEfficient Attention

    Sapient Intelligence · University of Alberta · Tsinghua University

  5. Beyond Pairwise Attention: Higher-Order Modular Attention for Efficient Sequence Learning

    Mar 11, 2026Shirin Amiraslani, Xin GaoSelf-AttentionTransformer Attention

    Department of Mathematics and Statistics York University Toronto, ON M3J 1P3, Canada

  6. HoloByte: Continuous Hyperspherical Distillation for Tokenizer-Free Modeling

    Mar 10, 2026Vladimer KhasiaByte-Level Language ModelLanguage Modeling

    Independent Researcher

  7. SVG-EAR: Parameter-Free Linear Compensation for Sparse Video Generation via Error-aware Routing

    Mar 9, 2026Xuanyi Zhou, Qiuyang Mang, Shuo Yang +7Video Diffusion ModelsEfficient Attention

    UC Berkeley, USA

  8. LinearARD: Linear-Memory Attention Distillation for RoPE Restoration

    Mar 9, 2026Ning Yang, Hengyu Zhong, Wentao Wang +4Attention-Guided DistillationMemory-Efficient Attention

    Institute of Automation, Chinese Academy of Sciences, Beijing, China · Southwest University, Chongqing, China · Dalian University of Technology, Dalian, Liaoning, China +3

  9. SketchGraphNet: A Memory-Efficient Hybrid Graph Transformer for Large-Scale Sketch Corpora Recognition

    Mar 8, 2026Shilong Chen, Mingyuan Li, Zhaoyang Wang +2Memory-Efficient AttentionGraph Transformer

    College of Computer, Qinghai Normal University, Xining, 810001, Qinghai, China · College of Computer, Nanjing University, Nanjing, 210023, Jiangsu, China · College of Computer, Qinghai Minzu University, Xining, 810007, Qinghai, China

  10. FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling

    Mar 5, 2026Ted Zadouri, Markus Hoehnerbach, Jay Shah +3Efficient AttentionGPU Kernel Optimization

    Princeton University · Together AI · Meta +3

  11. TiledAttention: a CUDA Tile SDPA Kernel for PyTorch

    Mar 2, 2026Taimur KhanGPU Kernel OptimizationSelf-Attention

    Helmholtz Centre for Environmental Research - UFZ

  12. Spectral Attention Steering for Prompt Highlighting

    Mar 1, 2026Weixian Waylon Li, Yuchen Niu, Yongxin Yang +3Language Model SteeringEfficient Attention

    University of Edinburgh, UK · RayNeo, China · Queen Mary University of London, UK +1

  13. An Evaluation of Context Length Extrapolation in Long Code via Positional Embeddings and Efficient Attention

    Feb 25, 2026Madhusudan Ghosh, Rishabh GuptaLong-Context ModelingLong-Context Language Model Evaluation

    Indian Association for the Cultivation of Science, Kolkata, India · Bosch Research and Technology Centre, Bangalore, India

  14. Untied Ulysses: Memory-Efficient Context Parallelism via Headwise Chunking

    Feb 24, 2026Ravi Ghadia, Maksim Abraham, Sergei Vorobyov +1Memory-Efficient AttentionContext Parallelism

    Together AI

  15. FlashBlock: Attention Caching for Efficient Long-Context Block Diffusion

    Feb 5, 2026Zhuokun Chen, Jianfei Cai, Bohan ZhuangVideo Diffusion ModelsSelf-Attention

    1Monash University · 2Zhejiang University

  16. Transform Trained Transformer for Accelerating Native 4K Video Generation

    Dec 15, 2025Jiangning Zhang, Junwei Zhu, Teng Hu +9Efficient Transformer InferenceHigh-Resolution Video Generation

    Youtu Lab, Tencent · Zhejiang University

  17. Accelerating Attention with Basis Decomposition

    Oct 2, 2025Jialin ZhaoSelf-AttentionTransformer Attention

    Department of Computer Science Tsinghua University

  18. SEMA: a Scalable and Efficient Mamba like Attention via Token Localization and Averaging

    Jun 10, 2025Nhat Thanh Tran, Fanghui Xue, Shuai Zhang +4Self-AttentionVision Transformer

    Department of Mathematics; University of California, Irvine; Irvine, CA 92697, USA · Qualcomm AI Research; San Diego, CA 92121, USA