Sparse Attention

Ask about this topic

Overview

Sparse attention studies how Transformers can focus computation on a subset of token interactions instead of comparing every query with every key, reducing compute and memory costs as sequences grow. Approaches include fixed local or block patterns, content-adaptive top-k routing, hierarchical KV-cache retrieval, and approximations based on clustering, low-rank structure, or feature sparsity; specialized kernels and cache systems help turn these savings into practical speedups. Methods are used in long-context language models and in video, image, and other settings, with evaluation often measuring both task quality on benchmarks such as LongBench and RULER and latency, throughput, or memory use. A central challenge is balancing accuracy with real end-to-end efficiency, since selecting important tokens can itself be costly and aggressive sparsity may miss information or behave unpredictably across inputs, layers, and hardware.

Latest papers

CardsList

244 papers52 papers in the last four weeks, up 100% on the four weeks before. 0.3% of all new papers.

  1. Dynamic Sparse Attention: Access Patterns and Architecture

    Mar 13, 2026Noam LevyLLM Inference AccelerationSparse Attention

    Intel Corporation.

  2. IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse

    Mar 12, 2026Yushi Bai, Qian Dong, Ting Jiang +5LLM Inference AccelerationSparse Attention

    Tsinghua University

  3. Why Attend to Everything? Focus is the Key

    Mar 12, 2026Hengshuai Yao, Xing Chen, Ahmed Murtadha +8Attention MechanismsEfficient Attention

    Sapient Intelligence · University of Alberta · Tsinghua University

  4. SVG-EAR: Parameter-Free Linear Compensation for Sparse Video Generation via Error-aware Routing

    Mar 9, 2026Xuanyi Zhou, Qiuyang Mang, Shuo Yang +7Video Diffusion ModelsEfficient Attention

    UC Berkeley, USA

  5. Speed3R: Sparse Feed-forward 3D Reconstruction Models

    Mar 9, 2026Weining Ren, Xiao Tan, Kai Han3D ReconstructionFeed-Forward 3D Reconstruction

    The University of Hong Kong · Baidu AMU

  6. Stem: Rethinking Causal Information Flow in Sparse Attention

    Mar 6, 2026Lin Niu, Xin Luo, Linchuan Xie +4Causal AttentionSparse Attention

    Tencent · School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, Anhui, 230026, China · Suzhou Institute for Advanced Research, University of Science and Technology of China, Suzhou, Jiangsu, 215123, China +5

  7. FlashPrefill: Instantaneous Pattern Discovery and Thresholding for Ultra-Fast Long-Context Prefilling

    Mar 6, 2026Qihang Fan, Huaibo Huang, Zhiying Wu +3Long-Context Language Model InferenceLLM Inference Acceleration

    MAIS&NLPR, CASIA · UCAS · WeChat, Tencent

  8. MSA: Memory Sparse Attention for Efficient End-to-End Memory Model Scaling to 100M Tokens

    Mar 6, 2026Yu Chen, Runkai Chen, Sheng Yi +9Memory-Augmented Language ModelsLong-Context Language Modeling

    Evermind · Shanda Group · Peking University

  9. Accelerating Text-to-Video Generation with Calibrated Sparse Attention

    Mar 5, 2026Shai Yehezkel, Shahar Yadin, Noam Elata +2Diffusion Model Inference AccelerationT2V Generation

    Apple · Tel Aviv University

  10. ProSMA-UNet: Decoder Conditioning for Proximal-Sparse Skip Feature Selection

    Mar 3, 2026Chun-Wun Cheng, Yanqi Cheng, Peiyuan Jing +4U-NetMedical Image Segmentation

    Department of Applied Mathematics and Theoretical Physics, University of Cambridge, UK · School of Engineering, Zurich University of Applied Sciences, CH · Bioengineering Department and ImperialX, Imperial College London, UK +2

  11. VSPrefill: Vertical-Slash Sparse Attention with Lightweight Indexing for Long-Context Prefilling

    Mar 3, 2026Chen GuanzhongLong-Context Language Model InferenceLLM Inference Acceleration

    School of Computer Science and Technology, Tongji University

  12. S2O: Early Stopping for Sparse Attention via Online Permutation

    Feb 26, 2026Yu Zhang, Songwei Liu, Chenqian Yan +4LLM Inference AccelerationSparse Attention

    ByteDance · Xiamen University

  13. LWM-Temporal: Sparse Spatio-Temporal Attention for Wireless Channel Representation Learning

    Feb 22, 2026Sadjad Alikhani, Akshay Malhotra, Shahab Hamidi-Rad +1Spatio-Temporal AttentionWireless Communications

  14. Incremental Learning of Sparse Attention Patterns in Transformers

    Feb 22, 2026Oğuz Kaan Yüksel, Rodrigo Alvarez Lucendo, Nicolas FlammarionSelf-AttentionTransformer Training Dynamics

    TML Lab, EPFL

  15. RAT+: Train Dense, Infer Sparse -- Recurrence Augmented Attention for Dilated Inference

    Feb 20, 2026Xiuying Wei, Caglar GulcehreLLM Inference AccelerationSparse Attention

    CLAIRE lab at EPFL, Lausanne, Switzerland.

  16. Selective Synchronization Attention

    Feb 16, 2026Hasi HaysKuramoto ModelSelf-Attention

    Department of Chemical Engineering, University of Arkansas, Fayetteville, AR 72701, USA

  17. SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuning

    Feb 13, 2026Jintao Zhang, Kai Jiang, Chendong Xiang +5Sparse AttentionDiffusion Model Acceleration

    Tsinghua University · UC Berkeley

  18. SLA2: Sparse-Linear Attention with Learnable Routing and Quantization-aware Training

    Feb 13, 2026Jintao Zhang, Haoxu Wang, Kai Jiang +6Sparse AttentionDiffusion Model Inference Acceleration

    Tsinghua University · UC Berkeley

  19. Routing Absorption in Sparse Attention: Why Random Gates Are Hard to Beat

    Feb 11, 2026Keston Aquino-MichaelsTransformerSelf-Attention

    No Way Labs

  20. Near-Oracle KV Selection via Pre-hoc Sparsity for Long-Context Inference

    Feb 9, 2026Yifei Gao, Lei Wang, Rong-Cheng Tu +3Self-AttentionKV Caching

    University College London, London, UK · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences (CAS), Shenzhen 518055, China · The Chinese University of Hong Kong +1

  21. Attention-Mass Condensation for Sparse Decoding

    Feb 6, 2026Jorge L. Ruiz WilliamsLanguage Model DecodingSparse Attention

    NaNZeta LLC

  22. Sparse Attention as Compact Kernel Regression

    Jan 30, 2026Saul Santos, Nuno Gonçalves, Daniel C. McNamee +2Transformer AttentionSparse Attention

    Instituto Superior Técnico & Instituto de Telecomunicações, Universidade de Lisboa, Portugal · Carnegie Mellon University, Pittsburgh, USA · Champalimaud Research, Lisbon, Portugal +1

  23. Threshold Differential Attention for Sink-Free, Ultra-Sparse, and Non-Dispersive Language Modeling

    Jan 17, 2026Xingyue Huang, Xueying Ding, Mingxuan Ju +3Self-AttentionLong-Context Language Modeling

    University of Oxford · Carnegie Mellon University · Snap Inc.

  24. Mixture of Distributions Matters: Dynamic Sparse Attention for Efficient Video Diffusion Transformers

    Jan 14, 2026Yuxi Liu, Yipeng Hu, Zekun Zhang +2Video Diffusion ModelsDiffusion Transformer

    Peking University · University of Electronic Science and Technology of China · University of Science and Technology of China

  25. From Local Windows to Adaptive Candidates via Individualized Exploratory: Rethinking Attention for Image Super-Resolution

    Jan 13, 2026Chunyu Meng, Wei Long, Shuhang GuEfficient ViTsSparse Attention

    University of Electronic Science and Technology of China

  26. Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference

    Dec 18, 2025Dhruv Deshmukh, Saurabh Goyal, Nipun Kwatra +1Self-AttentionLong-Context Language Model Inference

    Microsoft Research India.

  27. Block Sparse Flash Attention

    Dec 7, 2025Daniel Ohayon, Itay Lamprecht, Itay Hubara +3GPU AccelerationSelf-Attention

    Technion – Israel Institute of Technology, Haifa, Israel · Intel - Habana Labs · Intel – Habana Labs, Tel Aviv, Israel

  28. NOSA: Native and Offloadable Sparse Attention

    Oct 15, 2025Yuxiang Huang, Pengjie Wang, Jicheng Han +9KV-Cache OffloadingGPU Acceleration

    NLP Group, DCST, IAI, BNRIST, Tsinghua University, Beijing, China. · OpenBMB, China. · BUPT, Beijing, China. +1

  29. Multipole Semantic Attention: A Fast Approximation of Softmax Attention for Pretraining

    Sep 12, 2025Rupert Mitchell, Kristian KerstingLanguage Model PretrainingSoftmax Attention

    Department of Computer Science, TU Darmstadt, Darmstadt, Germany · Hessian Center for AI (hessian.AI), Darmstadt, Germany · German Research Center for Artificial Intelligence (DFKI), Darmstadt, Germany +1

  30. FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernel

    Aug 25, 2025Ran Yan, Youhe Jiang, Zhuoming Chen +3GPU Kernel OptimizationGPU Acceleration

    1. The Hong Kong University of Science and Technology · 2. Carnegie Mellon University