Sparse Attention

Ask about this topic

Overview

Sparse attention studies how Transformers can focus computation on a subset of token interactions instead of comparing every query with every key, reducing compute and memory costs as sequences grow. Approaches include fixed local or block patterns, content-adaptive top-k routing, hierarchical KV-cache retrieval, and approximations based on clustering, low-rank structure, or feature sparsity; specialized kernels and cache systems help turn these savings into practical speedups. Methods are used in long-context language models and in video, image, and other settings, with evaluation often measuring both task quality on benchmarks such as LongBench and RULER and latency, throughput, or memory use. A central challenge is balancing accuracy with real end-to-end efficiency, since selecting important tokens can itself be costly and aggressive sparsity may miss information or behave unpredictably across inputs, layers, and hardware.

Latest papers

CardsList

244 papers52 papers in the last four weeks, up 100% on the four weeks before. 0.3% of all new papers.

  1. SpikingBrain2.0: Brain-Inspired Foundation Models for Efficient Long-Context and Cross-Platform Inference

    Apr 24, 2026Yuqi Pan, Jinghao Zhuang, Yupeng Feng +16Long-Context Language Model InferenceSpiking Neural Networks

    Institute of Automation, Chinese Academy of Sciences · The Hong Kong Polytechnic University · Beijing Key Laboratory of Brain-Inspired General Intelligence Large Model +1

  2. HubRouter: A Pluggable Sub-Quadratic Routing Primitive for Hybrid Sequence Models

    Apr 24, 2026Abhinaba BasuTransformer AttentionLanguage Modeling

    National Institute of Electronics and Information Technology (NIELIT)

  3. Sculpt4D: Generating 4D Shapes via Sparse-Attention Diffusion Transformers

    Apr 23, 2026Minghao Yin, Wenbo Hu, Jiale Xu +2Diffusion Transformer3D Shape Generation

    1The University of Hong Kong · 2ARC Lab, Tencent PCG

  4. Sparse Forcing: Native Trainable Sparse Attention for Real-time Autoregressive Diffusion Video Generation

    Apr 23, 2026Boxun Xu, Yuming Du, Zichang Liu +7Autoregressive DiffusionSelf-Attention

    Meta Superintelligence Labs · University of California, Santa Barbara · Work done at Meta

  5. DynamicRad: Content-Adaptive Sparse Attention for Long Video Diffusion

    Apr 22, 2026Yongji Long, Shijun Liang, Jintao Li +1Video Diffusion ModelsLong-Horizon Video Generation

    University of Electronic Science and Technology of China · Shenzhen Institute for Advanced Study, UESTC · Computational Mathematics, Science, & Engineering at Michigan State University (MSU)

  6. Simplified Sparse Attention via Gist Tokens

    Apr 22, 2026Yuzhen Mao, Michael Y. Li, Emily B. FoxSelf-AttentionLong-Context Language Model Inference

    Stanford University

  7. AdaCluster: Adaptive Query-Key Clustering for Sparse Attention in Video Generation

    Apr 20, 2026Haoyue Tan, Shengnan Wang, Yulin Qiao +5Video Diffusion ModelsDiffusion Transformer

    University of Science and Technology of China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center · 3Independent Researcher +2

  8. AdaSplash-2: Faster Differentiable Sparse Attention

    Apr 16, 2026Nuno Gonçalves, Hugo Pitorro, Vlad Niculae +4GPU Kernel OptimizationSelf-Attention

    Instituto Superior Técnico, Universidade de Lisboa · Instituto de Telecomunicações · Carnegie Mellon University +4

  9. Improving Sparse Autoencoder with Dynamic Attention

    Apr 16, 2026Dongsheng Wang, Jinsen Zhang, Dawei Su +1Sparse AutoencodersNeural Network Interpretability

    College of Computer Science and Software Engineering, Shenzhen University, China

  10. SparseBalance: Load-Balanced Long Context Training with Dynamic Sparse Attention

    Apr 15, 2026Hongtao Xu, Jianchao Tan, Yuxuan Hu +8Sparse AttentionDistributed Training

    School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing, China · Meituan, Beijing, China · University of Chinese Academy of Sciences, Beijing, China

  11. Ride the Wave: Precision-Allocated Sparse Attention for Smooth Video Generation

    Apr 14, 2026Wentai Zhang, Ronghui Xi, Shiyao Peng +4Sparse AttentionDiffusion Model Inference Acceleration

    Beijing University of Posts and Telecommunications, Beijing, China · Nanyang Technological University, Singapore, Singapore

  12. Saccade Attention Networks: Using Transfer Learning of Attention to Reduce Network Sizes

    Apr 13, 2026Marc EstafanousEfficient ViTsNeural Network Compression

    Johns Hopkins University · Neurobaby Corporation

  13. Modality-Aware Zero-Shot Pruning and Sparse Attention for Efficient Multimodal Edge Inference

    Apr 10, 2026Yueyuan Sui, Payal Mohapatra, Doğaç Eldenk +5Efficient Multimodal InferenceNeural Network Pruning

    Northwestern University, Evanston, Illinois, USA

  14. AdaSpark: Adaptive Sparsity for Efficient Long-Video Understanding

    Apr 9, 2026Handong Li, Zikang Liu, Longteng Guo +10Long-Video UnderstandingEfficient VLM Inference

    School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Alibaba Group Holding Limited +1

  15. AsyncTLS: Efficient Generative LLM Inference with Asynchronous Two-level Sparse Attention

    Apr 9, 2026Yuxuan Hu, Jianchao Tan, Jiaqi Zhang +7LLM Inference AccelerationLong-Context Language Model Inference

    School of Information, Renmin University of China, Beijing, China · Meituan, Beijing, China

  16. Flux Attention: Context-Aware Hybrid Attention for Efficient LLMs Inference

    Apr 8, 2026Quantong Qiu, Zhiyi Hong, Yi Yang +5Self-AttentionLong-Context Language Model Inference

    School of Computer Science and Technology, Soochow University · Baidu Inc, China

  17. LSRM: High-Fidelity Object-Centric Reconstruction via Scaled Context Windows

    Apr 6, 2026Zhengqin Li, Cheng Zhang, Jakob Engel +1Object-Centric 3D ReconstructionInverse Rendering

    Meta Reality Labs Research

  18. k-Maximum Inner Product Attention for Graph Transformers and the Expressive Power of GraphGPS

    Apr 4, 2026Jonas De Schouwer, Haitz Sáez de Ocáriz Borde, Xiaowen DongTransformer AttentionGraph Transformer

    Stanford University · University of Oxford

  19. VecAttention: Vector-wise Sparse Attention for Accelerating Long Context Inference

    Mar 31, 2026Anmin Liu, Ruixuan Yang, Huiqiang Jiang +5Transformer AttentionSparse Attention

    SCS, Peking University, Beijing, China · Key Lab of HCST (PKU), MOE, Beijing, China · Fudan University, Shanghai, China +2

  20. HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention

    Mar 30, 2026Yufei Xu, Fanxu Meng, Fan Jiang +11Self-AttentionLLM Inference Acceleration

    Xiaohongshu

  21. CSAttention: Centroid-Scoring Attention for Accelerating LLM Inference

    Mar 30, 2026Chuxu Song, Zhencan Peng, Jiuqi Wei +1LLM Inference AccelerationLong-Context Language Model Inference

    Department of Computer Science, Rutgers University · Oceanbase, Ant Group.

  22. HeSS: Head Sensitivity Score for Sparsity Redistribution in VGGT

    Mar 26, 2026Yongsung Kim, Wooseok Song, Jaihyun Lew +3Sparse AttentionEfficient ViTs

    IPAI, Seoul National University · ECE, Seoul National University · AIIS, ASRI, INMC, ISRC, Seoul National University

  23. Free-Lunch Long Video Generation via Layer-Adaptive O.O.D Correction

    Mar 26, 2026Jiahao Tian, Chenxi Song, Wei Cheng +1Long-Horizon Video GenerationVideo Diffusion Models

    Westlake University

  24. Accurate and Efficient Multi-Channel Time Series Forecasting via Sparse Attention Mechanism

    Mar 19, 2026Lei Gao, Hengda Bao, Jingfei Fang +3Multivariate Time Series ForecastingSparse Attention

    Department of Machine Learning, SF Express, Shanghai, China · Department of Management, Zhejiang University of Technology, Hangzhou, China · Department of Management, Zhejiang University, Hangzhou, China

  25. Self-Tuning Sparse Attention: Multi-Fidelity Hyperparameter Optimization for Transformer Acceleration

    Mar 19, 2026Arundhathi Dev, Justin ZhanSparse AttentionHyperparameter Optimization

    Department of Computer Science, University of Cincinnati, Cincinnati, USA

  26. Learning When to Attend: Conditional Memory Access for Long-Context LLMs

    Mar 18, 2026Sakshi Choudhary, Aditya Chattopadhyay, Luca Zancato +4Self-AttentionLong-Context Language Modeling

    Department of Electrical and Computer Engineering, Purdue University, USA · AWS Agentic AI, USA

  27. Scaling Attention via Feature Sparsity

    Mar 17, 2026Yan Xie, Tiansheng Wen, Tangda Huang +4Transformer AttentionEfficient Attention

    School of Electronic Engineering, Xidian University · Stony Brook University · TUM +2

  28. WAND: Windowed Attention and Knowledge Distillation for Efficient Autoregressive Text-to-Speech Models

    Mar 17, 2026Hanna Lee, Tan Dat Nguyen, Jaehoon Kang +1TTS SynthesisSelf-Attention

    Korea Advanced Institute of Science and Technology, Republic of Korea · Sungkyunkwan University, Republic of Korea

  29. When Does Sparsity Mitigate the Curse of Depth in LLMs

    Mar 16, 2026Dilxat Muhtar, Xinyuan Song, Sebastian Pokutta +4Grouped-Query AttentionLanguage Model Scaling

    Max Planck Institute for Intelligent Systems · ELLIS Institute Tübingen · Tübingen AI Center +4

  30. Self-Indexing KVCache: Predicting Sparse Attention from Compressed Keys

    Mar 15, 2026Xu Yang, Jiapeng Zhang, Dongyang Zhao +2Binary QuantizationLLM Inference Acceleration

    College of Computer Science and Electronic Engineering, Hunan University, Changsha, China · The Ministry of Education Key Laboratory of “Fusion Computing of Supercomputing and Artificial Intelligence” China · Shenzhen Research Institute, Hunan University, Shenzhen, China