Overview

Sparse attention studies how Transformers can focus computation on a subset of token interactions instead of comparing every query with every key, reducing compute and memory costs as sequences grow. Approaches include fixed local or block patterns, content-adaptive top-k routing, hierarchical KV-cache retrieval, and approximations based on clustering, low-rank structure, or feature sparsity; specialized kernels and cache systems help turn these savings into practical speedups. Methods are used in long-context language models and in video, image, and other settings, with evaluation often measuring both task quality on benchmarks such as LongBench and RULER and latency, throughput, or memory use. A central challenge is balancing accuracy with real end-to-end efficiency, since selecting important tokens can itself be costly and aggressive sparsity may miss information or behave unpredictably across inputs, layers, and hardware.

Latest papers

CardsList

274 papers34 papers in the last four weeks, down 32% on the four weeks before. 0.3% of all new papers.

  1. XM-Sparse: High-Throughput Sparse-Attention LLM Serving over Scale-up Memory Pools

    Nov 17, 2026Yijun Sun, Xudong Liao, Zhuobin Huang +3KV-Cache OffloadingLLM Inference Acceleration

  2. Sparse Attention for Visual Generation: Toward Algorithm-System Co-Design

    Nov 16, 2026Majun Zhang, Yuan Guo, Decheng Wu +6Sparse AttentionImage Generation

  3. QASK-Edit: Adaptive Sparse Attention Across Queries and Key Sources for Efficient Video Editing

    Nov 16, 2026Hengrui Liu, Wenbo Xu, Ziqi Sheng +1Sparse AttentionDiffusion-Based Video Editing

  4. Eco3R: Pinned-Key Inference for Efficient and Streamable Feed-Forward 3D Reconstruction from Long Videos

    Nov 13, 2026Hongsen Cao, Shanxin Yuan, Mona Jaber +13D ReconstructionMulti-View 3D Reconstruction

  5. NearFormer: Linear-Size Sparse Attention via Randomized Near Neighbors

    Nov 9, 2026Ali Parviz, Alex Cloninger, Gal MishneGraph TransformerLong-Range Graph Learning

  6. ADIndex: Layer-Adaptive Sparse Attention Indexing via Group-Level Distillation

    Nov 7, 2026Jin-Fei Qi, Yiduo Li, Jiang-Xin Shi +4LLM Inference AccelerationSparse Attention

  7. Attend at the Right Resolution: Head-Adaptive Sparse Attention via Query-Key Interaction Structure

    Nov 6, 2026Wajih Hassan Raza, Ahsan Bilal, Yidan Shen +3Sparse AttentionEfficient Attention

  8. DUET: Dual-Sparse Inference for Block Diffusion Language Models

    Nov 6, 2026Siyuan Yan, Shen Gu, Xiaoxing Ma +2Diffusion Language Model InferenceDiffusion Model Inference Acceleration

  9. Recursive Sparse Attention: Million-Token Length Extrapolation at Logarithmic Cost

    Nov 4, 2026Ruitian Wang, Xinyu Yang, Minshen Zhang +1Sparse AttentionLong-Context Language Modeling

  10. Accelerating Long-Context Decoding in Large-Scale Hybrid LLMs Without Retraining

    Nov 4, 2026João Eduardo Batista, Emil Vatai, Mohamed WahibLLM Inference AccelerationLong-Context Language Model Inference

  11. DecoupAttn: Decoupled Sparse Attention for Training-Free Acceleration of Video Diffusion

    Nov 4, 2026Yanjin Li, Bingbing Qiu, Qiang Huang +5Diffusion Model Inference AccelerationSparse Attention

  12. A Parametric Linear Conic Projection for Structured Sparse Attention

    Nov 3, 2026Michel Barlaud, Clement Genninasca, Maxime Zanella +2Structured SparsitySparse Attention

  13. Efficient and Faithful High-Resolution Image Editing with Sparse Cross-Resolution Denoising

    Nov 3, 2026Maosen Zhao, Xianfang Zeng, Pengtao Chen +4Image EditingDiffusion-Based Image Editing

  14. Coarse Semantic Blocks for Efficient Training-Free Sparse Attention

    Nov 3, 2026Sosuke Kobayashi, Shoichiro Yamaguchi, Seiya TokuiSparse AttentionLLM Inference Acceleration

  15. When Should an Agent Refresh? Adaptive Index Reuse for Efficient Long-Horizon Reasoning

    Nov 1, 2026Guangyuan Wang, Yifan Zhang, Shilong LiuLLM Agent EfficiencyLLM Inference Scheduling

  16. TileWeave: Heterogeneous Tiling for Sparse Prefill on Mobile SoCs

    Oct 30, 2026Wentao Ni, Hong Cai, Mingu Lee +7LLM Inference AccelerationOn-Device Language Model Inference

  17. Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention

    Oct 30, 2026Ales Kubicek, Andrea Caciolai, Liangzu Peng +7Sparse AttentionLong-Context Language Modeling

  18. Subquadratic Attention with a Hierarchical Tree-Structured Indexer

    Oct 30, 2026Leon Lufkin, Cutter Dawes, Alexander Speicher +3Efficient AttentionLong-Context Language Modeling

  19. Sparse Text Diffusion under Dual Budgets: Context Connectivity and Expert Activation

    Oct 30, 2026Alexandros Christoforos, Chadbourne Davis, Paul RajDiffusion Language ModelsLong-Context Modeling

  20. Unlimited OCR Works 2

    Oct 29, 2026Shu Wei, Jingjing Wu, Qunyi Xie +11Optical Character RecognitionLong-Context Modeling

  21. INTRA: Interleaved Non-contiguous Token spaRse Attention

    Oct 25, 2026Yuxuan Xia, Wenbo Lu, Shaoyi Zheng +1Sparse AttentionTransformer Inference

  22. Cross-Scale Sparse Attention for Referring Segmentation in Remote Sensing

    Oct 24, 2026Yuxi Li, Lu Si, Xia Liao +4Referring Image SegmentationRemote Sensing Image Segmentation