Overview

Block-sparse attention studies how transformers can skip less useful groups of query-key interactions to reduce computation and memory use, especially for long contexts and high-resolution video. Methods select blocks using attention scores, learned or query-adaptive routers, structured neighborhoods, or summaries of key-value content, then execute the selected work with hardware-optimized sparse kernels. Applications include language-model prefill and decoding, video generation, and other large-grid models. The central challenge is to preserve useful information and output quality while keeping selection overhead low enough for sparsity to deliver real end-to-end speedups.

Latest papers

CardsList

56 papers12 papers in the last four weeks, up 50% on the four weeks before. 0.1% of all new papers.

  1. dit.xpu : Taming DiT Sparse Inference on Mobile SoCs

    Nov 14, 2026Jiayu Chen, Jinye Hu, Maoliang Li +8Diffusion Model Inference AccelerationOn-Device Inference

  2. InstaGen: Group-Shared Importance Sampling for Fast Video Generation

    Nov 5, 2026Kislay Aditya Oj, Kumar Krishna Agrawal, Sri Surya Varma Datla +1Efficient Video GenerationVideo Diffusion Models

  3. TriAttention: Unified Sparse Mixed-Precision Attention for Long-Context LLM Inference

    Nov 5, 2026Hanzhi Zhang, Qiao Zhang, Qinglei Cao +4LLM Inference AccelerationAttention Quantization

  4. DecoupAttn: Decoupled Sparse Attention for Training-Free Acceleration of Video Diffusion

    Nov 4, 2026Yanjin Li, Bingbing Qiu, Qiang Huang +5Diffusion Model Inference AccelerationSparse Attention

  5. Coarse Semantic Blocks for Efficient Training-Free Sparse Attention

    Nov 3, 2026Sosuke Kobayashi, Shoichiro Yamaguchi, Seiya TokuiSparse AttentionLLM Inference Acceleration

  6. Step-dLLM: Adaptive Step-aware Sparse Attention for Efficient Diffusion LLM Inference

    Oct 31, 2026Zhichen Zeng, Xichong Zhang, Junpan Wu +7Diffusion Language Model InferenceDiffusion Language Models

  7. Jet-QSparse: Accelerating Agentic LLM Inference with 4-Bit Sparse Attention

    Oct 31, 2026Haozhan Tang, Hao Kang, Meng Lu +7LLM Inference AccelerationKV-Cache Quantization

  8. TileWeave: Heterogeneous Tiling for Sparse Prefill on Mobile SoCs

    Oct 30, 2026Wentao Ni, Hong Cai, Mingu Lee +7LLM Inference AccelerationOn-Device Language Model Inference

  9. AB-Attention: Accurate Block Attention for Accelerating Video Diffusion Transformers

    Oct 29, 2026Bin Xu, Mo Zhang, Zhengfeiyang Zhang +6Diffusion Model Inference AccelerationBlock-Sparse Attention

  10. PHBA: Prefix-State Hybrid Block Attention

    Oct 6, 2026Ruijie Li, Jiaxi Hu, Shiyu Wang +1Long-Context ModelingLinear Attention

  11. Prism: Dynamic Sparse Attention for Native 2K Joint Video-Audio Generation Model Training

    Oct 4, 2026Shuyuan Tu, Qi Tian, Yinming Huang +8Audio-Video GenerationHigh-Resolution Video Generation

  12. PARK: Accurate Block Retrieval for Sparse Attention in Video Diffusion Transformers

    Sep 30, 2026Yun Dai, Jiarui Wen, Huiping Zhuang +2Video Diffusion ModelsDiffusion Transformer

  13. ReSS: Residual-Restoring Sparse Attention for 3D Vision Transformers

    Sep 28, 2026Yongsung Kim, Jaehoon Lee, Minjun Park +33D ViTsEfficient ViTs

  14. DraftAttention2: Fast Video Diffusion with Low-Resolution-Guided Mixed-Precision Attention

    Sep 26, 2026Rui Ding, Haopeng Li, Weize Ma +10Video Diffusion ModelsDiffusion Model Inference Acceleration

  15. Block Sparse Attention with Log-Linear Complexity

    Sep 25, 2026Bohao Tang, Zhen Qin, Yuqi Pan +2Efficient AttentionBlock-Sparse Attention

  16. Attention Routing Stabilizes Early: Working-Set Inference for Recurrent Language Models

    Sep 23, 2026Ke Wan, Chen ChenLong-Context Language Model InferenceRecurrent Transformers

  17. Block-Sparse Attention with Semantic-Geometric Decoupled Routing

    Sep 19, 2026Xinwei Long, Weigao Sun, Weibo Gao +6Attention RoutingSparse Attention

  18. RBS-Attention: Radius-Bounded Sparse Prefill for Long-Context Large Language Models

    Sep 17, 2026Chuxu Song, Jiuqi Wei, Zhencan PengLong-Context Language Model InferenceLLM Inference Acceleration

  19. Elastic Threshold Attention: Learned Contextual Sparsity for Long-Context Decoding

    Sep 16, 2026Themistoklis Haris, Henry Li, Maryam KarimzadehganLong-Context Language Model InferenceSparse Attention

  20. Faster Than Flash: Exploiting Attention Sparsity for Efficient Long-Context Decoding

    Aug 31, 2026Zhigeng Liu, Zhiyuan Ning, Ruixiao Li +5Long-Context Language Model InferenceLLM Inference Acceleration

  21. FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving

    Aug 20, 2026Qihang Fan, Huaibo Huang, Zhiying Wu +2LLM Inference AccelerationLong-Context Language Model Inference

  22. LoSA: Near-Lossless Sparse Attention for Training-Free Video Diffusion Acceleration

    Aug 12, 2026Enhuai Liu, Yunke Wang, Yutong Wang +2Video Diffusion ModelsDiffusion Transformer

  23. SparSTAR: Sparse Attention for SpaceTime AutoRegressive Video Synthesis

    Aug 11, 2026Jongbeom Lee, Hyunwoo Yu, Jincheol Yang +2Image-to-Video GenerationSelf-Attention

  24. SPADE: An Input-Adaptive Sparse Attention Engine for Fast Video Diffusion Models Inference

    Aug 4, 2026Shanghao Liu, Renze Chen, Size Zheng +4Video Diffusion ModelsDiffusion Transformer