Block-Sparse Attention

Follow this topic Ask about this topic

Block-sparse attention studies how transformers can skip less useful groups of query-key interactions to reduce computation and memory use, especially for long contexts and high-resolution video. Methods select blocks using attention scores, learned or query-adaptive routers, structured neighborhoods, or summaries of key-value content, then execute the selected work with hardware-optimized sparse kernels. Applications include language-model prefill and decoding, video generation, and other large-grid models. The central challenge is to preserve useful information and output quality while keeping selection overhead low enough for sparsity to deliver real end-to-end speedups.

Latest papers

44 papers7 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

LazyFlash: Forward-Dense, Backward-Sparse Attention for Accelerated Long-Context Fine-Tuning
Oct 22, 2026Keqi Deng, Ruchao Fan, Jinyu LiSparse Fine-Tuning LLM Fine-Tuning
Microsoft, USA
Sparsity Should Be ALLOTted by Output: Training-Free Allocation of Block-Sparse Attention for Video Diffusion Transformers
Oct 21, 2026Tianze Xia, Xiaoling Zhou, Ruya Lan +3Sparse Attention Diffusion Model Inference Acceleration
UCAS · CASIA · KlingAI Research +1
SPIN: Shadow Predictive Indexer for Sparse Attention
Oct 6, 2026Yao Fu, Jiahan Chang, Ritchie Zhao +11LLM Inference Acceleration Sparse Attention
PHBA: Prefix-State Hybrid Block Attention
Oct 6, 2026Ruijie Li, Jiaxi Hu, Shiyu Wang +1Long-Context Modeling Linear Attention
The Hong Kong University of Science and Technology (Guangzhou) · Independent researcher
Prism: Dynamic Sparse Attention for Native 2K Joint Video-Audio Generation Model Training
Oct 4, 2026Shuyuan Tu, Qi Tian, Yinming Huang +8Audio-Video Generation High-Resolution Video Generation
Fudan University · Tencent Hunyuan Foundation Model Team · Zhejiang University
PARK: Accurate Block Retrieval for Sparse Attention in Video Diffusion Transformers
Sep 30, 2026Yun Dai, Jiarui Wen, Huiping Zhuang +2Video Diffusion Models Diffusion Transformer
South China University of Technology
ReSS: Residual-Restoring Sparse Attention for 3D Vision Transformers
Sep 28, 2026Yongsung Kim, Jaehoon Lee, Minjun Park +33D ViTs Efficient ViTs
IPAI, 2ECE, 3AIIS, ASRI, INMC, ISRC Seoul National University · ECE Seoul National University · AIIS Seoul National University
DraftAttention2: Fast Video Diffusion with Low-Resolution-Guided Mixed-Precision Attention
Sep 26, 2026Rui Ding, Haopeng Li, Weize Ma +10Video Diffusion Models Diffusion Model Inference Acceleration
Nanjing University · HKUST(GZ) · Duke University +3
Block Sparse Attention with Log-Linear Complexity
Sep 25, 2026Bohao Tang, Zhen Qin, Yuqi Pan +2Efficient Attention Block-Sparse Attention
Shanghai Jiao Tong University · Shanghai Innovation Institute · ByteDance Seed
Attention Routing Stabilizes Early: Working-Set Inference for Recurrent Language Models
Sep 23, 2026Ke Wan, Chen ChenLong-Context Language Model Inference Recurrent Transformers
Department of Computer Science University of Virginia Charlottesville, VA, USA
Block-Sparse Attention with Semantic-Geometric Decoupled Routing
Sep 19, 2026Xinwei Long, Weigao Sun, Weibo Gao +6Attention Routing Sparse Attention
Department of Electronic Engineering, Tsinghua University, Beijing, China · Intern at Tongyi Lab, Alibaba Group · Tongyi Lab, Alibaba Group, Shanghai, China +2
RBS-Attention: Radius-Bounded Sparse Prefill for Long-Context Large Language Models
Sep 17, 2026Chuxu Song, Jiuqi Wei, Zhencan PengLong-Context Language Model Inference LLM Inference Acceleration
Department of Computer Science, Rutgers University · OceanBase, Ant Group
Elastic Threshold Attention: Learned Contextual Sparsity for Long-Context Decoding
Sep 16, 2026Themistoklis Haris, Henry Li, Maryam KarimzadehganLong-Context Language Model Inference Sparse Attention
Google · Boston University
Faster Than Flash: Exploiting Attention Sparsity for Efficient Long-Context Decoding
Aug 31, 2026Zhigeng Liu, Zhiyuan Ning, Ruixiao Li +5Long-Context Language Model Inference LLM Inference Acceleration
Fudan University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China · Harbin Institute of Technology, Harbin, China
FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving
Aug 20, 2026Qihang Fan, Huaibo Huang, Zhiying Wu +2LLM Inference Acceleration Long-Context Language Model Inference
MAIS&NLPR, CASIA · UCAS · WeChat, Tencent
Partition the Support, Reconstruct the Residual: Training-Free Sparse Attention for Video Generation and World Models
Aug 19, 2026Pardis Taghavi, Reza Langari, Gaurav PandeySparse Attention Block-Sparse Attention
Texas A&M University
LoSA: Near-Lossless Sparse Attention for Training-Free Video Diffusion Acceleration
Aug 12, 2026Enhuai Liu, Yunke Wang, Yutong Wang +2Video Diffusion Models Diffusion Transformer
1The University of Sydney · 2CSIRO
SparSTAR: Sparse Attention for SpaceTime AutoRegressive Video Synthesis
Aug 11, 2026Jongbeom Lee, Hyunwoo Yu, Jincheol Yang +2Image-to-Video Generation Self-Attention
Sogang University Seoul, Republic of Korea
SPADE: An Input-Adaptive Sparse Attention Engine for Fast Video Diffusion Models Inference
Aug 4, 2026Shanghao Liu, Renze Chen, Size Zheng +4Video Diffusion Models Diffusion Transformer
Eric · Beihang University · Peking University
Token Radius Attention for Efficient Video Generation
Aug 3, 2026Jiayu Chen, Zhikun Jiang, Maoliang Li +6Video Diffusion Models Sparse Attention
School of Computer Science, Peking University · School of Electronics Engineering and Computer Science, Peking University · School of Computer Science and Engineering, Beihang University +2
Understanding Sparse Attention Selectivity in Long-Context Foundation Models via Counterfactual Evaluation
Aug 3, 2026Xingyu Ren, Youran Sun, Chugang Yi +1LLM Auditing Sparse Attention
The Chinese University of Hong Kong, Hong Kong, China · University of Maryland, College Park, MD, USA
CoSA: Accelerating Long-Context Inference via Proxy-Kernel Co-Designed Sparse Attention
Jul 28, 2026Yufei Xue, Lin Niu, Hong Liu +6Long-Context Language Model Inference LLM Inference Acceleration
1Tencent · 2The Hong Kong University of Science and Technology
Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsification
Jul 27, 2026Haopeng Li, Yitong Li, Junsong Chen +8Video Diffusion Models Sparse Attention
NVIDIA
FVAttn: Adaptive Sparse Attention with Runtime Load Balancing for Video Generation
Jul 17, 2026Hao Liu, Chenghuan Huang, Ye Huang +7Video Diffusion Models Diffusion Transformer
Sun Yat-sen University · WeChat HPC, Tencent Inc. · Peking University +1
COBS: Cumulant Order Block Sparse Attention
Jul 10, 2026Alexander Tian, Aditya Ghai, Sanjit Neelam +2KV Caching LLM Inference Acceleration
MatX
Uncertainty-gated selection for block-sparse attention
Jul 4, 2026Thomas RossiLong-Context Language Model Inference LLM Inference Acceleration
Eonpass
ScalingAttention: Discovering Intrinsic Sparse Attention Topology for Video Diffusion Transformers
Jun 22, 2026Ruiliang Zhou, Xuecheng Wu, Kang He +6Video Diffusion Models Diffusion Transformer
KlingAI Research · Beijing Institute of Technology · NVIDIA +1
SpotAttention: Plug-In Block-Sparse Routing for Pretrained Long-Context Transformers
Jun 22, 2026Huzama Ahmad, Se-Young YunSparse Attention Block-Sparse Attention
KAIST
MiniMax Sparse Attention
Jun 11, 2026Xunhao Lai, Weiqi Xu, Yufeng Yang +14GPU Kernel Optimization Long-Context Language Modeling
MiniMax · Peking University · NVIDIA +4
Light Interaction: Training-Free Inference Acceleration for Interactive Video World Models
May 29, 2026Jiacheng Lu, Haoyi Zhu, Sipei Yi +3Real-Time Interactive Video Generation Interactive World Models
1Zhejiang University · 2NVIDIA

Oct 22, 2026J/K move · Enter open · S save

LazyFlash: Forward-Dense, Backward-Sparse Attention for Accelerated Long-Context Fine-Tuning

Keqi Deng, Ruchao Fan, Jinyu Li

Microsoft, USA

Sparse Fine-Tuning LLM Fine-Tuning Block-Sparse Attention Long-Context Language Modeling

Open paper AskPDF arXiv

As large language models (LLMs) process increasingly long sequences, attention becomes a major computational bottleneck. Sparse attention mostly targets inference, yet many methods still train densely or cannot support sparse training, and their added complexity makes sparsity hard to turn into wall-clock speedups, leaving dense attention widely used in practice. This paper introduces LazyFlash, a forward-dense, backward-sparse attention method built on the observation that attention training is dominated by the backward pass. LazyFlash keeps the forward pass exactly dense, so the trained model remains a standard dense model that existing systems such as vLLM can serve without any change. For the backward pass, LazyFlash proposes a gradient-aware block sparsification design that ranks query blocks by their attention-output gradient norms and skips those contributing negligibly to parameter updates, turning the dense backward into a block-sparse one. A gradient-mass budget adapts the amount of skipped computation to each head and training step, while importance sampling preserves the dense gradient in expectation. These designs are implemented inside the tiled FlashAttention backward kernel, where skipped blocks are never loaded or computed, translating gradient sparsity into real wall-clock acceleration. Experiments on long-context text and speech LLMs show that LazyFlash maintains downstream performance while achieving up to 5.7x attention-backward speedup over FlashAttention and 1.7x end-to-end training speedup.