Block-Sparse Attention

Follow this topic Ask about this topic

Block-sparse attention studies how transformers can skip less useful groups of query-key interactions to reduce computation and memory use, especially for long contexts and high-resolution video. Methods select blocks using attention scores, learned or query-adaptive routers, structured neighborhoods, or summaries of key-value content, then execute the selected work with hardware-optimized sparse kernels. Applications include language-model prefill and decoding, video generation, and other large-grid models. The central challenge is to preserve useful information and output quality while keeping selection overhead low enough for sparsity to deliver real end-to-end speedups.

Latest papers

44 papers7 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

LVSA: Training-Free Sparse Attention for Long Video Diffusion
May 29, 2026Gael Glorian, Ioannis Lamprou, Zhen Zhang +2Video Diffusion Models Long-Horizon Video Generation
Distributed Parallel Technology Laboratory, Paris Research Center, Huawei Technologies France · AI Framework and Data Technology Lab, Huawei Technologies Co., Ltd.
DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation
May 22, 2026Jie Hu, Zixiang Gao, Yutong He +1Video Diffusion Models Diffusion Transformer
1Peking University.
Efficient Long-Context Modeling in Diffusion Language Models via Block Approximate Sparse Attention
May 19, 2026Wenhu Zhang, Yiming Wu, Huanyu Wang +6Self-Attention Long-Context Language Modeling
The Hong Kong University of Science and Technology · The University of Hong Kong · Zhejiang University +1
An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inference
May 8, 2026Feiyu Yao, Zhixiong Niu, Xiaqing Li +3GPU Acceleration Self-Attention
Beijing University of Technology Beijing, China · Microsoft Research Beijing, China · Beijing Jiaotong University Beijing, China
Sparse Forcing: Native Trainable Sparse Attention for Real-time Autoregressive Diffusion Video Generation
Apr 23, 2026Boxun Xu, Yuming Du, Zichang Liu +7Autoregressive Diffusion Self-Attention
Meta Superintelligence Labs · University of California, Santa Barbara · Work done at Meta
AsyncTLS: Efficient Generative LLM Inference with Asynchronous Two-level Sparse Attention
Apr 9, 2026Yuxuan Hu, Jianchao Tan, Jiaqi Zhang +7LLM Inference Acceleration Long-Context Language Model Inference
School of Information, Renmin University of China, Beijing, China · Meituan, Beijing, China
(Sparse) Attention to the Details: Preserving Spectral Fidelity in ML-based Weather Forecasting Models
Apr 6, 2026Maksim Zhdanov, Ana Lucic, Max Welling +1Weather Forecasting Probabilistic Forecasting
AMLab · University of Amsterdam · CuspAI
Attention Sparsity is Input-Stable: Training-Free Sparse Attention for Video Generation via Offline Sparsity Profiling and Online QK Co-Clustering
Mar 19, 2026Jiayi Luo, Jiayu Chen, Jiankun Wang +6Efficient Video Generation Video Generation
SKLCCSE, School of Computer Science and Engineering, Beihang University · Zhongguancun Academy · School of Computer Science, Peking University +4
SVG-EAR: Parameter-Free Linear Compensation for Sparse Video Generation via Error-aware Routing
Mar 9, 2026Xuanyi Zhou, Qiuyang Mang, Shuo Yang +7Video Diffusion Models Efficient Attention
UC Berkeley, USA
FlashPrefill: Instantaneous Pattern Discovery and Thresholding for Ultra-Fast Long-Context Prefilling
Mar 6, 2026Qihang Fan, Huaibo Huang, Zhiying Wu +3Long-Context Language Model Inference LLM Inference Acceleration
MAIS&NLPR, CASIA · UCAS · WeChat, Tencent
Attention-Mass Condensation for Sparse Decoding
Feb 6, 2026Jorge L. Ruiz WilliamsLanguage Model Decoding Sparse Attention
NaNZeta LLC
PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformers
Feb 1, 2026Haopeng Li, Shitong Shao, Wenliang Zhong +4Diffusion Model Inference Acceleration Attention Approximation
The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology
Block Sparse Flash Attention
Dec 7, 2025Daniel Ohayon, Itay Lamprecht, Itay Hubara +3GPU Acceleration Self-Attention
Technion – Israel Institute of Technology, Haifa, Israel · Intel - Habana Labs · Intel – Habana Labs, Tel Aviv, Israel
NABLA: Neighborhood Adaptive Block-Level Attention
Jul 17, 2025Dmitrii Mikhailov, Aleksey Letunovskiy, Maria Kovaleva +6Video Diffusion Models Diffusion Transformer
Kandinsky Lab, Moscow, Russia

May 29, 2026J/K move · Enter open · S save

LVSA: Training-Free Sparse Attention for Long Video Diffusion

Gael Glorian, Ioannis Lamprou, Zhen Zhang, Yujie Yuan+1

Distributed Parallel Technology Laboratory, Paris Research Center, Huawei Technologies France · AI Framework and Data Technology Lab, Huawei Technologies Co., Ltd.

Video Diffusion Models Long-Horizon Video Generation Sparse Attention Diffusion Model Inference Acceleration Temporal Consistency in Video Generation Block-Sparse Attention

Open paper AskPDF arXiv

Dense self-attention is the compute and quality bottleneck of long-video diffusion inference: cost grows quadratically with the sequence length, and beyond the training horizon the model converges to near-static output, that is, "frozen" repetitive video. State of the art approaches are either too costly, e.g., they require retraining, or fail to satisfy both performance and quality objectives in a scalable manner. To this end, we introduce Long Video Sparse Attention (LVSA), a training-free model-agnostic block-sparse attention for video diffusion transformers that combines a structured window pattern with rotating global anchors, thus removing the fixed-grid bias which causes long-range temporal artifacts. LVSA, combined with a FlashInfer kernel, reduces compute up to 3.17x on Wan 2.1 1.3B at a 6x horizon, 2.98x on Wan 2.1 14B at a 6x horizon, and 3.33x on HunyuanVideo 1.5 at a 1.5x horizon, compared to dense attention. Beyond reducing compute, LVSA enables HunyuanVideo 1.5 generation at a 2x horizon, which is otherwise out-of-memory on a single GPU. Moreover, LVSA provides speedups up to 2.41x compared to RIFLEx and 3.27x compared to UltraViCo on Wan 2.1 1.3B. To demonstrate applicability across diverse platforms, we apply LVSA on NPUs and achieve speedups up to 2.71x on Wan 2.2 A14B and 3.24x on Wan 2.1 1.3B compared to dense attention. To evaluate quality in a fair way, we introduce VQeval, a tool properly scoring loopy video failures, which instead are rewarded in state of the art evaluators like VBench-Long. LVSA is quality-neutral for generation at training horizon length and quality-positive at extended lengths.