Block-Sparse Attention

Follow this topic Ask about this topic

Momentum

4 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 30

All topics

SPIN: Shadow Predictive Indexer for Sparse Attention
Oct 6, 2026Yao Fu, Jiahan Chang, Ritchie Zhao +11LLM Inference Acceleration Sparse Attention
PHBA: Prefix-State Hybrid Block Attention
Oct 6, 2026Ruijie Li, Jiaxi Hu, Shiyu Wang +1Long-Context Modeling Linear Attention
The Hong Kong University of Science and Technology (Guangzhou) · Independent researcher
Prism: Dynamic Sparse Attention for Native 2K Joint Video-Audio Generation Model Training
Oct 4, 2026Shuyuan Tu, Qi Tian, Yinming Huang +8Audio-Video Generation Video Generation
Fudan University · Tencent Hunyuan Foundation Model Team · Zhejiang University
PARK: Accurate Block Retrieval for Sparse Attention in Video Diffusion Transformers
Sep 30, 2026Yun Dai, Jiarui Wen, Huiping Zhuang +2Video Diffusion Models Diffusion Transformer
South China University of Technology
ReSS: Residual-Restoring Sparse Attention for 3D Vision Transformers
Sep 28, 2026Yongsung Kim, Jaehoon Lee, Minjun Park +33D ViTs Efficient ViTs
IPAI, 2ECE, 3AIIS, ASRI, INMC, ISRC Seoul National University · ECE Seoul National University · AIIS Seoul National University
Attention Routing Stabilizes Early: Working-Set Inference for Recurrent Language Models
Sep 23, 2026Ke Wan, Chen ChenLong-Context Language Model Inference Recurrent Transformers
Department of Computer Science University of Virginia Charlottesville, VA, USA
Faster Than Flash: Exploiting Attention Sparsity for Efficient Long-Context Decoding
Aug 31, 2026Zhigeng Liu, Zhiyuan Ning, Ruixiao Li +5Long-Context Language Model Inference LLM Inference Acceleration
Fudan University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China · Harbin Institute of Technology, Harbin, China
LoSA: Near-Lossless Sparse Attention for Training-Free Video Diffusion Acceleration
Aug 12, 2026Enhuai Liu, Yunke Wang, Yutong Wang +2Video Diffusion Models Diffusion Transformer
1The University of Sydney · 2CSIRO
SPADE: An Input-Adaptive Sparse Attention Engine for Fast Video Diffusion Models Inference
Aug 4, 2026Shanghao Liu, Renze Chen, Size Zheng +4Video Diffusion Models Diffusion Transformer
Eric · Beihang University · Peking University
Token Radius Attention for Efficient Video Generation
Aug 3, 2026Jiayu Chen, Zhikun Jiang, Maoliang Li +6Video Diffusion Models Sparse Attention
School of Computer Science, Peking University · School of Electronics Engineering and Computer Science, Peking University · School of Computer Science and Engineering, Beihang University +2
Understanding Sparse Attention Selectivity in Long-Context Foundation Models via Counterfactual Evaluation
Aug 3, 2026Xingyu Ren, Youran Sun, Chugang Yi +1LLM Auditing Sparse Attention
The Chinese University of Hong Kong, Hong Kong, China · University of Maryland, College Park, MD, USA
CoSA: Accelerating Long-Context Inference via Proxy-Kernel Co-Designed Sparse Attention
Jul 28, 2026Yufei Xue, Lin Niu, Hong Liu +6Long-Context Language Model Inference LLM Inference Acceleration
1Tencent · 2The Hong Kong University of Science and Technology
Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsification
Jul 27, 2026Haopeng Li, Yitong Li, Junsong Chen +8Video Diffusion Models Sparse Attention
NVIDIA
FVAttn: Adaptive Sparse Attention with Runtime Load Balancing for Video Generation
Jul 17, 2026Hao Liu, Chenghuan Huang, Ye Huang +7Video Diffusion Models Diffusion Transformer
Sun Yat-sen University · WeChat HPC, Tencent Inc. · Peking University +1
COBS: Cumulant Order Block Sparse Attention
Jul 10, 2026Alexander Tian, Aditya Ghai, Sanjit Neelam +2KV Caching LLM Inference Acceleration
MatX
Uncertainty-gated selection for block-sparse attention
Jul 4, 2026Thomas RossiLong-Context Language Model Inference LLM Inference Acceleration
Eonpass
ScalingAttention: Discovering Intrinsic Sparse Attention Topology for Video Diffusion Transformers
Jun 22, 2026Ruiliang Zhou, Xuecheng Wu, Kang He +6Video Diffusion Models Diffusion Transformer
KlingAI Research · Beijing Institute of Technology · NVIDIA +1
SpotAttention: Plug-In Block-Sparse Routing for Pretrained Long-Context Transformers
Jun 22, 2026Huzama Ahmad, Se-Young YunLong-Context Language Model Inference LLM Inference Acceleration
KAIST
MiniMax Sparse Attention
Jun 11, 2026Xunhao Lai, Weiqi Xu, Yufeng Yang +14GPU Kernel Optimization Long-Context Language Modeling
MiniMax · Peking University · NVIDIA +4
Light Interaction: Training-Free Inference Acceleration for Interactive Video World Models
May 29, 2026Jiacheng Lu, Haoyi Zhu, Sipei Yi +3Real-Time Interactive Video Generation Interactive World Models
1Zhejiang University · 2NVIDIA
LVSA: Training-Free Sparse Attention for Long Video Diffusion
May 29, 2026Gael Glorian, Ioannis Lamprou, Zhen Zhang +2Video Diffusion Models Long-Horizon Video Generation
Distributed Parallel Technology Laboratory, Paris Research Center, Huawei Technologies France · AI Framework and Data Technology Lab, Huawei Technologies Co., Ltd.
DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation
May 22, 2026Jie Hu, Zixiang Gao, Yutong He +1Video Diffusion Models Diffusion Transformer
1Peking University.
Efficient Long-Context Modeling in Diffusion Language Models via Block Approximate Sparse Attention
May 19, 2026Wenhu Zhang, Yiming Wu, Huanyu Wang +6Self-Attention Long-Context Language Modeling
The Hong Kong University of Science and Technology · The University of Hong Kong · Zhejiang University +1
An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inference
May 8, 2026Feiyu Yao, Zhixiong Niu, Xiaqing Li +3GPU Acceleration Self-Attention
Beijing University of Technology Beijing, China · Microsoft Research Beijing, China · Beijing Jiaotong University Beijing, China
Sparse Forcing: Native Trainable Sparse Attention for Real-time Autoregressive Diffusion Video Generation
Apr 23, 2026Boxun Xu, Yuming Du, Zichang Liu +7Autoregressive Diffusion Self-Attention
Meta Superintelligence Labs · University of California, Santa Barbara · Work done at Meta
SVG-EAR: Parameter-Free Linear Compensation for Sparse Video Generation via Error-aware Routing
Mar 9, 2026Xuanyi Zhou, Qiuyang Mang, Shuo Yang +7Video Diffusion Models Efficient Attention
UC Berkeley, USA
Attention-Mass Condensation for Sparse Decoding
Feb 6, 2026Jorge L. Ruiz WilliamsLanguage Model Decoding Sparse Attention
NaNZeta LLC
Block Sparse Flash Attention
Dec 7, 2025Daniel Ohayon, Itay Lamprecht, Itay Hubara +3GPU Acceleration Self-Attention
Technion – Israel Institute of Technology, Haifa, Israel · Intel - Habana Labs · Intel – Habana Labs, Tel Aviv, Israel
NABLA: Neighborhood Adaptive Block-Level Attention
Jul 17, 2025Dmitrii Mikhailov, Aleksey Letunovskiy, Maria Kovaleva +6Video Diffusion Models Diffusion Transformer
Kandinsky Lab, Moscow, Russia
SparSTAR: Sparse Attention for SpaceTime AutoRegressive Video Synthesis
Date pendingJongbeom Lee, Hyunwoo Yu, Jincheol Yang +2Image-to-Video Generation Self-Attention
Sogang University Seoul, Republic of Korea

Oct 6, 2026, cs.LGJ/K move · Enter open · S save

SPIN: Shadow Predictive Indexer for Sparse Attention

Yao Fu, Jiahan Chang, Ritchie Zhao, Bryce Long+10

LLM Inference Acceleration Sparse Attention Long-Context Language Model Inference Block-Sparse Attention

Open paper AskPDF arXiv

Indexer-based sparse attention reduces the cost of core attention by passing only a fixed, small number of important tokens to it. However, the indexer must still score the entire KV cache at every decoding step. This scoring overhead becomes a major bottleneck as the context length grows. We propose SPIN (Shadow Predictive Indexer) to reduce this indexer overhead. SPIN uses lightweight, history-based prediction to identify important KV blocks, avoiding the need to score the full KV cache at every decoding step. SPIN treats KV blocks and speculative decoding as first-class design and implementation considerations. Across extensive evaluations on long-context and agentic benchmarks, SPIN achieves 30-40% sparsity while preserving task quality. In end-to-end vLLM serving, SPIN improves output throughput by up to 14.9% and reduces median inter-token latency by up to 13.2%.

Momentum

Related topics

Latest papers 30