Sparse Attention

Follow this topic Ask about this topic

Momentum

24 papers in the last four weeks, up 200% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 160

All topics

Veda: Scalable Video Diffusion via Distilled Sparse Attention
May 28, 2026Shihao Han, Hao Yang, Xinting Hu +3Video Diffusion Models Diffusion Transformer
ByteDance Inc. · University of Science and Technology of China · The University of Hong Kong
OSP-Next: Efficient High-Quality Video Generation with Sparse Sequence Parallelism, HiF8 Quantization, and Reinforcement Learning
May 27, 2026Yunyang Ge, Xianyi He, Zezhong Zhang +4Video Diffusion Models Text-to-Video Generation
1Peking University · 2Nanyang Technological University, Singapore · 3Rabbitpre AI
Augmenting Attention with Exponentially Decaying Memory Improves Query-Aware KV Sparsity
May 27, 2026Xiuying Wei, Caglar GulcehreMemory-Augmented Language Models Self-Attention
EPFL, CLAIRE Lausanne, Switzerland
UNIQUE: Universal Top-k Sparse Attention for Training-free Inference and Sparsity-aware Training
May 26, 2026Keqi Deng, Shaoshi Ling, Ruchao Fan +1Self-Attention Long-Context Language Model Inference
Microsoft, USA
Grammatically-Guided Sparse Attention for Efficient and Interpretable Transformers
May 23, 2026Spandan PratyushTransformer Interpretability Sparse Attention
Independent Researcher, New Delhi 110070, India
Inference Time Context Sparsity: Illusion or Opportunity?
May 22, 2026Sahil Joshi, Prithvi Dixit, Agniva Chowdhury +5LLM Inference Efficiency Structured Sparsity
α Rice University · β UC Berkeley · γ IIT Bombay
Vision Transformers Need Better Token Interaction
May 22, 2026Linxiang SuVisual Representation Learning Vision Transformer
University of Szeged
DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation
May 22, 2026Jie Hu, Zixiang Gao, Yutong He +1Video Diffusion Models Diffusion Transformer
1Peking University.
Structured-Sparse Attention for Entity Tracking with Subquadratic Sequence Complexity
May 21, 2026Hangyue Zhao, Paul Caillon, Erwan Fagnou +1Self-Attention Structured Sparsity
ESPCI PSL, Paris, France · LAMSADE, Université Paris Dauphine - PSL, Paris, France
EntmaxKV: Support-Aware Decoding for Entmax Attention
May 20, 2026Gonçalo Duarte, Miguel Couceiro, Marcos V. TrevisoSelf-Attention KV Caching
Instituto Superior Técnico, Universidade de Lisboa. · ELLIS Unit Lisbon. · INESC-ID. +1
PulseCol: Periodically Refreshed Column-Sparse Attention for Accelerating Diffusion Language Models
May 20, 2026Yanyi Lyu, Letian Chen, Futing Sun +3Self-Attention Diffusion Language Model Inference
Harbin Institute of Technology (Shenzhen)
LT2: Linear-Time Looped Transformers
May 20, 2026Chunyuan Deng, Yizhe Zhang, Rui-Jie Zhu +4Efficient Transformer Inference Recurrent Transformers
1Rice University · 2Apple · 3UC Santa Cruz +1
RoPeSLR: 3D RoPE-driven Sparse-LowRank Attention for Efficient Diffusion Transformers
May 20, 2026Yuxi Liu, Zekun Zhang, Yixiang Cai +3Rotary Positional Embeddings Low-Rank Attention
Peking University · University of Electronic Science and Technology of China · Beijing University of Posts and Telecommunications
Efficient Long-Context Modeling in Diffusion Language Models via Block Approximate Sparse Attention
May 19, 2026Wenhu Zhang, Yiming Wu, Huanyu Wang +6Self-Attention Long-Context Language Modeling
The Hong Kong University of Science and Technology · The University of Hong Kong · Zhejiang University +1
DashAttention: Differentiable and Adaptive Sparse Hierarchical Attention
May 18, 2026Yuxiang Huang, Nuno M. T. Gonçalves, Federico Alvetreti +5Self-Attention Long-Context Modeling
1Tsinghua University · 2Instituto Superior Técnico, Universidade de Lisboa. · 3Instituto de Telecomunicações. +6
SparseSAM: Structured Sparsification of Activations in Segment Anything Models
May 17, 2026Hoai-Chau Tran, Chi H. Nguyen, Duy M. H. Nguyen +3Efficient Transformer Inference Image Segmentation
University of Illinois at Urbana-Champaign · VinUni-Illinois Smart Health Center, VinUniversity · College of Engineering & Computer Science, VinUniversity +3
Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps
May 16, 2026Yanke Zhou, Yiduo Li, Hanlin Tang +6Self-Attention Long-Context Language Model Inference
Nanjing University · Alibaba Group
CompactAttention: Accelerating Chunked Prefill with Block-Union KV Selection
May 16, 2026Jiwon Song, Dongwon Jo, Beomseok Kang +1Self-Attention KV-Cache Management
Seoul National University
From Sparsity to Simplicity: Enabling Simpler Sequential Replacements via Sparse Attention Distillation
May 15, 2026Yuxin Ren, Maxwell D Collins, Miao Hu +1Self-Attention Vision Transformer
University of Arizona, Tucson AZ, USA · TetraMem, Inc., San Jose CA, USA
ElasticDiT: Efficient Diffusion Transformers via Elastic Architecture and Sparse Attention for High-Resolution Image Generation on Mobile Devices
May 15, 2026Kunpeng Du, Haizhen Xie, Sen Lu +11Diffusion Transformer Diffusion-Based Image Generation
Huawei Technologies
STS: Efficient Sparse Attention with Speculative Token Sparsity
May 15, 2026Ceyu Xu, Jiangnan Yu, Yongji Wu +1Self-Attention Long-Context Language Model Inference
The Hong Kong University of Science and Technology · UC Berkeley
HASTE: Training-Free Video Diffusion Acceleration via Head-Wise Adaptive Sparse Attention
May 14, 2026Xuzhe Zheng, Yuexiao Ma, Jing Xu +3Video Diffusion Models Diffusion Transformer
1Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China
TurboVGGT: Fast Visual Geometry Reconstruction with Adaptive Alternating Attention
May 14, 2026David Huang, Guile Wu, Chengjie Huang +23D ViTs Multi-View 3D Reconstruction
Huawei Noah’s Ark Lab · University of Toronto · Foundation Model Department, Huawei
Learning to See What You Need: Gaze Attention for Multimodal Large Language Models
May 13, 2026Junha Song, Byeongho Heo, Geonmo Gu +3Efficient Multimodal Inference Visual Attention
1KAIST · 2NAVER AI Lab
Do Vision Transformers Need All-to-All Attention? Global Communication Through Elastic Learned Cores
May 12, 2026Alan Z. Song, Yinjie Chen, Mu Nan +3Visual Representation Learning Vision Transformer
Carnegie Mellon University · University of Hong Kong
Learning Subspace-Preserving Sparse Attention Graphs from Heterogeneous Multiview Data
May 12, 2026Jie Chen, Yuanbiao Gou, Chuanbin Liu +2Graph Structure Learning Multi-View Learning
College of Computer Science Sichuan University Chengdu, China · School of Economics and Management China University of Petroleum (Beijing) Beijing, China · Law School Sichuan University Chengdu, China +1
An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inference
May 8, 2026Feiyu Yao, Zhixiong Niu, Xiaqing Li +3GPU Acceleration Self-Attention
Beijing University of Technology Beijing, China · Microsoft Research Beijing, China · Beijing Jiaotong University Beijing, China
MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inference
May 8, 2026Ruijie Zhou, Fanxu Meng, Yufei Xu +4Self-Attention Long-Context Language Model Inference
Sparse Attention as a Range Searching Problem: Towards an Inference-Efficient Index for KV Cache
May 7, 2026Mohsen Dehghankar, Abolfazl AsudehSelf-Attention KV Caching
Department of Computer Science University of Illinois Chicago
LIVEditor-14B: Lightning Unified Video Editing via In-Context Sparse Attention
May 6, 2026Shitong Shao, Zikai Zhou, Haopeng Li +4Video Editing In-Context Learning
Hong Kong University of Science and Technology (GZ), Hong Kong · University of Arizona, USA

May 28, 2026, cs.CVJ/K move · Enter open · S save

Veda: Scalable Video Diffusion via Distilled Sparse Attention

Shihao Han, Hao Yang, Xinting Hu, Xiaofeng Mei+2

ByteDance Inc. · University of Science and Technology of China · The University of Hong Kong

Video Diffusion Models Diffusion Transformer Sparse Attention Diffusion Model Inference Acceleration

Open paper AskPDF arXiv

Scaling Diffusion Transformers to generate high-resolution, long videos is constrained by the quadratic cost of self-attention, and existing sparse attention methods degrade under high sparsity. We show empirically that generation quality is determined not by the sparsity ratio itself, but by how well the sparse mask aligns with the tile-wise geometry of full attention. Based on this insight, we propose Veda, a distilled sparse attention framework that formulates tile selection as an explicit reconstruction problem from full attention. Veda integrates statistics-aware tile scoring with head-aware tiling to reduce estimation error and structural mismatch, enabling aggressive sparsity. A hardware-efficient tile-skipping kernel converts theoretical sparsity into practical wall-clock speedups. Experiments on large video diffusion models, including Waver and Wan2.1, demonstrate substantial acceleration with no noticeable degradation in generation quality. To generate 720P 10-second videos on Waver-T2V-12B, Veda achieves a 5.1

\times

end-to-end speedup and a 10.5

\times

self-attention speedup, reducing attention overhead from 92% to 50%. Notably, the gains increase with sequence length, indicating that Veda scales favorably with spatiotemporal resolution across models.

Momentum

Related topics

Latest papers 160