Sparse Attention

Follow this topic Ask about this topic

Sparse attention studies how Transformers can focus computation on a subset of token interactions instead of comparing every query with every key, reducing compute and memory costs as sequences grow. Approaches include fixed local or block patterns, content-adaptive top-k routing, hierarchical KV-cache retrieval, and approximations based on clustering, low-rank structure, or feature sparsity; specialized kernels and cache systems help turn these savings into practical speedups. Methods are used in long-context language models and in video, image, and other settings, with evaluation often measuring both task quality on benchmarks such as LongBench and RULER and latency, throughput, or memory use. A central challenge is balancing accuracy with real end-to-end efficiency, since selecting important tokens can itself be costly and aggressive sparsity may miss information or behave unpredictably across inputs, layers, and hardware.

Latest papers

244 papers52 papers in the last four weeks, up 100% on the four weeks before. 0.3% of all new papers.

SPARTA: A Sparse Ternary LLM Accelerator with Adaptive Attention and Unified Datapath for Long-Context Edge Inference
Oct 26, 2026Yanzhen Zhou, Zhengyu Han, Qunyou Liu +3AI Accelerator Inference LLM Inference Acceleration
Zhejiang University · Veloxis AI
INTRA: Interleaved Non-contiguous Token spaRse Attention
Oct 25, 2026Yuxuan Xia, Wenbo Lu, Shaoyi Zheng +1Sparse Attention Transformer Inference
New York University
Cross-Scale Sparse Attention for Referring Segmentation in Remote Sensing
Oct 24, 2026Yuxi Li, Lu Si, Xia Liao +4Referring Image Segmentation Remote Sensing Image Segmentation
Westlake Institute for Advanced Study, Hangzhou, Zhejiang, China · School of Engineering, Westlake University, Hangzhou, Zhejiang, China · Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing, China +2
HyScale: Scaling Long-Context Training for Learned Sparse Attention
Oct 23, 2026Yangshen Deng, Xinjing Huang, Yulong Wang +8Efficient Language Model Training Sparse Attention
University of Edinburgh · Tencent Hunyuan
Nested Sparsity for Scalable Pretraining and Continual Learning
Oct 22, 2026Tianzhe Chu, Druv Pai, Yi MaConditional Computation Sparse Attention
University of Hong Kong · UC Berkeley
OpenAttention: One Percent Is Enough with Tail Proxies
Oct 22, 2026Hyunwoo Yu, Jongbeom Lee, Hyunbin Kim +8Efficient Attention Sparse Attention
Sogang University · Samsung Electronics · Pusan National University
BlazeVGGT: 13x Faster VGGT at 2,000 Frames via Camera Token without Training
Oct 21, 2026Shujun Yao, Junhui Lin, Congying Han +3Multi-View 3D Reconstruction Feed-Forward 3D Reconstruction
University of Chinese Academy of Sciences, Beijing, China · the University of Maryland, College Park, MD, USA · Beijing University of Posts and Telecommunications, Beijing, China
Sparsity Should Be ALLOTted by Output: Training-Free Allocation of Block-Sparse Attention for Video Diffusion Transformers
Oct 21, 2026Tianze Xia, Xiaoling Zhou, Ruya Lan +3Sparse Attention Diffusion Model Inference Acceleration
UCAS · CASIA · KlingAI Research +1
Stratified Sparse Attention: Reducing the Sparse-to-Dense Gap at Extreme Sparsity
Oct 20, 2026Sahil Joshi, Prithvi Dixit, Aryan Katiyar +5Sparse Attention Long-Context Language Model Inference
Department of Computer Science, Rice University, TX, USA. · Department of Electrical Engineering and Computer Sciences, UC Berkeley, CA, USA. · Department of Computer Science, Stanford University, CA, USA. +1
MLIPS: Multi-Layer Indexer Pool Sharing
Oct 20, 2026Samkit Jain, Sai Kishan Pampana, Shlok Agarwal +4LLM Inference Acceleration Sparse Attention
NVIDIA Corporation
This is SParTA: A modular, efficient and scalable transformer architecture for jet-tagging
Oct 19, 2026Rajneil Baruah, Subhadeep Mondal, Sunando Kumar Patra +1Transformer Jet Tagging
Department of Physics, SEAS, Bennett University, Greater Noida, Uttar Pradesh, 201310, India · Department of Physics, Bangabasi Evening College, Kolkata, 700009, West Bengal, India
Dynamics-Aware Sparse Attention for Efficient Autoregressive Video Diffusion
Oct 19, 2026Yuanyu He, Zhuokun Chen, Yefei He +5Video Diffusion Models Efficient Video Generation
Zhejiang University · Alibaba Group · Monash University
Certified Sparse Attention via Perturbed Retrieval
Oct 18, 2026Runguo LiSparse Attention LLM Inference Acceleration
University of Illinois Urbana-Champaign
Prism Attention: Proposal-Refined Index Sharing Mechanism for Efficient LLMs Inference
Oct 16, 2026Zhenxu Tian, Kebin Liu, Zhengwu Yang +8Long-Context Language Model Inference LLM Inference Acceleration
Soochow University · Baidu Inc.
Sink-Sparse Hybrid Attention for Efficient Inference in T2I/T2V Diffusion Transformers
Oct 16, 2026Bingxin Xing, Yu Zhang, Jiatong Ding +2Diffusion Model Inference Acceleration Sparse Attention
Xi’an Jiaotong University · Xiamen University · Shanghai Jiao Tong University
Velox: Spatial Memory Makes World Models Efficient
Oct 15, 2026Qiuhong Shen, Shizun Wang, Xinchao WangAttention Routing Video World Models
National University of Singapore
Sparse Attention for Video Generation Acceleration via Growing Sparsity and Reduced Search
Oct 15, 2026Yun Dai, Yuyun Chen, Jiarui Wen +3Sparse Attention Diffusion Model Inference Acceleration
South China University of Technology
Joint Subspace Sparse Attention for Efficient Long-Context Decoding
Oct 13, 2026Jiwon Song, Yoongon Kim, Jae-Joon KimLLM Inference Acceleration Sparse Attention
Seoul National University
Kwai Summary Attention V2 Technical Report
Oct 13, 2026OneRec Team, Chenglong Chu, Guohong Mu +30KV-Cache Compression Hybrid Attention
FlashMask-3: Efficient and Expressive Mask-Aware Distributed Attention
Oct 13, 2026Guoxia Wang, Qianyue He, Siming Wu +6Context Parallelism Attention Mechanisms
Baidu Inc.
Prefilling-dLLM v2: Learned Chunk Routing for Long-Context Diffusion Language Models
Oct 13, 2026Qi Han, Jing Xiong, Ziyuan Zhang +1Diffusion Language Model Inference Diffusion Language Models
University of Hong Kong · Zhejiang University
FreeSolo: Linear-time Sparse Attention with Zero-Shot Length Extrapolation
Oct 12, 2026Zekun Shi, Alastair James Arthur PriceLong-Context Language Modeling Long-Context Modeling
National University of Singapore · Sea AI Lab, Singapore · University of Toronto
Competition Domains Matter: Selective Head Globalization for Training-Free Sparse Attention in Video Diffusion Transformers
Oct 10, 2026Tiantian Liu, Meng Wan, Zhiqiang Liang +3Sparse Attention Diffusion Model Acceleration
University of Chinese Academy of Sciences, Beijing, China
S $^3$ R: Fast and Faithful RL Rollouts via Self-Sparse Speculative Decoding
Oct 9, 2026Yuhang He, Hongyu Ge, Zihan Lin +2Speculative Decoding LLM Inference Acceleration
The Hong Kong University of Science and Technology (Guangzhou)
Zebra3D: Efficient Hybrid Foundation Models For Scalable 3D Reconstruction
Oct 8, 2026Utkarsh Saxena, Ruijing Yang, Mehdi Rezagholizadeh +33D Foundation Models Feed-Forward 3D Reconstruction
Advanced Micro Devices Inc. (AMD)
PageWeaver: KV-Guided Query Unions for Sparse Attention
Oct 8, 2026Zhiyuan Li, Zihan Li, Zefang Yuan +2Sparse Attention GPU Kernel Optimization
SenseTime
QUILT: Rethinking Sparse-Attention Prefill through Shared Query Execution
Oct 8, 2026Zhenduo Zhao, Qihui Zhou, Mingcong Song +7LLM Inference Acceleration Sparse Attention
Huawei Technologies Co., Ltd. China
Sparse Attention Is Matrix Approximation, Not Choosing from a Bag of Values
Oct 7, 2026Fang Wan, Xufeng Liu, Fan Li +1Sparse Attention Efficient Attention
Stony Brook University · Duke University
Attention via Black-Box Vector Search
Oct 7, 2026Stepan Zharkov, Krish Singal, Ashwin Padaki +1Attention Approximation Efficient Attention
University of Pennsylvania
Evaluating Trajectory Features for Routing Final-Layer Attention
Oct 7, 2026Yupeng YaoAttention Routing LLM Routing

Oct 26, 2026J/K move · Enter open · S save

SPARTA: A Sparse Ternary LLM Accelerator with Adaptive Attention and Unified Datapath for Long-Context Edge Inference

Yanzhen Zhou, Zhengyu Han, Qunyou Liu, Zheyu Yan+2

Zhejiang University · Veloxis AI

AI Accelerator Inference LLM Inference Acceleration On-Device Language Model Inference Sparse Attention

Open paper AskPDF arXiv

LLMs on edge devices are constrained by limited memory and bandwidth. Although 1.58-bit ternary weights greatly reduce model storage, long-context inference is increasingly bottlenecked by KV-cache traffic and attention. Existing accelerators also use separate datapaths for ternary linear layers and attention, incurring hardware and execution overhead. We present SPARTA, a sparse ternary LLM accelerator for efficient long-context edge inference. SPARTA integrates: (1) a phase-aware sparse dataflow exploiting heterogeneous sparsity across compute-bound prefill and memory-bound decode; (2) a unified compute fabric mapping ternary SpMM/SpMV and bit-serial attention onto shared processing elements to exploit zero-weight and zero-bit sparsity; and (3) an adaptive two-stage attention engine combining page-level Top-k selection to reduce K-cache accesses with token-level Top-p pruning to reduce V-cache traffic. SPARTA reduces KV traffic by 4.1-4.8x over full attention and 1.3-1.5x over a state-of-the-art sparse-attention scheme, with task-dependent quality loss. Compared with NVIDIA Jetson AGX Orin, SPARTA achieves 3.9x higher generation throughput and 8.3x higher end-to-end energy efficiency on average; versus AMD Ryzen 7 9700X, the gains are 5.0x and 81.7x. It also delivers 1.6-12.0x higher throughput, 2.0-4.2x higher energy efficiency, and 1.3-51.4x higher area efficiency than FIGLUT, TeLLMe, and MCBP.