Sparse Attention

Follow this topic Ask about this topic

Sparse attention studies how Transformers can focus computation on a subset of token interactions instead of comparing every query with every key, reducing compute and memory costs as sequences grow. Approaches include fixed local or block patterns, content-adaptive top-k routing, hierarchical KV-cache retrieval, and approximations based on clustering, low-rank structure, or feature sparsity; specialized kernels and cache systems help turn these savings into practical speedups. Methods are used in long-context language models and in video, image, and other settings, with evaluation often measuring both task quality on benchmarks such as LongBench and RULER and latency, throughput, or memory use. A central challenge is balancing accuracy with real end-to-end efficiency, since selecting important tokens can itself be costly and aggressive sparsity may miss information or behave unpredictably across inputs, layers, and hardware.

Latest papers

244 papers52 papers in the last four weeks, up 100% on the four weeks before. 0.3% of all new papers.

DeltaLLM: Delta-Key Sparse Attention for Edge LLMs
Jul 25, 2025Jiawen Qi, Jiayu Liu, Chang Gao +2On-Device Language Model Inference LLM Inference Acceleration
Leiden Institute of Advanced Computer Science (LIACS), Leiden University, 2300 RA Leiden, The Netherlands · Department of Microelectronics, Delft University of Technology, 2628 CD Delft, The Netherlands
NABLA: Neighborhood Adaptive Block-Level Attention
Jul 17, 2025Dmitrii Mikhailov, Aleksey Letunovskiy, Maria Kovaleva +6Video Diffusion Models Diffusion Transformer
Kandinsky Lab, Moscow, Russia
The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs
Apr 24, 2025Piotr Nawrot, Robert Li, Renjie Huang +3Transformer Inference LLM Evaluation
University of Edinburgh · Cohere · Meta
Fewer yet critical: Reducing Redundant Token Dependencies for Transformer-based Time Series Forecasting
Mar 10, 2025Jianqi Zhang, Yuchan Liu, Zeen Song +2Time Series Forecasting Transformer Attention
National Key Laboratory of Space Integrated Information Systems, Institute of Software, Chinese Academy of Sciences, Beijing, China 1 · University of Chinese Academy of Sciences, Beijing, China 2 · Digital-Intelligent Operations Center, Guizhou Power Grid Co., Ltd., Guizhou, China 3 +1

Jul 25, 2025J/K move · Enter open · S save

DeltaLLM: Delta-Key Sparse Attention for Edge LLMs

Jiawen Qi, Jiayu Liu, Chang Gao, Zhaochun Ren+1

Leiden Institute of Advanced Computer Science (LIACS), Leiden University, 2300 RA Leiden, The Netherlands · Department of Microelectronics, Delft University of Technology, 2628 CD Delft, The Netherlands

On-Device Language Model Inference LLM Inference Acceleration Sparse Attention

Open paper AskPDF arXiv

On-device LLM inference is increasingly important for edge platforms but remains challenging due to high computational and memory demands and tight resource constraints. Modern applications often involve long prompts. Although compact models reduce linear-layer costs through weight compression, attention remains largely unoptimized and becomes the dominant bottleneck, especially during prefilling. Existing attention pruning methods primarily target highly parallel hardware (e.g., GPUs) and long context windows, limiting their suitability for edge scenarios. We present DeltaLLM, a training-free LLM framework based on algorithm-hardware-dataflow co-design to accelerate prefilling attention. DeltaLLM introduces a causal-attention-aware K-only delta formulation that enables Delta-Key reuse across query heads under Grouped-Query Attention (GQA). A context-aware hybrid strategy preserves exact attention near the diagonal while applying delta approximation to nonlocal regions. For score-value multiplication, we propose a vectorized structured sparse online softmax (VSSOS) that converts fine-grained spatial score sparsity into hardware-friendly column-level skipping. A specialized vector-processor architecture and unified sparse/dense attention dataflow exploit the resulting sparsity. Evaluated on BitNet-b1.58-2B-4T and LLaMA3.2-1B-Instruct, DeltaLLM achieves up to 60% Delta-Key sparsity and 63% structured attention-score sparsity with negligible average accuracy loss. On the KV260 FPGA board, it delivers a 1.62x speedup over TeLLMe-v2 while reducing DSP and BRAM usage. These results show that DeltaLLM supports efficient edge LLM deployment and integration with existing pipelines without fine-tuning.

Latest papers

DeltaLLM: Delta-Key Sparse Attention for Edge LLMs

NABLA: Neighborhood Adaptive Block-Level Attention

The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs

Fewer yet critical: Reducing Redundant Token Dependencies for Transformer-based Time Series Forecasting