Sparse Attention

Follow this topic Ask about this topic

Momentum

25 papers in the last four weeks, up 213% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 164

All topics

An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inference
May 8, 2026Feiyu Yao, Zhixiong Niu, Xiaqing Li +3GPU Acceleration Self-Attention
Beijing University of Technology Beijing, China · Microsoft Research Beijing, China · Beijing Jiaotong University Beijing, China
MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inference
May 8, 2026Ruijie Zhou, Fanxu Meng, Yufei Xu +4Self-Attention Long-Context Language Model Inference
Sparse Attention as a Range Searching Problem: Towards an Inference-Efficient Index for KV Cache
May 7, 2026Mohsen Dehghankar, Abolfazl AsudehSelf-Attention KV Caching
Department of Computer Science University of Illinois Chicago
LIVEditor-14B: Lightning Unified Video Editing via In-Context Sparse Attention
May 6, 2026Shitong Shao, Zikai Zhou, Haopeng Li +4Video Editing In-Context Learning
Hong Kong University of Science and Technology (GZ), Hong Kong · University of Arizona, USA
StreamIndex: Memory-Bounded Compressed Sparse Attention via Streaming Top-k
May 4, 2026Jaber Jaber, Osama JaberGPU Kernel Optimization Self-Attention
RightNow AI
Stochastic Sparse Attention for Memory-Bound Inference
May 3, 2026Kyle Lee, Corentin Delacour, Kevin Callahan-Coray +5GPU Acceleration Self-Attention
Department of Electrical and Computer Engineering, University of California, Santa Barbara, CA, USA · Department of Electrical and Computer Engineering, University of Michigan, Ann Arbor, MI, USA · Department of Electrical and Computer Engineering, Carnegie Mellon University, Pittsburgh, PA, USA
Spectral Dynamic Attention Network for Hyperspectral Image Super-Resolution
Apr 30, 2026Tengya Zhang, Feng Gao, Lin Qi +2Hyperspectral Image Super-Resolution Hyperspectral Imaging
Sanya Oceanographic Institution, Ocean University of China, Sanya, China · State Key Laboratory of Physical Oceanography, Ocean University of China, Qingdao 266100, China · Department of Electrical and Computer Engineering, Mississippi State University, Starkville, MS 39762 USA
Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving
Apr 29, 2026Zihan Zhao, Baotong Lu, Shengjie Lin +8LLM Serving KV-Cache Offloading
University of Virginia · Microsoft Research · Georgia Institute of Technology +1
Salca: A Sparsity-Aware Hardware Accelerator for Efficient Long-Context Attention Decoding
Apr 27, 2026Wang Fan, Wei Cao, Xi Zha +5Self-Attention Long-Context Language Model Inference
Fudan University
SparseContrast: Dynamic Sparse Attention for Efficient and Accurate Contrastive Learning in Medical Imaging
Apr 27, 2026Paarth Prasad, Ruchika MalhotraContrastive Learning Chest X-Ray Classification
Department of Software Engineering, Delhi Technological University
DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
Apr 26, 2026DeepSeek-AI, Anyi Xu, Bangcai Lin +316Mixture-of-Experts Language Models Long-Context Language Model Inference
BSViT: A Burst Spiking Vision Transformer for Expressive and Efficient Visual Representation Learning
Apr 25, 2026Hongxiang Peng, Dewei Bai, Hong QuSpiking Transformers Efficient ViTs
School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu 610054, China
SpikingBrain2.0: Brain-Inspired Foundation Models for Efficient Long-Context and Cross-Platform Inference
Apr 24, 2026Yuqi Pan, Jinghao Zhuang, Yupeng Feng +16Long-Context Language Model Inference Spiking Neural Networks
Institute of Automation, Chinese Academy of Sciences · The Hong Kong Polytechnic University · Beijing Key Laboratory of Brain-Inspired General Intelligence Large Model +1
HubRouter: A Pluggable Sub-Quadratic Routing Primitive for Hybrid Sequence Models
Apr 24, 2026Abhinaba BasuTransformer Attention Language Modeling
National Institute of Electronics and Information Technology (NIELIT)
Sculpt4D: Generating 4D Shapes via Sparse-Attention Diffusion Transformers
Apr 23, 2026Minghao Yin, Wenbo Hu, Jiale Xu +2Diffusion Transformer 3D Shape Generation
1The University of Hong Kong · 2ARC Lab, Tencent PCG
Sparse Forcing: Native Trainable Sparse Attention for Real-time Autoregressive Diffusion Video Generation
Apr 23, 2026Boxun Xu, Yuming Du, Zichang Liu +7Autoregressive Diffusion Self-Attention
Meta Superintelligence Labs · University of California, Santa Barbara · Work done at Meta
DynamicRad: Content-Adaptive Sparse Attention for Long Video Diffusion
Apr 22, 2026Yongji Long, Shijun Liang, Jintao Li +1Video Diffusion Models Long-Horizon Video Generation
University of Electronic Science and Technology of China · Shenzhen Institute for Advanced Study, UESTC · Computational Mathematics, Science, & Engineering at Michigan State University (MSU)
Simplified Sparse Attention via Gist Tokens
Apr 22, 2026Yuzhen Mao, Michael Y. Li, Emily B. FoxSelf-Attention Long-Context Language Model Inference
Stanford University
AdaCluster: Adaptive Query-Key Clustering for Sparse Attention in Video Generation
Apr 20, 2026Haoyue Tan, Shengnan Wang, Yulin Qiao +5Video Diffusion Models Diffusion Transformer
University of Science and Technology of China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center · 3Independent Researcher +2
AdaSplash-2: Faster Differentiable Sparse Attention
Apr 16, 2026Nuno Gonçalves, Hugo Pitorro, Vlad Niculae +4GPU Kernel Optimization Self-Attention
Instituto Superior Técnico, Universidade de Lisboa · Instituto de Telecomunicações · Carnegie Mellon University +4
Improving Sparse Autoencoder with Dynamic Attention
Apr 16, 2026Dongsheng Wang, Jinsen Zhang, Dawei Su +1Sparse Autoencoders Neural Network Interpretability
College of Computer Science and Software Engineering, Shenzhen University, China
Flux Attention: Context-Aware Hybrid Attention for Efficient LLMs Inference
Apr 8, 2026Quantong Qiu, Zhiyi Hong, Yi Yang +5Self-Attention Long-Context Language Model Inference
School of Computer Science and Technology, Soochow University · Baidu Inc, China
LSRM: High-Fidelity Object-Centric Reconstruction via Scaled Context Windows
Apr 6, 2026Zhengqin Li, Cheng Zhang, Jakob Engel +1Inverse Rendering 3D ViTs
Meta Reality Labs Research
HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention
Mar 30, 2026Yufei Xu, Fanxu Meng, Fan Jiang +11Self-Attention LLM Inference Acceleration
Xiaohongshu
Learning When to Attend: Conditional Memory Access for Long-Context LLMs
Mar 18, 2026Sakshi Choudhary, Aditya Chattopadhyay, Luca Zancato +4Self-Attention Long-Context Language Modeling
Department of Electrical and Computer Engineering, Purdue University, USA · AWS Agentic AI, USA
WAND: Windowed Attention and Knowledge Distillation for Efficient Autoregressive Text-to-Speech Models
Mar 17, 2026Hanna Lee, Tan Dat Nguyen, Jaehoon Kang +1TTS Synthesis Self-Attention
Korea Advanced Institute of Science and Technology, Republic of Korea · Sungkyunkwan University, Republic of Korea
When Does Sparsity Mitigate the Curse of Depth in LLMs
Mar 16, 2026Dilxat Muhtar, Xinyuan Song, Sebastian Pokutta +4Grouped-Query Attention Language Model Scaling
Max Planck Institute for Intelligent Systems · ELLIS Institute Tübingen · Tübingen AI Center +4
SVG-EAR: Parameter-Free Linear Compensation for Sparse Video Generation via Error-aware Routing
Mar 9, 2026Xuanyi Zhou, Qiuyang Mang, Shuo Yang +7Video Diffusion Models Efficient Attention
UC Berkeley, USA
Incremental Learning of Sparse Attention Patterns in Transformers
Feb 22, 2026Oğuz Kaan Yüksel, Rodrigo Alvarez Lucendo, Nicolas FlammarionSelf-Attention Transformer Attention
TML Lab, EPFL
Routing Absorption in Sparse Attention: Why Random Gates Are Hard to Beat
Feb 11, 2026Keston Aquino-MichaelsTransformer Self-Attention
No Way Labs

May 8, 2026, cs.LGJ/K move · Enter open · S save

An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inference

Feiyu Yao, Zhixiong Niu, Xiaqing Li, Yongqiang Xiong+2

Beijing University of Technology Beijing, China · Microsoft Research Beijing, China · Beijing Jiaotong University Beijing, China

GPU Acceleration Self-Attention Long-Context Language Model Inference KV-Cache Management LLM Inference Acceleration Heterogeneous Computing+2

Open paper AskPDF arXiv

Long-context inference increasingly operates over CPU-resident KV caches, either because decoding-time KV states exceed GPU memory capacity or because disaggregated prefill-decode systems place KV data in host memory. Although block-sparse attention reduces attention cost in this setting, sparsity alone is insufficient for end-to-end efficiency. GPU-only designs remain constrained by PCIe bandwidth and metadata memory overhead, while CPU-GPU hybrid designs still suffer from substantial GPU idle time and bottlenecks in CPU-side top-k selection and sparse attention computation. Fluxion is built on three key insights: output-aware KV budgeting, head-specific and granularity-aware sparse configuration, and cross-device coordinated execution for sparse attention over CPU-resident KV caches. Guided by these insights, Fluxion combines a lightweight head-property predictor, a granularity-budget selector, and a priority-based scheduler to jointly optimize budget allocation, sparse configuration, and CPU-GPU execution overlap. This co-design enables hybrid sparse attention to achieve both accuracy and system efficiency in long-context inference. Across 2 models, 3 benchmarks, and 40 tasks, Fluxion preserves quality well -- the worst average degradation is only -0.26 relative to FULL, while delivering 1.5

\times

-3.7

\times

speedup over the strongest fixed sparse hybrid baseline, whose KV budget is only 0.05.

Momentum

Related topics

Latest papers 164