Sparse Attention

Follow this topic Ask about this topic

Momentum

24 papers in the last four weeks, up 200% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 160

All topics

StreamIndex: Memory-Bounded Compressed Sparse Attention via Streaming Top-k
May 4, 2026Jaber Jaber, Osama JaberGPU Kernel Optimization Self-Attention
RightNow AI
Stochastic Sparse Attention for Memory-Bound Inference
May 3, 2026Kyle Lee, Corentin Delacour, Kevin Callahan-Coray +5GPU Acceleration Self-Attention
Department of Electrical and Computer Engineering, University of California, Santa Barbara, CA, USA · Department of Electrical and Computer Engineering, University of Michigan, Ann Arbor, MI, USA · Department of Electrical and Computer Engineering, Carnegie Mellon University, Pittsburgh, PA, USA
Spectral Dynamic Attention Network for Hyperspectral Image Super-Resolution
Apr 30, 2026Tengya Zhang, Feng Gao, Lin Qi +2Hyperspectral Image Super-Resolution Hyperspectral Imaging
Sanya Oceanographic Institution, Ocean University of China, Sanya, China · State Key Laboratory of Physical Oceanography, Ocean University of China, Qingdao 266100, China · Department of Electrical and Computer Engineering, Mississippi State University, Starkville, MS 39762 USA
Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving
Apr 29, 2026Zihan Zhao, Baotong Lu, Shengjie Lin +8LLM Serving KV-Cache Offloading
University of Virginia · Microsoft Research · Georgia Institute of Technology +1
Salca: A Sparsity-Aware Hardware Accelerator for Efficient Long-Context Attention Decoding
Apr 27, 2026Wang Fan, Wei Cao, Xi Zha +5Self-Attention Long-Context Language Model Inference
Fudan University
SparseContrast: Dynamic Sparse Attention for Efficient and Accurate Contrastive Learning in Medical Imaging
Apr 27, 2026Paarth Prasad, Ruchika MalhotraContrastive Learning Chest X-Ray Classification
Department of Software Engineering, Delhi Technological University
DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
Apr 26, 2026DeepSeek-AI, Anyi Xu, Bangcai Lin +316Mixture-of-Experts Language Models Long-Context Language Model Inference
BSViT: A Burst Spiking Vision Transformer for Expressive and Efficient Visual Representation Learning
Apr 25, 2026Hongxiang Peng, Dewei Bai, Hong QuSpiking Transformers Efficient ViTs
School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu 610054, China
SpikingBrain2.0: Brain-Inspired Foundation Models for Efficient Long-Context and Cross-Platform Inference
Apr 24, 2026Yuqi Pan, Jinghao Zhuang, Yupeng Feng +16Long-Context Language Model Inference Spiking Neural Networks
Institute of Automation, Chinese Academy of Sciences · The Hong Kong Polytechnic University · Beijing Key Laboratory of Brain-Inspired General Intelligence Large Model +1
HubRouter: A Pluggable Sub-Quadratic Routing Primitive for Hybrid Sequence Models
Apr 24, 2026Abhinaba BasuTransformer Attention Language Modeling
National Institute of Electronics and Information Technology (NIELIT)
Sculpt4D: Generating 4D Shapes via Sparse-Attention Diffusion Transformers
Apr 23, 2026Minghao Yin, Wenbo Hu, Jiale Xu +2Diffusion Transformer 3D Shape Generation
1The University of Hong Kong · 2ARC Lab, Tencent PCG
Sparse Forcing: Native Trainable Sparse Attention for Real-time Autoregressive Diffusion Video Generation
Apr 23, 2026Boxun Xu, Yuming Du, Zichang Liu +7Autoregressive Diffusion Self-Attention
Meta Superintelligence Labs · University of California, Santa Barbara · Work done at Meta
DynamicRad: Content-Adaptive Sparse Attention for Long Video Diffusion
Apr 22, 2026Yongji Long, Shijun Liang, Jintao Li +1Video Diffusion Models Long-Horizon Video Generation
University of Electronic Science and Technology of China · Shenzhen Institute for Advanced Study, UESTC · Computational Mathematics, Science, & Engineering at Michigan State University (MSU)
Simplified Sparse Attention via Gist Tokens
Apr 22, 2026Yuzhen Mao, Michael Y. Li, Emily B. FoxSelf-Attention Long-Context Language Model Inference
Stanford University
AdaCluster: Adaptive Query-Key Clustering for Sparse Attention in Video Generation
Apr 20, 2026Haoyue Tan, Shengnan Wang, Yulin Qiao +5Video Diffusion Models Diffusion Transformer
University of Science and Technology of China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center · 3Independent Researcher +2
AdaSplash-2: Faster Differentiable Sparse Attention
Apr 16, 2026Nuno Gonçalves, Hugo Pitorro, Vlad Niculae +4GPU Kernel Optimization Self-Attention
Instituto Superior Técnico, Universidade de Lisboa · Instituto de Telecomunicações · Carnegie Mellon University +4
Improving Sparse Autoencoder with Dynamic Attention
Apr 16, 2026Dongsheng Wang, Jinsen Zhang, Dawei Su +1Sparse Autoencoders Neural Network Interpretability
College of Computer Science and Software Engineering, Shenzhen University, China
Flux Attention: Context-Aware Hybrid Attention for Efficient LLMs Inference
Apr 8, 2026Quantong Qiu, Zhiyi Hong, Yi Yang +5Self-Attention Long-Context Language Model Inference
School of Computer Science and Technology, Soochow University · Baidu Inc, China
LSRM: High-Fidelity Object-Centric Reconstruction via Scaled Context Windows
Apr 6, 2026Zhengqin Li, Cheng Zhang, Jakob Engel +1Inverse Rendering 3D ViTs
Meta Reality Labs Research
HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention
Mar 30, 2026Yufei Xu, Fanxu Meng, Fan Jiang +11Self-Attention LLM Inference Acceleration
Xiaohongshu
Learning When to Attend: Conditional Memory Access for Long-Context LLMs
Mar 18, 2026Sakshi Choudhary, Aditya Chattopadhyay, Luca Zancato +4Self-Attention Long-Context Language Modeling
Department of Electrical and Computer Engineering, Purdue University, USA · AWS Agentic AI, USA
WAND: Windowed Attention and Knowledge Distillation for Efficient Autoregressive Text-to-Speech Models
Mar 17, 2026Hanna Lee, Tan Dat Nguyen, Jaehoon Kang +1TTS Synthesis Self-Attention
Korea Advanced Institute of Science and Technology, Republic of Korea · Sungkyunkwan University, Republic of Korea
When Does Sparsity Mitigate the Curse of Depth in LLMs
Mar 16, 2026Dilxat Muhtar, Xinyuan Song, Sebastian Pokutta +4Grouped-Query Attention Language Model Scaling
Max Planck Institute for Intelligent Systems · ELLIS Institute Tübingen · Tübingen AI Center +4
SVG-EAR: Parameter-Free Linear Compensation for Sparse Video Generation via Error-aware Routing
Mar 9, 2026Xuanyi Zhou, Qiuyang Mang, Shuo Yang +7Video Diffusion Models Efficient Attention
UC Berkeley, USA
Incremental Learning of Sparse Attention Patterns in Transformers
Feb 22, 2026Oğuz Kaan Yüksel, Rodrigo Alvarez Lucendo, Nicolas FlammarionSelf-Attention Transformer Attention
TML Lab, EPFL
Routing Absorption in Sparse Attention: Why Random Gates Are Hard to Beat
Feb 11, 2026Keston Aquino-MichaelsTransformer Self-Attention
No Way Labs
Near-Oracle KV Selection via Pre-hoc Sparsity for Long-Context Inference
Feb 9, 2026Yifei Gao, Lei Wang, Rong-Cheng Tu +3Self-Attention KV Caching
University College London, London, UK · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences (CAS), Shenzhen 518055, China · The Chinese University of Hong Kong +1
Attention-Mass Condensation for Sparse Decoding
Feb 6, 2026Jorge L. Ruiz WilliamsLanguage Model Decoding Sparse Attention
NaNZeta LLC
Threshold Differential Attention for Sink-Free, Ultra-Sparse, and Non-Dispersive Language Modeling
Jan 17, 2026Xingyue Huang, Xueying Ding, Mingxuan Ju +3Self-Attention Long-Context Language Modeling
University of Oxford · Carnegie Mellon University · Snap Inc.
Mixture of Distributions Matters: Dynamic Sparse Attention for Efficient Video Diffusion Transformers
Jan 14, 2026Yuxi Liu, Yipeng Hu, Zekun Zhang +2Video Diffusion Models Diffusion Transformer
Peking University · University of Electronic Science and Technology of China · University of Science and Technology of China

May 4, 2026, cs.LGJ/K move · Enter open · S save

StreamIndex: Memory-Bounded Compressed Sparse Attention via Streaming Top-k

Jaber Jaber, Osama Jaber

RightNow AI

GPU Kernel Optimization Self-Attention Memory-Efficient Optimization Memory-Efficient Inference Efficient Attention Sparse Attention

Open paper AskPDF arXiv

DeepSeek-V3.2 and V4 introduce Compressed Sparse Attention (CSA): a lightning indexer (a learned scoring projection over compressed keys) scores them, the top-k are selected per query, and a sparse attention kernel reads only those. Public CSA implementations materialize a [B, S, H_I, T] FP32 score tensor before the top-k reduction. With H_I=64 indexer heads and the V4-Flash compression ratio m=4, that intermediate is 256 GB at sequence length S=65,536, exceeding any single-GPU high-bandwidth-memory (HBM) budget. We present StreamIndex, a Triton implementation of the CSA pipeline whose central component is a chunked partition-merge top-k driver that never materializes the full intermediate. On synthetic-but-realistic V4-shaped inputs at the indexer-step (layer) level on a single NVIDIA H200, the materialize path runs out of memory (OOMs) at S=65,536 with V4-Flash dimensions; StreamIndex runs the same indexer to S=1,048,576 with 6.21 GB peak HBM, a 32x regime extension. Set-overlap recall against the materialize ground truth is bit-exact at small S where both fit; across three 5-point design-space sweeps (chunk size, key-tile size, top-k), mean recall rounds to 1.0000 with min recall at least 0.9980 in every cell. The chunked driver composes with TileLang's pipelined attention kernel: at S=262,144 with V4-Flash dimensions, the materialize indexer paired with TileLang attention OOMs while the chunked indexer paired with the same attention runs in 1.97 s at 18.56 GB peak. Our contribution targets the indexer step; we make no claim of a faster attention kernel or of real-checkpoint end-to-end behavior. Code: https://github.com/RightNow-AI/StreamIndex.

Momentum

Related topics

Latest papers 160