Sparse Attention

Follow this topic Ask about this topic

Momentum

24 papers in the last four weeks, up 200% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 160

All topics

Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling
Jul 3, 2026Xiang Hu, Xinyu Wei, Hao Gu +10Long-Context Language Model Inference Long-Context Language Modeling
Tencent HY Team · ShanghaiTech University · The Hong Kong University of Science and Technology +1
Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scale
Jul 1, 2026Siddharth Gollapudi, Nilesh Gupta, Prasann Singhal +1In-Context Retrieval Length Generalization
UC Berkeley · UT Austin
RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference
Jun 30, 2026Wenhao Li, Jinhao Dong, Hailin Zhang +3KV Caching Long-Context Language Model Inference
School of Information, Renmin University of China, Beijing, China · Peking University, Beijing, China
Dual Sparse Aggregation Transformer for Multispectral Object Detection
Jun 30, 2026Wencong Wu, Xiuwei Zhang, Hanlin Yin +2Multispectral Object Detection Detection Transformer
School of Computer Science, Northwestern Polytechnical University, Xi’an 710072, China · School of Computer Science, the Shaanxi Provincial Key Laboratory of Speech and Image Information Processing and the National Engineering Laboratory for Integrated Aerospace-Ground-Ocean Big Data Application Technology, Northwestern Polytechnical University, Xi’an 710072, China · School of Cybersecurity, Northwestern Polytechnical University, Xi’an 710072, China
Hierarchical Global Attention (HGA)
Jun 29, 2026Woernle Frank, Fedosov Vladimir, Grinenko ArtemiyKV-Cache Offloading Long-Context Modeling
BMW Group
Predict, Reuse, and Repair: Accelerating Dynamic Sparse Attention for Long-Context LLM Decoding
Jun 29, 2026Tianyu Wang, Gourav Rattihalli, Aditya Dhakal +4Long-Context Language Model Inference LLM Inference Acceleration
University of Pittsburgh, Pittsburgh, PA, USA · HPE Labs, Milpitas, CA, USA
MATCH: Modulating Attention via In-Context Retrieval for Long-Context Transformers
Jun 29, 2026Linrui Ma, Chun Hei Lo, Xinyu Wang +12In-Context Retrieval Long-Context Language Modeling
Huawei Canada · McGill University · Université de Montréal +1
Depth-Staggered Fibonacci Spacing for Sparse Attention: Static Schedules Beat Learned Dilation and Extrapolate Where Dense Attention Fails
Jun 26, 2026Chad A. CappsStructured Sparsity Long-Context Modeling
Independent Researcher
LCG: Long-Context Consistent Image Generation with Sparse Relational Attention
Jun 24, 2026Zihao Wang, Yijia Xu, Haoze Zheng +3T2I Generation Image Generation
Huazhong University of Science and Technology, China · Peking University, China · Hong Kong University of Science and Technology, Hong Kong
Chorus II: Cross-Request Sparsity Reuse for Efficient Image-to-Video Generation
Jun 23, 2026Hao Liu, Chenghuan Huang, Hao Liu +6Image-to-Video Generation Video Diffusion Models
Sun Yat-sen University · WeChat HPC, Tencent Inc. · WeChat Vision, Tencent Inc.
Emergent Capabilities Arise Randomly from Learning Sparse Attention Patterns
Jun 23, 2026Vatsal Baherwani, Zixi Chen, Shikai Qiu +2Language Model Scaling Laws Transformer Attention
New York University
VSANet: View-aware Sparse Attention Network for Light Field Image Denoising
Jun 23, 2026Gargi Panda, Soumitra Kundu, Saumik Bhattacharya +1Sparse Attention Image Denoising
Department of EE, IIT Kharagpur, India · Rekhi Centre of Excellence for the Science of Happiness, IIT Kharagpur, India · Department of E&ECE, IIT Kharagpur, India
Dustin: Draft-Augmented Sparse Verification for Efficient Long-Context Generation with Speculative Decoding
Jun 23, 2026WenHung Lee, Jian-Jia Chen, Xiaolin Lin +6Long-Context Language Model Inference KV-Cache Management
National Yang Ming Chiao Tung University · Technische Universität Darmstadt · Cornell University
ScalingAttention: Discovering Intrinsic Sparse Attention Topology for Video Diffusion Transformers
Jun 22, 2026Ruiliang Zhou, Xuecheng Wu, Kang He +6Video Diffusion Models Diffusion Transformer
KlingAI Research · Beijing Institute of Technology · NVIDIA +1
SpotAttention: Plug-In Block-Sparse Routing for Pretrained Long-Context Transformers
Jun 22, 2026Huzama Ahmad, Se-Young YunLong-Context Language Model Inference LLM Inference Acceleration
KAIST
ConSA: Controllable Sparsity in Hybrid Attention via Learnable Allocation
Jun 16, 2026Yao Chen, Yinqi Yang, Junyuan Shang +6LLM Inference Acceleration Sparse Attention
Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · Baidu Inc.
Training-free sparse attention based on cumulative energy filtering
Jun 15, 2026Chunlu Li, Yixuan Pan, Bai Du +5Video Diffusion Models Diffusion Transformer
Huawei Technologies
MiniMax Sparse Attention
Jun 11, 2026Xunhao Lai, Weiqi Xu, Yufeng Yang +14GPU Kernel Optimization Long-Context Language Modeling
MiniMax · Peking University · NVIDIA +4
From Rigid to Dynamic: Entropy-Guided Adaptive Inference for Long-Context LLMs
Jun 8, 2026Zhanchao Xu, Haoyang Li, Qingfa Xiao +4Long-Context Language Model Inference LLM Inference Acceleration
Department of Computing, PolyU · DSA, HKUST(GZ) · CSE, HKUST
SIFT: Selective-Index For Fast Compute of RAG Prefill by Exploiting Attention Invariance
Jun 8, 2026Rya Sanovar, Srikant Bharadwaj, Hritvik Taneja +1Retrieval-Augmented Generation LLM Inference Acceleration
Georgia Institute of Technology Atlanta, GA, USA · Microsoft Redmond, WA, USA
FlashMemory-DeepSeek-V4: Lightning Index Ultra-Long Context via Lookahead Sparse Attention
Jun 8, 2026Yan Wang, Qifan Zhang, Jiachen Yu +12KV Caching Long-Context Language Model Inference
Independent Researchers · Tencent · The Hong Kong University of Science and Technology (Guangzhou) +1
Look Less, Reason More: Block-wise Attention Skipping for Efficient Multimodal LLMs
Jun 7, 2026Jie Ma, Zhike Qiu, Jiayi Ji +2Efficient Multimodal Inference Multimodal Large Language Models
Xiamen University
Sparrow: Sparse Rollout for Stable and Efficient Long-context RL of Large Language Models
Jun 7, 2026Yang Zhou, Ranajoy Sadhukhan, Zhaofeng Sun +7RL for Language Model Reasoning Sparse Attention
Carnegie Mellon University · Cornell University · Intel +1
How Much Dense Attention is Necessary? Oracle-Guided Sparse Prefill for Full/GQA Layers in Hybrid Long-Context Models
Jun 5, 2026Hongxing Wang, Harenome Razanajato, Zhen Zhang +2Grouped-Query Attention Long-Context Language Modeling
You Only Index Once: Cross-Layer Sparse Attention with Shared Routing
Jun 4, 2026Yutao Sun, Yanqi Zhang, Li Dong +2Long-Context Language Model Inference KV-Cache Management
Microsoft Research · Tsinghua University
Vortex: Efficient and Programmable Sparse Attention Serving for AI Agents
Jun 4, 2026Zhuoming Chen, Xinrui Zhong, Qilong Feng +5LLM Serving LLM Inference Acceleration
1Carnegie Mellon University · 2Rice University · 3National University of Singapore
SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference
Jun 3, 2026Yaosheng Fu, Guangxuan Xiao, Xin Dong +2KV-Cache Offloading Long-Context Language Model Inference
1NVIDIA · 2Thinking Machines Lab · 3ByteDance Seed +1
Move the Query, Not the Cache: Characterizing Cross-Instance Latent Attention Redistribution Across GPU Fabrics
May 31, 2026Bole Ma, Jan Eitzinger, Harald Köstler +1LLM Inference KV Caching
LVSA: Training-Free Sparse Attention for Long Video Diffusion
May 29, 2026Gael Glorian, Ioannis Lamprou, Zhen Zhang +2Video Diffusion Models Long-Horizon Video Generation
Distributed Parallel Technology Laboratory, Paris Research Center, Huawei Technologies France · AI Framework and Data Technology Lab, Huawei Technologies Co., Ltd.
OmniMem: Scalable and Adaptive Memory Retrieval for Long Video Generation
May 28, 2026Lin Zhao, Yushu Wu, Yifan Gong +2KV Caching Long Video Generation
Northeastern University · Adobe Research

Jul 3, 2026, cs.CLJ/K move · Enter open · S save

Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling

Xiang Hu, Xinyu Wei, Hao Gu, Minshen Zhang+9

Tencent HY Team · ShanghaiTech University · The Hong Kong University of Science and Technology+1

Long-Context Language Model Inference Long-Context Language Modeling Long-Context Modeling Sparse Attention

Open paper AskPDF arXiv

Scaling modern large language models (LLMs) to long contexts is limited by the quadratic computation cost, and poor length extrapolation of dense attention. Chunk-wise sparse attention offers a promising alternative, but all existing methods fall short of full attention because of their inaccurate chunk selection. We propose Hierarchical Landmark Sparse (HiLS) Attention, a chunk-wise sparse attention mechanism that learns chunk selection end-to-end under the language-modeling (LM) loss. HiLS factorizes attention hierarchically: each query performs attention independently with each retrieved chunk to extract chunk-specific information, and the resulting outputs are fused according to chunk retrieval scores. By incorporating retrieval scores into the forward attention computation, HiLS optimizes them directly with the LM loss, enabling end-to-end retrieval learning and native sparse training. Experimental results show that HiLS-Attention achieves performance comparable to, and in some cases better than, full attention at in-domain context lengths. Meanwhile, HiLS-Attention extrapolates more than

64\times

the training context length with 90% retrieval accuracy, far beyond full attention. Moreover, existing full-attention models can be converted to HiLS-Attention with lightweight continued pretraining, preserving in-domain performance while acquiring ultra-long-context extrapolation. Together with its sparse KV access and computation, HiLS-Attention breaks the usual efficiency-performance trade-off, enabling long-context LLMs that are both more efficient and more effective on general long-context tasks than their full-attention counterparts.

Momentum

Related topics

Latest papers 160