Sparse Attention

Follow this topic Ask about this topic

Momentum

25 papers in the last four weeks, up 213% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 164

All topics

Fathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Caches
Sep 15, 2026Vivek KalyanaranganKV-Cache Offloading KV Caching
SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking
Sep 14, 2026Zhiwei Li, Lei Zhu, Hao Gu +6Long-Context Language Modeling Sparse Attention
Tencent HY LLM Frontier · Hong Kong University of Science and Technology (Guangzhou) · Hong Kong University of Science and Technology
RouteRelay: Event-Triggered Cross-Layer Route Reuse for Efficient Dynamic Sparse Attention
Sep 7, 2026Bin Li, Sisi Liu, Chenyang Hu +3Adaptive Model Routing Efficient Attention
Department of Computer Science, Xiamen University of Technology · School of Mathematics and Computer Science, Fuzhou University
CEDAR: Error-Bounded Residual Routing for Efficient Long-Context Attention
Sep 7, 2026Siyu Li, Dong Wang, Jie Zhou +3Long-Context Language Modeling Efficient Attention
School of Computer Science & Technology, Shanghai University · School of Computer Science & Technology, East China Normal University · School of Computer Engineering, Jiangsu Ocean University
VestigeKV: The NoPE-MLA KV Cache Carries Its Own Sparse-Attention Signal in a Vestigial Branch
Sep 3, 2026WenJie FanKV Caching KV-Cache Management
Yotta Labs
Language Models Can Control Their Own Attention
Sep 2, 2026Namgyu Ho, Huzama Ahmad, Woosung Koh +3Efficient Language Model Inference Long-Context Language Modeling
KAIST AI · Google DeepMind
CRISP: Cliff-awaRe Input-adaptive Sparse Prefilling with Structural-Mass-Motivated Routing
Sep 1, 2026Huu Huy Nguyen, Chien Van Nguyen, Franck Dernoncourt +4Long-Context Language Model Inference LLM Inference Acceleration
University of Oregon · Adobe Research · Hanoi University of Science and Technology
On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
Aug 31, 2026Zihan Qiu, Zekun Wang, Xiao Li +33Efficient Language Model Training Sparse Attention
Qwen Team
A.X K2 Technical Report
Aug 31, 2026Cheolseung Baek, Dhammiko Arya, Eunki Kim +40Mixture-of-Experts Language Models Long-Context Language Modeling
Learning how to Forget: Fine-tuning for Long-Context Sparse Attention
Aug 20, 2026Matthias Seeger, Zeyu Zhang, Vihang Patil +2Fine-Tuning KV Caching
Amazon Web Services · University of Amsterdam · Amazon +1
Self-Indexing Attention for Compression-Compatible Sparse Long-Context LLM Inference
Aug 16, 2026Xu Yang, Jiapeng Zhang, Yuxin Chen +4Long-Context Language Model Inference LLM Inference Acceleration
Hunan University · Tencent
SCOPE: Subspace Clustering with Online Per-Head Top-K Estimation for Sparse Video Attention
Aug 13, 2026Qi Zhao, Qirui Li, Hanlin Tang +10Video Diffusion Models Diffusion Transformer
Zhejiang University · Alibaba Group
Sparse Attention to Emotion: Efficient Facial Emotion Recognition via Token Reduction
Aug 9, 2026Aya Manel Zitouni, Aicha Zenakhri, Karim Haroun +1Efficient ViTs Facial Expression Recognition
LIASD Laboratory, University of Paris 8, France
Autonomy-of-Heads: Data-Free Sparse Attention from Frozen Query-Key Geometry
Aug 7, 2026Yehan Yang, Junyuan Shang, Yang Li +3Attention Head Analysis LLM Inference Acceleration
Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Baidu Inc. +1
Training-Free Hashing-Based Attention via Binary Principal Components
Aug 5, 2026Daohai Yu, Zhanpeng Zeng, Keyu Chen +6Long-Context Language Modeling LLM Inference Acceleration
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China. · Tencent YouTu Lab, Shenzhen, China · Sino-Russian Research Center for Digital Economy.
SPADE: An Input-Adaptive Sparse Attention Engine for Fast Video Diffusion Models Inference
Aug 4, 2026Shanghao Liu, Renze Chen, Size Zheng +4Video Diffusion Models Diffusion Transformer
Eric · Beihang University · Peking University
ATFlash: Per-RoPE-Wavelength Attention Windows for Compute/Memory-Efficient LLM Inference
Aug 3, 2026Shun-ichiro Hayashi, Daichi Mukunoki, Tetsuya Hoshino +1Long-Context Language Model Inference LLM Inference Acceleration
Graduate School of Informatics, Nagoya University · Information Technology Center, Nagoya University
Token Radius Attention for Efficient Video Generation
Aug 3, 2026Jiayu Chen, Zhikun Jiang, Maoliang Li +6Video Diffusion Models Sparse Attention
School of Computer Science, Peking University · School of Electronics Engineering and Computer Science, Peking University · School of Computer Science and Engineering, Beihang University +2
Understanding Sparse Attention Selectivity in Long-Context Foundation Models via Counterfactual Evaluation
Aug 3, 2026Xingyu Ren, Youran Sun, Chugang Yi +1LLM Auditing Sparse Attention
The Chinese University of Hong Kong, Hong Kong, China · University of Maryland, College Park, MD, USA
LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexing
Aug 3, 2026Wen Zan, Jiaqi Zhang, Jianchao Tan +11Long-Context Modeling LLM Inference Acceleration
Meituan LongCat Team
Recall Before You Rank: Similarity-Guided Top- $K$ Reuse for Efficient Long-Context Attention
Jul 30, 2026Wenshuai Yao, Wenyong Zhou, Hanyong Shao +5Long-Context Language Model Inference Efficient Attention
School of Integrated Circuits, Peking University, Beijing, China · Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong SAR, China · School of Integrated Circuit Science and Engineering, Beihang University, Beijing, China
CoSA: Accelerating Long-Context Inference via Proxy-Kernel Co-Designed Sparse Attention
Jul 28, 2026Yufei Xue, Lin Niu, Hong Liu +6Long-Context Language Model Inference LLM Inference Acceleration
1Tencent · 2The Hong Kong University of Science and Technology
PIVOT: Efficient Query-Group Indexing for Token-Level Sparse Attention
Jul 27, 2026Hong Liu, Yuan Cheng, Lin Niu +5LLM Inference Acceleration Efficient Attention
Tencent
LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decoding
Jul 27, 2026Junsung HwangLow-Rank Attention KV Caching
Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsification
Jul 27, 2026Haopeng Li, Yitong Li, Junsong Chen +8Video Diffusion Models Sparse Attention
NVIDIA
RIS-Kernel: A Model-Agnostic Architecture for Long-Context LLM Inference via Sparse Attention
Jul 24, 2026Anderson R. SantosLong-Context Language Model Inference Memory-Efficient Inference
Faculty of Computing, Federal University of Uberlândia (UFU), Brazil.
Parameter-free Adaptive Sparse Attention via Compression-Based Content Selection
Jul 23, 2026Debarshi Kundu, Swaroop Ghosh, Vasant HonavarLong-Context Language Modeling Sparse Attention
The Pennsylvania State University
ELSAA: Efficient Low-Rank and Sparse Attention Approximation for Training Transformers
Jul 22, 2026Mahdi Heidari, Mohammad Mahdi Rahimi, Jaekyun MoonLow-Rank Attention Transformer Attention
School of Electrical Engineering, KAIST, Daejeon 34141, Republic of Korea · Department of Electrical Engineering and Computer Science, Daegu Gyeongbuk Institute of Science and Technology (DGIST), Daegu 42988, Republic of Korea
FVAttn: Adaptive Sparse Attention with Runtime Load Balancing for Video Generation
Jul 17, 2026Hao Liu, Chenghuan Huang, Ye Huang +7Video Diffusion Models Diffusion Transformer
Sun Yat-sen University · WeChat HPC, Tencent Inc. · Peking University +1
LiteTopK: Exploiting the Curse of Dimensionality for a Fused Indexer-TopK Kernel in Long-Context Sparse Attention
Jul 13, 2026Ziqi Yin, Jianyang Gao, Peiqi Yin +2GPU Kernel Optimization LLM Inference Acceleration
Nanyang Technological University · ETH Zurich · The Chinese University of Hong Kong

Sep 15, 2026, cs.LGJ/K move · Enter open · S save

Fathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Caches

Vivek Kalyanarangan

KV-Cache Offloading KV Caching LLM Inference Acceleration Sparse Attention

Open paper AskPDF arXiv

When agentic sessions run to a million tokens with many sessions resident at once, the KV cache and the index that ranks it live in host memory, and the scan that ranks all n keys for a top-k step becomes the traffic that bounds decoding. We present Fathom, a key scan in which each query decides how many bits of each key channel to read. The 4-bit K cache is stored channel-major as bit planes, so a prefix of t planes is exactly the channel's t-bit quantizer, and the query spends its bit budget by reverse water-filling over the variance-weighted importance of its channels. At one million tokens on Qwen3-8B a decode step is 1.67x faster in GPU time than with the 136-bit scans of Double Sparsity, Loki and SparQ r=32, and in the same GPU time as SparQ's 68-bit read (r=16) Fathom reads 18% fewer bytes with lower attention error on six of seven model and context settings. On RULER-style tasks every per-token scan matches exact top-k decoding, and on real coding-agent sessions Fathom reaches the step agreement of the most accurate 136-bit scan at 92 bits. The store is the 4-bit K copy a quantized serving stack already holds, and the method is not faster when the index is resident in GPU memory.

Momentum

Related topics

Latest papers 164