Sparse Attention

Follow this topic Ask about this topic

Momentum

24 papers in the last four weeks, up 200% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 160

All topics

VestigeKV: The NoPE-MLA KV Cache Carries Its Own Sparse-Attention Signal in a Vestigial Branch
Sep 3, 2026WenJie FanKV Caching KV-Cache Management
Yotta Labs
Language Models Can Control Their Own Attention
Sep 2, 2026Namgyu Ho, Huzama Ahmad, Woosung Koh +3Efficient Language Model Inference Long-Context Language Modeling
KAIST AI · Google DeepMind
CRISP: Cliff-awaRe Input-adaptive Sparse Prefilling with Structural-Mass-Motivated Routing
Sep 1, 2026Huu Huy Nguyen, Chien Van Nguyen, Franck Dernoncourt +4Long-Context Language Model Inference LLM Inference Acceleration
University of Oregon · Adobe Research · Hanoi University of Science and Technology
On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
Aug 31, 2026Zihan Qiu, Zekun Wang, Xiao Li +33Efficient Language Model Training Sparse Attention
Qwen Team
A.X K2 Technical Report
Aug 31, 2026Cheolseung Baek, Dhammiko Arya, Eunki Kim +40Mixture-of-Experts Language Models Long-Context Language Modeling
Learning how to Forget: Fine-tuning for Long-Context Sparse Attention
Aug 20, 2026Matthias Seeger, Zeyu Zhang, Vihang Patil +2Fine-Tuning KV Caching
Amazon Web Services · University of Amsterdam · Amazon +1
Self-Indexing Attention for Compression-Compatible Sparse Long-Context LLM Inference
Aug 16, 2026Xu Yang, Jiapeng Zhang, Yuxin Chen +4Long-Context Language Model Inference LLM Inference Acceleration
Hunan University · Tencent
SCOPE: Subspace Clustering with Online Per-Head Top-K Estimation for Sparse Video Attention
Aug 13, 2026Qi Zhao, Qirui Li, Hanlin Tang +10Video Diffusion Models Diffusion Transformer
Zhejiang University · Alibaba Group
Sparse Attention to Emotion: Efficient Facial Emotion Recognition via Token Reduction
Aug 9, 2026Aya Manel Zitouni, Aicha Zenakhri, Karim Haroun +1Efficient ViTs Facial Expression Recognition
LIASD Laboratory, University of Paris 8, France
Autonomy-of-Heads: Data-Free Sparse Attention from Frozen Query-Key Geometry
Aug 7, 2026Yehan Yang, Junyuan Shang, Yang Li +3Attention Head Analysis LLM Inference Acceleration
Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Baidu Inc. +1
Training-Free Hashing-Based Attention via Binary Principal Components
Aug 5, 2026Daohai Yu, Zhanpeng Zeng, Keyu Chen +6Long-Context Language Modeling LLM Inference Acceleration
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China. · Tencent YouTu Lab, Shenzhen, China · Sino-Russian Research Center for Digital Economy.
SPADE: An Input-Adaptive Sparse Attention Engine for Fast Video Diffusion Models Inference
Aug 4, 2026Shanghao Liu, Renze Chen, Size Zheng +4Video Diffusion Models Diffusion Transformer
Eric · Beihang University · Peking University
ATFlash: Per-RoPE-Wavelength Attention Windows for Compute/Memory-Efficient LLM Inference
Aug 3, 2026Shun-ichiro Hayashi, Daichi Mukunoki, Tetsuya Hoshino +1Long-Context Language Model Inference LLM Inference Acceleration
Graduate School of Informatics, Nagoya University · Information Technology Center, Nagoya University
Token Radius Attention for Efficient Video Generation
Aug 3, 2026Jiayu Chen, Zhikun Jiang, Maoliang Li +6Video Diffusion Models Sparse Attention
School of Computer Science, Peking University · School of Electronics Engineering and Computer Science, Peking University · School of Computer Science and Engineering, Beihang University +2
Understanding Sparse Attention Selectivity in Long-Context Foundation Models via Counterfactual Evaluation
Aug 3, 2026Xingyu Ren, Youran Sun, Chugang Yi +1LLM Auditing Sparse Attention
The Chinese University of Hong Kong, Hong Kong, China · University of Maryland, College Park, MD, USA
LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexing
Aug 3, 2026Wen Zan, Jiaqi Zhang, Jianchao Tan +11Long-Context Modeling LLM Inference Acceleration
Meituan LongCat Team
Recall Before You Rank: Similarity-Guided Top- $K$ Reuse for Efficient Long-Context Attention
Jul 30, 2026Wenshuai Yao, Wenyong Zhou, Hanyong Shao +5Long-Context Language Model Inference Efficient Attention
School of Integrated Circuits, Peking University, Beijing, China · Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong SAR, China · School of Integrated Circuit Science and Engineering, Beihang University, Beijing, China
CoSA: Accelerating Long-Context Inference via Proxy-Kernel Co-Designed Sparse Attention
Jul 28, 2026Yufei Xue, Lin Niu, Hong Liu +6Long-Context Language Model Inference LLM Inference Acceleration
1Tencent · 2The Hong Kong University of Science and Technology
PIVOT: Efficient Query-Group Indexing for Token-Level Sparse Attention
Jul 27, 2026Hong Liu, Yuan Cheng, Lin Niu +5LLM Inference Acceleration Efficient Attention
Tencent
LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decoding
Jul 27, 2026Junsung HwangLow-Rank Attention KV Caching
Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsification
Jul 27, 2026Haopeng Li, Yitong Li, Junsong Chen +8Video Diffusion Models Sparse Attention
NVIDIA
RIS-Kernel: A Model-Agnostic Architecture for Long-Context LLM Inference via Sparse Attention
Jul 24, 2026Anderson R. SantosLong-Context Language Model Inference Memory-Efficient Inference
Faculty of Computing, Federal University of Uberlândia (UFU), Brazil.
Parameter-free Adaptive Sparse Attention via Compression-Based Content Selection
Jul 23, 2026Debarshi Kundu, Swaroop Ghosh, Vasant HonavarLong-Context Language Modeling Sparse Attention
The Pennsylvania State University
ELSAA: Efficient Low-Rank and Sparse Attention Approximation for Training Transformers
Jul 22, 2026Mahdi Heidari, Mohammad Mahdi Rahimi, Jaekyun MoonLow-Rank Attention Transformer Attention
School of Electrical Engineering, KAIST, Daejeon 34141, Republic of Korea · Department of Electrical Engineering and Computer Science, Daegu Gyeongbuk Institute of Science and Technology (DGIST), Daegu 42988, Republic of Korea
FVAttn: Adaptive Sparse Attention with Runtime Load Balancing for Video Generation
Jul 17, 2026Hao Liu, Chenghuan Huang, Ye Huang +7Video Diffusion Models Diffusion Transformer
Sun Yat-sen University · WeChat HPC, Tencent Inc. · Peking University +1
LiteTopK: Exploiting the Curse of Dimensionality for a Fused Indexer-TopK Kernel in Long-Context Sparse Attention
Jul 13, 2026Ziqi Yin, Jianyang Gao, Peiqi Yin +2GPU Kernel Optimization LLM Inference Acceleration
Nanyang Technological University · ETH Zurich · The Chinese University of Hong Kong
Sparse Attention for Dense Open-Vocabulary Prediction in CLIP
Jul 8, 2026Fatimah Zohra, Chen Zhao, Shuming Liu +1Visual Attention Fine-Grained Image Retrieval
King Abdullah University of Science and Technology (KAUST) · Harbin Institute of Technology, Shenzhen
Uncertainty-gated selection for block-sparse attention
Jul 4, 2026Thomas RossiLong-Context Language Model Inference LLM Inference Acceleration
Eonpass
SAF3R: Dynamic Sparse Attention for Feed-Forward 3D Reconstruction Transformers
Jul 3, 2026Jianing Deng, Yuanzhe Li, Jialu Wang +4Efficient Transformer Inference 3D ViTs
University of Pittsburgh, Pittsburgh, PA, USA · University of Arizona, Tucson, AZ, USA · Tongji University, Shanghai, China +2
HyperVAttention: Efficient Sparse Attention with Spatio-Temporal Clustering for Video Diffusion
Jul 3, 2026Dongyeun Lee, Amir Zandieh, Vahab Mirrokni +2Video Diffusion Models Diffusion Transformer
KAIST · Google Research

Sep 3, 2026, cs.LGJ/K move · Enter open · S save

VestigeKV: The NoPE-MLA KV Cache Carries Its Own Sparse-Attention Signal in a Vestigial Branch

WenJie Fan

Yotta Labs

KV Caching KV-Cache Management LLM Inference Acceleration Sparse Attention KV-Cache Compression Attention Mechanisms+1

Open paper AskPDF arXiv

A long-lived KV cache must be compressed before the queries that will read it exist. Selection by observed attention collapses there: on a NoPE-MLA model, H2O and SnapKV retrieve 0.00 and 0.33 of needles at 8x compression, because a token's importance has not yet been observed. VestigeKV instead derives a sparse attention pattern from a signal the cache already carries, occupying the sparse-attention literature's one unoccupied quadrant: training-free and query-independent. In NoPE-MLA the 64-dimensional decoupled branch is a vestige of RoPE that training repurposes into a salience channel; reading 11% of each row, it partitions the cache into an attended tier and a GPU-resident archive that no row ever leaves, reachable each step by a certified, query-adaptive trigger. Nothing is trained and cache rows are never quantized, so every quality effect attributes to selection and scheduling. On Kimi Linear 48B, retrieval holds at 1.00 under 8x and 0.96 under 32x from 8k to 65k context, with zero gap to full-row selection, and the recall tier holds 128x at 1.00 (8k). Both tiers stay on the GPU, so the win is speed, not memory: the per-step scan reads ~26% of the bytes dense attention would, and on a two-node sglang deployment the crossover sits at ~40k context, reaching 1.18x at 256k and 1.39x at 496k. The mechanism is exclusive to NoPE: the identical operator on a RoPE MLA collapses to 0.08, query-independent salience exists only without rotation, and query-universal exact merging is provably impossible under RoPE. All thresholds were frozen before their data; 20 archived verdicts and 8 closed routes accompany the paper.

Momentum

Related topics

Latest papers 160