Dynamic Sparse Attention

Ask about this topic

Momentum

18 papers in the last four weeks, up 157% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 123

All topics
CardsList

  1. MWOP: Modality-aware Width-wise Operation Pruning for Efficient MLLMs

    Oct 1, 2026Xudong Wang, Hao Wu, Haozhe Hu +5Visual Token PruningLarge Language Model Compression

    EIT-NLP Lab, Eastern Institute of Technology, Ningbo · Shanghai Jiao Tong University · The Hong Kong Polytechnic University +1

  2. HHR: Hierarchical Hash Retrieval for Efficient LLM Generation

    Oct 1, 2026Lianjun Liu, Tiantian Zheng, You Huang +5LLM Inference OptimizationDynamic Sparse Attention

    Hainan University, Haikou, China · Xiamen University, Xiamen, China · Zhejiang Normal University, Jinhua, China

  3. CommunityKV: Efficient Long-Context Decoding via Graph Partitioning

    Sep 30, 2026Joe McKenna, Anastasios Alexandridis, Nathan Susanj +1Dynamic Sparse AttentionEfficient Long-Context Inference

    Amazon AGI

  4. SparseEngine: Sparse-First Inference Engine

    Sep 30, 2026Jitai Hao, Quansheng Gu, Qiang Huang +1LLM Inference OptimizationDynamic Sparse Attention

  5. SparLeak: Privacy Leakage from Sparse Attention in LLM Inference on Shared GPUs

    Sep 30, 2026Fahao Chen, Linkang Du, Jinhao Zhou +2Dynamic Sparse AttentionLLM Inference Optimization

    Shandong University China · Xi’an Jiaotong University China · Waseda University Japan

  6. SANTA++: Sampling Attention through Representative Keys

    Sep 28, 2026Kyle Lee, Christian Z. Pratt, Ruoyu Fang +4Dynamic Sparse AttentionBounded-Memory

    Department of Electrical and Computer Engineering University of California, Santa Barbara, CA, USA · Flucta, San Francisco, CA, USA

  7. ReSS: Residual-Restoring Sparse Attention for 3D Vision Transformers

    Sep 28, 2026Yongsung Kim, Jaehoon Lee, Minjun Park +3Visual Geometry Grounded TransformerDynamic Sparse Attention

    IPAI, 2ECE, 3AIIS, ASRI, INMC, ISRC Seoul National University · ECE Seoul National University · AIIS Seoul National University

  8. WorldAttention: An Efficient Attention Architecture for Interactive Video World Models

    Sep 28, 2026Zeyu Zhang, Jinyuan Mao, Dakai An +6Video World ModelsDynamic Sparse Attention

    DAMO Academy, Alibaba Group · Zhejiang University · Hong Kong University of Science and Technology +2

  9. Counterexamples to Local Reconstruction Gain as a Proxy for Final Fidelity in Residual Completion

    Sep 28, 2026Yasuto Hoshi, Daisuke Miyashita, Jun DeguchiCompletionDynamic Sparse Attention

    Kioxia Corporation

  10. Where Activation Sparsity and KV-Cache Sparsity Cross in LLM Decoding

    Sep 27, 2026Jungseob Lee, Seungyoon Lee, Seongtae Hong +2Dynamic Sparse AttentionLarge Language Model Decoding

    Korea University · Yonsei University Mirae Campus

  11. PQ-HSA: Reusing Product-Quantized Scores for Hybrid Sparse-Approximate Attention

    Sep 27, 2026Kunming Shao, Jierun Chen, Yanli Wang +4Dynamic Sparse AttentionKey-Value Cache Quantization

    The Hong Kong University of Science and Technology · Huawei Technologies Ltd. · Sun Yat-sen University +1

  12. HySparse2: Hybrid Sparse Attention with Two-Level KV Sharing

    Sep 22, 2026Jianyu Wei, Yizhao Gao, Qihao Zhang +12Dynamic Sparse AttentionEfficient Long-Context Inference

    LLM-Core Xiaomi

  13. CompKV: Compensation-Aware KV Selection for Long-Context LLM Inference

    Sep 22, 2026Zhen Huang, Ruizhe Yao, Danyi Liu +8Dynamic Sparse AttentionLLM Inference Optimization

    Tsinghua University · Harbin Institute of Technology, Shenzhen · Northeastern University

  14. Understanding and Exploiting Diagonal Attention Sparsity in Autoregressive Image Generation

    Sep 17, 2026Daeun Kim, Junwha Hong, Changhun Oh +3Autoregressive Image GenerationDynamic Sparse Attention

    KAIST · Agency for Defense Development · Seoul National University

  15. Reaching Every Position Without Searching: Rotating Sparse Wiring on the Hypercube as a Substitute for Attention

    Sep 16, 2026Yoshiaki TakashitaDynamic Sparse AttentionAttention Layers

    School of Law, Waseda University

  16. SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking

    Sep 14, 2026Zhiwei Li, Lei Zhu, Hao Gu +6Dynamic Sparse AttentionBlock Sparse Flash Attention

    Tencent HY LLM Frontier · Hong Kong University of Science and Technology (Guangzhou) · Hong Kong University of Science and Technology

  17. RouteRelay: Event-Triggered Cross-Layer Route Reuse for Efficient Dynamic Sparse Attention

    Sep 7, 2026Bin Li, Sisi Liu, Chenyang Hu +3Dynamic Sparse AttentionCross-Layer Interactions

    Department of Computer Science, Xiamen University of Technology · School of Mathematics and Computer Science, Fuzhou University

  18. CEDAR: Error-Bounded Residual Routing for Efficient Long-Context Attention

    Sep 7, 2026Siyu Li, Dong Wang, Jie Zhou +3Dynamic Sparse AttentionEfficient Long-Context Inference

    School of Computer Science & Technology, Shanghai University · School of Computer Science & Technology, East China Normal University · School of Computer Engineering, Jiangsu Ocean University

  19. CRISP: Cliff-awaRe Input-adaptive Sparse Prefilling with Structural-Mass-Motivated Routing

    Sep 1, 2026Huu Huy Nguyen, Chien Van Nguyen, Franck Dernoncourt +4Dynamic Sparse AttentionLLM Inference Optimization

    University of Oregon · Adobe Research · Hanoi University of Science and Technology

  20. Event-Driven Language Models with Sparse Neural Activity for Neuromorphic Hardware

    Aug 31, 2026Simon Richter, Ruhai Lin, Jason Yik +4Dynamic Sparse AttentionNeuromorphic Computing

    Department of Electrical and Computer Engineering, Aarhus University, Aarhus, Denmark · Department of Computer Science and Engineering, University of California, Santa Cruz, Santa Cruz, USA · School of Engineering and Applied Sciences, Harvard University, Cambridge, USA +1

  21. On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability

    Aug 31, 2026Zihan Qiu, Zekun Wang, Xiao Li +33Autoregressive Token PredictionUnigram Tokenizer

    Qwen Team

  22. ClusterAttention: A training-free speedup of bidirectional attention

    Aug 27, 2026Kasper Nordenram, Amelie DittmannDynamic Sparse AttentionLinear Attention

    Independent researcher

  23. Learning how to Forget: Fine-tuning for Long-Context Sparse Attention

    Aug 20, 2026Matthias Seeger, Zeyu Zhang, Vihang Patil +2Dynamic Sparse AttentionEfficient Long-Context Inference

    Amazon Web Services · University of Amsterdam · Amazon +1

  24. SCOPE: Subspace Clustering with Online Per-Head Top-K Estimation for Sparse Video Attention

    Aug 13, 2026Qi Zhao, Qirui Li, Hanlin Tang +10Dynamic Sparse AttentionSpatio-Temporal Attention

    Zhejiang University · Alibaba Group

  25. LoSA: Near-Lossless Sparse Attention for Training-Free Video Diffusion Acceleration

    Aug 12, 2026Enhuai Liu, Yunke Wang, Yutong Wang +2Dynamic Sparse AttentionSpatio-Temporal Attention

    1The University of Sydney · 2CSIRO

  26. Autonomy-of-Heads: Data-Free Sparse Attention from Frozen Query-Key Geometry

    Aug 7, 2026Yehan Yang, Junyuan Shang, Yang Li +3Dynamic Sparse AttentionLLM Inference Optimization

    Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Baidu Inc. +1

  27. Training-Free Hashing-Based Attention via Binary Principal Components

    Aug 5, 2026Daohai Yu, Zhanpeng Zeng, Keyu Chen +6Dynamic Sparse AttentionBlock Sparse Flash Attention

    Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China. · Tencent YouTu Lab, Shenzhen, China · Sino-Russian Research Center for Digital Economy.

  28. SPADE: An Input-Adaptive Sparse Attention Engine for Fast Video Diffusion Models Inference

    Aug 4, 2026Shanghao Liu, Renze Chen, Size Zheng +4Dynamic Sparse AttentionSparsity

    Eric · Beihang University · Peking University

  29. Understanding Sparse Attention Selectivity in Long-Context Foundation Models via Counterfactual Evaluation

    Aug 3, 2026Xingyu Ren, Youran Sun, Chugang Yi +1Dynamic Sparse AttentionBlock Sparse Flash Attention

    The Chinese University of Hong Kong, Hong Kong, China · University of Maryland, College Park, MD, USA

  30. LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexing

    Aug 3, 2026Wen Zan, Jiaqi Zhang, Jianchao Tan +11Dynamic Sparse AttentionEfficient Long-Context Inference

    Meituan LongCat Team

  31. Celty: SpMspV GPU Kernel and SIMT Co-Design for Efficient Dual-Sparse LLM Inference

    Aug 2, 2026Ruokai Yin, Priyadarshini PandaLLM Inference OptimizationDynamic Sparse Attention

    Yale University New Haven, USA · University of Southern California Los Angeles, USA

  32. A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding

    Jul 30, 2026Yuesong Liu, Yuan Zeng, Min Lyu +3Speculative DecodingSelf-Speculative

    University of Science and Technology of China

  33. Recall Before You Rank: Similarity-Guided Top-KK Reuse for Efficient Long-Context Attention

    Jul 30, 2026Wenshuai Yao, Wenyong Zhou, Hanyong Shao +5Dynamic Sparse AttentionTop-K

    School of Integrated Circuits, Peking University, Beijing, China · Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong SAR, China · School of Integrated Circuit Science and Engineering, Beihang University, Beijing, China

  34. At-the-Roofline Sparse Tensor Contractions on Vector Processors for Transformer Inference

    Jul 28, 2026Bowen Wang, Chi Zhang, Diyou Shen +3Dynamic Sparse AttentionVectorization

    *IIS, ETH Zürich, Switzerland · ‡Computing Systems Lab, Huawei, Switzerland · †DEI, Universit`a di Bologna, Italy

  35. CoSA: Accelerating Long-Context Inference via Proxy-Kernel Co-Designed Sparse Attention

    Jul 28, 2026Yufei Xue, Lin Niu, Hong Liu +6Dynamic Sparse AttentionEfficient Long-Context Inference

    1Tencent · 2The Hong Kong University of Science and Technology

  36. PIVOT: Efficient Query-Group Indexing for Token-Level Sparse Attention

    Jul 27, 2026Hong Liu, Yuan Cheng, Lin Niu +5Dynamic Sparse AttentionTop-K

    Tencent

  37. Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsification

    Jul 27, 2026Haopeng Li, Yitong Li, Junsong Chen +8Dynamic Sparse AttentionLong Video Generation

    NVIDIA

  38. Neuromorphic Diffusion Language Models: Addressing Compute and Memory Bottlenecks via Sparsity and Block Denoising

    Jul 24, 2026Dengyu Wu, Clement Ruah, Jiechen Chen +2Diffusion Language ModelsLLM Inference Optimization

    Department of Engineering, King’s College London, London, UK · Institute for Intelligent Networked Systems (INSI), Northeastern University London, London, UK

  39. Unified Static-Dynamic Pruning for Efficient LLM Inference

    Jul 24, 2026Jinhyeok Kim, Yejoon Lee, Jaeyoung DoLLM Inference OptimizationDynamic Sparse Attention

    Seoul National University Seoul, South Korea

  40. RIS-Kernel: A Model-Agnostic Architecture for Long-Context LLM Inference via Sparse Attention

    Jul 24, 2026Anderson R. SantosDynamic Sparse AttentionLLM Inference Optimization

    Faculty of Computing, Federal University of Uberlândia (UFU), Brazil.

  41. Parameter-free Adaptive Sparse Attention via Compression-Based Content Selection

    Jul 23, 2026Debarshi Kundu, Swaroop Ghosh, Vasant HonavarDynamic Sparse AttentionEfficient Long-Context Inference

    The Pennsylvania State University

  42. Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMs

    Jul 23, 2026Yidu Wu, Xiang Wang, Kejie Zhao +3LLM Inference OptimizationDynamic Sparse Attention

    Department of Electronic and Electrical Engineering, Southern University of Science and Technology, Shenzhen, China · ACS Lab, Huawei Technologies Co., Ltd., Shenzhen, China · Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China

  43. FVAttn: Adaptive Sparse Attention with Runtime Load Balancing for Video Generation

    Jul 17, 2026Hao Liu, Chenghuan Huang, Ye Huang +7Dynamic Sparse AttentionSpatio-Temporal Attention

    Sun Yat-sen University · WeChat HPC, Tencent Inc. · Peking University +1

  44. LiteTopK: Exploiting the Curse of Dimensionality for a Fused Indexer-TopK Kernel in Long-Context Sparse Attention

    Jul 13, 2026Ziqi Yin, Jianyang Gao, Peiqi Yin +2Dynamic Sparse AttentionTop-K

    Nanyang Technological University · ETH Zurich · The Chinese University of Hong Kong

  45. COBS: Cumulant Order Block Sparse Attention

    Jul 10, 2026Alexander Tian, Aditya Ghai, Sanjit Neelam +2Dynamic Sparse AttentionDeepseek

    MatX

  46. Uncertainty-gated selection for block-sparse attention

    Jul 4, 2026Thomas RossiDynamic Sparse AttentionEfficient Long-Context Inference

    Eonpass

  47. SAF3R: Dynamic Sparse Attention for Feed-Forward 3D Reconstruction Transformers

    Jul 3, 2026Jianing Deng, Yuanzhe Li, Jialu Wang +4Feed-Forward 3D ReconstructionDynamic Sparse Attention

    University of Pittsburgh, Pittsburgh, PA, USA · University of Arizona, Tucson, AZ, USA · Tongji University, Shanghai, China +2

  48. HyperVAttention: Efficient Sparse Attention with Spatio-Temporal Clustering for Video Diffusion

    Jul 3, 2026Dongyeun Lee, Amir Zandieh, Vahab Mirrokni +2Dynamic Sparse AttentionSpatio-Temporal Attention

    KAIST · Google Research

  49. Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling

    Jul 3, 2026Xiang Hu, Xinyu Wei, Hao Gu +10Efficient Long-Context InferenceDynamic Sparse Attention

    Tencent HY Team · ShanghaiTech University · The Hong Kong University of Science and Technology +1

  50. Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scale

    Jul 1, 2026Siddharth Gollapudi, Nilesh Gupta, Prasann Singhal +1In-ContextText Corpora

    UC Berkeley · UT Austin

  51. MosaicKV: Serving Long-Context LLM with Dynamic Two-D KV Cache Compression

    Jul 1, 2026Sheng Qiang, Ruiwei Chen, Yinpeng Wu +5Key-Value Cache CompressionLarge Language Model Compression

    Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University, Shanghai, China

  52. RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference

    Jun 30, 2026Wenhao Li, Jinhao Dong, Hailin Zhang +3LLM Inference OptimizationDynamic Sparse Attention

    School of Information, Renmin University of China, Beijing, China · Peking University, Beijing, China

  53. Predict, Reuse, and Repair: Accelerating Dynamic Sparse Attention for Long-Context LLM Decoding

    Jun 29, 2026Tianyu Wang, Gourav Rattihalli, Aditya Dhakal +4Dynamic Sparse AttentionLarge Language Model Decoding

    University of Pittsburgh, Pittsburgh, PA, USA · HPE Labs, Milpitas, CA, USA

  54. MATCH: Modulating Attention via In-Context Retrieval for Long-Context Transformers

    Jun 29, 2026Linrui Ma, Chun Hei Lo, Xinyu Wang +12Efficient Long-Context InferenceDynamic Sparse Attention

    Huawei Canada · McGill University · Université de Montréal +1

  55. Depth-Staggered Fibonacci Spacing for Sparse Attention: Static Schedules Beat Learned Dilation and Extrapolate Where Dense Attention Fails

    Jun 26, 2026Chad A. CappsDynamic Sparse AttentionLinear Attention

    Independent Researcher

  56. VSANet: View-aware Sparse Attention Network for Light Field Image Denoising

    Jun 23, 2026Gargi Panda, Soumitra Kundu, Saumik Bhattacharya +1Sparse-ViewIterative Denoising Methods

    Department of EE, IIT Kharagpur, India · Rekhi Centre of Excellence for the Science of Happiness, IIT Kharagpur, India · Department of E&ECE, IIT Kharagpur, India

  57. ScalingAttention: Discovering Intrinsic Sparse Attention Topology for Video Diffusion Transformers

    Jun 22, 2026Ruiliang Zhou, Xuecheng Wu, Kang He +6Dynamic Sparse AttentionDiffusion Transformers

    KlingAI Research · Beijing Institute of Technology · NVIDIA +1

  58. SpotAttention: Plug-In Block-Sparse Routing for Pretrained Long-Context Transformers

    Jun 22, 2026Huzama Ahmad, Se-Young YunDynamic Sparse AttentionEfficient Long-Context Inference

    KAIST

  59. ConSA: Controllable Sparsity in Hybrid Attention via Learnable Allocation

    Jun 16, 2026Yao Chen, Yinqi Yang, Junyuan Shang +6Dynamic Sparse AttentionSparsity

    Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · Baidu Inc.

  60. Training-free sparse attention based on cumulative energy filtering

    Jun 15, 2026Chunlu Li, Yixuan Pan, Bai Du +5Dynamic Sparse AttentionSpatio-Temporal Attention

    Huawei Technologies