Sparse Attention

Ask about this topic

Momentum

24 papers in the last four weeks, up 200% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 160

All topics
CardsList

  1. Attention via Black-Box Vector Search

    Oct 7, 2026Stepan Zharkov, Krish Singal, Ashwin Padaki +1Efficient AttentionSparse Attention

    University of Pennsylvania

  2. Evaluating Trajectory Features for Routing Final-Layer Attention

    Oct 7, 2026Yupeng YaoLLM RoutingLLM Inference Acceleration

  3. SPIN: Shadow Predictive Indexer for Sparse Attention

    Oct 6, 2026Yao Fu, Jiahan Chang, Ritchie Zhao +11LLM Inference AccelerationSparse Attention

  4. Backend-Agnostic Sparse Attention for Fast High-Resolution Visual Generation

    Oct 6, 2026Liao Ma, Jiayi Song, Yunfeng Wu +2Video Diffusion ModelsDiffusion Transformer

    School of Artificial Intelligence, Shanghai Jiao Tong University · School of Data Science, Fudan University · School of Computing and Data Science, The University of Hong Kong

  5. MC-Sparse: Deconstructing and Closing the Dense-Sparse Attention Gap in Diffusion Transformers

    Oct 5, 2026Jiarui Chen, Zeqiang Lai, Jiangshan Wang +5Video Diffusion ModelsDiffusion Transformer

    Fudan University · Tencent HY · Shanghai Innovation Institute +4

  6. OVAL: Output-Aware Local Page Bases for KV Cache Retrieval

    Oct 5, 2026Ashkan Shahbazi, Chayne Thrash, Soheil KolouriLong-Context Language Model InferenceLLM Inference Acceleration

    College of Connected Computing Vanderbilt University Nashville, TN, USA

  7. Prism: Dynamic Sparse Attention for Native 2K Joint Video-Audio Generation Model Training

    Oct 4, 2026Shuyuan Tu, Qi Tian, Yinming Huang +8Audio-Video GenerationVideo Generation

    Fudan University · Tencent Hunyuan Foundation Model Team · Zhejiang University

  8. More Value per Key: Asymmetric Sparse Attention for Faster LLM Decoding

    Oct 3, 2026Noam Elata, Itay Lamprecht, Mikey Shechter +3LLM Inference AccelerationSparse Attention

    Technion – Haifa, Israel · Intel, Israel · Corma +1

  9. HHR: Hierarchical Hash Retrieval for Efficient LLM Generation

    Oct 1, 2026Lianjun Liu, Tiantian Zheng, You Huang +5Long-Context Language Model InferenceLLM Inference Acceleration

    Hainan University, Haikou, China · Xiamen University, Xiamen, China · Zhejiang Normal University, Jinhua, China

  10. VASC: Value-Aware Sparse Attention with Cross-Layer Memory for Efficient 3D Reconstruction

    Oct 1, 2026Junyi Wu, Fanqing Kong, Leyang Chen +23D ViTs3D Reconstruction

    Shanghai Jiao Tong University

  11. CommunityKV: Efficient Long-Context Decoding via Graph Partitioning

    Sep 30, 2026Joe McKenna, Anastasios Alexandridis, Nathan Susanj +1Long-Context Language Model InferenceKV-Cache Management

    Amazon AGI

  12. SparseEngine: Sparse-First Inference Engine

    Sep 30, 2026Jitai Hao, Quansheng Gu, Qiang Huang +1LLM InferenceLong-Context Language Model Inference

  13. SparLeak: Privacy Leakage from Sparse Attention in LLM Inference on Shared GPUs

    Sep 30, 2026Fahao Chen, Linkang Du, Jinhao Zhou +2LLM InferencePrivacy Leakage in Language Models

    Shandong University China · Xi’an Jiaotong University China · Waseda University Japan

  14. Retrieval Capacity of Self-Attention Under Competition

    Sep 29, 2026Timur Mudarisov, Mikhail Burtsev, Radu StateSelf-AttentionLong-Context Language Modeling

    University of Luxembourg, Luxembourg

  15. Parameterized Stripe Attention for Efficient Video Generation

    Sep 29, 2026Xingyu Jia, Baole Ai, Ang Wang +2Video Diffusion ModelsDiffusion Transformer

    Alibaba Group

  16. ReSS: Residual-Restoring Sparse Attention for 3D Vision Transformers

    Sep 28, 2026Yongsung Kim, Jaehoon Lee, Minjun Park +33D ViTsEfficient ViTs

    IPAI, 2ECE, 3AIIS, ASRI, INMC, ISRC Seoul National University · ECE Seoul National University · AIIS Seoul National University

  17. SpikeLite: Lightweight Spiking Neural Networks for Time-Series Forecasting

    Sep 28, 2026Bang Hu, Changze Lv, Mingjie Li +3Multivariate Time Series ForecastingEfficient Neural Network Inference

    School of Computer Science, Fudan University, Shanghai, China

  18. WorldAttention: An Efficient Attention Architecture for Interactive Video World Models

    Sep 28, 2026Zeyu Zhang, Jinyuan Mao, Dakai An +6Real-Time Interactive Video GenerationLong-Horizon Video Generation

    DAMO Academy, Alibaba Group · Zhejiang University · Hong Kong University of Science and Technology +2

  19. Counterexamples to Local Reconstruction Gain as a Proxy for Final Fidelity in Residual Completion

    Sep 28, 2026Yasuto Hoshi, Daisuke Miyashita, Jun DeguchiSparse Attention

    Kioxia Corporation

  20. PQ-HSA: Reusing Product-Quantized Scores for Hybrid Sparse-Approximate Attention

    Sep 27, 2026Kunming Shao, Jierun Chen, Yanli Wang +4Self-AttentionLLM Inference Acceleration

    The Hong Kong University of Science and Technology · Huawei Technologies Ltd. · Sun Yat-sen University +1

  21. RelaxKV: Recomputation Guided by the Query with Sparse Context Attention for Efficient KV Cache Reuse

    Sep 27, 2026Ruoling Qi, Yirui Liu, Xuaner Wu +5Retrieval-Augmented GenerationKV-Cache Management

    Shanghai Jiao Tong University · Institute of Artificial Intelligence, China Telecom (TeleAI) · State University of New York at Buffalo

  22. HySparse2: Hybrid Sparse Attention with Two-Level KV Sharing

    Sep 22, 2026Jianyu Wei, Yizhao Gao, Qihao Zhang +12Long-Context RetrievalKV-Cache Management

    LLM-Core Xiaomi

  23. CompKV: Compensation-Aware KV Selection for Long-Context LLM Inference

    Sep 22, 2026Zhen Huang, Ruizhe Yao, Danyi Liu +8Long-Context Language Model InferenceKV-Cache Management

    Tsinghua University · Harbin Institute of Technology, Shenzhen · Northeastern University

  24. Opinion Leader Dynamics: How Sparse Attention Shapes Token Clustering

    Sep 21, 2026Jingkun Liu, Yue SongOpinion DynamicsClustering

    Tsinghua University · Shanghai Jiao Tong University

  25. Understanding and Exploiting Diagonal Attention Sparsity in Autoregressive Image Generation

    Sep 17, 2026Daeun Kim, Junwha Hong, Changhun Oh +3Autoregressive Image GenerationSparse Attention

    KAIST · Agency for Defense Development · Seoul National University

  26. Reaching Every Position Without Searching: Rotating Sparse Wiring on the Hypercube as a Substitute for Attention

    Sep 16, 2026Yoshiaki TakashitaEfficient AttentionLanguage Modeling

    School of Law, Waseda University

  27. Fathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Caches

    Sep 15, 2026Vivek KalyanaranganKV-Cache OffloadingKV Caching

  28. SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking

    Sep 14, 2026Zhiwei Li, Lei Zhu, Hao Gu +6Long-Context Language ModelingSparse Attention

    Tencent HY LLM Frontier · Hong Kong University of Science and Technology (Guangzhou) · Hong Kong University of Science and Technology

  29. RouteRelay: Event-Triggered Cross-Layer Route Reuse for Efficient Dynamic Sparse Attention

    Sep 7, 2026Bin Li, Sisi Liu, Chenyang Hu +3Adaptive Model RoutingEfficient Attention

    Department of Computer Science, Xiamen University of Technology · School of Mathematics and Computer Science, Fuzhou University

  30. CEDAR: Error-Bounded Residual Routing for Efficient Long-Context Attention

    Sep 7, 2026Siyu Li, Dong Wang, Jie Zhou +3Long-Context Language ModelingEfficient Attention

    School of Computer Science & Technology, Shanghai University · School of Computer Science & Technology, East China Normal University · School of Computer Engineering, Jiangsu Ocean University