Sparse Attention

Ask about this topic

Momentum

24 papers in the last four weeks, up 300% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 154

All topics
CardsList

  1. Backend-Agnostic Sparse Attention for Fast High-Resolution Visual Generation

    Oct 6, 2026Liao Ma, Jiayi Song, Yunfeng Wu +2Video Diffusion ModelsDiffusion Transformer

    School of Artificial Intelligence, Shanghai Jiao Tong University · School of Data Science, Fudan University · School of Computing and Data Science, The University of Hong Kong

  2. MC-Sparse: Deconstructing and Closing the Dense-Sparse Attention Gap in Diffusion Transformers

    Oct 5, 2026Jiarui Chen, Zeqiang Lai, Jiangshan Wang +5Video Diffusion ModelsDiffusion Transformer

    Fudan University · Tencent HY · Shanghai Innovation Institute +4

  3. OVAL: Output-Aware Local Page Bases for KV Cache Retrieval

    Oct 5, 2026Ashkan Shahbazi, Chayne Thrash, Soheil KolouriLong-Context Language Model InferenceLLM Inference Acceleration

    College of Connected Computing Vanderbilt University Nashville, TN, USA

  4. Prism: Dynamic Sparse Attention for Native 2K Joint Video-Audio Generation Model Training

    Oct 4, 2026Shuyuan Tu, Qi Tian, Yinming Huang +8Audio-Video GenerationVideo Generation

    Fudan University · Tencent Hunyuan Foundation Model Team · Zhejiang University

  5. More Value per Key: Asymmetric Sparse Attention for Faster LLM Decoding

    Oct 3, 2026Noam Elata, Itay Lamprecht, Mikey Shechter +3LLM Inference AccelerationSparse Attention

    Technion – Haifa, Israel · Intel, Israel · Corma +1

  6. HHR: Hierarchical Hash Retrieval for Efficient LLM Generation

    Oct 1, 2026Lianjun Liu, Tiantian Zheng, You Huang +5Long-Context Language Model InferenceLLM Inference Acceleration

    Hainan University, Haikou, China · Xiamen University, Xiamen, China · Zhejiang Normal University, Jinhua, China

  7. VASC: Value-Aware Sparse Attention with Cross-Layer Memory for Efficient 3D Reconstruction

    Oct 1, 2026Junyi Wu, Fanqing Kong, Leyang Chen +23D ViTs3D Reconstruction

    Shanghai Jiao Tong University

  8. CommunityKV: Efficient Long-Context Decoding via Graph Partitioning

    Sep 30, 2026Joe McKenna, Anastasios Alexandridis, Nathan Susanj +1Long-Context Language Model InferenceKV-Cache Management

    Amazon AGI

  9. SparseEngine: Sparse-First Inference Engine

    Sep 30, 2026Jitai Hao, Quansheng Gu, Qiang Huang +1LLM InferenceLong-Context Language Model Inference

  10. SparLeak: Privacy Leakage from Sparse Attention in LLM Inference on Shared GPUs

    Sep 30, 2026Fahao Chen, Linkang Du, Jinhao Zhou +2LLM InferencePrivacy Leakage in Language Models

    Shandong University China · Xi’an Jiaotong University China · Waseda University Japan

  11. Retrieval Capacity of Self-Attention Under Competition

    Sep 29, 2026Timur Mudarisov, Mikhail Burtsev, Radu StateSelf-AttentionLong-Context Language Modeling

    University of Luxembourg, Luxembourg

  12. Parameterized Stripe Attention for Efficient Video Generation

    Sep 29, 2026Xingyu Jia, Baole Ai, Ang Wang +2Video Diffusion ModelsDiffusion Transformer

    Alibaba Group

  13. ReSS: Residual-Restoring Sparse Attention for 3D Vision Transformers

    Sep 28, 2026Yongsung Kim, Jaehoon Lee, Minjun Park +33D ViTsEfficient ViTs

    IPAI, 2ECE, 3AIIS, ASRI, INMC, ISRC Seoul National University · ECE Seoul National University · AIIS Seoul National University

  14. SpikeLite: Lightweight Spiking Neural Networks for Time-Series Forecasting

    Sep 28, 2026Bang Hu, Changze Lv, Mingjie Li +3Multivariate Time Series ForecastingEfficient Neural Network Inference

    School of Computer Science, Fudan University, Shanghai, China

  15. WorldAttention: An Efficient Attention Architecture for Interactive Video World Models

    Sep 28, 2026Zeyu Zhang, Jinyuan Mao, Dakai An +6Real-Time Interactive Video GenerationLong-Horizon Video Generation

    DAMO Academy, Alibaba Group · Zhejiang University · Hong Kong University of Science and Technology +2

  16. Counterexamples to Local Reconstruction Gain as a Proxy for Final Fidelity in Residual Completion

    Sep 28, 2026Yasuto Hoshi, Daisuke Miyashita, Jun DeguchiSparse Attention

    Kioxia Corporation

  17. PQ-HSA: Reusing Product-Quantized Scores for Hybrid Sparse-Approximate Attention

    Sep 27, 2026Kunming Shao, Jierun Chen, Yanli Wang +4Self-AttentionLLM Inference Acceleration

    The Hong Kong University of Science and Technology · Huawei Technologies Ltd. · Sun Yat-sen University +1

  18. RelaxKV: Recomputation Guided by the Query with Sparse Context Attention for Efficient KV Cache Reuse

    Sep 27, 2026Ruoling Qi, Yirui Liu, Xuaner Wu +5Retrieval-Augmented GenerationKV-Cache Management

    Shanghai Jiao Tong University · Institute of Artificial Intelligence, China Telecom (TeleAI) · State University of New York at Buffalo

  19. HySparse2: Hybrid Sparse Attention with Two-Level KV Sharing

    Sep 22, 2026Jianyu Wei, Yizhao Gao, Qihao Zhang +12Long-Context RetrievalKV-Cache Management

    LLM-Core Xiaomi

  20. CompKV: Compensation-Aware KV Selection for Long-Context LLM Inference

    Sep 22, 2026Zhen Huang, Ruizhe Yao, Danyi Liu +8Long-Context Language Model InferenceKV-Cache Management

    Tsinghua University · Harbin Institute of Technology, Shenzhen · Northeastern University

  21. Opinion Leader Dynamics: How Sparse Attention Shapes Token Clustering

    Sep 21, 2026Jingkun Liu, Yue SongOpinion DynamicsClustering

    Tsinghua University · Shanghai Jiao Tong University

  22. Understanding and Exploiting Diagonal Attention Sparsity in Autoregressive Image Generation

    Sep 17, 2026Daeun Kim, Junwha Hong, Changhun Oh +3Autoregressive Image GenerationSparse Attention

    KAIST · Agency for Defense Development · Seoul National University

  23. Reaching Every Position Without Searching: Rotating Sparse Wiring on the Hypercube as a Substitute for Attention

    Sep 16, 2026Yoshiaki TakashitaEfficient AttentionLanguage Modeling

    School of Law, Waseda University

  24. Fathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Caches

    Sep 15, 2026Vivek KalyanaranganKV-Cache OffloadingKV Caching

  25. SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking

    Sep 14, 2026Zhiwei Li, Lei Zhu, Hao Gu +6Long-Context Language ModelingSparse Attention

    Tencent HY LLM Frontier · Hong Kong University of Science and Technology (Guangzhou) · Hong Kong University of Science and Technology

  26. RouteRelay: Event-Triggered Cross-Layer Route Reuse for Efficient Dynamic Sparse Attention

    Sep 7, 2026Bin Li, Sisi Liu, Chenyang Hu +3Adaptive Model RoutingEfficient Attention

    Department of Computer Science, Xiamen University of Technology · School of Mathematics and Computer Science, Fuzhou University

  27. CEDAR: Error-Bounded Residual Routing for Efficient Long-Context Attention

    Sep 7, 2026Siyu Li, Dong Wang, Jie Zhou +3Long-Context Language ModelingEfficient Attention

    School of Computer Science & Technology, Shanghai University · School of Computer Science & Technology, East China Normal University · School of Computer Engineering, Jiangsu Ocean University

  28. VestigeKV: The NoPE-MLA KV Cache Carries Its Own Sparse-Attention Signal in a Vestigial Branch

    Sep 3, 2026WenJie FanKV CachingKV-Cache Management

    Yotta Labs

  29. CRISP: Cliff-awaRe Input-adaptive Sparse Prefilling with Structural-Mass-Motivated Routing

    Sep 1, 2026Huu Huy Nguyen, Chien Van Nguyen, Franck Dernoncourt +4Long-Context Language Model InferenceLLM Inference Acceleration

    University of Oregon · Adobe Research · Hanoi University of Science and Technology

  30. On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability

    Aug 31, 2026Zihan Qiu, Zekun Wang, Xiao Li +33Efficient Language Model TrainingSparse Attention

    Qwen Team

  31. A.X K2 Technical Report

    Aug 31, 2026Cheolseung Baek, Dhammiko Arya, Eunki Kim +40Mixture-of-Experts Language ModelsLong-Context Language Modeling

  32. Learning how to Forget: Fine-tuning for Long-Context Sparse Attention

    Aug 20, 2026Matthias Seeger, Zeyu Zhang, Vihang Patil +2Fine-TuningKV Caching

    Amazon Web Services · University of Amsterdam · Amazon +1

  33. SCOPE: Subspace Clustering with Online Per-Head Top-K Estimation for Sparse Video Attention

    Aug 13, 2026Qi Zhao, Qirui Li, Hanlin Tang +10Video Diffusion ModelsDiffusion Transformer

    Zhejiang University · Alibaba Group

  34. Sparse Attention to Emotion: Efficient Facial Emotion Recognition via Token Reduction

    Aug 9, 2026Aya Manel Zitouni, Aicha Zenakhri, Karim Haroun +1Efficient ViTsFacial Expression Recognition

    LIASD Laboratory, University of Paris 8, France

  35. Autonomy-of-Heads: Data-Free Sparse Attention from Frozen Query-Key Geometry

    Aug 7, 2026Yehan Yang, Junyuan Shang, Yang Li +3Attention Head AnalysisLLM Inference Acceleration

    Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Baidu Inc. +1

  36. Training-Free Hashing-Based Attention via Binary Principal Components

    Aug 5, 2026Daohai Yu, Zhanpeng Zeng, Keyu Chen +6Long-Context Language ModelingLLM Inference Acceleration

    Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China. · Tencent YouTu Lab, Shenzhen, China · Sino-Russian Research Center for Digital Economy.

  37. SPADE: An Input-Adaptive Sparse Attention Engine for Fast Video Diffusion Models Inference

    Aug 4, 2026Shanghao Liu, Renze Chen, Size Zheng +4Video Diffusion ModelsDiffusion Transformer

    Eric · Beihang University · Peking University

  38. ATFlash: Per-RoPE-Wavelength Attention Windows for Compute/Memory-Efficient LLM Inference

    Aug 3, 2026Shun-ichiro Hayashi, Daichi Mukunoki, Tetsuya Hoshino +1Long-Context Language Model InferenceLLM Inference Acceleration

    Graduate School of Informatics, Nagoya University · Information Technology Center, Nagoya University

  39. Token Radius Attention for Efficient Video Generation

    Aug 3, 2026Jiayu Chen, Zhikun Jiang, Maoliang Li +6Video Diffusion ModelsSparse Attention

    School of Computer Science, Peking University · School of Electronics Engineering and Computer Science, Peking University · School of Computer Science and Engineering, Beihang University +2

  40. Understanding Sparse Attention Selectivity in Long-Context Foundation Models via Counterfactual Evaluation

    Aug 3, 2026Xingyu Ren, Youran Sun, Chugang Yi +1LLM AuditingSparse Attention

    The Chinese University of Hong Kong, Hong Kong, China · University of Maryland, College Park, MD, USA

  41. LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexing

    Aug 3, 2026Wen Zan, Jiaqi Zhang, Jianchao Tan +11Long-Context ModelingLLM Inference Acceleration

    Meituan LongCat Team

  42. Recall Before You Rank: Similarity-Guided Top-KK Reuse for Efficient Long-Context Attention

    Jul 30, 2026Wenshuai Yao, Wenyong Zhou, Hanyong Shao +5Long-Context Language Model InferenceEfficient Attention

    School of Integrated Circuits, Peking University, Beijing, China · Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong SAR, China · School of Integrated Circuit Science and Engineering, Beihang University, Beijing, China

  43. CoSA: Accelerating Long-Context Inference via Proxy-Kernel Co-Designed Sparse Attention

    Jul 28, 2026Yufei Xue, Lin Niu, Hong Liu +6Long-Context Language Model InferenceLLM Inference Acceleration

    1Tencent · 2The Hong Kong University of Science and Technology

  44. PIVOT: Efficient Query-Group Indexing for Token-Level Sparse Attention

    Jul 27, 2026Hong Liu, Yuan Cheng, Lin Niu +5LLM Inference AccelerationEfficient Attention

    Tencent

  45. LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decoding

    Jul 27, 2026Junsung HwangLow-Rank AttentionKV Caching

  46. Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsification

    Jul 27, 2026Haopeng Li, Yitong Li, Junsong Chen +8Video Diffusion ModelsSparse Attention

    NVIDIA

  47. RIS-Kernel: A Model-Agnostic Architecture for Long-Context LLM Inference via Sparse Attention

    Jul 24, 2026Anderson R. SantosLong-Context Language Model InferenceMemory-Efficient Inference

    Faculty of Computing, Federal University of Uberlândia (UFU), Brazil.

  48. Parameter-free Adaptive Sparse Attention via Compression-Based Content Selection

    Jul 23, 2026Debarshi Kundu, Swaroop Ghosh, Vasant HonavarLong-Context Language ModelingSparse Attention

    The Pennsylvania State University

  49. ELSAA: Efficient Low-Rank and Sparse Attention Approximation for Training Transformers

    Jul 22, 2026Mahdi Heidari, Mohammad Mahdi Rahimi, Jaekyun MoonLow-Rank AttentionTransformer Attention

    School of Electrical Engineering, KAIST, Daejeon 34141, Republic of Korea · Department of Electrical Engineering and Computer Science, Daegu Gyeongbuk Institute of Science and Technology (DGIST), Daegu 42988, Republic of Korea

  50. FVAttn: Adaptive Sparse Attention with Runtime Load Balancing for Video Generation

    Jul 17, 2026Hao Liu, Chenghuan Huang, Ye Huang +7Video Diffusion ModelsDiffusion Transformer

    Sun Yat-sen University · WeChat HPC, Tencent Inc. · Peking University +1

  51. LiteTopK: Exploiting the Curse of Dimensionality for a Fused Indexer-TopK Kernel in Long-Context Sparse Attention

    Jul 13, 2026Ziqi Yin, Jianyang Gao, Peiqi Yin +2GPU Kernel OptimizationLLM Inference Acceleration

    Nanyang Technological University · ETH Zurich · The Chinese University of Hong Kong

  52. Sparse Attention for Dense Open-Vocabulary Prediction in CLIP

    Jul 8, 2026Fatimah Zohra, Chen Zhao, Shuming Liu +1Visual AttentionFine-Grained Image Retrieval

    King Abdullah University of Science and Technology (KAUST) · Harbin Institute of Technology, Shenzhen

  53. Uncertainty-gated selection for block-sparse attention

    Jul 4, 2026Thomas RossiLong-Context Language Model InferenceLLM Inference Acceleration

    Eonpass

  54. SAF3R: Dynamic Sparse Attention for Feed-Forward 3D Reconstruction Transformers

    Jul 3, 2026Jianing Deng, Yuanzhe Li, Jialu Wang +4Efficient Transformer Inference3D ViTs

    University of Pittsburgh, Pittsburgh, PA, USA · University of Arizona, Tucson, AZ, USA · Tongji University, Shanghai, China +2

  55. HyperVAttention: Efficient Sparse Attention with Spatio-Temporal Clustering for Video Diffusion

    Jul 3, 2026Dongyeun Lee, Amir Zandieh, Vahab Mirrokni +2Video Diffusion ModelsDiffusion Transformer

    KAIST · Google Research

  56. Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling

    Jul 3, 2026Xiang Hu, Xinyu Wei, Hao Gu +10Long-Context Language Model InferenceLong-Context Language Modeling

    Tencent HY Team · ShanghaiTech University · The Hong Kong University of Science and Technology +1

  57. Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scale

    Jul 1, 2026Siddharth Gollapudi, Nilesh Gupta, Prasann Singhal +1In-Context RetrievalLength Generalization

    UC Berkeley · UT Austin

  58. RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference

    Jun 30, 2026Wenhao Li, Jinhao Dong, Hailin Zhang +3KV CachingLong-Context Language Model Inference

    School of Information, Renmin University of China, Beijing, China · Peking University, Beijing, China

  59. Dual Sparse Aggregation Transformer for Multispectral Object Detection

    Jun 30, 2026Wencong Wu, Xiuwei Zhang, Hanlin Yin +2Multispectral Object DetectionDetection Transformer

    School of Computer Science, Northwestern Polytechnical University, Xi’an 710072, China · School of Computer Science, the Shaanxi Provincial Key Laboratory of Speech and Image Information Processing and the National Engineering Laboratory for Integrated Aerospace-Ground-Ocean Big Data Application Technology, Northwestern Polytechnical University, Xi’an 710072, China · School of Cybersecurity, Northwestern Polytechnical University, Xi’an 710072, China

  60. Hierarchical Global Attention (HGA)

    Jun 29, 2026Woernle Frank, Fedosov Vladimir, Grinenko ArtemiyKV-Cache OffloadingLong-Context Modeling

    BMW Group