Block Sparse Flash Attention

Ask about this topic

Momentum

8 papers in the last four weeks, up 60% on the four weeks before. 0.1% of all new papers.

Jul 6Week of Sep 21
View on map

Latest papers 29

All topics
CardsList

  1. LampAttention: Look-Ahead Mixed-Precision FlashAttention for Dedicated Accelerators

    Sep 30, 2026Stanislav Budzinskiy, Marian Gloser, Tolunay Yilmaz +5Block Sparse Flash Attention

    Faculty of Mathematics University of Vienna, Austria · Huawei Heisenberg Research Center, Munich, Germany · Huawei Technologies Co. Ltd

  2. Parameterized Stripe Attention for Efficient Video Generation

    Sep 29, 2026Xingyu Jia, Baole Ai, Ang Wang +2Spatio-Temporal AttentionVideo Diffusion Models

    Alibaba Group

  3. DORA: Dynamic Online Reinforcement Agent for Token Pruning in Vision Transformers

    Sep 28, 2026Kaixuan He, Song Chen, Yi KangVisual Token PruningVision Transformer

    University of Science and Technology of China, Hefei, China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Hefei, China

  4. Broken Symmetry in BF16 Attention: Why FlashAttention Gradients Blow Up Late in Training

    Sep 28, 2026Junlin Chen, Daize Dong, Huanwei Di +9Block Sparse Flash AttentionMatched Fp16 Intermediate

    Rutgers University · Carnegie Mellon University · Oracle +2

  5. FoldAttention: Declared-Reference Softmax for Fast Decode and Deterministic Backward

    Sep 27, 2026Sriman AchantaBlock Sparse Flash AttentionAutoregressive Decoding

    Virginia Commonwealth University

  6. FlashLoop: Fast and Memory-Efficient Looped Transformers via Lazy Updates

    Sep 24, 2026Wanqi Yang, Shiwei LiuBlock Sparse Flash AttentionTransformer Architectures

    ELLIS Institute Tübingen Max Planck Institute for Intelligent Systems Tübingen AI Center

  7. FlashBoB: I/O-Efficient Exact Backward-over-Backward for Softmax Attention

    Sep 21, 2026Anthony Givans, Michael Crawshaw, Mingrui LiuBlock Sparse Flash AttentionGumbel-Softmax Relaxation

  8. LumoTree: Path-Parallel Speculative Verification for Hybrid Language Models

    Sep 20, 2026Zhiyuan MaGated DeltanetBlock Sparse Flash Attention

  9. DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression

    Sep 17, 2026DeepSeek-AI, :, Anyi Xu +590Key-Value Cache CompressionKey-Value Cache

  10. SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking

    Sep 14, 2026Zhiwei Li, Lei Zhu, Hao Gu +6Dynamic Sparse AttentionBlock Sparse Flash Attention

    Tencent HY LLM Frontier · Hong Kong University of Science and Technology (Guangzhou) · Hong Kong University of Science and Technology

  11. EFQ-Softmax: Exp-Free Quantization for Softmax

    Sep 9, 2026Haohui Han, Yuming Wan, Hongni Wang +4Gumbel-Softmax RelaxationBlock Sparse Flash Attention

    Xi'an Jiaotong University · Xi’an Jiaotong University · Huawei Technologies Co., Ltd +1

  12. Hardware-Aware FP4 FlashAttention-4

    Sep 3, 2026Robert HuBlock Sparse Flash AttentionCausal Inferences

    GRAPHCORE RESEARCH

  13. Ask Self, Ask Others: Relation Is All You Need

    Aug 20, 2026Yuting Ge, Pengju Yang, Mingkai NieLinear AttentionBlock Sparse Flash Attention

    Department of Electrical Engineering City University of Hong Kong · Jilin University · National University of Singapore

  14. Training-Free Hashing-Based Attention via Binary Principal Components

    Aug 5, 2026Daohai Yu, Zhanpeng Zeng, Keyu Chen +6Dynamic Sparse AttentionBlock Sparse Flash Attention

    Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China. · Tencent YouTu Lab, Shenzhen, China · Sino-Russian Research Center for Digital Economy.

  15. ATFlash: Per-RoPE-Wavelength Attention Windows for Compute/Memory-Efficient LLM Inference

    Aug 3, 2026Shun-ichiro Hayashi, Daichi Mukunoki, Tetsuya Hoshino +1Block Sparse Flash AttentionRotary Position Embedding

    Graduate School of Informatics, Nagoya University · Information Technology Center, Nagoya University

  16. Output-Aware Rotation for INT2 KV-Cache Quantization

    Aug 3, 2026Vincent-Daniel Yun, Woosang Lim, Minsoo Cheong +4Key-Value Cache QuantizationKv-Cache Management

    University of Southern California · Seoul National University · Inha University

  17. Understanding Sparse Attention Selectivity in Long-Context Foundation Models via Counterfactual Evaluation

    Aug 3, 2026Xingyu Ren, Youran Sun, Chugang Yi +1Dynamic Sparse AttentionBlock Sparse Flash Attention

    The Chinese University of Hong Kong, Hong Kong, China · University of Maryland, College Park, MD, USA

  18. Flash EQ-Linear: Accelerating Equivariant Linear Layers via Group-wise Discrete Fourier Transform

    Jul 23, 2026Zhongchen Zhao, Jixin Wang, Qi Xie +4Equivariant Neural NetworksRotation Equivariance

    Xi’an Jiaotong University · The Hong Kong Polytechnic University · OPPO Research

  19. HiFA4: Training-Free 4-bit FlashAttention on Ascend HIF4 NPUs for LLM Inference

    Jul 5, 2026Hui Dong, Yanzhao Li, Jie Gao +5Block Sparse Flash AttentionLLM Inference Optimization

    Huawei Technologies

  20. RotateAttention: RoPE-Aware Rotation and Range Rectification for INT4 Quantized Attention in Video Generation

    Jul 1, 2026Yaofu Liu, Wanli Lan, Jinxi Li +2Block Sparse Flash AttentionVideo Diffusion Models

    The Hong Kong University of Science and Technology, Hong Kong SAR, China · Independent Researcher, China

  21. Reweighting Framewise Attention in Video Transformers for Facial Expression Understanding

    Jun 29, 2026Seongro Yoon, Donghyeon Cho, Jinsun Park +1Facial Expression RecognitionSelf-Supervised Vision Transformers

    Inria, Université Côte d’Azur, France · Hanyang University, South Korea · Pusan National University, South Korea

  22. EpiKV: Epiphany-Aware KV Cache Eviction Without the Attention Matrix

    Jun 25, 2026Steven Kolawole, Virginia SmithKey-Value Cache EvictionKey-Value Cache

    Language Technologies Institute Carnegie Mellon University · Machine Learning Department Carnegie Mellon University

  23. P-Cast Precision in FP8 Attention: Sink-Induced Collapse and the Optimality of S=2^8

    Jun 2, 2026Reed LauFull-Precision PerformanceFp8

    Tencent

  24. Quantized Keys Steal Attention: Bias Correction for KV-Cache Compression in Video Diffusion

    May 25, 2026Tuna Tuncer, Felix Becker, Thomas PfeilAutoregressive Video Diffusion ModelsKey-Value Cache Quantization

    1Technical University of Munich · 2Tensordyne

  25. ThriftAttention: Selective Mixed Precision for Long-Context FP4 Attention

    May 21, 2026Joe SharrattBlock Sparse Flash AttentionEfficient Long-Context Inference

  26. DualKV: Shared-Prompt Flash Attention for Efficient RL Training with Large Rollouts and Long Contexts

    May 14, 2026Jiading Gai, Shuai Zhang, Xiang Song +2Block Sparse Flash AttentionEfficient Long-Context Inference

    1Amazon Web Services · 2Google · University of Minnesota

  27. QFlash: Bridging Quantization and Memory Efficiency in Vision Transformer Attention

    Apr 28, 2026Sehyeon Oh, Yongin Kwon, Jemin LeeBlock Sparse Flash AttentionQuantized

    University of Science and Technology, Daejeon, Republic of Korea · Electronics and Telecommunications Research Institute, Daejeon, Republic of Korea · Pusan National University, Busan, Republic of Korea +1

  28. Dispatch-Aware Ragged Attention for Pruned Vision Transformers

    Apr 16, 2026Seifeldin Abdellatif, Ahmad AlmasriVisual Token PruningBlock Sparse Flash Attention

    College of Engineering Al Ain University Abu Dhabi, United Arab Emirates

  29. Block Sparse Flash Attention

    Dec 7, 2025Daniel Ohayon, Itay Lamprecht, Itay Hubara +3Block Sparse Flash AttentionEfficient Long-Context Inference

    Technion – Israel Institute of Technology, Haifa, Israel · Intel - Habana Labs · Intel – Habana Labs, Tel Aviv, Israel