Efficient Attention

Follow this topic Ask about this topic

Momentum

15 papers in the last four weeks, up 114% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 96

All topics

MoA-Structured Decode Attention DNF Derivation, KV-Cache Accumulation, GQA/MQA, and OpenACC Kernel
Jul 21, 2026Lenore Mulin, Gaetan HainsEfficient Transformer Inference Grouped-Query Attention
University at Albany, SUNY · LACL, Universit´e Paris-Est Cr´eteil
Kernelized Linear Attention: Breaking the Capacity Wall with Symmetric Cones
Jul 19, 2026Ayoub Ghriss, Sourav ChakrabortyEfficient Attention Linear Attention
Department of Computer Science, University of Colorado Boulder
DiTango: Cost-Effective Parallel Diffusion Generation with Selective Attention State Reuse
Jul 17, 2026Yuyang Chen, Runxin Zhong, Zan Zong +3Diffusion Transformer Efficient Attention
Shanghai Jiao Tong University, Shanghai AI Laboratory Shanghai, China · Tsinghua University Beijing, China · University of Science and Technology Beijing Beijing, China +1
VideoSEMA: a scalable and efficient Mamba-like attention for video understanding
Jul 16, 2026Nhat Thanh Tran, Fanghui Xue andShuai Zhang, Jiancheng Lyu +3Efficient ViTs Efficient Attention
Department of Mathematics University of California, Irvine Irvine, USA · Qualcomm AI Research San Diego, USA
AVQ-Attention: Adaptive Vector-Quantized Attention
Jul 14, 2026Winfried van den dool, Patrick Forré, Amir Habibian +2Transformer Attention Vector Quantization
QUVA Lab, University of Amsterdam, The Netherlands · AMLab, Informatics Institute, University of Amsterdam, The Netherlands · AI4Science Lab, University of Amsterdam, The Netherlands +3
FastTPS: An Optimized Method for LLM Token Phase for AI accelerators
Jul 13, 2026Wenzong Yang, Danyang Zhang, Kun Cao +17AI Accelerator Inference LLM Inference
Advanced Micro Devices, Inc., Santa Clara, California, USA.
STEEL: Sparsity-Aware Fused Attention for Energy-Efficient Long-Sequence Inference on AMD's XDNA NPU
Jul 10, 2026Victor J. B. Jung, Gagandeep Singh, Joseph Melber +3Efficient Transformer Inference Energy-Efficient ML
Integrated Systems Laboratory (IIS), ETH Zürich, Switzerland. · AMD Research and Advanced Development (RAD). · Department of Electrical, Electronic and Information Engineering (DEI), University of Bologna, Italy.
Morphing into Hybrid Attention Models
Jun 29, 2026Disen Lan, Jianbin Zheng, Yuxi Ren +5Efficient Attention Neural Architecture Search
1Fudan University · 2ByteDance Seed · Work done at ByteDance Seed +1
Hierarchical Global Attention (HGA)
Jun 29, 2026Woernle Frank, Fedosov Vladimir, Grinenko ArtemiyKV-Cache Offloading Long-Context Modeling
BMW Group
Enhancing Layer Interaction Using Key-Correlated Layer Attention
Jun 24, 2026Jianlong Xiong, ChuanBo Xie, Le Yu +2Efficient Attention Linear Attention
College of Computer Science, Sichuan University, Chengdu, 610065, China · Department of Ultrasound, Zigong Maternal and Child Health Care Hospital, China
Scalable Peptide Design via Memory-Efficient Equivariant Transformer
Jun 23, 2026Rui Jiao, Xiangzhe Kong, Yinjun Jia +4Equivariant Neural Networks Efficient Attention
Department of Computer Science and Technology, Tsinghua University · Institute for AI Industry Research, Tsinghua University · Department of Electronic Engineering, Tsinghua University +1
Keyless Attention: Value-Space Routing and Value-Only Caching for Efficient Transformers
Jun 20, 2026Xin GaoTransformer Inference KV Caching
Department of Mathematics and Statistics York University Toronto, M3J 1P3
Grouped Query Experts: Mixture-of-Experts on GQA Self-Attention
Jun 18, 2026Vishesh Tripathi, Abhay KumarGrouped-Query Attention Transformer Attention
FrontiersMind
CSWinUNETR: Segmentation of Thin Anatomical Structures in Medical Images
Jun 18, 2026Junho Moon, Haejun Chung, Ikbeom JangHybrid CNN-Transformer Architectures Efficient Attention
Hanyang University, Seoul, Republic of Korea · Hankuk University of Foreign Studies, Yongin, Republic of Korea
Rethinking the Role of Efficient Attention in Hybrid Architectures
Jun 13, 2026Ziqing Qiao, Yinuo Xu, Chaojun Xiao +6Attention Head Analysis Long-Context Modeling
1Tsinghua University · 2OpenBMB
Semidirect Fourier Delta Attention: Phase-Controlled Delta Memory with Constructive Chunk-WY Kernels
Jun 12, 2026Tiantian ZhangEfficient Attention Linear Attention
Columbia University
MiniMax Sparse Attention
Jun 11, 2026Xunhao Lai, Weiqi Xu, Yufeng Yang +14GPU Kernel Optimization Long-Context Language Modeling
MiniMax · Peking University · NVIDIA +4
Gaussian Mixture Attention: Linear-Time Sequence Mixing via Probabilistic Latent Routing
Jun 9, 2026Yongchao Huang, Hassan RazaLong-Context Language Modeling Efficient Attention
Inside the Latent Flow: Causal Deciphering of Attention Dynamics in Audio Separation Foundation Models
Jun 8, 2026Yuxuan Chen, Haoyuan Yu, Peize HeCross-Attention Audio Diffusion Models
Jilin University · Hunan University · University of Electronic Science and Technology of China
Look Less, Reason More: Block-wise Attention Skipping for Efficient Multimodal LLMs
Jun 7, 2026Jie Ma, Zhike Qiu, Jiayi Ji +2Efficient Multimodal Inference Multimodal Large Language Models
Xiamen University
Attention at the Theoretical Minimum: A Mathematics of Arrays Framework for Memory-Optimal Transformer Kernels
Jun 5, 2026Lenore Mullin, Gaetan HainsGPU Kernel Optimization Energy-Efficient ML
University at Albany, SUNY, Albany, NY, USA · LACL, Université Paris-Est Créteil, Créteil, France
Towards Tight Bounds for Streaming Attention
Jun 5, 2026Justin Y. Chen, Ying Feng, Piotr Indyk +3Efficient Attention KV-Cache Compression
MIT · EPFL
How Much Dense Attention is Necessary? Oracle-Guided Sparse Prefill for Full/GQA Layers in Hybrid Long-Context Models
Jun 5, 2026Hongxing Wang, Harenome Razanajato, Zhen Zhang +2Grouped-Query Attention Long-Context Language Modeling
Do Transformers Need Three Projections? Systematic Study of QKV Variants
Jun 1, 2026Ali Kayyam, Anusha Madan Gopal, M Anthony LewisEfficient Transformer Inference Transformer Attention
BrainChip Inc., Laguna Hills, CA, USA.
Post-Boundary Bridge: Must Local Attention Go Global Between Global Layers?
Jun 1, 2026Zhibo YangEfficient Attention Local Attention
Ocean University of China
Interdomain Attention: Beyond Token-Level Key-Value Memory
May 23, 2026Naoki Kiyohara, Harrison Bo Hua Zhu, Riccardo El Hassanin +4Self-Attention Long-Context Language Modeling
Approaching I/O-optimality for Approximate Attention
May 22, 2026Pál András Papp, Aleksandros Sobczyk, Anastasios ZouziasSelf-Attention Efficient Attention
Computing Systems Lab, Huawei Technologies, Zurich, Switzerland
Preisach Attention: A Hysteretic Model of Sequential Memory
May 22, 2026Piotr FrydrychTransformer Efficient Attention
The Metrology and Biomedical Engineering Institute · Faculty of Mechatronics, Warsaw University of Technology
ThriftAttention: Selective Mixed Precision for Long-Context FP4 Attention
May 21, 2026Joe SharrattMixed-Precision Quantization Self-Attention
Structured-Sparse Attention for Entity Tracking with Subquadratic Sequence Complexity
May 21, 2026Hangyue Zhao, Paul Caillon, Erwan Fagnou +1Self-Attention Structured Sparsity
ESPCI PSL, Paris, France · LAMSADE, Université Paris Dauphine - PSL, Paris, France

Jul 21, 2026, cs.LGJ/K move · Enter open · S save

MoA-Structured Decode Attention DNF Derivation, KV-Cache Accumulation, GQA/MQA, and OpenACC Kernel

Lenore Mulin, Gaetan Hains

University at Albany, SUNY · LACL, Universit´e Paris-Est Cr´eteil

Efficient Transformer Inference Grouped-Query Attention Transformer Inference GPU Kernel Optimization KV Caching Memory-Efficient Optimization+2

Open paper AskPDF arXiv

We derive four memory-optimal inference artifacts for transformer attention using the Mathematics of Arrays (MoA), each following directly from the forward-pass Denotational Normal Form (DNF) of with the query-row index fixed to the current decode step. The artifacts are: (1)~a single-query decode DNF in which the

ψ

-reduction eliminates the

K^\top

buffer algebraically, achieving

(d_k + nd_k+ nd_v+ d_v)\times4\,{B}

Dynamic Random Access Memory (DRAM) traffic result numerically verified to

\|{err}\|_\leq2\times10^{-7}

; (2)~a C/OpenACC Graphics Processing Unit (GPU) kernel with Operational Normal Form (ONF) stride arithmetic and hardware-coalesced memory access, verified to

\|\mathrm{err}\|_\infty=0

(exact IEEE-754 floating-point arithmetic); (3)~a multi-step KV-cache with

O(d_k+d_v)

per-step append via MoA concatenation

\#

; and (4)~Grouped-Query Attention (GQA) and Multi-Query Attention (MQA) derived via

ψ

-selection, achieving a proven

\frac {h_q} { h_{kv} }

reduction in KV traffic. All programs are verified against PyTorch scaled_dot_product_attention.

Momentum

Related topics

Latest papers 96