Efficient Attention

Follow this topic Ask about this topic

Efficient attention studies how to reduce the cost of comparing queries with keys, especially for long sequences and constrained hardware. Approaches include IO-aware fused kernels, sparse or hierarchical retrieval, linear and recurrent attention, and KV-cache compression; many combine algorithm design with GPU or accelerator-specific kernels. Evaluation spans language modeling and long-context retrieval as well as vision, video, and other sequence tasks, where quality, latency, memory, and energy can trade off differently. A central tension is that practical methods can work well on structured inputs, while preserving full-attention quality across arbitrary inputs—and providing strong guarantees—remains difficult.

Latest papers

138 papers17 papers in the last four weeks, down 6% on the four weeks before. 0.1% of all new papers.

MixPT: Point Transformer with Interleaved Window and Global Attention
Oct 22, 2026Minkyun Seo, Hyungtae Lim, Jaesik ParkPoint Transformer Point Cloud Understanding
Seoul National University · Massachusetts Institute of Technology
OpenAttention: One Percent Is Enough with Tail Proxies
Oct 22, 2026Hyunwoo Yu, Jongbeom Lee, Hyunbin Kim +8Efficient Attention Sparse Attention
Sogang University · Samsung Electronics · Pusan National University
Flow-based Spectral Kernel Learning for Nonstationary Attention
Oct 21, 2026Zi Yang, Ying Li, Alejandro Lancho +3Kernel Methods Efficient Attention
School of Computing and Data Science, The University of Hong Kong · Department of Statistics and Data Science, National University of Singapore · Department of Signal Theory and Communications, Universidad Carlos III de Madrid
Transformers as Generic Deep Neural Networks: Exact Compilation, the Cost of Removing Activation-Activation Products, and Multi-Agent Attention Clustering
Oct 20, 2026Sergey Salishev, Zeev Volkovich, Oleg GranichinEfficient Transformer Inference Transformer Attention
St. Petersburg State University, Saint Petersburg 199034, Russia · Department of Software Engineering, Braude College of Engineering, Karmiel 2161002, Israel
Not All Dimensions Are Equal: Outlier-Aware Attention Linearization with Grouped Quadratic Kernels
Oct 9, 2026Weiqi Sun, Qianyue Wu, Wenjia Wang +1Attention Approximation Efficient Attention
The Chinese University of Hong Kong, Shenzhen · Beijing Normal University, Zhuhai · National University of Singapore
High-Throughput Generative Video Super-Resolution on a Single Consumer GPU
Oct 9, 2026Georgios Panaretou, Sotirios ChatzisVideo Super-Resolution Temporal Consistency in Video Generation
Ethical AI Novelties Limassol, Cyprus · Cyprus University of Technology Limassol, Cyprus
Sparse Attention Is Matrix Approximation, Not Choosing from a Bag of Values
Oct 7, 2026Fang Wan, Xufeng Liu, Fan Li +1Sparse Attention Efficient Attention
Stony Brook University · Duke University
Attention via Black-Box Vector Search
Oct 7, 2026Stepan Zharkov, Krish Singal, Ashwin Padaki +1Attention Approximation Efficient Attention
University of Pennsylvania
LatentIndex: Cross-Layer Sharing with Layer-Specific Selection for Sparse Attention
Oct 3, 2026Zhaohui Wang, Zhixin Pan, Fanxu Meng +1Cross-Layer Attention LLM Inference Acceleration
Institute for Artificial Intelligence, Peking University · Dots Studio, Xiaohongshu Inc. · Beijing Institute of Technology
Retrieval-Centric Deep Learning in Growing Nonparametric Neural Networks
Oct 2, 2026Maximilian Schlegel, Rajai Nasser, Seijin Kobayashi +6Deep Learning Optimization Memory-Augmented Neural Networks
Google, Paradigms of Intelligence Team, Zurich, Switzerland · MIT CSAIL, Cambridge, MA, USA · Department of Computer Science and Wu Tsai Institute, Yale University, New Haven, CT, USA
Page-EntroKV: Hardware-Aligned, Entropy-Weighted KV-Cache Eviction under Grouped-Query Attention
Oct 2, 2026Inbasekaran SKV-Cache Eviction Memory-Efficient Inference
Department of Computer Science and Engineering SRM Institute of Science and Technology, India
LampAttention: Look-Ahead Mixed-Precision FlashAttention for Dedicated Accelerators
Sep 30, 2026Stanislav Budzinskiy, Marian Gloser, Tolunay Yilmaz +5AI Accelerator Inference LLM Inference Acceleration
Faculty of Mathematics University of Vienna, Austria · Huawei Heisenberg Research Center, Munich, Germany · Huawei Technologies Co. Ltd
Switching Linear Attention
Sep 30, 2026Hyun Dong Lee, Xavier Gonzalez, Nicolas Zucchet +3Efficient Attention Linear Attention
Stanford University · Unconventional AI. Work done at Stanford University.
Efficiently Approximating Attention Is Hard
Sep 29, 2026Lukas Haverbeck, Carmen Amo Alonso, Andres Felipe Posada-Moreno +2Attention Approximation Efficient Attention
Institute for Data Science in Mechanical Engineering, RWTH Aachen University · Autonomous Systems Lab, Stanford University
SANTA++: Sampling Attention through Representative Keys
Sep 28, 2026Kyle Lee, Christian Z. Pratt, Ruoyu Fang +4Memory-Efficient Inference LLM Inference Acceleration
Department of Electrical and Computer Engineering University of California, Santa Barbara, CA, USA · Flucta, San Francisco, CA, USA
When Can Attention Heads Be Statically Defined?
Sep 28, 2026Weixian Waylon Li, Yintao Tai, Marcio Fonseca +1Attention Head Analysis Efficient Attention
University of Edinburgh, United Kingdom · Chamber of Deputies, Brazil
Broken Symmetry in BF16 Attention: Why FlashAttention Gradients Blow Up Late in Training
Sep 28, 2026Junlin Chen, Daize Dong, Huanwei Di +9Softmax Attention Efficient Attention
Rutgers University · Carnegie Mellon University · Oracle +2
FoldAttention: Declared-Reference Softmax for Fast Decode and Deterministic Backward
Sep 27, 2026Sriman AchantaSoftmax Attention LLM Inference Acceleration
Virginia Commonwealth University
MassAlloc Attention: Let Attention Allocate Its Own Compute
Sep 26, 2026Jingze Shi, Zhangyang Peng, Xianduo Li +6Attention Mechanisms Efficient Attention
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Beijing Academy of Artificial Intelligence, Beijing, China · Université Paris Cité, Paris, France
CoWindow Attention: Full Causal Coverage Is a Collective Property
Sep 26, 2026Jingze Shi, Zhangyang Peng, Xianduo Li +6Efficient Attention Causal Attention
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China. · Beijing Academy of Artificial Intelligence, Beijing, China. · Université Paris Cité, Paris, France.
Benchmarking Attention for Tabular Foundation Models
Sep 25, 2026Maximilian Schambach, Clemens Biehl, Sam ThelinTabular Foundation Models Efficient Attention
SAP SE, Germany
MoSAR: Mixture of Semantic Attention Regimes for Learning Adaptive and Approximable Attention Geometries
Sep 25, 2026Michele Paolicelli, Alessandro Petruzzelli, Alessandro Franceso Maria Martina +2Attention Mechanisms Efficient Attention
Department of Computer Science Università degli Studi di Bari Aldo Moro Bari, 70121, Italy
Block Sparse Attention with Log-Linear Complexity
Sep 25, 2026Bohao Tang, Zhen Qin, Yuqi Pan +2Efficient Attention Block-Sparse Attention
Shanghai Jiao Tong University · Shanghai Innovation Institute · ByteDance Seed
Attention Routing Stabilizes Early: Working-Set Inference for Recurrent Language Models
Sep 23, 2026Ke Wan, Chen ChenLong-Context Language Model Inference Recurrent Transformers
Department of Computer Science University of Virginia Charlottesville, VA, USA
FlashBoB: I/O-Efficient Exact Backward-over-Backward for Softmax Attention
Sep 21, 2026Anthony Givans, Michael Crawshaw, Mingrui LiuSoftmax Attention Automatic Differentiation
Programming AMD XDNA NPUs with Open-source Compiler Tools: A FlashAttention Case Study
Sep 18, 2026Erwei Wang, Ephrem Wu, Victor J. B. Jung +4Roofline Modeling AI Accelerator Inference
Advanced Micro Devices, Inc., USA. · Integrated Systems Laboratory (IIS), ETH Zürich, Switzerland. · Cornell University, Ithaca, NY, USA.
Reaching Every Position Without Searching: Rotating Sparse Wiring on the Hypercube as a Substitute for Attention
Sep 16, 2026Yoshiaki TakashitaEfficient Attention Language Modeling
School of Law, Waseda University
AttnFuse: A Composable DSL for Compiling Attentions to Fused GPU Kernels
Sep 11, 2026Varun Kumar Dasoju, Tian ZhaoEfficient Attention GPU Kernel Optimization
Department of Computer Science University of Wisconsin–Milwaukee
Rethinking Heterogeneous System Disaggregation for Subquadratic Attention
Sep 11, 2026Arya Tschand, Yaosheng Fu, Vikram Sharma Mailthody +6Disaggregated LLM Serving LLM Inference Acceleration
Harvard University Cambridge, Massachusetts, USA · NVIDIA Santa Clara, California, USA
Grouped Value Attention: Efficient KV Caching via On-Demand Key Reconstruction
Sep 8, 2026Vishesh Tripathi, Abhay Kumar, Ramsha KhanEfficient Transformer Inference Grouped-Query Attention
FrontiersMind

Oct 22, 2026J/K move · Enter open · S save

MixPT: Point Transformer with Interleaved Window and Global Attention

Minkyun Seo, Hyungtae Lim, Jaesik Park

Seoul National University · Massachusetts Institute of Technology

Point Transformer Point Cloud Understanding Efficient Attention Hybrid Attention

Open paper AskPDF arXiv

Point cloud transformers aim to capture detailed local geometry and relationships across a scene at a manageable computational cost. Two common approaches are window attention and global attention. They differ in how they enable scene-wide interaction, which refers to information exchange among tokens across a scene. Window attention reduces computational cost by restricting self-attention to groups of tokens called windows; in serialized designs, each window contains a fixed number of consecutive tokens from a spatially ordered sequence. Because each block only directly connects tokens within the same window, information exchange across windows relies on changes in window membership across blocks and hierarchical pooling. Global attention, in contrast, directly connects all tokens within a scene, but computing attention over all token pairs can be costly when the number of tokens is large. To balance computational cost and scene-wide interaction, we introduce MixPT, a point transformer that uses a mix of window and global attention blocks in an interleaved schedule, while retaining a single fixed serialization pattern for its window blocks. For scenes with a large number of tokens, MixPT applies global attention to spatially pooled tokens and returns residual updates to the original-resolution stream. Across indoor and outdoor benchmarks, MixPT achieves competitive accuracy with lower latency than representative window and global attention baselines. We further demonstrate the feasibility of scaling the attention stage in depth by repeating the interleaved pattern of window and global attention blocks.