Memory-Efficient Attention

Follow this topic Ask about this topic

Latest papers

32 papers5 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Fast and Exact Relative-Position Attention via Structure-Aware Kernels
Oct 31, 2026Delyan Boychev, Ivo Petrov, Dimitar I. DimitrovEfficient Attention Memory-Efficient Attention
INSAIT, Sofia University “St. Kliment Ohridski”.
Attention, Dynamic Wave and Hybrid Sequence Mixers in Language Models
Oct 29, 2026Dmytro Babarytskyi, Oleksandr BabarytskyiRecurrent Language Models Hybrid Attention
Kharkiv National University of Radio Electronics Kharkiv, Ukraine · Military Law Institute Yaroslav Mudryi National Law University Kharkiv, Ukraine
Folding Attention Biases into Keys Makes Compacted KV Caches Run on Standard Kernels
Oct 27, 2026Utkarsh KanwatKV-Cache Compression Memory-Efficient Attention
Independent Researcher
INTRA: Interleaved Non-contiguous Token spaRse Attention
Oct 25, 2026Yuxuan Xia, Wenbo Lu, Shaoyi Zheng +1Sparse Attention Transformer Inference
New York University
Efficient Edge-aware Attention Network for Graph Generation
Oct 20, 2026Vincent Jung, Alba Carballo-Castro, Yiming Qin +2Graph Generation Graph Transformer
LTS4, EPFL, Lausanne, Switzerland · Università della Svizzera italiana, Lugano, Switzerland
SANTA++: Sampling Attention through Representative Keys
Sep 28, 2026Kyle Lee, Christian Z. Pratt, Ruoyu Fang +4Memory-Efficient Inference LLM Inference Acceleration
Department of Electrical and Computer Engineering University of California, Santa Barbara, CA, USA · Flucta, San Francisco, CA, USA
Sliding-window beats linear attention
Aug 28, 2026Alexia Jolicoeur-Martineau, Rhea Sanjay Sukthanker, Pashmina Cameron +1Linear Attention KV Caching
Microsoft Applied Sciences Group (ASG) · Independent
Q-Interference: Memory-Efficient Phase-Aware Quantum-Inspired Attention
Aug 18, 2026Emama Nahid, Tahmid Imtiaz Imu, Huayue Gu +3Memory-Efficient Attention Transformer Attention
Kennesaw State University Marietta, GA, USA · Miami University Oxford, OH, USA · Georgia State University Atlanta, GA, USA
Efficient Reinforcement Learning for Long-Horizon Tool-Use Agentic Tasks
Aug 11, 2026Zelei Cheng, Amritansh Mishra, Sambit Sahu +1Memory-Efficient Attention Efficient Attention
AI Foundations, Capital One
MoA-Structured Decode Attention DNF Derivation, KV-Cache Accumulation, GQA/MQA, and OpenACC Kernel
Jul 21, 2026Lenore Mulin, Gaetan HainsAttention Mechanisms Efficient Transformer Inference
University at Albany, SUNY · LACL, Universit´e Paris-Est Cr´eteil
A Hippocampus for Linear Attention: An Exact Memory for What the Recurrent State Forgets
Jul 2, 2026Wanyun CuiHybrid Attention Long-Context Retrieval
Shanghai University of Finance and Economics
Scalable Peptide Design via Memory-Efficient Equivariant Transformer
Jun 23, 2026Rui Jiao, Xiangzhe Kong, Yinjun Jia +4Memory-Efficient Attention Equivariant Neural Networks
Department of Computer Science and Technology, Tsinghua University · Institute for AI Industry Research, Tsinghua University · Department of Electronic Engineering, Tsinghua University +1
SpotAttention: Plug-In Block-Sparse Routing for Pretrained Long-Context Transformers
Jun 22, 2026Huzama Ahmad, Se-Young YunSparse Attention Block-Sparse Attention
KAIST
Keyless Attention: Value-Space Routing and Value-Only Caching for Efficient Transformers
Jun 20, 2026Xin GaoTransformer Inference KV Caching
Department of Mathematics and Statistics York University Toronto, M3J 1P3
Express Language Modeling
Jun 9, 2026Albert Gong, Annabelle Michael Carrell, Raaz Dwivedi +1Long-Context Language Model Inference Attention Mechanisms
Cornell Tech · University of Cambridge · Microsoft Research
Attention at the Theoretical Minimum: A Mathematics of Arrays Framework for Memory-Optimal Transformer Kernels
Jun 5, 2026Lenore Mullin, Gaetan HainsMemory-Efficient Attention GPU Kernel Optimization
University at Albany, SUNY, Albany, NY, USA · LACL, Université Paris-Est Créteil, Créteil, France
Augmenting Attention with Exponentially Decaying Memory Improves Query-Aware KV Sparsity
May 27, 2026Xiuying Wei, Caglar GulcehreKV Caching Memory-Augmented Language Models
EPFL, CLAIRE Lausanne, Switzerland
Approaching I/O-optimality for Approximate Attention
May 22, 2026Pál András Papp, Aleksandros Sobczyk, Anastasios ZouziasSelf-Attention Efficient Attention
Computing Systems Lab, Huawei Technologies, Zurich, Switzerland
Variational Linear Attention: Stable Associative Memory for Long-Context Transformers
May 11, 2026Vishal Pandey, Gopal SinghEfficient Attention Linear Attention
Independent Researcher London, UK · Metriqual Athens, GR
Nectar: Neural Estimation of Cached-Token Attention via Regression
May 10, 2026João Monteiro, Michal Klein, Pierre Ablin +1Attention Approximation Softmax Attention
Apple
Attention Itself Could Retrieve.RetrieveVGGT: Training-Free Long Context Streaming 3D Reconstruction via Query-Key Similarity Retrieval
May 10, 2026Zichen Zou, Xiaosong Jia, Zuxuan Wu +1Memory-Efficient Attention 3D ViTs
Institute of Trustworthy Embodied AI (TEAI), Fudan University · 2Shanghai Key Laboratory of Multimodal Embodied AI
Nearly Optimal Attention Coresets
May 7, 2026Edo Liberty, Alexandr Andoni, Eldar KleinerSoftmax Attention Efficient Attention
Tucker Attention: A generalization of approximate attention mechanisms
Mar 31, 2026Timon Klein, Jonas Kusch, Sebastian Sager +2Tucker Decomposition Attention Mechanisms
Department of Mathematics; Otto von Guericke University Magdeburg; 39106 Magdeburg; Germany. · Scientific Computing, Norwegian University of Life Sciences, ˚As, Norway · Max Planck Institute for Dynamics of Complex Technical Systems; 39106 Magdeburg; Germany. +1
MKA: Memory-Keyed Attention for Efficient Long-Context Reasoning
Mar 21, 2026Dong Liu, Yanxuan Yu, Ben Lengerich +1Hierarchical Attention Attention Mechanisms
University of California, Los Angeles, Los Angeles, California, USA · Columbia University, New York, New York, USA · University of Wisconsin-Madison, Madison, Wisconsin, USA
LinearARD: Linear-Memory Attention Distillation for RoPE Restoration
Mar 9, 2026Ning Yang, Hengyu Zhong, Wentao Wang +4Attention-Guided Distillation Memory-Efficient Attention
Institute of Automation, Chinese Academy of Sciences, Beijing, China · Southwest University, Chongqing, China · Dalian University of Technology, Dalian, Liaoning, China +3
SketchGraphNet: A Memory-Efficient Hybrid Graph Transformer for Large-Scale Sketch Corpora Recognition
Mar 8, 2026Shilong Chen, Mingyuan Li, Zhaoyang Wang +2Memory-Efficient Attention Graph Transformer
College of Computer, Qinghai Normal University, Xining, 810001, Qinghai, China · College of Computer, Nanjing University, Nanjing, 210023, Jiangsu, China · College of Computer, Qinghai Minzu University, Xining, 810007, Qinghai, China
Untied Ulysses: Memory-Efficient Context Parallelism via Headwise Chunking
Feb 24, 2026Ravi Ghadia, Maksim Abraham, Sergei Vorobyov +1Memory-Efficient Attention Context Parallelism
Together AI
2Mamba2Furious: Linear in Complexity, Competitive in Accuracy
Feb 19, 2026Gabriel Mongaras, Eric C. LarsonLinear Attention Memory-Efficient Attention
Lyle School of Engineering Southern Methodist University Dallas, TX 75205
Thin Keys, Full Values: Reducing KV Cache via Low-Dimensional Attention Selection
Feb 16, 2026Hengshuai Yao, Xing Chen, Ahmed Murtadha +1Memory-Efficient Attention KV-Cache Compression
Sapient Intelligence · Department of Computing Science, University of Alberta
NoiseFormer -- Noise Diffused Symmetric Attention Transformer
Jan 10, 2026Phani Kumar, Nyshadham, Jyothendra Varma +1Transformer Attention Attention Mechanisms
Intel Corporation

Oct 31, 2026J/K move · Enter open · S save

Fast and Exact Relative-Position Attention via Structure-Aware Kernels

Delyan Boychev, Ivo Petrov, Dimitar I. Dimitrov

INSAIT, Sofia University “St. Kliment Ohridski”.

Efficient Attention Memory-Efficient Attention Relative Positional Encoding

Open paper AskPDF arXiv

Relative-position attention mechanisms offer compelling alternatives to RoPE across various transformer architectures, delivering superior performance in domains such as audio. These mechanisms typically modify attention scores by adding content-position or bias-position terms. Unlike RoPE, which encodes position via per-token rotations directly on the queries and keys, relative-position terms generally require pairwise offset-dependent computation or lookups, complicating their integration into fused attention kernels and introducing severe memory, bandwidth, and compute overheads as sequences scale. To address this bottleneck, we propose an exact attention algorithm that incorporates additive relative-position terms into FlashAttention-style tiled computation. Our method (1) uses offset-aware tiling to avoid redundant indexing and memory accesses, (2) computes only the necessary key/query-relation products and reuses them across token pairs in both forward and backward passes, and (3) uses offset-based gradient sharding and aggregation, eliminating atomic contention. We provide a unified, efficient Triton implementation across diverse relative-position attention mechanisms and evaluate it against naive and specialized implementations. For 8K-token sequences, our method speeds up the forward (backward) pass by 1.8-6.4

\times

(2.0-21

\times

) and reduces peak memory up to 2.9-128

\times

(1.4-54

\times

) over the strongest baseline for each mechanism, enabling long-context end-to-end transcription that is 1.8

\times

faster and uses 2.4

\times

less memory. Our method thus provides a drop-in, hardware-efficient solution for scaling relative-position attention transformers.