Memory-Efficient Attention

Latest papers

24 papers1 paper in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

SANTA++: Sampling Attention through Representative Keys
Sep 28, 2026Kyle Lee, Christian Z. Pratt, Ruoyu Fang +4Memory-Efficient Inference LLM Inference Acceleration
Department of Electrical and Computer Engineering University of California, Santa Barbara, CA, USA · Flucta, San Francisco, CA, USA
Sliding-window beats linear attention
Aug 28, 2026Alexia Jolicoeur-Martineau, Rhea Sanjay Sukthanker, Pashmina Cameron +1Linear Attention KV Caching
Microsoft Applied Sciences Group (ASG) · Independent
Q-Interference: Memory-Efficient Phase-Aware Quantum-Inspired Attention
Aug 18, 2026Emama Nahid, Tahmid Imtiaz Imu, Huayue Gu +3Memory-Efficient Attention Transformer Attention
Kennesaw State University Marietta, GA, USA · Miami University Oxford, OH, USA · Georgia State University Atlanta, GA, USA
Efficient Reinforcement Learning for Long-Horizon Tool-Use Agentic Tasks
Aug 11, 2026Zelei Cheng, Amritansh Mishra, Sambit Sahu +1Memory-Efficient Attention Efficient Attention
AI Foundations, Capital One
MoA-Structured Decode Attention DNF Derivation, KV-Cache Accumulation, GQA/MQA, and OpenACC Kernel
Jul 21, 2026Lenore Mulin, Gaetan HainsAttention Mechanisms Efficient Transformer Inference
University at Albany, SUNY · LACL, Universit´e Paris-Est Cr´eteil
A Hippocampus for Linear Attention: An Exact Memory for What the Recurrent State Forgets
Jul 2, 2026Wanyun CuiHybrid Attention Long-Context Retrieval
Shanghai University of Finance and Economics
Scalable Peptide Design via Memory-Efficient Equivariant Transformer
Jun 23, 2026Rui Jiao, Xiangzhe Kong, Yinjun Jia +4Memory-Efficient Attention Equivariant Neural Networks
Department of Computer Science and Technology, Tsinghua University · Institute for AI Industry Research, Tsinghua University · Department of Electronic Engineering, Tsinghua University +1
SpotAttention: Plug-In Block-Sparse Routing for Pretrained Long-Context Transformers
Jun 22, 2026Huzama Ahmad, Se-Young YunSparse Attention Block-Sparse Attention
KAIST
Keyless Attention: Value-Space Routing and Value-Only Caching for Efficient Transformers
Jun 20, 2026Xin GaoTransformer Inference KV Caching
Department of Mathematics and Statistics York University Toronto, M3J 1P3
Express Language Modeling
Jun 9, 2026Albert Gong, Annabelle Michael Carrell, Raaz Dwivedi +1Long-Context Language Model Inference Attention Mechanisms
Cornell Tech · University of Cambridge · Microsoft Research
Attention at the Theoretical Minimum: A Mathematics of Arrays Framework for Memory-Optimal Transformer Kernels
Jun 5, 2026Lenore Mullin, Gaetan HainsMemory-Efficient Attention GPU Kernel Optimization
University at Albany, SUNY, Albany, NY, USA · LACL, Université Paris-Est Créteil, Créteil, France
Augmenting Attention with Exponentially Decaying Memory Improves Query-Aware KV Sparsity
May 27, 2026Xiuying Wei, Caglar GulcehreKV Caching Memory-Augmented Language Models
EPFL, CLAIRE Lausanne, Switzerland
Approaching I/O-optimality for Approximate Attention
May 22, 2026Pál András Papp, Aleksandros Sobczyk, Anastasios ZouziasSelf-Attention Efficient Attention
Computing Systems Lab, Huawei Technologies, Zurich, Switzerland
Variational Linear Attention: Stable Associative Memory for Long-Context Transformers
May 11, 2026Vishal Pandey, Gopal SinghEfficient Attention Linear Attention
Independent Researcher London, UK · Metriqual Athens, GR
Nectar: Neural Estimation of Cached-Token Attention via Regression
May 10, 2026João Monteiro, Michal Klein, Pierre Ablin +1Attention Approximation Softmax Attention
Apple
Attention Itself Could Retrieve.RetrieveVGGT: Training-Free Long Context Streaming 3D Reconstruction via Query-Key Similarity Retrieval
May 10, 2026Zichen Zou, Xiaosong Jia, Zuxuan Wu +1Memory-Efficient Attention 3D ViTs
Institute of Trustworthy Embodied AI (TEAI), Fudan University · 2Shanghai Key Laboratory of Multimodal Embodied AI
Nearly Optimal Attention Coresets
May 7, 2026Edo Liberty, Alexandr Andoni, Eldar KleinerSoftmax Attention Efficient Attention
Tucker Attention: A generalization of approximate attention mechanisms
Mar 31, 2026Timon Klein, Jonas Kusch, Sebastian Sager +2Tucker Decomposition Attention Mechanisms
Department of Mathematics; Otto von Guericke University Magdeburg; 39106 Magdeburg; Germany. · Scientific Computing, Norwegian University of Life Sciences, ˚As, Norway · Max Planck Institute for Dynamics of Complex Technical Systems; 39106 Magdeburg; Germany. +1
MKA: Memory-Keyed Attention for Efficient Long-Context Reasoning
Mar 21, 2026Dong Liu, Yanxuan Yu, Ben Lengerich +1Attention Mechanisms Memory-Efficient Attention
University of California, Los Angeles, Los Angeles, California, USA · Columbia University, New York, New York, USA · University of Wisconsin-Madison, Madison, Wisconsin, USA
LinearARD: Linear-Memory Attention Distillation for RoPE Restoration
Mar 9, 2026Ning Yang, Hengyu Zhong, Wentao Wang +3Attention-Guided Distillation Memory-Efficient Attention
Institute of Automation, Chinese Academy of Sciences, Beijing, China · Southwest University, Chongqing, China · Dalian University of Technology, Dalian, Liaoning, China +3
SketchGraphNet: A Memory-Efficient Hybrid Graph Transformer for Large-Scale Sketch Corpora Recognition
Mar 8, 2026Shilong Chen, Mingyuan Li, Zhaoyang Wang +2Memory-Efficient Attention Graph Transformer
College of Computer, Qinghai Normal University, Xining, 810001, Qinghai, China · College of Computer, Nanjing University, Nanjing, 210023, Jiangsu, China · College of Computer, Qinghai Minzu University, Xining, 810007, Qinghai, China
Untied Ulysses: Memory-Efficient Context Parallelism via Headwise Chunking
Feb 24, 2026Ravi Ghadia, Maksim Abraham, Sergei Vorobyov +1Memory-Efficient Attention Context Parallelism
Together AI
2Mamba2Furious: Linear in Complexity, Competitive in Accuracy
Feb 19, 2026Gabriel Mongaras, Eric C. LarsonLinear Attention Memory-Efficient Attention
Lyle School of Engineering Southern Methodist University Dallas, TX 75205
Thin Keys, Full Values: Reducing KV Cache via Low-Dimensional Attention Selection
Feb 16, 2026Hengshuai Yao, Xing Chen, Ahmed Murtadha +1Memory-Efficient Attention KV-Cache Compression
Sapient Intelligence · Department of Computing Science, University of Alberta

Sep 28, 2026J/K move · Enter open · S save

SANTA++: Sampling Attention through Representative Keys

Kyle Lee, Christian Z. Pratt, Ruoyu Fang, Heekyung Lee+3

Department of Electrical and Computer Engineering University of California, Santa Barbara, CA, USA · Flucta, San Francisco, CA, USA

Memory-Efficient Inference LLM Inference Acceleration Efficient Attention Memory-Efficient Attention

Open paper AskPDF arXiv

Attention often concentrates on a small subset of tokens in the context, but which subset matters changes from one query to the next. To exploit this changing structure, we introduce SANTA++, a training-free stochastic attention method that uses representative keys for memory-efficient selection without scanning the entire key-value (KV) cache. Cached keys are organized into teams, and the query scores one representative from each team to decide which teams to sample. We compute exact attention scores within the sampled teams and reweight each team's contribution by the inverse of its inclusion probability. This importance sampling correction estimates attention over the full cache, with a sampling budget that lets us trade memory reads for accuracy. Remarkably, with 32 or 64 sampled teams, SANTA++ uses 16% to 22% of dense attention's KV reads and retains 94% to 99% of the dense-attention baseline's scores on LongBench v2 and HELMET's retrieval-augmented generation subset, and 85% to 91% on RULER, with Qwen2.5-7B-Instruct at 32K context. With 31 sampled teams, our GPU implementation delivers a

1.69\times

attention speedup over the dense FlashAttention baseline at 32K context. By reducing the number of cache entries read, SANTA++ in principle complements architectures with compressed KV representations, such as multi-head latent attention. Our kernels are available at: https://github.com/OPUSLab/santapp-kernel-demo.git.