Attention Approximation

Follow this topic Ask about this topic

Latest papers

17 papers4 papers in the last four weeks, up 33% on the four weeks before. 0.0% of all new papers.

Not All Dimensions Are Equal: Outlier-Aware Attention Linearization with Grouped Quadratic Kernels
Oct 9, 2026Weiqi Sun, Qianyue Wu, Wenjia Wang +1Attention Approximation Efficient Attention
The Chinese University of Hong Kong, Shenzhen · Beijing Normal University, Zhuhai · National University of Singapore
Sparse Attention Is Matrix Approximation, Not Choosing from a Bag of Values
Oct 7, 2026Fang Wan, Xufeng Liu, Fan Li +1Sparse Attention Efficient Attention
Stony Brook University · Duke University
Attention via Black-Box Vector Search
Oct 7, 2026Stepan Zharkov, Krish Singal, Ashwin Padaki +1Attention Approximation Efficient Attention
University of Pennsylvania
Efficiently Approximating Attention Is Hard
Sep 29, 2026Lukas Haverbeck, Carmen Amo Alonso, Andres Felipe Posada-Moreno +2Attention Approximation Efficient Attention
Institute for Data Science in Mechanical Engineering, RWTH Aachen University · Autonomous Systems Lab, Stanford University
Exact Finite Attention Responses From RoPE Derivatives
Sep 12, 2026Julie Huang, Maggie Chlon, Gregory Gutin +1Attention Approximation Causal Interventions in Language Models
Hassana Labs · Department of Computer Science, Royal Holloway, University of London · University of Oxford
ClusterAttention: A training-free speedup of bidirectional attention
Aug 27, 2026Kasper Nordenram, Amelie DittmannEfficient Attention Attention Mechanisms
Independent researcher
How Much Rank Does LoRA Need? Rank-Error Bounds for Transformer Attention
Aug 26, 2026Gerard Conangla PlanesAttention Approximation Low-Rank Adaptation
Aily Labs
SpotAttention: Plug-In Block-Sparse Routing for Pretrained Long-Context Transformers
Jun 22, 2026Huzama Ahmad, Se-Young YunSparse Attention Block-Sparse Attention
KAIST
Express Language Modeling
Jun 9, 2026Albert Gong, Annabelle Michael Carrell, Raaz Dwivedi +1Long-Context Language Model Inference Attention Mechanisms
Cornell Tech · University of Cambridge · Microsoft Research
Variational Linear Attention: Stable Associative Memory for Long-Context Transformers
May 11, 2026Vishal Pandey, Gopal SinghEfficient Attention Linear Attention
Independent Researcher London, UK · Metriqual Athens, GR
Nectar: Neural Estimation of Cached-Token Attention via Regression
May 10, 2026João Monteiro, Michal Klein, Pierre Ablin +1Attention Approximation Softmax Attention
Apple
Nearly Optimal Attention Coresets
May 7, 2026Edo Liberty, Alexandr Andoni, Eldar KleinerSoftmax Attention Efficient Attention
Transformer Approximations from ReLUs
Apr 27, 2026Jerry Yao-Chieh Hu, Mingcheng Lu, Yi-Chen Lee +1Softmax Attention Neural Network Approximation Theory
Center for Foundation Models and Generative AI, Northwestern University, Evanston, IL 60208, USA · Department of Computer Science, Northwestern University, Evanston, IL 60208, USA · Department of Physics, National Taiwan University, Taipei 10617, Taiwan +1
Hierarchical Kernel Transformer: Multi-Scale Attention with an Information-Theoretic Approximation Analysis
Apr 10, 2026Giansalvo CirrincioneAttention Approximation Transformer Attention
Laboratoire LTI, Université de Picardie Jules Verne, 33 rue Saint-Leu, Amiens, 80039, France
Tucker Attention: A generalization of approximate attention mechanisms
Mar 31, 2026Timon Klein, Jonas Kusch, Sebastian Sager +2Tucker Decomposition Attention Mechanisms
Department of Mathematics; Otto von Guericke University Magdeburg; 39106 Magdeburg; Germany. · Scientific Computing, Norwegian University of Life Sciences, ˚As, Norway · Max Planck Institute for Dynamics of Complex Technical Systems; 39106 Magdeburg; Germany. +1
Why Attend to Everything? Focus is the Key
Mar 12, 2026Hengshuai Yao, Xing Chen, Ahmed Murtadha +8Attention Mechanisms Efficient Attention
Sapient Intelligence · University of Alberta · Tsinghua University
SVG-EAR: Parameter-Free Linear Compensation for Sparse Video Generation via Error-aware Routing
Mar 9, 2026Xuanyi Zhou, Qiuyang Mang, Shuo Yang +7Video Diffusion Models Efficient Attention
UC Berkeley, USA

Oct 9, 2026J/K move · Enter open · S save

Not All Dimensions Are Equal: Outlier-Aware Attention Linearization with Grouped Quadratic Kernels

Weiqi Sun, Qianyue Wu, Wenjia Wang, Tianyang Hu

The Chinese University of Hong Kong, Shenzhen · Beijing Normal University, Zhuhai · National University of Singapore

Attention Approximation Efficient Attention Linear Attention Transformer Inference

Open paper AskPDF arXiv

Linearizing softmax attention offers a route to efficient Transformer inference, but preserving pretrained attention remains challenging. We identify an important source of approximation error overlooked by standard linearization methods: a small number of outlier dimensions contribute disproportionately to query-key dot products and account for a large fraction of the approximation error. Motivated by this heterogeneity, we propose Outlier-Aware Attention Linearization (OAL), a dimension-aware method for constructing quadratic attention approximations. Starting from a second-order Taylor approximation, we introduce a Hadamard-product formulation that makes individual query-key contributions explicit. This formulation relaxes globally shared coefficients to support dimension-specific linear terms and cross-dimensional quadratic interactions while preserving separable evaluation. To retain this flexibility with fewer parameters, OAL groups dimensions with similar contribution scales to share coefficients, while allowing distinct treatment of the few outliers. The resulting positive quadratic kernel is calibrated against pretrained attention outputs and admits linear-time evaluation with respect to sequence length for fixed head dimensions. On Qwen2.5-0.5B, OAL reduces frozen-attention output relative MSE by approximately 8.9% compared with DiJiang. Replacing middle-layer attention modules and jointly fine-tuning the kernel coefficients with LoRA further lowers WikiText-103 perplexity from 31.71 to 27.50 compared with a learned global quadratic, while improving accuracy on PIQA, HellaSwag, and ARC-E. These results demonstrate the value of outlier-aware dimension grouping for both attention approximation and pretrained-model conversion. The code is available at https://github.com/wuqianyue09/OAL