Low-Rank Attention

Ask about this topic

Overview

Low-rank attention studies how to represent the interactions between tokens with fewer dimensions, making long-context models and large attention layers cheaper to train and run. Common approaches factorize query-key or attention matrices with SVD and Tucker decompositions, route through compact latent states, or combine low-rank global mixing with sparse local attention. These methods are being tested in language-model decoding and KV-cache compression, as well as video generation, recommendation, and neural operators, where benchmarks often measure both task quality and latency or memory use. A central tension is that compact representations can preserve broad attention patterns and perplexity yet lose retrieval or task-specific detail, while the rank needed for reliable approximation can depend strongly on the data and setting.

Latest papers

CardsList

24 papers1 paper in the last four weeks, down 75% on the four weeks before. 0.0% of all new papers.

  1. SVD-Attention: Efficiently Analyzing Attention Blocks in Transformers

    Oct 26, 2026Debopam Sanyal, Hongjie Chen, Riya Elizabeth Kuriyan +5Transformer AttentionAttention Head Analysis

    Georgia Institute of Technology · Dolby Laboratories

  2. Tucker Bottleneck Attention for Multi-Dimensional Sequence Modeling

    Oct 6, 2026Ryan Solgi, Parsa Madinei, Zheng ZhangAttention MechanismsLow-Rank Attention

    University of California-Santa Barbara Santa Barbara, CA, USA

  3. Prescriptive SVD-Inspired Attention via Spectral Energy Retention

    Sep 21, 2026Vasileios Arampatzakis, Vasileios Sevetlidis, George PavlidisTransformer InterpretabilityLow-Rank Attention

    Athena Research Center, Greece

  4. Nyström Attention Matches Full Attention for Cross-Sectional Stock Prediction

    Sep 8, 2026Kunhan GuoLow-Rank AttentionFinancial Forecasting

    The University of Hong Kong

  5. RoLA: Rotary-Positioned Low-Rank Linear Attention for Efficient Diffusion Transformers

    Sep 6, 2026Zekun Zhang, Yixiang Cai, Yuxi Liu +9Rotary Positional EmbeddingsLow-Rank Attention

    Peking University, Melon Group · Alibaba Group · University of Electronic Science and Technology of China +1

  6. Quality Recovery for Quantized KV Caches via Low-Rank Attention Adaptation

    Sep 2, 2026Seifeldin AbdellatifKV-Cache QuantizationQuantization-Aware LoRA

    College of Engineering, Al Ain University

  7. Intrinsic Interaction Geometry Controls the Low-Rank Complexity of Softmax Attention

    Aug 28, 2026Yuhe Sui, Jianing Zhang, Yingzhi TangSoftmax AttentionLow-Rank Attention

    Nanyang Technological University, Singapore · Carnegie Mellon University

  8. FLARE++: Low-rank attention with dynamic attention routing

    Aug 12, 2026Vedant Puri, Yongjie Jessica Zhang, Levent Burak KaraLow-Rank AttentionPDE Surrogate Modeling

    Department of Mechanical Engineering Carnegie Mellon University Pittsburgh, Pennsylvania, USA

  9. Three Tokens Force Exponential Feature Rank in Nonnegative Kernel Attention

    Aug 11, 2026Vicente OpazoLow-Rank AttentionAttention Mechanisms

    CENIA

  10. SAKI: Score-Aware Low-Rank Key Indexing with Random-Matrix Noise Correction for KV Retrieval

    Aug 4, 2026Lin ZhangLow-Rank AttentionKV Caching

    Data Science and Platform, Cotality

  11. LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decoding

    Jul 27, 2026Junsung HwangLow-Rank AttentionKV Caching

  12. Through the Bottleneck: How Multi-head Latent Attention Separates Content from Position in Language Models

    Jul 25, 2026Dhruvil S, Fenil Sojitra, Ravirajsinh ChauhanTransformer InterpretabilityLow-Rank Attention

    1Indian Institute of Technology Madras · 2P P Savani University

  13. ELSAA: Efficient Low-Rank and Sparse Attention Approximation for Training Transformers

    Jul 22, 2026Mahdi Heidari, Mohammad Mahdi Rahimi, Jaekyun MoonLow-Rank AttentionTransformer Attention

    School of Electrical Engineering, KAIST, Daejeon 34141, Republic of Korea · Department of Electrical Engineering and Computer Science, Daegu Gyeongbuk Institute of Science and Technology (DGIST), Daegu 42988, Republic of Korea

  14. Low-Rank Attention Residuals

    Jun 19, 2026Jonathan SuLow-Rank AttentionDecoder-Only Language Models

    Independent Researcher

  15. Hierarchical Attention via Domain Decomposition

    Jun 16, 2026Stephan Köhler, Oliver RheinbachLow-Rank AttentionPDE Operator Learning

    Faculty of Mathematics and Computer Science Technische Universität Bergakademie Freiberg, 09596 Freiberg, Germany

  16. High-Fidelity Industrial Crash Dynamics Prediction via Geometry-Aware Operator Learning with Memory-Efficient Low-Rank Attention

    May 26, 2026Deepak Akhare, Mohammad Amin Nabian, Corey Adams +2Low-Rank AttentionSurrogate Modeling

    Department of Aerospace and Mechanical Engineering, University of Notre Dame, Notre Dame, IN · NVIDIA, San Jose, CA · General Motors, Detroit, MI

  17. RoPeSLR: 3D RoPE-driven Sparse-LowRank Attention for Efficient Diffusion Transformers

    May 20, 2026Yuxi Liu, Zekun Zhang, Yixiang Cai +3Rotary Positional EmbeddingsLow-Rank Attention

    Peking University · University of Electronic Science and Technology of China · Beijing University of Posts and Telecommunications

  18. Compressible Softmax-Attended Language under Incompressible Attention

    Apr 6, 2026Wonsuk LeeSoftmax AttentionAttention Head Analysis

    Seoul National University & SK Hynix America

  19. Simple yet Effective: Low-Rank Spatial Attention for Neural Operators

    Apr 4, 2026Zherui Yang, Haiyang Xin, Tao Du +1Neural OperatorsPDE Operator Learning

    University of Science and Technology of China · Tsinghua University · Shanghai Qi Zhi Institute

  20. Tucker Attention: A generalization of approximate attention mechanisms

    Mar 31, 2026Timon Klein, Jonas Kusch, Sebastian Sager +2Tucker DecompositionAttention Mechanisms

    Department of Mathematics; Otto von Guericke University Magdeburg; 39106 Magdeburg; Germany. · Scientific Computing, Norwegian University of Life Sciences, ˚As, Norway · Max Planck Institute for Dynamics of Complex Technical Systems; 39106 Magdeburg; Germany. +1

  21. SOLAR: SVD-Optimized Lifelong Attention for Recommendation

    Mar 3, 2026Chenghao Zhang, Chao Feng, Yuanhao Pu +5Recommender SystemsSequential Recommendation

    Kuaishou Technology. · Unaffiliated.

  22. Multi-Head Low-Rank Attention

    Mar 2, 2026Songtao Liu, Hongwu Peng, Zhiwei Zhang +2Long-Context Language Model InferenceLLM Inference Acceleration

    The Pennsylvania State University · University of Connecticut · Carnegie Mellon University +1

  23. Thin Keys, Full Values: Reducing KV Cache via Low-Dimensional Attention Selection

    Feb 16, 2026Hengshuai Yao, Xing Chen, Ahmed Murtadha +1Memory-Efficient AttentionKV-Cache Compression

    Sapient Intelligence · Department of Computing Science, University of Alberta

  24. LIME: Link-based User-item Interaction Modeling with Decoupled XOR Attention for Efficient Test Time Scaling

    Oct 21, 2025Yang Liu, Yunjiang Jiang, Ayush Agarwal +3Recommender SystemsEfficient Inference

    Meta AI