Multi-Head Latent Attention

Ask about this topic

Overview

Multi-Head Latent Attention (MLA) replaces per-head key-value caches with a compact shared latent representation, aiming to reduce memory traffic and support longer contexts or larger batches. Work explores low-rank factorization and MHA/GQA-to-MLA conversion, alongside quantized caches, sparse attention, and serving kernels for tensor-parallel and long-context inference. Methods are evaluated on language-model accuracy, long-context retrieval and reasoning benchmarks such as LongBench and RULER, and serving measures such as throughput, latency, and cache use. The central tension is gaining memory and speed without harming attention quality: positional encoding, normalization, rank allocation, and hardware-specific data movement all shape how well compression works in practice.

Latest papers

CardsList

19 papers4 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

  1. RAQ-MLA: Role-Aware Quantization for Multi-Head Latent Attention Caches

    Oct 24, 2026Tianqi Chen, Xunchao Li, Linchuan Xie +4LLM QuantizationAttention Quantization

    C2DL, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Tencent

  2. MLAfication: Efficiently Enabling High-Fidelity MLA for Modern LLMs

    Oct 12, 2026Zile Wang, Zhide Xie, Tao JiMulti-Head Latent AttentionLLM Inference Acceleration

    Fudan University

  3. LatentIndex: Cross-Layer Sharing with Layer-Specific Selection for Sparse Attention

    Oct 3, 2026Zhaohui Wang, Zhixin Pan, Fanxu Meng +1Cross-Layer AttentionLLM Inference Acceleration

    Institute for Artificial Intelligence, Peking University · Dots Studio, Xiaohongshu Inc. · Beijing Institute of Technology

  4. QuantMLA: Function-Aligned Dual-Path Quantization for Low-Bit MLA KV Caching

    Sep 29, 2026Zunhai Su, Yuxuan Sun, Jianchao Tan +5LLM Inference AccelerationAttention Quantization

    The University of Hong Kong · Meituan LongCat Team · South China University of Technology +1

  5. RedKnot-MLA: Multi-Head Offline-Online Reuse for DeepSeek-V4 Long-Context Serving

    Sep 7, 2026Yang Liu, Zhaokai Luo, Huayi Jin +10KV CachingLLM Inference Acceleration

    Xiaohongshu Inc., China · Huawei Cloud · Shanghai Jiao Tong University +2

  6. VestigeKV: The NoPE-MLA KV Cache Carries Its Own Sparse-Attention Signal in a Vestigial Branch

    Sep 3, 2026WenJie FanKV CachingKV-Cache Management

    Yotta Labs

  7. An Exploratory Ablation of a Small MLA--SSM Hybrid Language Model

    Aug 28, 2026Christos KoutsiarisHybrid Language ModelsDecoder-Only Language Models

    Development Expert, SAP P&E, Cloud ERP, UX Foundation

  8. Beyond KV Reconstruction: Functional Reconstruction for MLA Draft Models in Speculative Decoding

    Jul 29, 2026Weiye Shi, Fanxu Meng, Muhan ZhangSpeculative DecodingLLM Inference Acceleration

    Institute for Artificial Intelligence, Peking University

  9. Through the Bottleneck: How Multi-head Latent Attention Separates Content from Position in Language Models

    Jul 25, 2026Dhruvil S, Fenil Sojitra, Ravirajsinh ChauhanTransformer InterpretabilityLow-Rank Attention

    1Indian Institute of Technology Madras · 2P P Savani University

  10. QK-Normed MLA: QK normalization without full key caching

    Jun 15, 2026Yizhou Han, Yao Zhao, Jun Zhou +2KV CachingTransformer Attention

    School of Data Science, The Chinese University of Hong Kong, Shenzhen.

  11. Move the Query, Not the Cache: Characterizing Cross-Instance Latent Attention Redistribution Across GPU Fabrics

    May 31, 2026Bole Ma, Jan Eitzinger, Harald Köstler +1LLM InferenceKV Caching

  12. VideoMLA: Low-Rank Latent KV Cache for Minute-Scale Autoregressive Video Diffusion

    May 28, 2026Hidir Yesiltepe, Jiazhen Hu, Tuna Han Salih Meral +4Video Diffusion ModelsSelf-Attention

    1Virginia Tech · 2fal

  13. GQLA: Group-Query Latent Attention for Hardware-Adaptive Decoding

    May 14, 2026Fanxu Meng, Zhixin PanGrouped-Query AttentionGPU Acceleration

    Institute for Artificial Intelligence, Peking University

  14. Irminsul: MLA-Native Position-Independent Caching for Agentic LLM Serving

    May 7, 2026Bole Ma, Jan Eitzinger, Harald KöstlerKV CachingLLM Inference Serving

    Erlangen National High Performance Computing Center · Erlangen, Germany

  15. Latent-Condensed Transformer for Efficient Long Context Modeling

    Apr 14, 2026Zeng You, Yaofo Chen, Qiuwu Chen +5Long-Context Language ModelingLLM Inference Acceleration

    South China University of Technology · Pengcheng Laboratory · AIGCode +2

  16. Tucker Attention: A generalization of approximate attention mechanisms

    Mar 31, 2026Timon Klein, Jonas Kusch, Sebastian Sager +2Tucker DecompositionAttention Mechanisms

    Department of Mathematics; Otto von Guericke University Magdeburg; 39106 Magdeburg; Germany. · Scientific Computing, Norwegian University of Life Sciences, ˚As, Norway · Max Planck Institute for Dynamics of Complex Technical Systems; 39106 Magdeburg; Germany. +1

  17. CARE: Covariance-Aware and Rank-Enhanced Decomposition for Enabling Multi-Head Latent Attention

    Mar 18, 2026Zhongzhu Zhou, Fengxiang Bie, Ziyan Chen +6Multi-Head Latent AttentionLow-Rank Approximation

    University of Sydney · Together AI · University of Texas at Austin +1

  18. Multi-Head Low-Rank Attention

    Mar 2, 2026Songtao Liu, Hongwu Peng, Zhiwei Zhang +2Long-Context Language Model InferenceLLM Inference Acceleration

    The Pennsylvania State University · University of Connecticut · Carnegie Mellon University +1

  19. Whisper-MLA: Reducing GPU Memory Consumption of ASR Models based on MHA2MLA Conversion

    Feb 28, 2026Sen Zhang, Jianguo Wei, Wenhuan Lu +6Automatic Speech RecognitionSpeech Language Models

    College of Intelligence and Computing, Tianjin University, Tianjin, China · Banma Network Technology Co., Ltd.