Grouped-Query Attention

Also known as GQA

Grouped-query attention studies how multiple query heads can share fewer key and value heads, trading some representational independence for smaller KV caches and less memory traffic during decoding. Work explores architectural variants such as latent, grouped-value, and virtual-head attention, alongside cache compression, sparse attention, hardware-aware kernels, and scaling rules. Researchers compare these methods with multi-head and multi-query attention using language-model accuracy, long-context retrieval, latency, and memory or energy use. The central tension is whether reduced cache and compute costs can be achieved without sacrificing quality, and how reliably gains transfer across model sizes, workloads, and hardware.

Latest papers

14 papers2 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Virtual Head Attention
Oct 13, 2026Xibo Ding, Guoxia Wang, Jinle Zeng +4Long-Context Language Modeling Grouped-Query Attention
Baidu Inc. · Peking University
CHASE: Channel-Aligned Structure Exploitation for Geometry-Aware Model Engineering
Oct 7, 2026Wei Wang, Wei Jiang, Ziran LiuNeural Network Compression KV-Cache Compression
Futurewei Technologies San Jose, CA 95131, USA · Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS) Shanghai, 200433, China
Grouped Value Attention: Efficient KV Caching via On-Demand Key Reconstruction
Sep 8, 2026Vishesh Tripathi, Abhay Kumar, Ramsha KhanEfficient Transformer Inference Grouped-Query Attention
FrontiersMind
Understanding the Energy Scaling of Large Language Model Inference Across Context Lengths and Attention Architectures
Aug 25, 2026Molka Chkir, Syed Muhammad Danish, Jos Höll +1Energy-Aware LLM Inference LLM Inference Efficiency
Algoma University, Brampton, Canada · School of Informatics, Reutlingen University, Reutlingen
MoA-Structured Decode Attention DNF Derivation, KV-Cache Accumulation, GQA/MQA, and OpenACC Kernel
Jul 21, 2026Lenore Mulin, Gaetan HainsAttention Mechanisms Efficient Transformer Inference
University at Albany, SUNY · LACL, Universit´e Paris-Est Cr´eteil
Grouped Query Experts: Mixture-of-Experts on GQA Self-Attention
Jun 18, 2026Vishesh Tripathi, Abhay KumarGrouped-Query Attention Transformer Attention
FrontiersMind
How Much Dense Attention is Necessary? Oracle-Guided Sparse Prefill for Full/GQA Layers in Hybrid Long-Context Models
Jun 5, 2026Hongxing Wang, Harenome Razanajato, Zhen Zhang +2Grouped-Query Attention Long-Context Language Modeling
GQA-μP: The maximal parameterization update for grouped query attention
May 14, 2026Kyle R. Chickering, Huijuan Wang, Mengxi Wu +7Grouped-Query Attention Representation Learning
UC Davis & MBZUAI IFM · USC & MBZUAI IFM · USC +4
GQLA: Group-Query Latent Attention for Hardware-Adaptive Decoding
May 14, 2026Fanxu Meng, Zhixin PanGrouped-Query Attention GPU Acceleration
Institute for Artificial Intelligence, Peking University
Architecture, Not Scale: Circuit Localization in Large Language Models
May 9, 2026Sohan VenkateshTransformer Interpretability Grouped-Query Attention
Manipal Institute of Technology Bengaluru.
Attention-Weighted Value Projection for KV-Cache Compression
Apr 13, 2026Samuel SalfatiGrouped-Query Attention KV Caching
fraQtl AI Research
Sensitivity-Positional Co-Localization in GQA Transformers
Apr 9, 2026Manoj Chandrashekar RaoTransformer Rotary Positional Embeddings
Independent Researcher
Tucker Attention: A generalization of approximate attention mechanisms
Mar 31, 2026Timon Klein, Jonas Kusch, Sebastian Sager +2Tucker Decomposition Attention Mechanisms
Department of Mathematics; Otto von Guericke University Magdeburg; 39106 Magdeburg; Germany. · Scientific Computing, Norwegian University of Life Sciences, ˚As, Norway · Max Planck Institute for Dynamics of Complex Technical Systems; 39106 Magdeburg; Germany. +1
When Does Sparsity Mitigate the Curse of Depth in LLMs
Mar 16, 2026Dilxat Muhtar, Xinyuan Song, Sebastian Pokutta +4Grouped-Query Attention Language Model Scaling
Max Planck Institute for Intelligent Systems · ELLIS Institute Tübingen · Tübingen AI Center +4

Oct 13, 2026J/K move · Enter open · S save

Virtual Head Attention

Xibo Ding, Guoxia Wang, Jinle Zeng, Jiabin Yang+3

Baidu Inc. · Peking University

Long-Context Language Modeling Grouped-Query Attention LLM Inference Acceleration KV-Cache Compression

Open paper AskPDF arXiv

The design of attention mechanisms in long-context large language models faces an ``impossible triangle'' among KV cache size, training and prefill FLOPs, and model quality. GQA achieves strong quality but incurs high KV cache overhead; alternative approaches such as MLA and MFA can compress the cache, but MLA suffers from quality loss while MFA substantially increases training and prefill compute. We propose Virtual Head Attention (VHA), which simultaneously improves all three vertices through two lightweight linear operations: Q Premix applies a near-identity transformation to queries within each KV group to recover virtual head diversity from halved physical query heads, and Linear Postmix fuses inter-head features via a low-rank

I + AB^\top

residual structure. Since VHA introduces no nonlinear operations, Premix can be folded into the query projection and Postmix into the output projection at inference time, reducing the model to a standard GQA-2 architecture that directly reuses all existing GQA inference optimizations. With only two KV heads, the group size in large models readily reaches 64, fully exploiting Tensor Core parallelism during decoding. Across four scales with 4K-context 50B-token pretraining, VHA consistently surpasses the standard GQA baseline at every scale (

+

0.05% to

+

0.81%), while reducing KV cache by

4\times

vs.\ GQA-8 at dense scales and

2\times

vs.\ GQA-4 at the MoE scale, and lowering training and prefill FLOPs by 7.9% (1.7B,

S{=}4096

). On an industrial 30B-A3B MoE model trained on 500B tokens at 8K context, VHA outperforms both MLA and GQA-4 by

+

0.70,/,

+

2.60,% on a 12-benchmark base-model evaluation suite; in a separate matched 10B-A1B MoE comparison after long-context mid-training, VHA attains an average RULER score of 84.4 over 8K--128K versus 73.0 for GQA, while using two rather than four KV heads in the full-attention layers.