Do Transformers Need Three Projections? Systematic Study of QKV Variants

Ali Kayyam; Anusha Madan Gopal; M Anthony Lewis

cs.LGJun 1, 2026

Do Transformers Need Three Projections? Systematic Study of QKV Variants

Authors: Ali Kayyam, Anusha Madan Gopal, M Anthony Lewis

Organizations: BrainChip Inc., Laguna Hills, CA, USA.

Abstract

Transformers have become the standard solution for various AI tasks, with the query, key, and value (QKV) attention formulation playing a central role. However, the individual contribution of these three projections and the impact of omitting some remain poorly understood. We systematically evaluate three projection sharing constraints: a) Q-K=V (shared key-value), b) Q=K-V (shared query-key), and c) Q=K=V (single projection). The last two variants produce symmetric attention maps; to address this, we also explore asymmetric attention via 2D positional encodings. Through experiments spanning synthetic tasks, vision (MNIST, CIFAR, TinyImageNet, anomaly), and language modeling (300M and 1.2B parameter models on 10B tokens), we discovered that our transformers perform on par or occasionally better than the QKV transformer. In language modeling, Q-K=V projection sharing achieves 50% KV cache reduction with only 3.1% perplexity degradation. Crucially, projection sharing is complementary to head sharing (GQA/MQA): combining Q-K=V with GQA-4 yields 87.5% cache reduction, while Q-K=V + MQA achieves 96.9%, enabling practical on-device inference. We show that Q-K=V preserves quality because keys and values can occupy similar representational spaces and attention operates in a low-rank regime, whereas Q=K-V breaks attention directionality. Our results systematically characterize projection sharing as an underexplored instance of weight tying in attention, with direct, quantifiable inference memory benefits, particularly valuable for edge deployment. The code is publicly available at https://github.com/Brainchip-Inc/Do-Transformers-Need-3-Projections

Explore similar work

Jun 20, 2026cs.CL

Keyless Attention: Value-Space Routing and Value-Only Caching for Efficient Transformers

We propose Keyless Attention, an attention mechanism that eliminates the key projection entirely, operating over queries and values only. This yields a Value-Only Cache that reduces KV cache memory and access overhead by exactly 50% over standard attention, while matching or exceeding standard attention's decode throughput. Beyond efficiency, we introduce Depth-

m

Attention Factorization: standard attention computes a depth-2 factorization of the attention bilinear form, while Keyless Attention realizes a depth-

m

instance of this family. At m=3, Keyless Attention matches the projection matrix count of standard attention via a value-space routing matrix that replaces the key projection and introduces a coupling between routing and retrieval. Experiments across five models and four architectures (GPT-2 280M, GPT-2 557M, Pythia 410M, Qwen2 1.5B, and Llama 3.2 1B) show that Keyless Attention matches or outperforms standard QKV attention on perplexity in 4 out of 5 models. On downstream zero-shot evaluation (GPT-2 557M), Keyless Attention outperforms on 4 out of 5 commonsense reasoning benchmarks, while achieving 50% KV cache reduction throughout.

Xin Gao

Quadratic Attention Transformer Architectures Reasoning Benchmark

Ask PDF

Jul 21, 2026cs.LG

MoA-Structured Decode Attention DNF Derivation, KV-Cache Accumulation, GQA/MQA, and OpenACC Kernel

We derive four memory-optimal inference artifacts for transformer attention using the Mathematics of Arrays (MoA), each following directly from the forward-pass Denotational Normal Form (DNF) of with the query-row index fixed to the current decode step. The artifacts are: (1)~a single-query decode DNF in which the

ψ

-reduction eliminates the

K^\top

buffer algebraically, achieving

(d_k + nd_k+ nd_v+ d_v)\times4\,{B}

Dynamic Random Access Memory (DRAM) traffic result numerically verified to

\|{err}\|_\leq2\times10^{-7}

; (2)~a C/OpenACC Graphics Processing Unit (GPU) kernel with Operational Normal Form (ONF) stride arithmetic and hardware-coalesced memory access, verified to

\|\mathrm{err}\|_\infty=0

(exact IEEE-754 floating-point arithmetic); (3)~a multi-step KV-cache with

O(d_k+d_v)

per-step append via MoA concatenation

\#

; and (4)~Grouped-Query Attention (GQA) and Multi-Query Attention (MQA) derived via

ψ

-selection, achieving a proven

\frac {h_q} { h_{kv} }

reduction in KV traffic. All programs are verified against PyTorch scaled_dot_product_attention.

Lenore Mulin, Gaetan Hains

Quadratic Attention Transformer Attention Key-Value Cache Quantization Graphics Processing Unit Kernels Vram

Ask PDF

Apr 13, 2026cs.LG

Attention-Weighted Value Projection for KV-Cache Compression

Rank reduction discards dimensions; quantization keeps them at lower precision. Comparing the two requires a choice of what compression should preserve. For attention values, we study reconstruction of the attention output rather than reconstruction of the values alone. With fixed attention weights, the optimal orthogonal rank-

r

projection uses the leading eigenvectors of

V^\topα^\topαV

, and its error is exactly the discarded eigenvalue sum. We extend this objective to calibration datasets and grouped query attention, and describe rank allocation under an additive local error budget. We also examine the limits of using local error to predict downstream loss. Historical weight-perturbation experiments favor coefficient rounding over the value projections tested, but do not establish a comparison at equal cache storage. The resulting distinction is practical: the projection objective has an exact solution, while a comparison with cache quantization requires separate activation-level experiments.

Samuel Salfati

Key-Value Cache Quantization Extreme Low-Bit Quantization Quadratic Attention

Ask PDF