Thin Keys, Full Values: Reducing KV Cache via Low-Dimensional Attention Selection

Hengshuai Yao; Xing Chen; Ahmed Murtadha; Guan Wang

cs.LGFeb 16, 2026

Thin Keys, Full Values: Reducing KV Cache via Low-Dimensional Attention Selection

Authors: Hengshuai Yao, Xing Chen, Ahmed Murtadha, Guan Wang

Organizations: Sapient Intelligence · Department of Computing Science, University of Alberta

Abstract

Standard Transformer attention uses identical dimensionality for queries, keys, and values, yet these components serve different roles: queries and keys produce scalar attention weights (selection), while values carry rich representations (value transfer). We show that selection requires only $O(\log N)$ dimensions to distinguish among $N$ relevant token categories (e.g., syntactic roles, semantic clusters, positional patterns) -- far fewer than value transfer needs. We introduce factored keys, which exploit this asymmetry to physically shrink the KV cache of any pretrained model without retraining from scratch -- unlike Grouped-Query Attention (GQA) and Multi-Head Latent Attention (MLA), which must be designed into the architecture before pretraining. We factorize each key projection $W_K \approx A_{d \times r} B_{r \times d}$ via truncated singular value decomposition (SVD) (where $r$ is the chosen compression dimension), set $W_K' = A$ as the new key projection producing compact $r$ -dimensional keys for the cache, and absorb $B^\top$ into the query projection ( $W_Q' = W_Q B^\top$ ) at zero cost -- since queries are never cached. At the 7B scale, training from scratch with $r = d/4$ (where $d$ is the model dimension) matches full-attention perplexity ( $9.24$ vs $9.25$ PPL after 20B tokens, mean over two seeds) while using 12% fewer parameters and training 8% faster. For existing models, SVD followed by QK fine-tuning (3 epochs, less than 1% of pretraining data) achieves 75% key cache savings at roughly 2% quality cost on both GPT-2 and Mistral-7B. The approach composes with GQA and quantization for up to $16\times$ combined key cache compression. For a 7B model serving a 128K context, factored keys save 25 GB of KV cache per user, enabling roughly 60% more concurrent users on identical hardware.

Figures & tables

Rank $r$	$r$ /head	Both $Q{+}K$	$K$ -only	$Q$ -only
128	10	67,132	57.37 (+130%)	149.04 (+498%)
192	16	90.95 (+265%)	31.32 (+26%)	70.05 (+181%)
256	21	46.15 (+85%)	27.49 (+10%)	40.42 (+62%)
384	32	—	25.40 (+2.0%)	27.58 (+11%)
512	42	25.69 (+3.1%)	25.23 (+1.3%)	25.47 (+2.2%)

Table 1: SVD compression of GPT-2 projections. Baseline PPL: 24.91.

\Delta

is relative PPL increase.

Rank $r$	Before FT	After FT	Control	vs Control	K cache saved
768 (none)	29.51	19.07	—	baseline	0%
384 ( $d_{\text{model}}/2$ )	30.15 (+2.2%)	19.14	19.07	+0.4%	50%
256 ( $d_{\text{model}}/3$ )	32.61 (+10.5%)	19.28	19.07	+1.1%	67%
192 ( $d_{\text{model}}/4$ )	37.64 (+27.6%)	19.42	19.07	+1.8%	75%
128 ( $d_{\text{model}}/6$ )	67.26 (+128%)	19.77	19.07	+3.7%	83%

Table 2: SVD compression + QK fine-tuning on WikiText-103. The “vs control” column measures the residual gap relative to the uncompressed model after both receive identical fine-tuning.

	Full Attention	Thin Keys ( $d_{\text{select}}=d_{\text{model}}/4$ )
Parameters	6.74B	5.93B ( $-$ 12%)
OWT val PPL	$13.82\pm 0.09$	$\mathbf{13.12\pm 0.04}$ ( $-$ 5.1%)
WT103 val PPL	$20.79\pm 0.35$	$\mathbf{19.60\pm 0.46}$ ( $-$ 5.7%)
Wall-clock time	25.9h	23.4h ( $-$ 9.4%)

Table 3: 7B LLaMA trained from scratch on OpenWebText (2B tokens), mean

\pm

std over 2 seeds.

Step Full Thin 5K 26.62 25.05 10K 20.81 19.21 15K 17.68 16.51 20K 15.66 14.74 25K 14.37 13.59 30K 13.79 13.12

Step Full Thin 5K 44.23 41.34 10K 31.99 29.20 15K 26.43 25.36 20K 24.20 22.73 25K 22.05 20.83 30K 21.19 19.96

Figure 1: Training trajectory for 7B from-scratch models (seed 137). Left: OWT validation PPL. Right: WikiText-103 validation PPL. Thin keys lead throughout.

	Full Attention	Thin Keys ( $d_{\text{select}}=d_{\text{model}}/4$ )
Parameters	6.74B	5.93B ( $-$ 12%)
OWT val PPL	$9.25\pm 0.00$	$\mathbf{9.24\pm 0.00}$ ( $-$ 0.1%)
WT103 val PPL	$13.26\pm 0.12$	$\mathbf{13.00\pm 0.07}$ ( $-$ 2.0%)
Wall-clock time	$261.5\pm 0.1$ h	$240.7\pm 0.9$ h ( $-$ 8.0%)

Table 4: 7B LLaMA on OpenWebText (20B tokens), mean

\pm

std over 2 seeds. Both architectures converge to the same quality; thin keys train 8% faster.

Figure 2: Training trajectory for 7B models over 305K steps (20B tokens). Left: PPL vs step; thin keys lead early but converge by

\sim

150K steps. Right: PPL vs wall-clock time; thin keys reach any target PPL

\sim

20 hours sooner.

Task	Full Attention	Thin Keys	$\Delta$
Hellaswag (acc_norm)	55.6	55.7	$+$ 0.1
ARC-Challenge (acc_norm)	29.8	29.4	$-$ 0.4
WinoGrande (acc)	60.6	59.1	$-$ 1.5
MMLU (acc)	23.4	23.3	$-$ 0.1
GSM8K (exact_match)	0.0	0.0	0.0

Table 5: Downstream evaluation of 7B from-scratch models (20B tokens, seed 137). Scores are low in absolute terms (OWT-only training) but thin keys matches full attention on all tasks.

Method	K cache (GB)	V cache (GB)	KV total (GB)	KV saved
MHA (baseline)	32.0	32.0	64.0	—
Thin keys ( $d_{\text{select}}=d_{\text{model}}/4$ )	8.0	32.0	40.0	37.5%
GQA-8	8.0	8.0	16.0	75.0%
MLA ( $d_{c}\!=\!512$ , $d_{h}^{R}\!=\!64$ )	4.5 (joint)		4.5	93.0%
GQA-8 + thin keys	2.0	8.0	10.0	84.4%

Table 6: Analytical KV cache comparison at LLaMA-7B scale (128K context, bf16). MLA stores a shared latent

d_{c}

plus a decoupled RoPE key (

d_{h}^{R}

). Thin keys compose with GQA by reducing

d_{\text{head}}^{QK}

independently.

Rank $r$	Before FT	After FT	Control	vs Control	K cache saved
1024 (none)	6.69	5.91	—	baseline	0%
512 ( $d_{K}/2$ )	6.81 (+1.7%)	5.93	5.91	+0.3%	50%
256 ( $d_{K}/4$ )	7.84 (+17.1%)	6.03	5.91	+2.0%	75%
128 ( $d_{K}/8$ )	12.34 (+84.4%)	6.10	5.91	+3.2%	88%

Table 7: Mistral-7B: SVD compression + QK fine-tuning on WikiText-103.

Task	Metric	Baseline	r512 +FT	r256 +FT	Ctrl+FT	$\Delta_{512}$	$\Delta_{256}$
Hellaswag	acc_norm	81.2	81.4	80.7	81.3	$+$ 0.1	$-$ 0.7
ARC-Challenge	acc_norm	54.0	54.1	53.4	54.4	$-$ 0.5	$-$ 1.7
WinoGrande	acc	75.4	73.2	72.1	73.2	$+$ 0.0	$-$ 1.6
MMLU	acc	60.1	55.2	54.4	55.7	$-$ 0.9	$-$ 2.3
GSM8K	exact_match	38.4	27.7	25.8	29.9	$-$ 7.4	$-$ 13.7

Table 8: Downstream evaluation of SVD-compressed Mistral-7B. “Ctrl+FT” is the uncompressed model with identical fine-tuning.

Benchmark	Metric	Control	r512	r256	$\Delta_{512}$	$\Delta_{256}$
GSM8K	exact_match	53.7	52.1	51.2	$-$ 1.6	$-$ 2.5
Minerva Algebra	math_verify	14.2	14.1	12.4	$-$ 0.2	$-$ 1.9
Minerva Pre-Algebra	math_verify	21.0	19.1	19.1	$-$ 2.0	$-$ 2.0
AGIEVAL AQuA-RAT	acc	15.7	17.3	17.3	$+$ 1.6	$+$ 1.6

Table 9: Math generalization of GSM8K-fine-tuned models on held-out benchmarks.

128K context
	Standard	$d_{\text{select}}=d_{\text{model}}/2$	$d_{\text{select}}=d_{\text{model}}/4$
		(SVD, no retrain)	(train or SVD+FT)
K cache	33.6 GB	16.8 GB	8.4 GB
V cache	33.6 GB	33.6 GB	33.6 GB
Total	67.2 GB	50.4 GB	42.0 GB
Savings/user	—	16.8 GB (25%)	25.2 GB (37.5%)

Table 10: KV cache memory per user (

d_{\text{model}}=4096

, 32 layers, fp16).

Measured throughput (tokens/s)
	Batch size
	1	4	8	16	32
Baseline ( $d_{k}\!=\!128$ )	49.5	124.0	160.6	181.5	194.7
r512 ( $d_{k}\!=\!64$ )	53.5	142.9	192.8	223.2	243.0
r256 ( $d_{k}\!=\!32$ )	56.1	155.7	216.4	254.3	280.2
Measured speedup

Table 11: Decode throughput for Mistral-7B with factored keys on H100 SXM (context 4096, 128 generated tokens). “Predicted” is the bandwidth roofline from Eq. 10 .

$d_{\text{select}}$	$d_{\text{select}}$ /head	Best Accuracy	Converge Epoch
4	1	100.0%	300
8	2	100.0%	200
16	4	100.0%	200
32	8	100.0%	200
64	16	100.0%	200

Table 12: Copy-back task: accuracy and convergence by

d_{\text{select}}

$d_{\text{select}}$	$d_{\text{select}}$ /head	Best Accuracy	Converge Epoch
4	1	65.2%	did not converge
8	2	100.0%	1900
16	4	100.0%	1900
32	8	100.0%	1400
64	16	100.0%	1000

Table 13: Key-value retrieval: accuracy and convergence by

d_{\text{select}}

$d_{\text{select}}$	$d_{\text{select}}$ /head	Val PPL	Test PPL	$\Delta$ PPL	QK Params	QK Saved
8	1	133.78	126.48	+3.5%	24,672	97%
16	2	132.67	125.49	+2.7%	49,344	94%
32	4	130.51	123.78	+1.3%	98,688	87%
64	8	129.34	122.24	+0.0%	197,376	75%
128	16	126.42	120.76	$-$ 1.2%	394,752	50%
256	32	126.95	122.22	—	789,504	0%

Table 14: WikiText-2 results. Model:

d_{\text{model}}=256

, 8 heads, 6 layers. Baseline

d_{\text{select}}=256

$d_{\text{select}}$	$d_{\text{select}}$ /head	Val PPL	Test PPL	$\Delta$ PPL	QK Saved
32	4	38.38	—	+7.6%	87%
64	8	37.22	—	+4.3%	75%
128	16	36.42	35.80	+2.1%	50%
256	32	35.67	—	—	0%

Table 15: WikiText-103 results. Model:

d_{\text{model}}=256

, 8 heads, 6 layers. Baseline

d_{\text{select}}=256

$d_{\text{select}}$	$d_{\text{select}}$ /head	Params	Val PPL	$\Delta$ PPL	QK saved
768 (full)	64	101.7M	22.80	—	0%
192 ( $d_{\text{model}}/4$ )	16	91.1M	23.77	+4.3%	75%
96 ( $d_{\text{model}}/8$ )	8	89.3M	24.45	+7.2%	87%
48 ( $d_{\text{model}}/16$ )	4	88.4M	25.30	+11.0%	94%

Table 16: LLaMA 125M with asymmetric attention on WikiText-103. Comparison with 10M vanilla transformer from Experiment 4.

Method	Config	Params	KV budget	KV saved	Test PPL
MHA	12 heads	101.7M	1536	0%	23.07
Thin keys	$d_{\text{select}}=384$	94.6M	1152	25%	23.22 (+0.7%)
Thin keys	$d_{\text{select}}=192$	91.1M	960	37.5%	24.09 (+4.4%)
GQA	6 KV heads	94.6M	768	50%	23.15 (+0.3%)
GQA	4 KV heads	92.3M	512	66.7%	23.32 (+1.1%)
MLA	$d_{c}=768$	108.8M	768	50%	23.11 (+0.2%)

Table 17: 125M LLaMA: comparison of KV compression methods trained from scratch on WikiText-103. KV budget = per-token, per-layer cache size (K + V dimensions stored).

Task	$N_{\text{effective}}$	Min $d_{\text{select}}$ /head	Prediction ( $\log_{2}N$ )
Positional (copy-back)	$\sim$ 10 offsets	1	$\log_{2}10\approx 3$
Content (16 keys)	16 keys	2	$\log_{2}16=4$ (total)
Language (WikiText)	$\sim$ 256 patterns	8	$\log_{2}256=8$

Table 18: Minimum effective

d_{\text{select}}

scales with task complexity.

Exp	FT Data	Control	r512	r256	$\Delta_{512}$	$\Delta_{256}$
–	None (baseline)	38.4	33.8	16.5	–	–
A	WikiText-103	29.9	27.7	25.8	$-$ 7.4	$-$ 13.7
F	C4 (10M tok)	30.6	29.4	22.8	$-$ 3.9	$-$ 25.5
F2	C4 + Math (10M tok)	31.7	28.2	24.1	$-$ 3.5	$-$ 7.6
F3	GSM8K CoT (1.5M tok)	53.7	52.5	52.0	$\mathbf{-0.7}$	$\mathbf{-1.2}$

Table 19: GSM8K exact-match accuracy across fine-tuning experiments. All models use the same QK-only fine-tuning protocol (3 epochs, lr=

5\times 10^{-5}

). “Control” = uncompressed model with identical fine-tuning.

Explore similar work

Jun 20, 2026

Keyless Attention: Value-Space Routing and Value-Only Caching for Efficient Transformers

We propose Keyless Attention, an attention mechanism that eliminates the key projection entirely, operating over queries and values only. This yields a Value-Only Cache that reduces KV cache memory and access overhead by exactly 50% over standard attention, while matching or exceeding standard attention's decode throughput. Beyond efficiency, we introduce Depth-

m

Attention Factorization: standard attention computes a depth-2 factorization of the attention bilinear form, while Keyless Attention realizes a depth-

m

instance of this family. At m=3, Keyless Attention matches the projection matrix count of standard attention via a value-space routing matrix that replaces the key projection and introduces a coupling between routing and retrieval. Experiments across five models and four architectures (GPT-2 280M, GPT-2 557M, Pythia 410M, Qwen2 1.5B, and Llama 3.2 1B) show that Keyless Attention matches or outperforms standard QKV attention on perplexity in 4 out of 5 models. On downstream zero-shot evaluation (GPT-2 557M), Keyless Attention outperforms on 4 out of 5 commonsense reasoning benchmarks, while achieving 50% KV cache reduction throughout.

Xin Gao

Department of Mathematics and Statistics York University Toronto, M3J 1P3

Transformer Inference KV Caching Memory-Efficient Optimization Memory-Efficient Inference Efficient Attention

Ask PDF

Oct 3, 2026

AttSVD:Prompt-Adaptive Low-Rank KV Cache Compression via Attention-Guided SVD

The key-value (KV) cache of autoregressive transformers grows linearly with context length and dominates memory at long context. Most training-free remedies evict low-importance tokens, an irreversible choice along the sequence axis. We instead keep every token and store it more cheaply along the "feature" axis. We therefore propose AttSVD, a new "interpretable" low-rank compression whose basis is derived from each prompt's own attention geometry: an online, per-prompt truncated SVD that keeps only the directions attention actually reads, cutting persistent per-head KV memory in proportion to the retained rank. We propose two decode-time caching strategies, accumulating and streaming, for short and long generation regimes. Furthermore, we propose two refinements that make compression adaptive. A per-matrix energy rule sizes the logit space and the attention mass independently. An attention-aware basis truncates only in the spaces attention actually reads, preserving both the attention logits and the attention output. The same factors also provide free, per-head interpretability insights into the effective rank and the geometry attention consumes. Across multiple models, on both an agentic benchmark and the full LongBench suite AttSVD stays on par with the dense cache while using up to 50% of the KV-cache memory.

Sara Abdali, Jongwoo Ko, Pashmina Cameron

Microsoft Applied Sciences Group (ASG)

KV-Cache Compression Low-Rank Compression Singular Value Decomposition Transformer Attention

Ask PDF

Jul 14, 2026

A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation

The key-value (KV) cache is the dominant memory bottleneck in long-context language model inference. Existing compression methods apply low-rank factorization or quantization independently, without jointly allocating rank and precision under a shared storage budget. We introduce JoLT, a training-free compressor that treats grouped prefill caches as fourth-order tensors and applies partial Tucker decomposition along the token and feature modes, the two axes that carry low-rank structure, while leaving the head and layer modes intact. A rotated low-bit quantizer captures the truncation residual, and a single Lagrangian dual allocates per-group Tucker ranks and residual bit-widths under a global byte constraint. FlashJoLT replaces the exact token-mode SVD with a randomized approximation that matches JoLT within the free zone at a fraction of the compression cost, and a fused Triton decode kernel evaluates attention directly over the stored factors without materializing dense KV tensors. Across five models from four architecture families, covering multi-head attention, grouped-query attention, and mixture-of-experts architecture, JoLT achieves 2 - 3x compression with less than 0.2% perplexity degradation, without retraining. On RULER at 64K context with LLaMA-3.1-8B, retrieval accuracy remains near-lossless through 3x and declines by only 0.90 and 2.40pp at 4x and 5x, respectively. JoLT demonstrates that tensor-aware low-rank decomposition and quantized residuals, unified under a single storage budget, achieve near-lossless KV-cache compression across diverse model architectures without retraining.

Rahul Krishnan, Volker Schulz

Universität Trier · Fachbereich IV, Mathematik, Universität Trier

KV Caching Low-Rank Matrix Decomposition Tensor Decomposition Low-Rank Compression Low-Rank Approximation KV-Cache Compression+1

Ask PDF