Thin Keys, Full Values: Reducing KV Cache via Low-Dimensional Attention Selection

Hengshuai Yao; Xing Chen; Ahmed Murtadha; Guan Wang

cs.LGFeb 16, 2026

Thin Keys, Full Values: Reducing KV Cache via Low-Dimensional Attention Selection

Authors: Hengshuai Yao, Xing Chen, Ahmed Murtadha, Guan Wang

Organizations: Sapient Intelligence · Department of Computing Science, University of Alberta

Abstract

Standard Transformer attention uses identical dimensionality for queries, keys, and values, yet these components serve different roles: queries and keys produce scalar attention weights (selection), while values carry rich representations (value transfer). We show that selection requires only $O(\log N)$ dimensions to distinguish among $N$ relevant token categories (e.g., syntactic roles, semantic clusters, positional patterns) -- far fewer than value transfer needs. We introduce factored keys, which exploit this asymmetry to physically shrink the KV cache of any pretrained model without retraining from scratch -- unlike Grouped-Query Attention (GQA) and Multi-Head Latent Attention (MLA), which must be designed into the architecture before pretraining. We factorize each key projection $W_K \approx A_{d \times r} B_{r \times d}$ via truncated singular value decomposition (SVD) (where $r$ is the chosen compression dimension), set $W_K' = A$ as the new key projection producing compact $r$ -dimensional keys for the cache, and absorb $B^\top$ into the query projection ( $W_Q' = W_Q B^\top$ ) at zero cost -- since queries are never cached. At the 7B scale, training from scratch with $r = d/4$ (where $d$ is the model dimension) matches full-attention perplexity ( $9.24$ vs $9.25$ PPL after 20B tokens, mean over two seeds) while using 12% fewer parameters and training 8% faster. For existing models, SVD followed by QK fine-tuning (3 epochs, less than 1% of pretraining data) achieves 75% key cache savings at roughly 2% quality cost on both GPT-2 and Mistral-7B. The approach composes with GQA and quantization for up to $16\times$ combined key cache compression. For a 7B model serving a 128K context, factored keys save 25 GB of KV cache per user, enabling roughly 60% more concurrent users on identical hardware.

Figures & tables

Rank $r$	$r$ /head	Both $Q{+}K$	$K$ -only	$Q$ -only
128	10	67,132	57.37 (+130%)	149.04 (+498%)
192	16	90.95 (+265%)	31.32 (+26%)	70.05 (+181%)
256	21	46.15 (+85%)	27.49 (+10%)	40.42 (+62%)
384	32	—	25.40 (+2.0%)	27.58 (+11%)
512	42	25.69 (+3.1%)	25.23 (+1.3%)	25.47 (+2.2%)

Table 1: SVD compression of GPT-2 projections. Baseline PPL: 24.91.

\Delta

is relative PPL increase.

Rank $r$	Before FT	After FT	Control	vs Control	K cache saved
768 (none)	29.51	19.07	—	baseline	0%
384 ( $d_{\text{model}}/2$ )	30.15 (+2.2%)	19.14	19.07	+0.4%	50%
256 ( $d_{\text{model}}/3$ )	32.61 (+10.5%)	19.28	19.07	+1.1%	67%
192 ( $d_{\text{model}}/4$ )	37.64 (+27.6%)	19.42	19.07	+1.8%	75%
128 ( $d_{\text{model}}/6$ )	67.26 (+128%)	19.77	19.07	+3.7%	83%

Table 2: SVD compression + QK fine-tuning on WikiText-103. The “vs control” column measures the residual gap relative to the uncompressed model after both receive identical fine-tuning.

	Full Attention	Thin Keys ( $d_{\text{select}}=d_{\text{model}}/4$ )
Parameters	6.74B	5.93B ( $-$ 12%)
OWT val PPL	$13.82\pm 0.09$	$\mathbf{13.12\pm 0.04}$ ( $-$ 5.1%)
WT103 val PPL	$20.79\pm 0.35$	$\mathbf{19.60\pm 0.46}$ ( $-$ 5.7%)
Wall-clock time	25.9h	23.4h ( $-$ 9.4%)

Table 3: 7B LLaMA trained from scratch on OpenWebText (2B tokens), mean

\pm

std over 2 seeds.

Step Full Thin 5K 26.62 25.05 10K 20.81 19.21 15K 17.68 16.51 20K 15.66 14.74 25K 14.37 13.59 30K 13.79 13.12

Step Full Thin 5K 44.23 41.34 10K 31.99 29.20 15K 26.43 25.36 20K 24.20 22.73 25K 22.05 20.83 30K 21.19 19.96

Figure 1: Training trajectory for 7B from-scratch models (seed 137). Left: OWT validation PPL. Right: WikiText-103 validation PPL. Thin keys lead throughout.

	Full Attention	Thin Keys ( $d_{\text{select}}=d_{\text{model}}/4$ )
Parameters	6.74B	5.93B ( $-$ 12%)
OWT val PPL	$9.25\pm 0.00$	$\mathbf{9.24\pm 0.00}$ ( $-$ 0.1%)
WT103 val PPL	$13.26\pm 0.12$	$\mathbf{13.00\pm 0.07}$ ( $-$ 2.0%)
Wall-clock time	$261.5\pm 0.1$ h	$240.7\pm 0.9$ h ( $-$ 8.0%)

Table 4: 7B LLaMA on OpenWebText (20B tokens), mean

\pm

std over 2 seeds. Both architectures converge to the same quality; thin keys train 8% faster.

Figure 2: Training trajectory for 7B models over 305K steps (20B tokens). Left: PPL vs step; thin keys lead early but converge by

\sim

150K steps. Right: PPL vs wall-clock time; thin keys reach any target PPL

\sim

20 hours sooner.

Task	Full Attention	Thin Keys	$\Delta$
Hellaswag (acc_norm)	55.6	55.7	$+$ 0.1
ARC-Challenge (acc_norm)	29.8	29.4	$-$ 0.4
WinoGrande (acc)	60.6	59.1	$-$ 1.5
MMLU (acc)	23.4	23.3	$-$ 0.1
GSM8K (exact_match)	0.0	0.0	0.0

Table 5: Downstream evaluation of 7B from-scratch models (20B tokens, seed 137). Scores are low in absolute terms (OWT-only training) but thin keys matches full attention on all tasks.

Method	K cache (GB)	V cache (GB)	KV total (GB)	KV saved
MHA (baseline)	32.0	32.0	64.0	—
Thin keys ( $d_{\text{select}}=d_{\text{model}}/4$ )	8.0	32.0	40.0	37.5%
GQA-8	8.0	8.0	16.0	75.0%
MLA ( $d_{c}\!=\!512$ , $d_{h}^{R}\!=\!64$ )	4.5 (joint)		4.5	93.0%
GQA-8 + thin keys	2.0	8.0	10.0	84.4%

Table 6: Analytical KV cache comparison at LLaMA-7B scale (128K context, bf16). MLA stores a shared latent

d_{c}

plus a decoupled RoPE key (

d_{h}^{R}

). Thin keys compose with GQA by reducing

d_{\text{head}}^{QK}

independently.

Rank $r$	Before FT	After FT	Control	vs Control	K cache saved
1024 (none)	6.69	5.91	—	baseline	0%
512 ( $d_{K}/2$ )	6.81 (+1.7%)	5.93	5.91	+0.3%	50%
256 ( $d_{K}/4$ )	7.84 (+17.1%)	6.03	5.91	+2.0%	75%
128 ( $d_{K}/8$ )	12.34 (+84.4%)	6.10	5.91	+3.2%	88%

Table 7: Mistral-7B: SVD compression + QK fine-tuning on WikiText-103.

Task	Metric	Baseline	r512 +FT	r256 +FT	Ctrl+FT	$\Delta_{512}$	$\Delta_{256}$
Hellaswag	acc_norm	81.2	81.4	80.7	81.3	$+$ 0.1	$-$ 0.7
ARC-Challenge	acc_norm	54.0	54.1	53.4	54.4	$-$ 0.5	$-$ 1.7
WinoGrande	acc	75.4	73.2	72.1	73.2	$+$ 0.0	$-$ 1.6
MMLU	acc	60.1	55.2	54.4	55.7	$-$ 0.9	$-$ 2.3
GSM8K	exact_match	38.4	27.7	25.8	29.9	$-$ 7.4	$-$ 13.7

Table 8: Downstream evaluation of SVD-compressed Mistral-7B. “Ctrl+FT” is the uncompressed model with identical fine-tuning.

Benchmark	Metric	Control	r512	r256	$\Delta_{512}$	$\Delta_{256}$
GSM8K	exact_match	53.7	52.1	51.2	$-$ 1.6	$-$ 2.5
Minerva Algebra	math_verify	14.2	14.1	12.4	$-$ 0.2	$-$ 1.9
Minerva Pre-Algebra	math_verify	21.0	19.1	19.1	$-$ 2.0	$-$ 2.0
AGIEVAL AQuA-RAT	acc	15.7	17.3	17.3	$+$ 1.6	$+$ 1.6

Table 9: Math generalization of GSM8K-fine-tuned models on held-out benchmarks.

128K context
	Standard	$d_{\text{select}}=d_{\text{model}}/2$	$d_{\text{select}}=d_{\text{model}}/4$
		(SVD, no retrain)	(train or SVD+FT)
K cache	33.6 GB	16.8 GB	8.4 GB
V cache	33.6 GB	33.6 GB	33.6 GB
Total	67.2 GB	50.4 GB	42.0 GB
Savings/user	—	16.8 GB (25%)	25.2 GB (37.5%)

Table 10: KV cache memory per user (

d_{\text{model}}=4096

, 32 layers, fp16).

Measured throughput (tokens/s)
	Batch size
	1	4	8	16	32
Baseline ( $d_{k}\!=\!128$ )	49.5	124.0	160.6	181.5	194.7
r512 ( $d_{k}\!=\!64$ )	53.5	142.9	192.8	223.2	243.0
r256 ( $d_{k}\!=\!32$ )	56.1	155.7	216.4	254.3	280.2
Measured speedup

Table 11: Decode throughput for Mistral-7B with factored keys on H100 SXM (context 4096, 128 generated tokens). “Predicted” is the bandwidth roofline from Eq. 10 .

$d_{\text{select}}$	$d_{\text{select}}$ /head	Best Accuracy	Converge Epoch
4	1	100.0%	300
8	2	100.0%	200
16	4	100.0%	200
32	8	100.0%	200
64	16	100.0%	200

Table 12: Copy-back task: accuracy and convergence by

d_{\text{select}}

$d_{\text{select}}$	$d_{\text{select}}$ /head	Best Accuracy	Converge Epoch
4	1	65.2%	did not converge
8	2	100.0%	1900
16	4	100.0%	1900
32	8	100.0%	1400
64	16	100.0%	1000

Table 13: Key-value retrieval: accuracy and convergence by

d_{\text{select}}

$d_{\text{select}}$	$d_{\text{select}}$ /head	Val PPL	Test PPL	$\Delta$ PPL	QK Params	QK Saved
8	1	133.78	126.48	+3.5%	24,672	97%
16	2	132.67	125.49	+2.7%	49,344	94%
32	4	130.51	123.78	+1.3%	98,688	87%
64	8	129.34	122.24	+0.0%	197,376	75%
128	16	126.42	120.76	$-$ 1.2%	394,752	50%
256	32	126.95	122.22	—	789,504	0%

Table 14: WikiText-2 results. Model:

d_{\text{model}}=256

, 8 heads, 6 layers. Baseline

d_{\text{select}}=256

$d_{\text{select}}$	$d_{\text{select}}$ /head	Val PPL	Test PPL	$\Delta$ PPL	QK Saved
32	4	38.38	—	+7.6%	87%
64	8	37.22	—	+4.3%	75%
128	16	36.42	35.80	+2.1%	50%
256	32	35.67	—	—	0%

Table 15: WikiText-103 results. Model:

d_{\text{model}}=256

, 8 heads, 6 layers. Baseline

d_{\text{select}}=256

$d_{\text{select}}$	$d_{\text{select}}$ /head	Params	Val PPL	$\Delta$ PPL	QK saved
768 (full)	64	101.7M	22.80	—	0%
192 ( $d_{\text{model}}/4$ )	16	91.1M	23.77	+4.3%	75%
96 ( $d_{\text{model}}/8$ )	8	89.3M	24.45	+7.2%	87%
48 ( $d_{\text{model}}/16$ )	4	88.4M	25.30	+11.0%	94%

Table 16: LLaMA 125M with asymmetric attention on WikiText-103. Comparison with 10M vanilla transformer from Experiment 4.

Method	Config	Params	KV budget	KV saved	Test PPL
MHA	12 heads	101.7M	1536	0%	23.07
Thin keys	$d_{\text{select}}=384$	94.6M	1152	25%	23.22 (+0.7%)
Thin keys	$d_{\text{select}}=192$	91.1M	960	37.5%	24.09 (+4.4%)
GQA	6 KV heads	94.6M	768	50%	23.15 (+0.3%)
GQA	4 KV heads	92.3M	512	66.7%	23.32 (+1.1%)
MLA	$d_{c}=768$	108.8M	768	50%	23.11 (+0.2%)

Table 17: 125M LLaMA: comparison of KV compression methods trained from scratch on WikiText-103. KV budget = per-token, per-layer cache size (K + V dimensions stored).

Task	$N_{\text{effective}}$	Min $d_{\text{select}}$ /head	Prediction ( $\log_{2}N$ )
Positional (copy-back)	$\sim$ 10 offsets	1	$\log_{2}10\approx 3$
Content (16 keys)	16 keys	2	$\log_{2}16=4$ (total)
Language (WikiText)	$\sim$ 256 patterns	8	$\log_{2}256=8$

Table 18: Minimum effective

d_{\text{select}}

scales with task complexity.

Exp	FT Data	Control	r512	r256	$\Delta_{512}$	$\Delta_{256}$
–	None (baseline)	38.4	33.8	16.5	–	–
A	WikiText-103	29.9	27.7	25.8	$-$ 7.4	$-$ 13.7
F	C4 (10M tok)	30.6	29.4	22.8	$-$ 3.9	$-$ 25.5
F2	C4 + Math (10M tok)	31.7	28.2	24.1	$-$ 3.5	$-$ 7.6
F3	GSM8K CoT (1.5M tok)	53.7	52.5	52.0	$\mathbf{-0.7}$	$\mathbf{-1.2}$

Table 19: GSM8K exact-match accuracy across fine-tuning experiments. All models use the same QK-only fine-tuning protocol (3 epochs, lr=

5\times 10^{-5}

). “Control” = uncompressed model with identical fine-tuning.

Explore similar work

Keyless Attention: Value-Space Routing and Value-Only Caching for Efficient Transformers
Jun 20, 2026Xin GaoTransformer Inference KV Caching
Department of Mathematics and Statistics York University Toronto, M3J 1P3
AttSVD:Prompt-Adaptive Low-Rank KV Cache Compression via Attention-Guided SVD
Oct 3, 2026Sara Abdali, Jongwoo Ko, Pashmina CameronKV-Cache Compression Low-Rank Compression
Microsoft Applied Sciences Group (ASG)
A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation
Jul 14, 2026Rahul Krishnan, Volker SchulzKV Caching Low-Rank Matrix Decomposition
Universität Trier · Fachbereich IV, Mathematik, Universität Trier

Jun 20, 2026J/K move · Enter open · S save

Keyless Attention: Value-Space Routing and Value-Only Caching for Efficient Transformers

Xin Gao

Department of Mathematics and Statistics York University Toronto, M3J 1P3

Transformer Inference KV Caching Memory-Efficient Optimization Memory-Efficient Inference Efficient Attention

Open paper AskPDF arXiv

We propose Keyless Attention, an attention mechanism that eliminates the key projection entirely, operating over queries and values only. This yields a Value-Only Cache that reduces KV cache memory and access overhead by exactly 50% over standard attention, while matching or exceeding standard attention's decode throughput. Beyond efficiency, we introduce Depth-

m

Attention Factorization: standard attention computes a depth-2 factorization of the attention bilinear form, while Keyless Attention realizes a depth-

m

instance of this family. At m=3, Keyless Attention matches the projection matrix count of standard attention via a value-space routing matrix that replaces the key projection and introduces a coupling between routing and retrieval. Experiments across five models and four architectures (GPT-2 280M, GPT-2 557M, Pythia 410M, Qwen2 1.5B, and Llama 3.2 1B) show that Keyless Attention matches or outperforms standard QKV attention on perplexity in 4 out of 5 models. On downstream zero-shot evaluation (GPT-2 557M), Keyless Attention outperforms on 4 out of 5 commonsense reasoning benchmarks, while achieving 50% KV cache reduction throughout.

Abstract

Explore similar work

Keyless Attention: Value-Space Routing and Value-Only Caching for Efficient Transformers

AttSVD:Prompt-Adaptive Low-Rank KV Cache Compression via Attention-Guided SVD

A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation