Folding Attention Biases into Keys Makes Compacted KV Caches Run on Standard Kernels

Utkarsh Kanwat

cs.LGOct 27, 2026

Folding Attention Biases into Keys Makes Compacted KV Caches Run on Standard Kernels

Authors: Utkarsh Kanwat

Organizations: Independent Researcher

Abstract

Attention Matching (AM) compacts a KV cache by keeping some keys per head, refitting their values and adding a bias to each kept key's logit. FlashAttention and vLLM's paged caches have no input for it. We fold the bias into the keys along one closed-form direction per head, fitted on context queries, giving a plain cache with AM's size and head dimension and the model's own query path; the output error is bounded by how far each query's projection on that direction is from 1. With equal per-head budgets, unmodified vLLM serves folded caches through a KV connector, matching Hugging Face's answers on all 51 Qwen3-4B questions and its greedy output on 180 of 200 Qwen3-14B LongBench samples. With equal budgets on QuALITY (Qwen3-1.7B, 4B, 8B, Llama-3.1-8B) and LongHealth (three Qwen3 models), the fold is within noise of AM's accuracy in 13 of 14 settings, with lower KL to the full cache than dropping the bias in all 12 where both ran. Under the published per-head budgets, on Qwen3-4B, its Instruct-2507 variant and Llama-3.1-8B, it is within noise of AM in 9 of 10, no longer beats dropping the bias in KL, and the two each win once in accuracy.

Figures & tables

Figure 1: The fold moves each retained key along one shared direction

u

\beta_{j}\sqrt{d}

, so a query near the hyperplane

q\cdot u=1

(shaded) sees the logit shift

\beta_{j}\,(q\cdot u)\approx\beta_{j}

. Left: AM, a scalar bias

\beta_{j}

per retained key

k_{j}

. Right: the fold. For the orange open circle, a query far from the hyperplane, AM still adds

\beta_{j}

while the fold scales it by

q\cdot u

Figure 2: On the median head the default fold stays within about 5% of AM at every distance (the mean, worse far out, is in Appendix A.12 ). Median relative attention-output error over heads on Pride and Prejudice , on 512-query windows starting

\Delta

tokens after the compacted context. Thin lines: interquartile range over heads for keep (dotted) and AM (solid). Shaded: median AM error above median keep error.

model	near	far	far (%)	far (%)	$\Delta=0$	$24$ k
	ratio		worse ${>}5\%$	worst 10%	AM $<$ keep (% heads)
Qwen3-1.7B	1.005	0.998	11	90	81	18
Qwen3-4B	1.011	1.021	35	92	85	17
Qwen3-8B	1.011	1.022	40	96	82	17

Table 1: On Pride and Prejudice the default fold tracks AM closely on the median head, and a tenth of the far (head, window) pairs carries most of its excess error. Ratio: median of

e_{\mathrm{fold}}/e_{\mathrm{AM}}

over (head, window) pairs, pooled over near or far windows. Worse

{>}5\%

: share of far pairs with

e_{\mathrm{fold}}/e_{\mathrm{AM}}>1.05

. Worst 10%: share of the total excess

\max(e_{\mathrm{fold}}-e_{\mathrm{AM}},0)

in the worst tenth of far pairs. AM

<

keep: share of key-value heads on which AM’s error is below keep’s, in the window at that

\Delta

accuracy (%)
	Qwen3-1.7B		Qwen3-4B		Qwen3-8B		Llama-3.1-8B
AM fit	AM-style		ours		ours		ours
budget	10%	5%	10%	5%	10%	5%	10%	5%
full cache	55.8		69.1		73.0		71.3
keep	45.5	42.3	62.2	58.8	70.2	62.5	65.7	62.4
drop	–	–	63.3	57.9	68.4	61.2	67.4	61.4

Table 2: On QuALITY no fold

-

AM accuracy difference in these settings is beyond noise. Fold

-

AM and drop

-

fold: paired differences with their 95% bootstrap intervals over the 30 articles below them, accuracy differences in points. Bold: beyond noise. KL

>

1: questions with KL above 1 under drop and under the fold. 534 questions. Each column’s AM fit was chosen on the test set for AM’s lower KL (Setup). Drop was not run with the AM-style fit on QuALITY, so the Qwen3-1.7B columns have none, and Table 13 gives drop under our fit.

accuracy (%)
	Qwen3-1.7B		Qwen3-4B		Qwen3-8B
budget	10%	5%	10%	5%	10%	5%
full cache	68.3		77.4		80.4
keep	36.2	30.4	46.7	39.7	44.7	37.2
drop	38.9	33.9	54.3	45.2	51.3	42.5
AM	39.9	31.9	54.5	43.5	55.8	45.0

Table 3: On LongHealth the fold keeps AM’s accuracy gain over keep on Qwen3-4B and 8B and loses it on Qwen3-1.7B at 10%. Our AM fit throughout. Layout as in Table 2 , with intervals over the 20 patients (398 questions).

	full	fold 10%	fold 5%
prompt tokens (mean)	6,533	781	462
KV per token (KiB)	144	144	144
capacity	4	31	44
decode (tokens/s)	29	200	297
same letter as HF	51/51	51/51	51/51
same 128 tokens as HF	46/51	42/51	44/51

Table 4: A folded Qwen3-4B cache served by unmodified vLLM 0.18.1 (Triton backend, fp16, one Tesla T4, 51 QuALITY questions, equal budgets per head). Capacity: concurrent requests that fit the KV pool with 256 generated tokens each. Decode: tokens/s at the largest batch that fits. Agreement with Hugging Face (HF) on the same cache: answer letter, and exact match of 128 greedy tokens. Correct letter: 51 questions are too few to measure accuracy, which Section 5 does.

	Qasper	MF-en	HotpotQA	2WikiMQA	mean
full cache (HF)	43.8	47.5	60.5	41.7	48.4
AM (HF)	24.1	30.7	37.8	30.4	30.8
drop (HF)	20.0	29.8	37.4	33.1	30.1
fold (HF)	28.8	29.9	29.1	27.0	28.7
fold (vLLM)	28.0	30.2	28.9	29.5	29.2
fold $-$ AM	$+4.7$	$-0.8$	$-8.8$	$-3.3$	$-2.0$

Table 5: Qwen3-14B with real weights on four LongBench tasks at 10%: the folded cache served by unmodified vLLM 0.18.1 scores as it does in Hugging Face (HF), and its mean is within noise of AM’s. F1

\times 100

with the official scorer, 50 samples per task, AM-style fit, equal budgets per head; bf16, one A100-80GB. Fold

-

AM: paired difference in the HF runs with its 95% bootstrap interval over samples.

model	context (max)	full	10%	5%	full	10%	5%
		capacity			decode (tokens/s)
Qwen3-32B	6,969	7	45	64	135 (7)	872 (45)	1,232 (64)
Qwen3-32B	13,754 ( $\times 2$ )	3	28	45	59 (3)	534 (28)	864 (45)
Qwen3-14B	6,969	39	255	256	766 (39)	3,442 (128)	3,831 (128)
Qwen3-14B	27,324 ( $\times 4$ )	10	90	158	196 (8)	1,000 (32)	1,182 (32)

Table 6: Folded caches of larger models served by unmodified vLLM 0.18.1 (default FlashAttention backend, bf16, one A100-80GB, random weights of the real architectures, 51 QuALITY questions). Capacity: concurrent requests that fit the KV pool with 256 generated tokens each. Decode: the highest rate measured, in tokens/s, at the batch in parentheses. At 5% the 14B model reaches vLLM’s limit of 256 concurrent sequences, and at 10% its pool holds 255, one short of it. The

\times 2

and

\times 4

rows repeat each article to lengthen the context. 14B peak rates are measured at batch

\leq 128

and are lower bounds. Fold columns at the native length are injected caches; at the longer lengths they are plain prompts of the fold’s length, which decode at the same rate.

method	per-entry logit term	fold applies
AM [ 33 ]	$\beta=\log w$ , NNLS fit, $\|\beta\|\leq 3$	yes (this paper)
KV-Surgeon, ARC-KV, TAM, online AM [ 23 , 21 , 16 , 17 ]	fitted $\log w$	yes
Latent Compaction [ 8 ]	learned per-slot bias	yes, bias only a
Still [ 19 ]	optional $\beta$ , off in all results	yes
ToMe, SemantiCache, KeepKV, ZSMerge, AsymKV [ 6 , 27 , 22 , 15 , 9 ]	$\log$ count or size, $\alpha$ -scaled in ZSMerge	yes
SelKV [ 7 ]	$0.5\log$ attention mass	yes

Table 7: Logit terms of compaction and merging methods. Static: one scalar per cache entry, the same for every query, which the fold handles. Fold applies: structurally, since the fold has been tested only on AM, one capped log-count merge and two of KV-Surgeon’s compactors.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Qwen3-4B, QuALITY; full cache 69.1%
	10%			5%
	acc	agree	KL	acc	agree	KL
keep	69.5	84.5	1.18	63.3	77.5	1.93
drop	68.7	86.0	1.14	64.4	79.4	1.65
AM	67.8	86.7	0.90	65.2	81.1	1.65
fold	68.5	86.5	0.96	64.8	80.9	1.80

Appendix

Table 8: The fold under each model’s published per-head budgets: level with AM in nine of the ten cells, and below it on Llama-3.1-8B LongHealth at 5%. AM-style fit with the published NNLS weighting, bf16; Qwen3-4B QuALITY on a Tesla T4, the rest on an A100. Below each paired difference, its 95% bootstrap interval over the 30 articles (534 questions) or 20 patients (398); accuracy and agreement in %, differences in points. Bold: beyond noise.

Llama-3.1-8B-Instruct, QuALITY; full cache 71.3%
	10%			5%
	acc	agree	KL	acc	agree	KL
keep	69.3	85.0	0.19	67.6	82.8	0.28
drop	68.9	86.7	0.16	66.9	81.8	0.28
AM	69.3	89.0	0.12	66.7	84.6	0.22
fold	68.9	88.4	0.14	66.7	83.0	0.23

Appendix

Table 8: (continued) The fold under the published per-head budgets on Llama-3.1-8B-Instruct, QuALITY; layout as above.

	accuracy (%)				fold $-$ AM
run	keep	drop	AM	fold	acc	KL
as in the main runs ( $\|\beta\|\leq 3$ )	65	63	62	57	$-$ 5 [ $-$ 9, $-$ 1]	+0.18 [+0.14, +0.23]
folded keys in fp32	65	–	62	58	$-$ 4 [ $-$ 7, $-$ 1]	+0.19 [+0.14, +0.24]
AM’s values	65	–	61	64	+3 [ $-$ 1, +5]	$-$ 0.01 [ $-$ 0.05, +0.03]
self-study fit queries	69	65	64	64	+0 [ $-$ 4, +4]	+0.03 [ $-$ 0.03, +0.11]
$\|\beta\|\leq 1$	65	63	71	70	$-$ 1 [ $-$ 6, +5]	$-$ 0.04 [ $-$ 0.09, +0.01]

Appendix

Table 9: Diagnosing the fold’s loss on Llama-3.1-8B LongHealth at 5%. Upper panel: patients 0–4 (100 questions, so accuracies are whole points), full-cache accuracy 79%. Lower panel: all 20 patients (398 questions), full-cache accuracy 80.2%; the

|\beta|\leq 3

row is the run of Table 8 , and the

|\beta|\leq 1

row ran on another A100 with identical full-cache, keep and drop log-probs. Published per-head budgets, AM-style fit with the published NNLS weighting, A100, bf16. AM’s values: AM and the fold both keep AM’s original values instead of refitting them. Fold

-

AM: paired difference with its 95% bootstrap interval over the patients. Bold: beyond noise.

QuALITY
	10%			5%
	acc	agree	KL	acc	agree	KL
merge-drop	69.1	84.3	1.15	62.9	76.4	1.95
merge-log	67.8	86.0	0.97	64.2	80.0	1.72
merge-fold	67.2	85.4	0.98	64.4	80.9	1.74
merge-refit	68.5	86.3	0.90	66.3	81.1	1.50

Appendix

Table 10: Log-count merging on Qwen3-4B: folding the count bias into the keys reproduces merge-log within noise on both benchmarks. Published per-head budgets, AM-style fit, cosine gate 0.9 on post-RoPE keys, at most 16 keys per cluster; bf16, QuALITY on a Tesla T4 and LongHealth on an A100. Differences: paired, with their 95% bootstrap intervals over the 30 articles or 20 patients; accuracy and agreement in %, differences in points. Bold: beyond noise. QuALITY: 534 questions, full-cache accuracy 69.1%; LongHealth: 398 questions, 77.4%. Keep, on the same keys, is in Table 8 .

	accuracy (%)				unparsed
base compactor	base	fold	fold $-$ base	gained/lost	base/fold
KV-Surgeon-fast	68.5	71.8	+3.3 [ $-$ 1.7, +8.2]	14/8	5/3
KVMerger, fitted $\beta$	68.0	65.2	$-$ 2.8 [ $-$ 6.8, +1.1]	11/16	5/11

Appendix

Table 11: Folding the bias of two compactors inside KV-Surgeon’s code and harness, Qwen3-4B QuALITY at 5%: neither difference is beyond noise. KV-Surgeon’s per-head budgets, answers generated with thinking on and parsed by its code, unparsed answers counted wrong; A100, bf16. Fold

-

base: paired difference in points with its 95% bootstrap interval over the 10 articles. Gained/lost: questions right only with the fold / only without it. 181 questions.

$\mathrm{KL}(\text{full}\,\\|\,\cdot)$
	Qwen3-1.7B		Qwen3-4B		Qwen3-8B
	10%	5%	10%	5%	10%	5%
AM: no refit $-$ refit	$-$ 0.26	$-$ 0.02	+0.37	+0.35	+0.09	+0.01
	[ $-$ 0.68, +0.14]	[ $-$ 0.42, +0.39]	[+0.12, +0.64]	[+0.09, +0.59]	[ $-$ 0.11, +0.30]	[ $-$ 0.27, +0.28]
fold: no refit $-$ refit	$-$ 0.53	$-$ 0.21	+0.43	+0.44	+0.36	+0.67
	[ $-$ 0.97, $-$ 0.08]	[ $-$ 0.67, +0.24]	[+0.09, +0.78]	[+0.13, +0.75]	[+0.08, +0.67]	[+0.33, +1.00]

Appendix

Table 12: On 4B and 8B the value refit and the covariance term in

u

each lower the fold’s KL. On 1.7B under our fit both raise it. Each row is a variant minus the cache it modifies, paired within one ablation run, with its 95% bootstrap interval over the 30 articles below it. QuALITY, our AM fit. Mean dir. :

u=\mu/\|\mu\|^{2}

instead of ( 6 ). Absolute values for the default caches are in Tables 2 and 13 .

		accuracy (%)				$\mathrm{KL}(\text{full}\,\\|\,\cdot)$
model	budget	keep	drop	AM	fold	keep	drop	AM	fold	$\Delta$ acc	$\Delta$ KL
Qwen3-1.7B	20%	50.4	45.7	48.1	46.6	1.74	3.00	2.07	2.41	$-$ 1.5	+0.34
										[ $-$ 4.4, +1.1]	[+0.13, +0.54]
	10%	48.3	45.3	43.1	47.2	2.82	3.62	2.57	2.88	+4.1	+0.31
										[+1.7, +6.8]	[+0.08, +0.53]
	5%	45.7	42.3	40.8	38.4	3.59	4.47	2.69	2.89	$-$ 2.4	+0.20

Appendix

Table 13: QuALITY under our AM fit, on all four models at 20% and 2% and on Qwen3-1.7B also at 10% and 5%: the fold’s KL is below AM’s beyond noise only on Qwen3-8B at 2%, above it on Qwen3-1.7B at 20% and 10%, and level elsewhere.

\Delta

: fold minus AM, with the 95% bootstrap interval over the 30 articles below it; accuracy in %,

\Delta

acc in points. 534 questions; full-cache accuracy 55.6, 69.1, 73.0 and 71.3% (55.6 here and 55.8 in Table 2 : run-to-run variation, Appendix A.10 ).

Qwen3	fit	budget	keep	AM	fold	$\Delta$ KL	$\Delta$ acc (points)
			$\mathrm{KL}(\text{full}\,\\|\,\cdot)$
1.7B	AM-style + SS	10%	3.78	2.11	2.26	+0.14 [ $-$ 0.00, +0.30]	+0.9 [ $-$ 0.7, +2.4]
		5%	4.97	2.44	2.65	+0.21 [+0.07, +0.37]	$-$ 0.9 [ $-$ 3.4, +1.7]
4B	AM-style	10%	2.10	2.40	2.38	$-$ 0.02 [ $-$ 0.20, +0.17]	$-$ 0.6 [ $-$ 2.5, +1.5]
		5%	2.88	3.20	3.21	+0.01 [ $-$ 0.16, +0.16]	+1.1 [ $-$ 0.8, +3.0]
8B	AM-style	10%	1.48	1.35	1.18	$-$ 0.16 [ $-$ 0.30, $-$ 0.03]	+0.4 [ $-$ 1.5, +2.2]

Appendix

Table 14: QuALITY under the AM-style fit, on Qwen3-4B and 8B and with self-study queries on 1.7B: the fold’s KL is lower than AM’s on 8B, at the edge of noise, level on 4B, and higher on 1.7B with self-study queries, beyond noise at 5%. SS (self-study) adds the queries of AM’s four self-study prompts to

Q

. Keep differs from Table 2 because the kept keys depend on the fit’s ranking and, with SS, on

Q

\Delta

: fold minus AM with 95% paired bootstrap interval over the 30 articles.

model	budget	keep	drop	AM	fold	keep	drop	AM	fold	$\Delta$ acc	$\Delta$ KL
		accuracy (%)				$\mathrm{KL}(\text{full}\,\\|\,\cdot)$
Qwen3-1.7B	10%	44.5	39.7	40.7	37.7	3.62	3.78	3.23	3.36	$-$ 3.0	+0.13
										[ $-$ 5.5, $-$ 0.5]	[ $-$ 0.07, +0.35]
	5%	35.9	35.2	32.4	34.4	4.95	4.72	4.04	4.25	+2.0	+0.21
										[ $-$ 1.0, +5.5]	[ $-$ 0.02, +0.46]
Qwen3-4B	10%	48.5	45.7	46.5	47.5	4.25	4.96	4.79	4.88	+1.0	+0.09

Appendix

Table 15: LongHealth under the AM-style fit (our fit: Table 3 ): AM’s KL is higher than under our fit at every size and budget, and its accuracy is at or below keep’s in every cell. Columns as in Table 13 ; intervals over the 20 patients, 398 questions. Full-cache accuracy 68.3, 77.4 and 80.4%.

		fold $-$ AM		drop $-$ fold
benchmark	model	10%	5%	10%	5%
QuALITY	Qwen3-1.7B	+0.7 [ $-$ 1.7, +3.3]	+0.6 [ $-$ 1.7, +3.1]	–	–
	Qwen3-4B	$-$ 1.3 [ $-$ 3.3, +0.7]	$-$ 1.1 [ $-$ 3.4, +1.1]	$-$ 3.6 [ $-$ 6.0, $-$ 1.1]	$-$ 3.0 [ $-$ 6.2, +0.0]
	Qwen3-8B	$-$ 0.7 [ $-$ 3.2, +1.7]	$-$ 0.6 [ $-$ 2.5, +1.3]	$-$ 0.6 [ $-$ 2.4, +1.1]	$-$ 1.3 [ $-$ 4.9, +2.1]
	Llama-3.1-8B	+0.6 [ $-$ 1.3, +2.6]	+0.0 [ $-$ 1.5, +1.5]	$-$ 0.4 [ $-$ 3.0, +2.4]	$-$ 0.7 [ $-$ 4.9, +3.0]
LongHealth	Qwen3-1.7B	$-$ 7.0 [ $-$ 12.2, $-$ 2.0]	$-$ 1.5 [ $-$ 4.8, +1.8]	+3.3 [ $-$ 0.5, +6.8]	$-$ 1.0 [ $-$ 5.1, +3.3]

Appendix

Table 16: Agreement with the full cache’s top option: paired differences in points, with 95% bootstrap intervals over contexts. AM fit as in Tables 2 and 3 ; drop was not run with the AM-style fit on QuALITY.

	fold		AM		stock run
model	acc (%)	same (%)	acc (%)	same (%)	fold $-$ AM KL	drop $-$ fold KL
Qwen3-1.7B	47.2 / 46.4	97.9	43.1 / 43.6	95.9	$+0.28$ [ $+0.06$ , $+0.49$ ]	$+0.71$ [ $+0.42$ , $+1.03$ ]
Qwen3-4B	62.7 / 63.3	98.7	63.7 / 63.9	99.6	$-0.01$ [ $-0.17$ , $+0.15$ ]	$+0.68$ [ $+0.44$ , $+0.96$ ]
Qwen3-8B	69.1 / 69.3	98.9	68.4 / 68.9	99.1	$-0.21$ [ $-0.39$ , $-0.05$ ]	$+0.59$ [ $+0.40$ , $+0.81$ ]
Llama-3.1-8B	67.6 / 67.2	99.4	66.1 / 66.5	99.1	$-0.00$ [ $-0.02$ , $+0.01$ ]	$+0.08$ [ $+0.04$ , $+0.12$ ]

Appendix

Table 17: QuALITY at 10%, our fit: the fp32 explicit-softmax path of the main text against stock PyTorch kernels in bf16 (flash without a bias, masked SDPA for AM). Accuracy and KL are fp32 / stock. Same: share of questions with the same top option in both runs. Last two columns: fold

-

AM KL and drop

-

fold KL in the stock run, with 95% bootstrap intervals.

$L=512$ , 1024, 2048, 4096 entries
	decode ( $q_{\text{len}}=1$ )		prefill ( $q_{\text{len}}=512$ )
	$B=1$	$B=8$	$B=1$	$B=8$
biased	1.06 (1.02–1.29)	1.58 (1.09–1.90)	1.27 (1.16–1.50)	1.26 (1.16–1.39)
Flex	1.00 (1.00–1.23)	1.29 (1.00–1.45)	1.16 (1.00–1.43)	1.16 (1.07–1.24)
Flex $+\beta$ / Flex	1.05 (1.00–1.29)	1.19 (1.09–1.33)	1.08 (1.06–1.31)	1.08 (1.06–1.30)
flash ( $\mu$ s)	13–22	19–92	52–344	130–1503

Appendix

Table 18: With launch cost removed by CUDA graphs, AM’s bias costs little in single-sequence decode and most at batch-8 decode. Top: 512–4096 cached entries, which covers our compacted caches. Bottom: 16k–32k. CUDA-graphed per-call times, one run per model. Cells: time over the fastest plain time (flash, cuDNN or FlexAttention) of the same model and setting, median over the four models and the

L

values of the panel, min–max in parentheses. Biased: fastest implementation that accepts the bias. Flex: FlexAttention without a bias. Flex

+\beta

/Flex: FlexAttention with the bias over FlexAttention without it. Flash: per-call time of plain flash SDPA in

\mu

s, min–max.

KV-Cache Compression

Key-value cache compression reduces the memory and bandwidth needed to retain attention states during long-context inference.

See all 269 papers

Minima-KV: Retention-Preserving KV Cache Compression with Mixed-Format Paged Attention
Aug 24, 2026Sergii Kozyrev, Davyd MaiborodaKV-Cache Compression LLM Inference Acceleration
Minima AI, Inc · Minima AI, Inc.
Adaptive Filtering of the KV Cache: Diagnosing and Correcting Structural-Role Bias in LLM Inference
Jul 14, 2026Soumil MandalKV Caching Long-Context Language Model Inference
ServiceNow
TriAttention: Efficient Long Reasoning with Trigonometric KV Compression
Apr 6, 2026Weian Mao, Xi Lin, Wei Huang +5Long-Context Reasoning in Language Models KV-Cache Compression
MIT · ZJU · NVIDIA

Aug 24, 2026J/K move · Enter open · S save

Minima-KV: Retention-Preserving KV Cache Compression with Mixed-Format Paged Attention

Sergii Kozyrev, Davyd Maiboroda

Minima AI, Inc · Minima AI, Inc.

KV-Cache Compression LLM Inference Acceleration Efficient Attention KV-Cache Quantization Mixed-Precision Quantization

Open paper AskPDF arXiv

The key-value (KV) cache is a primary capacity and bandwidth bottleneck in long-context LLM serving. We present Minima-KV, a retention-preserving hierarchy for mixed-format paged attention. Recent and protected Anchor pages remain in FP8, while older non-anchor pages move to packed TQ3; every live-request page remains addressable. Format-specific kernels compute partial attention states and combine them through a globally normalized online-softmax merge, enabling direct heterogeneous decode without a cache-sized dense shadow. Across separate, configuration-bound Qwen3.6-27B profiles on a single 96-GB NVIDIA RTX PRO 6000 Blackwell GPU, deployment accounting reports 18.3 KiB of attention KV per live token, corresponding to 3.50x compression relative to BF16 and 1.75x relative to FP8. A materializing quality profile matches its dense control on 16K RULER needle-in-a-haystack tasks. On the same 503-question LongBench v2 set, measured deltas are -0.80, -0.60, and -0.40 percentage points at 16K, 32K, and 64K. A separate single-pair direct-decode canary with two 59,008-token requests measures 3.625x active-KV compression and 0.9821x throughput relative to its control, routes all 16 full-attention layers without fallback, and retains no dense shadow. These results establish a practical mixed-format path for compressing long-context state without evicting live-request KV pages.

Abstract

KV-Cache Compression

Explore similar work

Minima-KV: Retention-Preserving KV Cache Compression with Mixed-Format Paged Attention

Adaptive Filtering of the KV Cache: Diagnosing and Correcting Structural-Role Bias in LLM Inference

TriAttention: Efficient Long Reasoning with Trigonometric KV Compression