Read What Matters: Query-Adaptive Quantization for KV Caches

Siddharth Bhandari; Lucas Gretta; Krishna Balasubramanian; Shiva Kasiviswanathan

cs.LGOct 8, 2026

Read What Matters: Query-Adaptive Quantization for KV Caches

Authors: Siddharth Bhandari, Lucas Gretta, Krishna Balasubramanian, Shiva Kasiviswanathan

Organizations: Amazon · University of California, Berkeley

Abstract

KV-cache entries are stored before their future queries are known, but each decoding query needs precision in different places. We study this mismatch using separate budgets for retained bits and bits fetched per query. ReadKV stores each key and value in a progressive code whose prefixes support different reconstruction precisions. For each query, it allocates key-channel prefixes using the query, computes attention from the reconstructed keys, and then allocates value-token prefixes using that attention. Stored entries remain unchanged. Each stage optimizes a calibrated distortion objective under a fixed budget; we prove exact allocation under diminishing refinement gains and relate these objectives to attention-output error. We also exhibit a finite-dimensional attention family where query-dependent access strictly outperforms every query-independent reader at the same read budget, even with unrestricted competing encoders and decoders. Across six base models, reading four bits on average from an eight-bit cache increases C4 perplexity by at most 0.66%, using about one quarter of the logical reads and half the retained capacity of a 16-bit cache. It is consistently more accurate than storing and fully reading four bits at the same payload-read budget. Retaining more bits than each query fetches is aimed at long-context decoding, where the cache bytes moved per step, rather than the weights, dominate cost. Long-context question answering and retrieval on two instruction-tuned models provide additional quality evidence. On the tested 8K-token, batch-one, single-layer workload on an NVIDIA A10G, a restricted eight-bit ReadKV reader with a two-bit mean payload-read budget has 39% lower latency than the tested TurboQuant codec.

Figures & tables

Panel of Figure 1 — Figure 1: Write-once, read-many KV-cache access. Token $i$ ’s key and value vectors are stored once and may be read by up to $T-i$ later queries. Fixed-precision quantization fetches the same full $W$ -bit code each time. ReadKV stores a progressive $W$ -bit code and lets each query fetch different prefixes under an average read budget $R<W$ ; unread refinements remain available.

Reader Class	Uses Query?	Use of Returned Bits in Address Selection
Query-oblivious fixed pattern	No	None; every address is fixed in advance
Query-independent branching	No	Each new address may use all previously returned bits
$\mathsf{staged}(1)$ : one batch	Yes	None; the complete batch is selected before reading
$\mathsf{staged}(J)$ , $J\geq 2$ : at most $J$ batches	Yes	A later batch may use bits returned by earlier batches
Fully branching	Yes	Each new address may use all previously returned bits

Table 1: Reader classes distinguished by whether address selection may use the current query or previously returned bits. All classes use the same storage and read budgets, and every final decoder may use the query.

Panel of Figure 2 — Figure 2: Query-time flow of ReadKV . The query selects key-prefix depths in the first batch. After the intermediate decoder reconstructs the keys, the resulting attention weights select value-prefix depths in the second batch. Both batches read from the same unchanged progressive $W$ -bit cache, and each batch’s addresses are fixed before any of its bits are returned.

Each method cell reports $\Delta\mathrm{PPL}$ (%) on top and logical reads / retained KV capacity (% of Dense ) below.
Panel A: Qwen2.5 Models
Method $(W,R)$	Qwen2.5 3B	Qwen2.5 7B	Qwen2.5 14B
Dense PPL	11.36	10.13	8.74
Full Reader $(4,4)$	+1.59 [1.5pt] 25.6 / 25.1	+1.34 [1.5pt] 25.4 / 25.1	+1.68 [1.5pt] 25.3 / 25.1
KIVI (2-bit K/V) [1pt] Liu et al.,2024	+4.84 [1.5pt] 26.2 / 23.8	+2.79 [1.5pt] 26.2 / 23.8	+2.21 [1.5pt] 26.2 / 23.8
SparQ $r32/k128$ [1pt] Ribar et al.,2024	+1.27 [1.5pt] 24.7 / 100.0	+1.44 [1.5pt] 24.7 / 100.0	+0.81 [1.5pt] 24.7 / 100.0

Table 2: C4: 32 held-out 2,048-token documents per model; three transform seeds where available. ReadKV

(8,4)

stays within 0.7% of Dense on all six models, using about a quarter of its logical reads and half its KV storage.

\dagger

: first/last two layers dense. Color marks the observed quality–read frontier: at shown precision, no other method row in the column improves loss or reads without worsening the other. Reads recur per query; retained capacity is shown but not ranked.

Each method cell reports $\Delta\mathrm{F1}$ (points) on top and logical reads / retained KV capacity (% of Dense ) below.
Panel A: Qwen2.5-7B-Instruct
Method $(W,R)$	Qasper	HotpotQA	2Wiki MultihopQA
Dense F1	41.13	48.34	45.30
KIVI (2-bit K/V) [1pt] Liu et al.,2024	+1.56 [1.5pt] 20.4 / 19.9	+0.08 [1.5pt] 20.0 / 19.9	+2.41 [1.5pt] 20.1 / 19.9
TurboQuant K4/V4 † [1pt] Zandieh et al.,2026	-0.06 [1.5pt] 36.7 / 36.7	+0.42 [1.5pt] 36.7 / 36.7	-0.55 [1.5pt] 36.7 / 36.7
SparQ $r32/k128$ [1pt] Ribar et al.,2024	+1.40 [1.5pt] 15.3 / 100.0	-0.44 [1.5pt] 14.2 / 100.0	+0.69 [1.5pt] 14.5 / 100.0

Table 3: LongBench QA at a 7,500-token prompt limit; 100 questions per condition. ReadKV

(8,2)

loses at most 1.46 F1 points across the six conditions while using about 13% of Dense logical reads.

\dagger

Figure 3: Comparison with representative baselines using Tables 2 and 3 . (a) On C4, ReadKV

(8,4)^{\dagger}

has lower PPL and slightly fewer logical reads than TurboQuant K4/V4 on all six models when the first and last two layers remain dense, while retaining more KV capacity. (b) On LongBench at a 7,500-token prompt limit, ReadKV

(8,2)

uses fewer logical reads than KIVI and SparQ. It has higher F1 than KIVI in all six conditions and mixed results against SparQ. Resource values are percentages of the 16-bit cache; lower quality loss is better.

Method	KV payload (MiB)	Latency (ms)	DRAM reads (MiB)	Total traffic (MiB)
Dense	16.00	0.051	17.73 / 17.86	20.85 / 21.06
TurboQuant K4/V4 ( Zandieh et al., 2026 )	4.19	0.154	7.28 / 7.92	9.38 / 10.84
ReadKV $(8,2)$	8.00	0.093	2.80 / 2.74	7.47 / 7.47

Table 4: Single-layer decoding on an NVIDIA A10G with 8,192 cached tokens and batch size one. Retained KV payload excludes temporary workspace. Latency is the median CUDA Graph replay time. DRAM columns report warm-cache / cache-evicted MiB from separately profiled eager calls and include the complete attention operation.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Domain or shape	Meaning
Partial-read model
$d$ ; $x,y$	$x,y\in\mathbb{B}_{2}^{d}$	Dimension and the stored and query vectors in the worst-case inner-product problem.
$W,R$ ; $w,r$	$w=\lfloor Wd\rfloor$ , $r=\lfloor Rd\rfloor$	Normalized storage and read rates, and their integer bit budgets.
$Z$	$\{0,1\}^{w}$	Stored binary record produced before the query is known.
$\theta$	Public state	Code and memory-layout description fixed independently of the stored vector and query.
$\Omega_{0},\Omega_{e},\Omega_{r}$	Random variables	Public, encoder-local, and reader-local randomness, all independent of $(x,y)$ .

Appendix

Table 5: Recurring notation. Dimensions are shown when the object is a vector, matrix, or array.

Method $(W,R)$	Qwen2.5 3B	Qwen2.5 7B	Qwen2.5 14B	Yi-1.5 6B	DeepSeek LLM-7B	Mistral 7B-v0.3
Dense PPL	11.36	10.13	8.74	8.96	8.49	7.62
Top: $\Delta\mathrm{PPL}$ (%)
Bottom: logical reads / retained KV capacity (% of Dense )
ReadKV $(8,2)^{\dagger}$	+3.05 [1.5pt]22.7 / 55.6	+1.93 [1.5pt]25.3 / 57.2	+2.37 [1.5pt]20.1 / 54.3	+7.82 [1.5pt]23.8 / 56.3	+4.39 [1.5pt]24.4 / 56.8	+8.39 [1.5pt]23.7 / 56.3
KIVI (4-bit K/V) [1pt] Liu et al.,2024	+0.07 [1.5pt]37.6 / 35.5	+0.09 [1.5pt]37.6 / 35.5	+0.06 [1.5pt]37.6 / 35.5	+0.07 [1.5pt]37.6 / 35.5	+0.10 [1.5pt]37.6 / 35.5	+0.09 [1.5pt]37.6 / 35.5
Lloyd-2 (norm-corrected) [1pt] Lloyd,1982	+40.99 [1.5pt]13.1 / 13.0	+33.83 [1.5pt]13.1 / 13.0	not run	+159.37 [1.5pt]13.1 / 13.0	+3766.75 [1.5pt]13.1 / 13.0	+1366.07 [1.5pt]13.1 / 13.0

Appendix

Table 6: Additional C4 language-modeling quality and KV costs on the held-out documents used in Table 2 .

\dagger

denotes configurations in which the first and last two layers remain uncompressed. The Qwen2.5-14B Lloyd-2 condition was not run. The low-read SparQ control uses

r16/k64

, except on Yi where it uses

r8/k64

. SparQ + nested code uses

r32/k128

	Qwen2.5-7B-Instruct			Mistral-7B-Instruct-v0.3
Method $(W,R)$	Qasper	HotpotQA	2Wiki MultihopQA	Qasper	HotpotQA	2Wiki MultihopQA
Dense F1	35.51	42.49	44.87	28.10	41.42	37.46
Top: $\Delta\mathrm{F1}$ (points) Color : quality–read frontier
Bottom: logical reads / retained KV capacity (% of Dense )
KIVI (2-bit K/V) [1pt] Liu et al.,2024	+0.51 [1.5pt]21.2 / 20.7	+2.59 [1.5pt]21.2 / 20.7	+0.90 [1.5pt]21.2 / 20.7	-0.39 [1.5pt]21.0 / 20.7	-0.17 [1.5pt]20.9 / 20.7	+0.32 [1.5pt]20.9 / 20.7
TurboQuant K4/V4 † [1pt] Zandieh et al.,2026	+1.14 [1.5pt]36.7 / 36.7	+1.81 [1.5pt]36.7 / 36.7	+1.00 [1.5pt]36.7 / 36.7	+1.01 [1.5pt]35.4 / 35.4	+0.17 [1.5pt]35.4 / 35.4	+0.87 [1.5pt]35.4 / 35.4

Appendix

Table 7: LongBench question-answering quality and KV costs at a 3,500-token prompt limit. The evaluation uses the same 100 questions and dense-prefill protocol as Table 3 .

\dagger

denotes configurations in which the first and last two layers remain uncompressed. Color marks the observed quality–read frontier: at shown precision, no other method row in the column improves loss or reads without worsening the other. Reads recur per query; retained capacity is shown but not ranked.

	Qwen2.5-7B-Instruct		Mistral-7B-Instruct-v0.3
Method $(W,R)$	4K	8K	4K	8K
Dense recovered	200	200	200	200
Top: recovered keys (out of 200) Color : quality–read frontier
Bottom: logical reads / retained KV capacity (% of Dense )
Full Reader $(4,4)$	200 [1.5pt]25.1 / 25.1	200 [1.5pt]25.1 / 25.0	200 [1.5pt]25.1 / 25.0	200 [1.5pt]25.0 / 25.0
ReadKV $(4,2)$	200 [1.5pt]12.6 / 25.1	200 [1.5pt]12.6 / 25.0	200 [1.5pt]12.6 / 25.0	200 [1.5pt]12.5 / 25.0

Appendix

Table 8: Passkey retrieval quality and KV costs at 4K and 8K context lengths, with 200 trials per condition. Every method uses dense prompt prefill.

\dagger

denotes configurations in which the first and last two layers remain uncompressed. SparQ + nested code uses

r32/k128

. ReadKV

(4,2)

recovers all keys with about half the logical reads of the matched-storage Full Reader

(4,4)

. Color marks the observed quality–read frontier: at shown precision, no other method row in the column improves recovered keys or reads without worsening the other (more recovered keys and fewer reads are better). Reads recur per query; retained capacity is shown but not ranked.

Allocation	Qwen3B	Qwen7B	Yi6B	DeepSeek7B
Uniform depth 2	21.66	19.91	39.34	568.35
Calibration-only	19.11	14.67	16.31	1 095.1
Current query	11.80	10.44	9.87	9.03

Appendix

Table 9: C4 perplexity under three allocation policies at mean payload depth

R=2

. Entries are arithmetic means over three transform seeds on the same 32 test documents. Calibration-only and current-query allocation retain the same

W=8

progressive code. Uniform reading retains a depth-two code. The policies use their development-selected K/V splits, so the table compares complete policies rather than isolating one factor.

Model	$R$	Equal K/V	Per head	Min. 1	Paired	Total
Qwen3B	2	+0.0729	+0.0050	+0.0012	+0.0004	+0.0795
Qwen7B	2	+0.0575	+0.0043	+0.0061	+0.0014	+0.0694
Qwen3B	4	+0.0032	+0.0002	+0.0004	-0.0003	+0.0035
Qwen7B	4	+0.0034	-0.0005	+0.0002	-0.0003	+0.0028

Appendix

Table 10: Effect of GPU-reader restrictions on C4 negative log likelihood. Entries give changes in nats per predicted token. Positive values are worse. The columns successively impose equal K/V rates, per-head budgets, minimum depth one, and paired values. The total compares the final restricted reader with the flexible reader. Because the restrictions are cumulative, each intermediate increment depends on their order.

	2K tokens		8K tokens		32K tokens
Method	$B=1$	$B=4$	$B=1$	$B=4$	$B=1$	$B=4$
Dense	0.020	0.048	0.051	0.146	0.168	0.585
TurboQuant K4/V4 codec [-1pt] Zandieh et al.,2026	0.055	0.143	0.154	0.518	0.625	1.990
Full Reader $(4,4)$	0.038	0.076	0.070	0.171	0.265	0.727
ReadKV $(4,2)$	0.063	0.095	0.092	0.179	0.287	0.719
ReadKV $(8,2)$	0.066	0.096	0.093	0.183	0.304	0.740

Appendix

Table 11: Median single-layer decode latency (ms) on an NVIDIA A10G under CUDA Graph replay with a warm cache. The 2K and 8K inputs use Qwen2.5-7B layer 0; the 32K input uses synthetic K/V with the same head geometry. All rows use the same timing boundary, including ReadKV ’s final bf16 conversion.

Method	DRAM reads	DRAM writes	Total
Dense	17.73 / 17.86	3.12 / 3.19	20.85 / 21.06
ReadKV $(4,2)$	2.72 / 2.71	4.63 / 4.71	7.35 / 7.42
Full Reader $(4,4)$	4.70 / 4.66	4.19 / 4.25	8.90 / 8.91
ReadKV $(4,4)$ , scheduled	4.81 / 4.85	4.64 / 4.73	9.44 / 9.58
ReadKV $(8,2)$	2.80 / 2.74	4.67 / 4.73	7.47 / 7.47
ReadKV $(8,4)$	4.99 / 4.98	4.72 / 4.75	9.71 / 9.74

Appendix

Table 12: Physical DRAM traffic for the same NVIDIA A10G single-layer benchmark used in the latency study, with 8,192 cached tokens and batch one. Each entry reports warm / cache-evicted traffic in MiB. Counters cover the complete eager attention call, including intermediate and output accesses. Graph-replay latency is measured separately.

Explore similar work

Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms
Aug 4, 2026Samuel Fernández-Menduiña, Amir Ziashahabi, Eduardo Pavez +2KV Caching Vector Quantization
Department of Electrical and Computer Engineering, University of Southern California
RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cache
May 8, 2026Junkai Zhang, Hang Guo, Luca Benini +1KV Caching Long-Context Language Model Inference
ETH Zurich · Tsinghua University
Dual-QK: Sharp Queries and Flat Keys for Prunable 2-bit KV Caches
Oct 7, 2026Sunjoo Whang, Jungjun Oh, Minsung Kim +5KV-Cache Compression KV-Cache Quantization
KAIST · GIST · Google Research

Aug 4, 2026, cs.LGJ/K move · Enter open · S save

Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms

Samuel Fernández-Menduiña, Amir Ziashahabi, Eduardo Pavez, Antonio Ortega+1

Department of Electrical and Computer Engineering, University of Southern California

KV Caching Vector Quantization KV-Cache Quantization Low-Bit Quantization

Open paper AskPDF arXiv

Long-context LLM decoding reads the key-value (KV) cache at every step. Loading it takes longer than computing attention over it, so throughput is bandwidth-bound. Hence, reducing the cache size can raise both decoding speed and serving capacity. The challenge is to reduce cache size while preserving the attention products, keeping reconstruction cheap, and using a fixed per-token bit count. At two bits per element, the most competitive methods rely on orthogonal transforms. However, existing techniques are either data-oblivious or use the query statistics without deriving the transform from a distortion criterion. Moreover, they rely on transforms built on top of random or Hadamard rotations, which equalize variances across entries rather than compacting energy, and fixed-width scalar quantizers, which are suboptimal at low rates. In this paper, we formulate KV cache quantization as a transform coding problem in which distortion is the error in the attention products. We derive closed-form optimal transforms for keys and values from calibration statistics, under a high-resolution model. We show that the optimal key transform is not orthogonal and satisfies a generalized Parseval relation: the attention-aware distortion becomes mean-squared error (MSE) in the transform domain. Thus, we can use MSE-optimal vector quantizers applied directly to the transformed key coefficients. To meet the fixed-width layout requirement, we show that grouping coefficients into equal-volume partitions makes equal-size codebooks attain the variable-rate optimum under the same high-resolution model. At two bits per element, our method, termed NOVA-KV, recovers most of the long-context retrieval accuracy lost by scalar quantization methods at comparable throughput.

Abstract

Explore similar work

Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms

RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cache

Dual-QK: Sharp Queries and Flat Keys for Prunable 2-bit KV Caches