1-bit key-value (KV) cache compression can combine quantizer calibration with selective high-precision storage to reduce reconstruction error. We propose Calibration and Fisher-guided Anchors for KV cache compression (CALFA-KV). Offline, we jointly fit a key center, radial normalization parameters, and K/V vector codebooks to match the frozen model's projected attention output. We also estimate channel-wise Fisher-type sensitivities from squared gradients of a reference next-token cross-entropy loss. Online, we use these sensitivities to weight the squared reconstruction errors of each KV state. Here, a KV state is one token's key-value pair within one layer and KV head. We retain the highest-scoring states at high precision under a per-layer, per-KV-head budget. Under exact restoration, we bound the excess risk of score-based anchor selection relative to the optimum at the same anchor budget, where risk is the conditional mean square of the first-order loss change. With a budget of 56 additional FP16 anchors per layer and KV head, CALFA-KV (d8b8) outperforms all evaluated nominal 1-bit baselines on the mean score across seven benchmarks for each of the three models. At the same anchor budget, CALFA-KV (d8b10) uses a nominal index rate of 1.25 bits per channel and narrows the gap to BF16 to 0.90--2.01 points on the same metric.
Figures & tables
Fig. 1: CALFA-KV for 1-bit KV cache compression. (a) Offline joint calibration proposes updates with fixed codeword assignments and tests them using deployment-precision replay after re-encoding. Channel sensitivities are estimated separately from reference-loss gradients. (b) During inference, sensitivity-weighted reconstruction errors rank historical KV states for FP16 retention, with up to N anchors per layer and KV head. Attention combines decoded history, restored anchors, and unquantized recent and live K/V. The nominal 1-bit rate refers to codeword indices.
Subset
Questions
Primary score
Output limit (tokens)
GSM8K
1,319
Numeric exact match with flexible answer extraction
8,192
IFEval
541
Strict prompt-level accuracy: all constraints in a prompt must pass
1,280
HumanEval+
164
pass@1 on extended tests
1,024
LiveCodeBench V5+V6
342 (167+175)
Official fast/lite code-generation pass@1
2,000
LongBench
4,750
Equal mean of 21 native subtask scores
32/64/128/512
MATH500
500
Mathematical answer equivalence using Math-Verify
8,192
TABLE 1: Benchmark subsets and evaluation settings.
Method
BPE
GSM8K
IFEval
HumanEval+
LiveCodeBench V5+V6
LongBench
MATH500
GPQA- Diamond
Mean score
BF16
16.000
92.34
81.15
69.51
28.36
42.57
92.60
62.12
66.95
VecInfer
1.000
1.06
34.20
21.95
0.00
24.05
1.60
13.64
13.79
KIVI
2.000
0.45
47.87
17.07
0.00
8.43
3.40
9.09
12.33
OSCAR
1.125
71.19
77.82
65.85
5.26
21.13
27.20
14.65
40.44
TaSQ
1.008
88.48
70.98
63.41
18.71
39.87
83.20
52.53
59.60
CALFA-KV (d8b10)
1.258
89.76
83.73
69.51
27.77
41.72
90.80
59.09
66.05
TABLE 2: Benchmark scores for Qwen3-4B-Instruct-2507
Method
GSM8K
IFEval
HumanEval+
LiveCodeBench V5+V6
LongBench
MATH500
GPQA- Diamond
Mean score
BF16
79.76
74.68
57.93
14.33
46.88
47.40
30.81
50.26
VecInfer
11.14
42.51
27.44
2.92
36.20
11.80
15.15
21.02
KIVI
2.05
40.30
18.29
0.00
10.53
1.60
11.62
12.06
OSCAR
76.80
68.95
56.71
7.60
21.32
37.60
13.13
40.30
TaSQ
73.01
50.65
46.34
7.89
43.55
39.20
22.22
40.41
CALFA-KV (d8b10)
79.45
77.08
54.88
14.33
46.01
45.20
25.76
48.96
TABLE 3: Benchmark scores for Llama-3.1-8B-Instruct
Method
GSM8K
IFEval
HumanEval+
LiveCodeBench V5+V6
LongBench
MATH500
GPQA- Diamond
Mean score
BF16
95.83
82.81
71.34
35.67
47.05
96.40
68.69
71.11
VecInfer
2.50
35.30
31.10
1.17
28.84
1.80
15.66
16.62
KIVI
0.08
47.32
20.12
0.00
9.14
2.40
14.65
13.39
OSCAR
59.74
74.68
68.29
6.43
22.44
24.40
11.62
38.23
TaSQ
93.18
69.69
65.24
22.22
43.33
87.40
55.05
62.30
CALFA-KV (d8b10)
95.45
81.52
68.90
33.92
46.46
93.80
63.64
69.10
TABLE 4: Benchmark scores for Qwen3-30B-A3B-Instruct-2507
Fig. 2: Layerwise errors relative to a BF16 reference on Qwen3-4B-Instruct-2507 with a fixed, jointly calibrated d8b8 quantizer. Panels show (a) attention-distribution KL divergence (nats), (b) relative MSE of the attention output after WO , and (c) hidden-state relative MSE. Relative MSE is the sum of squared differences divided by the reference signal energy. No anchors uses zero additional anchors; Random, Unweighted error, and Fisher-guided (ours) each use a budget of 56 additional K/V pairs per layer and KV head. Curves average 24 prompts, with Random averaged over three seeds. Shaded bands show 95% prompt-bootstrap confidence intervals. Lower is better.
Configuration
GSM8K
IFEval
HumanEval+
LiveCodeBench V5+V6
LongBench
MATH500
GPQA- Diamond
Mean score
BF16
92.34
81.15
69.51
28.36
42.57
92.60
62.12
66.95
JC
88.78
76.15
65.85
21.05
41.19
87.80
57.07
62.56
JC + Random
88.93
78.92
66.46
22.51
30.00
88.20
58.58
61.94
JC + Fisher-guided
89.84
82.99
67.07
22.80
41.15
89.80
54.54
64.03
TABLE 5: Anchor selection ablation on Qwen3-4B-Instruct-2507
What limits KV-cache compression at extreme bit-rates? We argue that it is not the choice of compression scheme, but how its budget is allocated across attention heads. Existing methods apply rank and bit-width uniformly, ignoring that each head has a different optimal mix of rank truncation and quantization. We show that co-optimizing rank and bit-width per head, using only standard low-rank projection and scalar quantization, dominates uniform allocation, with the largest gains at low bit-rates. Our method, KV-COBRA (Co-Optimized Bit-Rank Allocation), formalizes this as a resource-allocation problem: it balances rank-truncation loss against quantization loss within each head, then redistributes budget across heads to minimize total distortion. A fused Hadamard rotation equalizes per-channel variance, and reordering the SVD basis by attention-KL importance makes the solver query-aware. The same allocator extends to joint K+V compression. On perplexity, zero-shot, and long-context benchmarks from 0.5 to 4 bits per dimension (bpd), KV-COBRA shows the smallest accuracy degradation among evaluated methods at low bpd, with no per-token overhead.
Sihyeon Ha, Jaeho Lee, Yo-Seb Jeon
Pohang University of Science and Technology (POSTECH)
The key-value (KV) cache is the primary memory bottleneck in long-context LLM inference. Existing approaches attack it from opposite ends: eviction methods permanently discard tokens, degrading performance whenever a discarded token later proves essential, while quantization methods retain all tokens at low precision but offer limited compression. We propose AnchorKV, a compression scheme that shrinks the cache by 20× without discarding a single token. AnchorKV represents the cache using a small set of anchors stored exactly, expresses every other token through its most similar anchor, and refines only those whose approximation most affects the model's output. AnchorKV consistently preserves accuracy across models and datasets, retaining 99% of the full-cache score at the 70B scale, while keeping the entire context at a fraction of its cost.
Malik Khalaf, Yara Shamshoum, Nitzan Hodos +2
Department of Computer Science, Technion - Israel Institute of Technology
The key-value (KV) cache becomes a major memory bottleneck in long-context LLM inference, placing substantial pressure on memory capacity and bandwidth. To mitigate this bottleneck, vector quantization (VQ) has emerged as a promising approach for aggressive KV cache compression. However, existing VQ methods degrade substantially in the 1-bit regime. At such extreme compression, each codebook must represent a larger group of channels with a limited set of centroids, making effective use of its capacity increasingly challenging. To address this, we introduce TaSQ, which tailors the VQ target space by combining query-guided channel weighting, cross-head normalization, and covariance-aware channel grouping to better reflect the error sensitivity and statistical structure of cached activations. Since these transforms are RoPE-compatible and can be easily merged into projection weights and codebooks, TaSQ preserves the conventional VQ lookup structure and adds negligible serving overhead. Across general, long-chain-of-thought reasoning, and long-context retrieval benchmarks, TaSQ consistently outperforms existing low-bit KV cache VQ baselines while preserving reasoning stability. On a single RTX 6000 Ada GPU, its SGLang implementation supports up to 14× larger batch sizes and achieves 1.87× higher peak throughput compared to the BF16 baseline.