1-bit key-value (KV) cache compression can combine quantizer calibration with selective high-precision storage to reduce reconstruction error. We propose Calibration and Fisher-guided Anchors for KV cache compression (CALFA-KV). Offline, we jointly fit a key center, radial normalization parameters, and K/V vector codebooks to match the frozen model's projected attention output. We also estimate channel-wise Fisher-type sensitivities from squared gradients of a reference next-token cross-entropy loss. Online, we use these sensitivities to weight the squared reconstruction errors of each KV state. Here, a KV state is one token's key-value pair within one layer and KV head. We retain the highest-scoring states at high precision under a per-layer, per-KV-head budget. Under exact restoration, we bound the excess risk of score-based anchor selection relative to the optimum at the same anchor budget, where risk is the conditional mean square of the first-order loss change. With a budget of 56 additional FP16 anchors per layer and KV head, CALFA-KV (d8b8) outperforms all evaluated nominal 1-bit baselines on the mean score across seven benchmarks for each of the three models. At the same anchor budget, CALFA-KV (d8b10) uses a nominal index rate of 1.25 bits per channel and narrows the gap to BF16 to 0.90--2.01 points on the same metric.
Figures & tables
Fig. 1: CALFA-KV for 1-bit KV cache compression. (a) Offline joint calibration proposes updates with fixed codeword assignments and tests them using deployment-precision replay after re-encoding. Channel sensitivities are estimated separately from reference-loss gradients. (b) During inference, sensitivity-weighted reconstruction errors rank historical KV states for FP16 retention, with up to N anchors per layer and KV head. Attention combines decoded history, restored anchors, and unquantized recent and live K/V. The nominal 1-bit rate refers to codeword indices.
Subset
Questions
Primary score
Output limit (tokens)
GSM8K
1,319
Numeric exact match with flexible answer extraction
8,192
IFEval
541
Strict prompt-level accuracy: all constraints in a prompt must pass
1,280
HumanEval+
164
pass@1 on extended tests
1,024
LiveCodeBench V5+V6
342 (167+175)
Official fast/lite code-generation pass@1
2,000
LongBench
4,750
Equal mean of 21 native subtask scores
32/64/128/512
MATH500
500
Mathematical answer equivalence using Math-Verify
8,192
TABLE 1: Benchmark subsets and evaluation settings.
Method
BPE
GSM8K
IFEval
HumanEval+
LiveCodeBench V5+V6
LongBench
MATH500
GPQA- Diamond
Mean score
BF16
16.000
92.34
81.15
69.51
28.36
42.57
92.60
62.12
66.95
VecInfer
1.000
1.06
34.20
21.95
0.00
24.05
1.60
13.64
13.79
KIVI
2.000
0.45
47.87
17.07
0.00
8.43
3.40
9.09
12.33
OSCAR
1.125
71.19
77.82
65.85
5.26
21.13
27.20
14.65
40.44
TaSQ
1.008
88.48
70.98
63.41
18.71
39.87
83.20
52.53
59.60
CALFA-KV (d8b10)
1.258
89.76
83.73
69.51
27.77
41.72
90.80
59.09
66.05
TABLE 2: Benchmark scores for Qwen3-4B-Instruct-2507
Method
GSM8K
IFEval
HumanEval+
LiveCodeBench V5+V6
LongBench
MATH500
GPQA- Diamond
Mean score
BF16
79.76
74.68
57.93
14.33
46.88
47.40
30.81
50.26
VecInfer
11.14
42.51
27.44
2.92
36.20
11.80
15.15
21.02
KIVI
2.05
40.30
18.29
0.00
10.53
1.60
11.62
12.06
OSCAR
76.80
68.95
56.71
7.60
21.32
37.60
13.13
40.30
TaSQ
73.01
50.65
46.34
7.89
43.55
39.20
22.22
40.41
CALFA-KV (d8b10)
79.45
77.08
54.88
14.33
46.01
45.20
25.76
48.96
TABLE 3: Benchmark scores for Llama-3.1-8B-Instruct
Method
GSM8K
IFEval
HumanEval+
LiveCodeBench V5+V6
LongBench
MATH500
GPQA- Diamond
Mean score
BF16
95.83
82.81
71.34
35.67
47.05
96.40
68.69
71.11
VecInfer
2.50
35.30
31.10
1.17
28.84
1.80
15.66
16.62
KIVI
0.08
47.32
20.12
0.00
9.14
2.40
14.65
13.39
OSCAR
59.74
74.68
68.29
6.43
22.44
24.40
11.62
38.23
TaSQ
93.18
69.69
65.24
22.22
43.33
87.40
55.05
62.30
CALFA-KV (d8b10)
95.45
81.52
68.90
33.92
46.46
93.80
63.64
69.10
TABLE 4: Benchmark scores for Qwen3-30B-A3B-Instruct-2507
Fig. 2: Layerwise errors relative to a BF16 reference on Qwen3-4B-Instruct-2507 with a fixed, jointly calibrated d8b8 quantizer. Panels show (a) attention-distribution KL divergence (nats), (b) relative MSE of the attention output after WO , and (c) hidden-state relative MSE. Relative MSE is the sum of squared differences divided by the reference signal energy. No anchors uses zero additional anchors; Random, Unweighted error, and Fisher-guided (ours) each use a budget of 56 additional K/V pairs per layer and KV head. Curves average 24 prompts, with Random averaged over three seeds. Shaded bands show 95% prompt-bootstrap confidence intervals. Lower is better.
Configuration
GSM8K
IFEval
HumanEval+
LiveCodeBench V5+V6
LongBench
MATH500
GPQA- Diamond
Mean score
BF16
92.34
81.15
69.51
28.36
42.57
92.60
62.12
66.95
JC
88.78
76.15
65.85
21.05
41.19
87.80
57.07
62.56
JC + Random
88.93
78.92
66.46
22.51
30.00
88.20
58.58
61.94
JC + Fisher-guided
89.84
82.99
67.07
22.80
41.15
89.80
54.54
64.03
TABLE 5: Anchor selection ablation on Qwen3-4B-Instruct-2507