The key-value (KV) cache is the dominant memory bottleneck in long-context large language model (LLM) decoding: every step reads it entirely, so decoding is memory-bandwidth bound. Holding a quantized KV cache in dense on-chip non-volatile memory (NVM) removes the off-chip transfer. Existing KV quantization methods, however, were designed for GPU-style memory systems: KIVI attaches per-group metadata, adding about 25% to the stored KV cache; KVQuant keeps sparse full-precision outliers that a dense array cannot hold in place. This paper examines what these structures cost when the KV cache resides in NVM behind fixed-range converters, and designs a quantization scheme matched to that interface. The architecture stores the quantized KV cache in dense on-chip NVM, uses a small static analog crossbar only for the fixed rotation, and keeps attention in on-chip digital logic. A randomized rotation and per-vector normalization give every coordinate the same range, so one fixed codebook for keys and one for values, each shared across all tokens of the corresponding tensor type, serve the entire KV cache. The codebook thresholds are programmed once as the read converter's reference levels, enabling fixed-range digitization with no per-token converter reconfiguration. Dequantization is a sixteen-entry lookup and one norm multiply; the only per-vector metadata is one scalar, about 3%. Across models from 3B to 14B and contexts to 32k tokens, the four-bit KV cache maintains accuracy under storage and crossbar noise simulated at realistic device levels. KIVI and KVQuant remain more accurate in software; the advantage of our format lies at the memory interface: 3.1-3.6x lower KV read energy than both mapped to the same NVM, and 8x lower metadata overhead than KIVI. The contribution is a KV quantization co-designed with the NVM memory interface rather than a new accuracy record.
Figures & tables
Figure 1. Three ways to hold the KV cache. (a) GPU-style system: the KV cache and its per-group metadata (red; about 25% for KIVI) reside in off-chip DRAM and cross the off-chip boundary at every decoding step. (b) Prior CiM attention accelerators, drawn after the gain-cell design of Leroux et al. ( Leroux et al., 2025 ) (per-design variants in Table 1 ): keys and values live in volatile compute cells with a fixed window, one column rewritten at every token (dashed). (c) This work: dense NVM holds write-once 4-bit indices, read through sense-amplifier comparators (SA) whose references are the fixed codebook thresholds, then a sixteen-entry lookup and one norm multiply; a small static crossbar applies the rotation, and attention stays digital. Purple marks the KV cells, orange the analog crossbar, blue the digital logic; D/A blocks are converters; the bottom strip lists per-step costs.
Leroux ’25 ( Leroux et al., 2025 )
X-Former ( Sridharan et al., 2023 )
HARDSEA ( Liu et al., 2024a )
Ours
KV medium
gain cell
none ‡
SRAM
dense NVM
Non-volatile
no
–
no
yes
Attention
analog
analog
digital †
digital
KV write rate
every step
–
once/token
once/token
KV bits
3 (QAT)
8
8
4 + norm
Table 1. Existing CiM attention accelerators versus this work.
Figure 2. Proposed data path: one static write-once crossbar holds the rotation H (the only analog matrix-vector product); keys and values return from the NVM KV cache as codebook indices, looked up and rescaled by their stored norm; attention runs digitally in rotated space with the un-rotation folded into WO′ .
ARC-c ↑
WikiPPL ↓
RULER-MV † ↑
LongBench ‡ ↑
SAMSum ‡ ↑
Method
3B
8B
14B
3B
8B
14B
3B
8B
14B
3B
8B
14B
3B
8B
14B
half (ours)
43.3
52.4
69.1
12.5
7.9
6.3
80.4
98.1
94.9
27.2
15.7
11.7
26.5
29.1
29.8
theo (ours)
36.2
47.4
54.7
15.4
21.0
52.9
66.9
56.0
11.1
22.8
14.3
10.3
21.8
9.7
24.3
cali (ours)
42.5
53.3
63.2
12.5
7.9
9.8
81.8
98.8
73.9
26.5
15.6
10.5
26.4
29.7
27.1
KIVI
46.3
56.4
69.3
11.1
7.4
5.9
89.0
100
98.5
27.2
20.1
12.9
27.4
27.8
31.3
KVQuant
44.3
55.0
68.1
12.2
7.7
6.3
86.5
99.1
96.5
26.0
19.9
12.4
27.8
29.4
29.6
Table 2. Four-bit KV quantization at σ=0 across three model scales: ARC-c (acc ↑ ), WikiText perplexity ( ↓ ), RULER-MV (acc ↑ ), LongBench (avg F1 ↑ ), and SAMSum (ROUGE-L ↑ ). Half (data-free keys, calibrated values) is ours; theo / cali are its fully data-free and fully calibrated variants; on Qwen2.5-14B, half removes the projections’ bias (Section 3.5 ).
Figure 3. KV storage noise: multi-value retrieval by context length under σ with the rotation ideal (mean ± std, three runs). Accuracy stays stable up to 3% of full scale (FS, shaded).
Method
Meta- data
Eff. bits /coord
Read path
KV read energy (norm.)
Ours
3.1%
4.1
LUT + 1 mul
1.00 ×
KIVI
25%
5.0
fetch + d MAC
3.64 ×
KVQuant
7.1%
4.3
LUT + gather + MAC
3.11 ×
Table 3. Memory-interface cost of four-bit KV formats on the same NVM substrate at d=128 . Effective bits include each format’s metadata; KV read energy (Section 4.3 ) is per coordinate, normalized to ours; breakdown in the text.
Figure 4. Static rotation crossbar under realistic device noise (2% conductance, 6-bit DAC, 2% accumulation) versus the ideal rotation (mean ± std, three runs): the crossbar tracks the ideal.
Direct (fp16)
Ours (4-bit)
Reduction
KV per token
112 KB
28.9 KB
3.88×
KV at 8k context
939 MB
242 MB
3.88×
On-chip KV area
140 mm 2
62 mm 2
2.2×
KV read energy/step
0.16 mJ
0.041 mJ
3.9×
Table 4. Full-precision (fp16) versus four-bit on-chip KV storage at 8k context on the 3B model, same NVM and NeuroSim 22nm constants. Our area includes the 26 mm 2 norm SRAM; KV read energy per step follows Section 4.3 .