cs.ARSep 4, 2026

Interface-Aware KV Cache Quantization for Dense On-Chip NVM in Long-Context LLM Decoding

Authors: Jiahao Zheng, Yifan Qin, Xiaobo Sharon Hu, Yiyu Shi

Organizations: University of Notre Dame, Notre Dame, IN, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

The key-value (KV) cache is the dominant memory bottleneck in long-context large language model (LLM) decoding: every step reads it entirely, so decoding is memory-bandwidth bound. Holding a quantized KV cache in dense on-chip non-volatile memory (NVM) removes the off-chip transfer. Existing KV quantization methods, however, were designed for GPU-style memory systems: KIVI attaches per-group metadata, adding about 25% to the stored KV cache; KVQuant keeps sparse full-precision outliers that a dense array cannot hold in place. This paper examines what these structures cost when the KV cache resides in NVM behind fixed-range converters, and designs a quantization scheme matched to that interface. The architecture stores the quantized KV cache in dense on-chip NVM, uses a small static analog crossbar only for the fixed rotation, and keeps attention in on-chip digital logic. A randomized rotation and per-vector normalization give every coordinate the same range, so one fixed codebook for keys and one for values, each shared across all tokens of the corresponding tensor type, serve the entire KV cache. The codebook thresholds are programmed once as the read converter's reference levels, enabling fixed-range digitization with no per-token converter reconfiguration. Dequantization is a sixteen-entry lookup and one norm multiply; the only per-vector metadata is one scalar, about 3%. Across models from 3B to 14B and contexts to 32k tokens, the four-bit KV cache maintains accuracy under storage and crossbar noise simulated at realistic device levels. KIVI and KVQuant remain more accurate in software; the advantage of our format lies at the memory interface: 3.1-3.6x lower KV read energy than both mapped to the same NVM, and 8x lower metadata overhead than KIVI. The contribution is a KV quantization co-designed with the NVM memory interface rather than a new accuracy record.

Figures & tables

Figure 1. Three ways to hold the KV cache. (a) GPU-style system: the KV cache and its per-group metadata (red; about 25% for KIVI) reside in off-chip DRAM and cross the off-chip boundary at every decoding step. (b) Prior CiM attention accelerators, drawn after the gain-cell design of Leroux et al. ( Leroux et al., 2025 ) (per-design variants in Table 1 ): keys and values live in volatile compute cells with a fixed window, one column rewritten at every token (dashed). (c) This work: dense NVM holds write-once 4-bit indices, read through sense-amplifier comparators (SA) whose references are the fixed codebook thresholds, then a sixteen-entry lookup and one norm multiply; a small static crossbar applies the rotation, and attention stays digital. Purple marks the KV cells, orange the analog crossbar, blue the digital logic; D/A blocks are converters; the bottom strip lists per-step costs.
 Leroux ’25 ( Leroux et al., 2025 )X-Former ( Sridharan et al., 2023 )HARDSEA ( Liu et al., 2024a )Ours
KV mediumgain cellnone ‡SRAMdense NVM
Non-volatileno–noyes
Attentionanaloganalogdigital †digital
KV write rateevery step–once/tokenonce/token
KV bits3 (QAT)884 + norm
Table 1. Existing CiM attention accelerators versus this work.
Figure 2. Proposed data path: one static write-once crossbar holds the rotation H\mathbf{H} (the only analog matrix-vector product); keys and values return from the NVM KV cache as codebook indices, looked up and rescaled by their stored norm; attention runs digitally in rotated space with the un-rotation folded into WO′\mathbf{W}_{O}^{\prime} .
 ARC-c ↑\uparrow WikiPPL ↓\downarrow RULER-MV † ↑\uparrow LongBench ‡ ↑\uparrow SAMSum ‡ ↑\uparrow
Method3B8B14B 3B8B14B 3B8B14B 3B8B14B 3B8B14B
half (ours)43.352.469.1 12.57.96.3 80.498.194.9 27.215.711.7 26.529.129.8
theo (ours)36.247.454.7 15.421.052.9 66.956.011.1 22.814.310.3 21.89.724.3
cali (ours)42.553.363.2 12.57.99.8 81.898.873.9 26.515.610.5 26.429.727.1
KIVI46.356.469.3 11.17.45.9 89.010098.5 27.220.112.9 27.427.831.3
KVQuant44.355.068.1 12.27.76.3 86.599.196.5 26.019.912.4 27.829.429.6
Table 2. Four-bit KV quantization at σ=0\sigma{=}0 across three model scales: ARC-c (acc ↑\uparrow ), WikiText perplexity ( ↓\downarrow ), RULER-MV (acc ↑\uparrow ), LongBench (avg F1 ↑\uparrow ), and SAMSum (ROUGE-L ↑\uparrow ). Half (data-free keys, calibrated values) is ours; theo / cali are its fully data-free and fully calibrated variants; on Qwen2.5-14B, half removes the projections’ bias (Section 3.5 ).
Figure 3. KV storage noise: multi-value retrieval by context length under σ\sigma with the rotation ideal (mean ±\pm std, three runs). Accuracy stays stable up to 3%3\% of full scale (FS, shaded).
MethodMeta- dataEff. bits /coordRead pathKV read energy (norm.)
Ours3.1%4.1LUT + 1 mul1.00 ×\times
KIVI25%5.0fetch + dd MAC3.64 ×\times
KVQuant7.1%4.3LUT + gather + MAC3.11 ×\times
Table 3. Memory-interface cost of four-bit KV formats on the same NVM substrate at d=128d=128 . Effective bits include each format’s metadata; KV read energy (Section 4.3 ) is per coordinate, normalized to ours; breakdown in the text.
Figure 4. Static rotation crossbar under realistic device noise (2% conductance, 6-bit DAC, 2% accumulation) versus the ideal rotation (mean ±\pm std, three runs): the crossbar tracks the ideal.
 Direct (fp16)Ours (4-bit)Reduction
KV per token112 KB28.9 KB3.88×3.88\times
KV at 8k context939 MB242 MB3.88×3.88\times
On-chip KV area140 mm 262 mm 22.2×2.2\times
KV read energy/step0.16 mJ0.041 mJ3.9×3.9\times
Table 4. Full-precision (fp16) versus four-bit on-chip KV storage at 8k context on the 3B model, same NVM and NeuroSim 22nm constants. Our area includes the 26 mm 2 norm SRAM; KV read energy per step follows Section 4.3 .
More in this topic

Hardware-Aware Quantization

See all 17 papers

Explore similar work

CardsList
May 8, 2026

RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cache

Large language models (LLMs) have shown strong performance across diverse tasks, but their inference with long input contexts is bottlenecked by memory size and bandwidth. The Key-Value (KV) cache size grows linearly with sequence length and needs to be re-read from off-chip high-bandwidth memory (HBM) to on-chip memory at every decoding step, resulting in memory-bound inference. Existing methods reduce the cache by either eviction or quantization, but typically treat the two in isolation. In this paper, we cast KV cache compression as a rate-distortion problem, under which eviction and quantization are two end-points of the same bit allocation scheme. This exposes the need to optimize them jointly, motivating our method, RDKV (Rate-Distortion KV cache compression). RDKV derives the weight of each token or channel from the distortion that compression induces on the attention computation. Based on these weights, it assigns each token or channel a bit-width ranging from full precision down to zero bits guided by reverse water-filling, applied once after the prefilling stage. Experiments on LongBench, RULER, and InfiniteBench show that RDKV outperforms the best evaluated baseline by 9.1% on average. On LongBench it recovers 97.81% of full-cache accuracy with only 2.48% cache retention. Compared with full-cache FlashAttention-2 decoding, it achieves 4.5x decode speedup and 1.9x peak memory reduction with 128K context length, while maintaining comparable performance.

Junkai Zhang, Hang Guo, Luca Benini +1

ETH Zurich · Tsinghua University

KV CachingLong-Context Language Model InferenceKV-Cache EvictionRate-Distortion OptimizationKV-Cache CompressionKV-Cache Quantization
AskPDF
Aug 4, 2026

Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms

Long-context LLM decoding reads the key-value (KV) cache at every step. Loading it takes longer than computing attention over it, so throughput is bandwidth-bound. Hence, reducing the cache size can raise both decoding speed and serving capacity. The challenge is to reduce cache size while preserving the attention products, keeping reconstruction cheap, and using a fixed per-token bit count. At two bits per element, the most competitive methods rely on orthogonal transforms. However, existing techniques are either data-oblivious or use the query statistics without deriving the transform from a distortion criterion. Moreover, they rely on transforms built on top of random or Hadamard rotations, which equalize variances across entries rather than compacting energy, and fixed-width scalar quantizers, which are suboptimal at low rates. In this paper, we formulate KV cache quantization as a transform coding problem in which distortion is the error in the attention products. We derive closed-form optimal transforms for keys and values from calibration statistics, under a high-resolution model. We show that the optimal key transform is not orthogonal and satisfies a generalized Parseval relation: the attention-aware distortion becomes mean-squared error (MSE) in the transform domain. Thus, we can use MSE-optimal vector quantizers applied directly to the transformed key coefficients. To meet the fixed-width layout requirement, we show that grouping coefficients into equal-volume partitions makes equal-size codebooks attain the variable-rate optimum under the same high-resolution model. At two bits per element, our method, termed NOVA-KV, recovers most of the long-context retrieval accuracy lost by scalar quantization methods at comparable throughput.

Samuel Fernández-Menduiña, Amir Ziashahabi, Eduardo Pavez +2

Department of Electrical and Computer Engineering, University of Southern California

KV CachingVector QuantizationKV-Cache QuantizationLow-Bit Quantization
AskPDF
Apr 6, 2026

Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMs

Large Language Models (LLMs) have achieved remarkable progress across reasoning, generation, and decision-making tasks, yet deploying them on mobile, embedded, and edge devices remains particularly challenging. On-device LLM inference is heavily constrained by the memory and bandwidth overhead of the key-value (KV) cache, which grows linearly with context length and often dominates decoding cost. Existing KV-cache quantization schemes typically rely on fixed precision or hand-crafted heuristics, thereby wasting bits on low-impact tokens while over-compressing informative ones, leading to avoidable accuracy degradation. Inspired by Huffman coding's principle of variable-length allocation, we propose adaptive KV-cache quantization, a learned policy that assigns bit-width proportional to token importance, minimizing expected memory and latency without sacrificing competitive accuracy. Our framework extracts lightweight token-level features, including token frequency, quality score, attention variance, and entropy-based uncertainty, and feeds them into a compact data-driven controller that dynamically selects KV precision from {2-bit, 4-bit, 8-bit, FP16} during decoding. This adaptive precision policy reduces KV memory footprint and latency while improving accuracy compared to static KV quantization and rule-based baselines, and maintaining competitive accuracy close to FP16 inference across standard LLM benchmarks. Extensive experiments across multiple commonsense reasoning benchmarks using SmolLM-135M, SmolLM-360M, and SmolLM-1.7B demonstrate that our controller consistently improves the accuracy-latency trade-off. For instance, with SmolLM-360M on HellaSwag, our method reduces decoding latency (ms/token) by 17.75% relative to static KV quantization, improves accuracy by 7.60 points, and remains within only 0.30 points of FP16 inference.

Sayed Pedram Haeri Boroujeni, Niloufar Mehrabi, Patrick Woods +2

Clemson University, USA

KV-Cache QuantizationOn-Device Language Model InferenceLLM Inference AccelerationMixed-Precision Quantization
AskPDF