E8 Lattice Key Quantization for Low-Bit KV-Cache Compression

Sasho Nedelkoski; Alexander Acker; Soeren Becker; Dominik Scheinert; Odej Kao

cs.LGOct 11, 2026

E8 Lattice Key Quantization for Low-Bit KV-Cache Compression

Authors: Sasho Nedelkoski, Alexander Acker, Soeren Becker, Dominik Scheinert, Odej Kao

Organizations: Technical University of Berlin Berlin, Germany · logsight.ai Berlin, Germany

Abstract

Compressing the key-value (KV) cache is essential for serving long-context large language models (LLMs) under memory constraints. While current scalar quantization methods achieve significant compression, they typically face a quality-fidelity trade-off, either suffering from performance instability at 2-bit precision or necessitating 3-bit keys to maintain model quality. In this work, we demonstrate that a uniform low-bit k2v2 configuration (2-bit keys and 2-bit values) can preserve near-baseline quality by using vector quantization (VQ) based on the $E_8$ lattice for KV keys. We show that structured VQ exploits correlations across adjacent dimensions to achieve signal-to-quantization-noise ratios (SQNR) higher than those of independent scalar quantization at the same key payload rate. To ensure practical utility, we adapt a table-based asymmetric scoring algorithm that avoids explicit codebook dequantization during attention computation and preserves decode-throughput parity with scalar baselines. Across five LLM families, our $E_8$ k2v2 approach yields \textasciitilde $6\times$ KV compression, roughly 20% smaller than scalar k3v2 (3-bit keys and 2-bit values), while retaining near-baseline long-context perplexity. In our full-suite perplexity evaluation, $E_8$ improves over scalar k3v2 in 80% of model--dataset--context settings. These results indicate that $E_8$ lattice VQ improves quality per byte while keeping throughput at parity, suggesting that structured key quantization is a useful primitive for KV-compression stacks. Overall, our work provides a low-overhead primitive for expanding the quality-memory frontier in long-context LLM inference.

Figures & tables

Figure 1: Rate-distortion comparison on 128-dimensional Gaussian vectors.

E_{8}

lattice VQ achieves SQNR between scalar 3-bit and 4-bit quantization at a comparable low key rate.

Model	Ctx	WikiText-2							C4
		Base	KIVI	$\Delta$	Scalar	$\Delta$	E8	$\Delta$	Base	KIVI	$\Delta$	Scalar	$\Delta$	E8	$\Delta$
Qwen2.5-14B	512	5.90	7.16	+1.26	6.29	+0.39	6.13	+0.23	11.11	12.47	+1.36	11.59	+0.48	11.35	+0.24
	2K	5.01	7.32	+2.31	5.61	+0.60	5.29	+0.28	10.32	13.82	+3.50	11.31	+0.99	10.60	+0.29
	4K	4.84	7.88	+3.04	5.51	+0.68	5.13	+0.29	10.18	15.38	+5.20	11.59	+1.41	10.47	+0.29
Llama-3.1-8B	512	7.33	9.40	+2.07	7.70	+0.36	7.51	+0.18	11.34	15.00	+3.65	11.93	+0.59	11.71	+0.37
	2K	6.45	15.91	+9.45	6.97	+0.51	6.67	+0.21	10.90	32.72	+21.82	11.84	+0.94	11.35	+0.45

Table 1: Perplexity on WikiText-2 and C4 across context lengths. Each block reports compressed-method perplexity deltas against fp16 baseline; lower is better. Bold marks the best compressed configuration within each dataset/model/context.

Model	$L$	$H_{\text{kv}}$	Baseline	Scalar k3v2		E8 Lattice k2v2
			(B/tok)	(B/tok)	Ratio	(B/tok)	Ratio
Qwen2.5-14B	48	8	196,608	40,944	4.8 $\times$	32,592	6.0 $\times$
Llama-3.1-8B	32	8	131,072	27,296	4.8 $\times$	21,728	6.0 $\times$
Mistral-7B	32	8	131,072	27,296	4.8 $\times$	21,728	6.0 $\times$
Falcon3-7B	28	4	114,688	22,652	5.1 $\times$	18,452	6.2 $\times$
Granite-3.1-8B	40	8	163,840	34,120	4.8 $\times$	27,160	6.0 $\times$

Table 2: Effective compressed KV-cache bytes per token for the quantized cache region. The accounting includes packed key/value data and per-token metadata (key norms/QJL signs and value scale/zero groups), but excludes fixed model-level tables/rotations and the finite recent-token residual window.

Model	Context	Baseline	KIVI k2v2	Scalar k3v2	E8 Lattice k2v2
Qwen2.5-14B	1K	49.4	49.4	49.4	49.4
	4K	48.4	48.4	48.4	48.4
Llama-3.1-8B	1K	89.0	88.9	88.9	89.0
	4K	86.9	87.0	87.0	87.0
Mistral-7B	1K	92.9	92.9	92.9	93.0
	4K	90.7	90.7	90.8	90.8

Table 3: Decode throughput (tok/s) with table-based asymmetric scoring. E8 Lattice matches Scalar throughput despite using structured VQ, thanks to lookup-table scoring that avoids codebook dequantization.

Model	Baseline	KIVI k2v2	Scalar k3v2	E8 k2v2
Cell format: HellaSwag / ARC-C / WinoGrande / Avg
Qwen2.5-14B	65.5 / 60.5 / 75.1 / 67.1	65.7 / 60.3 / 75.3 / 67.1	65.7 / 60.4 / 75.3 / 67.1	65.7 / 60.3 / 75.3 / 67.1
Llama-3.1-8B	59.8 / 53.8 / 73.6 / 62.4	59.7 / 53.6 / 73.4 / 62.2	59.7 / 53.7 / 73.4 / 62.3	59.7 / 53.7 / 73.4 / 62.3
Mistral-7B	65.8 / 57.8 / 74.9 / 66.2	65.8 / 57.8 / 74.8 / 66.1	65.8 / 57.8 / 74.8 / 66.1	65.8 / 57.8 / 74.8 / 66.1
Falcon3-7B	60.3 / 55.8 / 70.3 / 62.1	60.3 / 55.9 / 70.4 / 62.2	60.3 / 55.9 / 70.4 / 62.2	60.3 / 56.0 / 70.4 / 62.2
Granite-3.1-8B	64.0 / 51.4 / 75.5 / 63.7	64.0 / 51.7 / 74.8 / 63.5	64.0 / 51.7 / 74.8 / 63.5	64.0 / 51.7 / 74.8 / 63.5

Table 4: Downstream task accuracy (%). Each cell reports HellaSwag 10-shot / ARC-Challenge 25-shot / WinoGrande 5-shot / average.

Figure 2: Average downstream accuracy across the core residual-sweep models versus residual window size. At

W=0

, compression quality differences are visible; by

W=128

, methods are nearly indistinguishable.

Figure 3: Left: per-layer cosine similarity between compressed and fp16 keys captured from model forward passes. Right: per-layer KL divergence in a controlled attention-distribution probe. Lines show mean over models and shaded bands show the min–max range after normalizing each architecture’s layer index to

[0,1]

$\alpha$	1	4	8	10	12	15	20	30
MSE	0.0577	0.0045	0.0011	0.0008	0.0007	0.0008	0.0013	0.0024
cos	0.292	0.727	0.913	0.940	0.953	0.960	0.959	0.952

Table 5: Compact ablations (mean across core models). (a) E8 scale calibration. (b) Initial-layer precision sweep.

KV-Cache Compression

Key-value cache compression reduces the memory and bandwidth needed to retain attention states during long-context inference.

See all 265 papers

Mar 17, 2026

VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization

The growing context length of Large Language Models (LLMs) enlarges the Key-Value (KV) cache, limiting deployment in resource-limited environments. Prior training-free approaches for KV cache compression typically rely on low-rank approximation or scalar quantization, which fail to simultaneously achieve high compression ratios and high reconstruction fidelity. We propose VQKV, a novel, training-free method introducing vector quantization (VQ) to obtain highly compressed KV representations while preserving high model fidelity, allowing for the representation of thousands of floating-point values with just a few integer indices. As a result, VQKV achieves an 82.8% compression ratio on LLaMA3.1-8B while retaining 98.6% of the baseline performance on LongBench and enabling 4.3x longer generation length on the same memory footprint.

Yixuan Wang, Qingyu Shi, Jiayu Zhou +3

LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai Innovation Institute · Shanghai Artificial Intelligence Laboratory +2

KV-Cache Compression KV-Cache Quantization Memory Compression LLM Inference Acceleration Vector Quantization

Ask PDF

Mar 24, 2026

EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction

The increasing memory demand of the Key-Value (KV) cache poses a significant bottleneck for Large Language Models (LLMs) in long-context applications. Existing low-rank KV compression methods reduce this footprint by modifying model projections, limiting the flexibility to switch back to standard full-cache inference when sufficient memory is available. In this paper, we propose EchoKV, a flexible KV cache compression framework that supports on-demand transitions from full KV caching to compressed caching. Unlike traditional compression-decompression paradigms, EchoKV utilizes a lightweight network to reconstruct the discarded KV components from a partial subset, exploiting intrinsic inter-layer and intra-layer similarities among attention heads. We further introduce a lightweight two-stage fine-tuning strategy, requiring only a few minutes on a single A100 GPU for a 7B model. Experimental results on LongBench and RULER demonstrate that EchoKV consistently outperforms existing methods across multiple compression ratios and backbone models while preserving the throughput of full-cache inference in short-context scenarios.

Shiyu Ji, Yixuan Wang, Yijun Liu +2

Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology, China

KV-Cache Compression LLM Inference Acceleration

Ask PDF

Oct 2, 2026

Tailoring the Quantization Space for 1-Bit KV Cache Compression

The key-value (KV) cache becomes a major memory bottleneck in long-context LLM inference, placing substantial pressure on memory capacity and bandwidth. To mitigate this bottleneck, vector quantization (VQ) has emerged as a promising approach for aggressive KV cache compression. However, existing VQ methods degrade substantially in the 1-bit regime. At such extreme compression, each codebook must represent a larger group of channels with a limited set of centroids, making effective use of its capacity increasingly challenging. To address this, we introduce

\textbf{TaSQ}

, which tailors the VQ target space by combining query-guided channel weighting, cross-head normalization, and covariance-aware channel grouping to better reflect the error sensitivity and statistical structure of cached activations. Since these transforms are RoPE-compatible and can be easily merged into projection weights and codebooks, TaSQ preserves the conventional VQ lookup structure and adds negligible serving overhead. Across general, long-chain-of-thought reasoning, and long-context retrieval benchmarks, TaSQ consistently outperforms existing low-bit KV cache VQ baselines while preserving reasoning stability. On a single RTX 6000 Ada GPU, its SGLang implementation supports up to

14\times

larger batch sizes and achieves

1.87\times

higher peak throughput compared to the BF16 baseline.

Minsoo Cheong, Donghyun Son, Sungjoo Yoo

Stanford University · Seoul National University

Binary Quantization KV-Cache Compression KV-Cache Quantization Vector Quantization LLM Inference Acceleration Neural Network Quantization

Ask PDF

Abstract

KV-Cache Compression

Explore similar work

VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization

EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction

Tailoring the Quantization Space for 1-Bit KV Cache Compression