PN-Mix: Pair-Norm Guided Mixed-Precision Key Cache Quantization for Efficient Long-Output LLM Inference

Gil Jae Song; Sungjoo Yoo

cs.LGOct 14, 2026

PN-Mix: Pair-Norm Guided Mixed-Precision Key Cache Quantization for Efficient Long-Output LLM Inference

Authors: Gil Jae Song, Sungjoo Yoo

Organizations: Seoul National University

Abstract

Long-output inference in large language models suffers from severe KV cache memory and bandwidth bottlenecks. While KV cache quantization alleviates this issue, low-bit quantization (e.g., 2-bit) severely degrades model accuracy. Recent mixed-precision key-cache quantization methods mitigate this degradation by allocating higher precision to important key channels, but estimate channel saliency using only key statistics. Through attention logit perturbation analysis at the RoPE channel-pair level, we show that quantization errors in channel pairs with large query--key pair-norm products contribute significantly to logit variation. Based on this insight, we propose PN-Mix, a mixed-precision key-cache quantization method for RoPE-based decoder-only LLMs, which allocates higher precision to salient channel pairs identified once during the prefill stage and reused throughout decoding. Across standard reasoning, code-generation, and AIME benchmarks, PN-Mix with 2.26-bit average precision achieves comparable or better average accuracy than prior mixed-precision methods using 2.32--2.39 bits. Furthermore, in our Pareto sweep, PN-Mix degrades gracefully down to 1.5 bits, whereas the compared baselines exhibit sharper performance drops below 2.25 bits.

Figures & tables

Figure 1: Post-RoPE key-cache layout of PN-Mix. Salient RoPE channel pairs selected during prefill are stored in higher precision across quantized pages, while the remaining pairs use lower precision.

Figure 2: Simulation results under H3(50%)L1 key-cache bit allocation. Each marker shows mean

D_{\mathrm{KL}}^{A}

; adjacent purple values show mean

\Delta_{\tilde{K}}

. For each criterion, we pseudo-quantize

\tilde{K}

with midpoint correction for 1-bit groups (Eq. 7 );

\Delta_{\tilde{K}}

is measured on the reconstructed

\tilde{K}^{\prime}

, while

D_{\mathrm{KL}}^{A}

is computed from

A^{\prime}_{h,t}

recomputed using the corresponding

\tilde{Q}

Model / Dataset	Spearman $\boldsymbol{\rho}$	Top-10% overlap	Top-50% overlap
R1-Llama-8B / LiveCodeBench v5	$0.996_{\pm 0.000}$	$96.2_{\pm 0.3}$	$97.9_{\pm 0.1}$
R1-Qwen-7B / AIME25	$0.990_{\pm 0.003}$	$95.3_{\pm 0.6}$	$96.2_{\pm 0.5}$

Table 1: Separable vs. causal-joint pair-norm score. The table reports the similarity in ranking and selection between

\sigma_{h,i}

(Eq. 5 ) and the causal-joint score computed over prefill states. Metrics are averaged over layers and KV heads and reported as mean ±std across samples.

Model	Method	K bits	GSM8K	MATH-Alg.	HumanEval	GPQA-Dia	Avg.
Llama3.1-8B-Instruct	FP16	16	$\mathrm{76.83}_{\pm 0.90}$	$\mathrm{48.17}_{\pm 0.89}$	$\mathrm{64.51}_{\pm 1.70}$	$\mathrm{26.63}_{\pm 1.22}$	$\mathrm{54.04}$
	KIVI	2.25	$\mathrm{66.97}_{\pm 1.07}$	$\mathrm{35.27}_{\pm 0.80}$	$\mathrm{55.49}_{\pm 3.18}$	$\mathrm{23.67}_{\pm 2.06}$	$\mathrm{45.35}$
	PolarQuant	2.25	$\mathrm{26.85}_{\pm 0.49}$	$\mathrm{10.38}_{\pm 0.85}$	$\mathrm{34.27}_{\pm 1.30}$	$\mathrm{20.41}_{\pm 2.35}$	$\mathrm{22.98}$
	ChanMix	2.32	$\mathrm{70.51}_{\pm 0.75}$	$\mathrm{37.57}_{\pm 0.70}$	$\mathrm{59.63}_{\pm 2.23}$	$\mathrm{23.37}_{\pm 1.38}$	$\mathrm{47.77}$
	Kitty	2.39	$\mathrm{74.51}_{\pm 0.62}$	$\mathrm{46.25}_{\pm 0.72}$	$\mathrm{62.68}_{\pm 1.78}$	$\mathrm{23.98}_{\pm 2.14}$	$\mathrm{51.86}$
	PN-Mix (1)	2.26	$\mathbf{75.66}_{\pm 0.94}$	$\mathbf{46.57}_{\pm 1.43}$	$\mathbf{64.39}_{\pm 2.10}$	$\mathrm{25.71}_{\pm 2.59}$	$\mathbf{53.08}$

Table 2: Accuracy comparison on standard reasoning and code-generation benchmarks. All baselines are configured to use an average key-cache precision in the 2.25–2.4-bit range, and the value cache is kept in FP16. Each cell reports the mean

\pm

standard deviation over five runs. Bold indicates the best non-FP16 result in each column.

\dagger

denotes the results obtained by applying the reproduced profiling procedure and the resulting profile to the model.

Model	Method	K bits	AIME24		AIME25
Model	Method	K bits	$b_{V}{=}16$	$b_{V}{=}2$	$b_{V}{=}16$	$b_{V}{=}2$
R1-Llama-8B	FP16	16	$\mathrm{48.33}_{\pm 6.19}$		$\mathrm{31.33}_{\pm 8.19}$
	KIVI	2.25	$\mathrm{28.67}_{\pm 7.02}$	$\mathrm{12.00}_{\pm 4.99}$	$\mathrm{26.67}_{\pm 5.16}$	$\mathrm{19.33}_{\pm 3.59}$
	PolarQuant	2.25	$\mathrm{0.00}_{\pm 0.00}$	$\mathrm{0.00}_{\pm 0.00}$	$\mathrm{0.00}_{\pm 0.00}$	$\mathrm{0.00}_{\pm 0.00}$
	ChanMix	2.32	$\mathrm{28.00}_{\pm 6.70}$	$\mathrm{19.00}_{\pm 5.59}$	$\mathrm{26.00}_{\pm 3.27}$	$\mathrm{16.67}_{\pm 4.47}$
	Kitty	2.39	$\mathrm{42.00}_{\pm 5.62}$	$\mathrm{42.00}_{\pm 5.42}$	$\mathrm{35.00}_{\pm 6.19}$	$\mathrm{30.00}_{\pm 5.96}$

Table 3: Accuracy comparison on AIME. For each model, we report both settings where the value cache is kept in FP16 (

b_{V}{=}16

) and where it is quantized to the 2-bit range (

b_{V}{=}2

). Each cell reports the mean

\pm

standard deviation over 10 responses per question. Bold indicates the best non-FP16 result in each column.

\dagger

denotes the results obtained by applying the reproduced profiling procedure and the resulting profile to the model.

Importance score	1-bit	sink	AIME24	HumanEval
FP16	–	–	$\mathrm{76.33}_{\pm 4.82}$	$\mathrm{85.85}_{\pm 0.46}$
$\tilde{Q}\tilde{K}$ pair norm	$\checkmark$	$\checkmark$	$\mathrm{76.00}_{\pm 4.67}$	$\mathrm{84.87}_{\pm 2.26}$
$\tilde{Q}\tilde{K}$ pair norm	$\checkmark$	$\times$	$72.00_{\pm 5.42}$	$84.02_{\pm 2.54}$
$\tilde{Q}\tilde{K}$ pair norm	$\times$	$\checkmark$	$69.67_{\pm 6.90}$	$80.36_{\pm 2.48}$
$\tilde{K}$ range	$\checkmark$	$\checkmark$	$70.67_{\pm 3.59}$	$83.78_{\pm 2.48}$
$\tilde{K}$ magnitude	$\checkmark$	$\checkmark$	$73.00_{\pm 5.04}$	$84.39_{\pm 1.95}$

Table 4: Ablation of PN-Mix design choices on Qwen3-8B. All variants use PN-Mix (2) with FP16 values; the highlighted row is the final configuration and (*) denotes the online variant.

Figure 3: Pareto frontier in the 1.5–3.0-bit range . Detailed configurations and model accuracies are provided in Appendix I .

$B$ / $P$ / $L_{\mathrm{gen}}$	Method	Mem (GB)	TTFT (ms)	Decode (s)	Total (s)
1/141/32K	FP16	19.80	46	1299	1299
	Kitty	16.03	55	2121	2121
	PN-Mix	15.95	71	2036	2036
1/16K/8K	FP16	20.26	1679	325	327
	Kitty	17.45	1702	590	592
	PN-Mix	17.39	1714	563	565

Table 5: End-to-end latency and peak memory on a single A100-SXM4-40GB.

B

P

, and

L_{\mathrm{gen}}

denote batch size, prompt length, and generation length.

P{=}141

corresponds to a single AIME24 problem, and

P{=}16\mathrm{K}

to a GSM8K 88-shot prompt.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Salient channel-pair prediction accuracy. Jaccard similarity between the top-

r\%

channel-pair predictions of four importance criteria and the decoding-time ground truth, averaged over layers, heads, and decoding steps. Results are shown for

r=10\%

and

r=50\%

Figure 5: Cumulative ratio of channel-pair importance scores

\sigma_{h,i}

. The figure reports the cumulative importance ratio captured by the top

r\%

channel pairs for R1-Qwen-7B, Qwen3-14B, and Qwen3-32B on AIME25, and for R1-Llama-8B on LiveCodeBench v5. For each layer and KV head, we compute the cumulative importance ratio from prefill-based scores and report the median of these ratios over layers, KV heads, and prompts.

Figure 6: Memory scaling and generation throughput on a single A100-SXM4-40GB GPU. We compare FP16, Kitty, and PN-Mix on Qwen3-8B by sweeping (a) generation length

L_{\mathrm{gen}}

at fixed batch size

B=32

, (b) batch size

B

at fixed generation length

L_{\mathrm{gen}}=8\mathrm{K}

, and (c) token generation throughput over

B

L_{\mathrm{gen}}=8\mathrm{K}

Figure 7: Per-step decode latency on a single A100-SXM4-40GB GPU. We measure Qwen3-8B decode latency with Kitty and PN-Mix. The shaded region (steps 11K–31K) indicates the KV-cache-read-dominated regime, and $\beta$ denotes the linear-regression slope in this region ( $\mu$ s/step).

Method	en	zh	fr	ja	bn
FP16	$\mathrm{82.80}$	$\mathrm{62.80}$	$\mathrm{64.00}$	$\mathrm{50.80}$	$\mathrm{58.00}$
KIVI	$\mathrm{73.20}$	$\mathrm{56.00}$	$\mathrm{58.80}$	$\mathrm{47.20}$	$\mathrm{44.40}$
ChanMix	$\mathrm{74.80}$	$\mathrm{54.40}$	$\mathrm{55.20}$	$\mathrm{45.60}$	$\mathrm{50.40}$
Kitty	$\mathbf{80.40}$	$\mathrm{57.20}$	$\mathrm{62.40}$	$\mathrm{48.80}$	$\mathbf{55.20}$
PN-Mix (1)	$\mathbf{80.40}$	$\mathrm{57.20}$	$\mathrm{63.60}$	$\mathrm{48.40}$	$\mathrm{52.40}$
PN-Mix (2)	$\mathrm{80.00}$	$\mathbf{59.20}$	$\mathbf{65.20}$	$\mathbf{53.60}$	$\mathrm{52.00}$

Appendix

Table 6: Accuracy comparison on MGSM with Llama-3.1-8B-Instruct. We evaluate the full 250-example test set using 8-shot native-language CoT. Performance is measured using flexible-extract accuracy. All compressed configurations (i.e., every method except FP16) are matched to an average key-cache precision of 2.25–2.39 bits, while the value cache is kept in full precision. Bold indicates the best non-FP16 result in each column.

Method	Avg. KV bits	AIME24	AIME25
R1-Llama-8B
FP16	16	$\mathrm{48.33}_{\pm 6.19}$	$\mathrm{31.33}_{\pm 8.19}$
PM-KVQ	2.38	$\mathrm{46.33}_{\pm 6.04}$	$\mathrm{31.66}_{\pm 4.53}$
PN-Mix (1)	2.26	$\mathrm{47.67}_{\pm 6.16}$	$\mathrm{31.00}_{\pm 4.23}$
PN-Mix (2)	2.28	$\mathrm{44.33}_{\pm 5.97}$	$\mathrm{33.00}_{\pm 5.04}$
R1-Qwen-7B

Appendix

Table 7: Comparison with PM-KVQ on AIME24 and AIME25. For PN-Mix, we use the same

b_{V}=2

configurations as in Table 3 , while PM-KVQ uses the lowest model-specific mixed-precision operating point within the 2.25–2.4 average-bit range. Each cell reports the mean

\pm

standard deviation over 10 responses per question.

Model	Task	K16V16	K2.25V16	K2.25*V16	K16V2.25	K16V2.25*
R1-Llama-8B	AIME24	$48.33_{\pm 6.19}$	$28.67_{\pm 7.02}$	$44.00_{\pm 7.27}$	$40.67_{\pm 5.33}$	$\mathbf{51.00_{\pm 6.51}}$
R1-Llama-8B	AIME25	$31.33_{\pm 8.19}$	$26.67_{\pm 5.16}$	$28.67_{\pm 3.71}$	$26.33_{\pm 4.58}$	$\mathbf{32.00_{\pm 5.81}}$
Qwen3-8B	AIME24	$76.33_{\pm 4.82}$	$64.00_{\pm 5.54}$	$70.00_{\pm 2.98}$	$68.33_{\pm 4.53}$	$\mathbf{73.33_{\pm 2.98}}$
Qwen3-8B	AIME25	$67.00_{\pm 4.58}$	$55.33_{\pm 4.99}$	$59.67_{\pm 4.07}$	$60.00_{\pm 2.11}$	$\mathbf{68.00_{\pm 5.42}}$

Appendix

Table 8: Sensitivity of key/value caches to quantization. Each cell reports mean

\pm

std over 10 responses per question. All quantized methods use KIVI ( Liu et al., 2024 ) (

G=128

N=128

); method names ending with (*) preserve the first 32 tokens in full precision. The best quantized method per row within each model block is in bold .

Method	R1-Qwen-7B (AIME25)		R1-Llama-8B (LiveCodeBench v5)
Method	$\Delta_{\tilde{K}}$	$D_{\mathrm{KL}}^{A}$	$\Delta_{\tilde{K}}$	$D_{\mathrm{KL}}^{A}$
Only 3-bit (Uniform precision)
Per-Channel	0.042	0.043	0.058	0.020
Per-Channel after $(r,\theta)$	0.098	0.101	0.142	0.043
Only 2-bit (Uniform precision)
Per-Channel	0.230	0.232	0.324	0.120

Appendix

Table 9: Key-cache quantization simulation. Table shows

\Delta_{\tilde{K}}

(key MSE) and

D_{\mathrm{KL}}^{A}

(attention KL divergence) for uniform-precision and H3(50%)L1 mixed-precision methods on R1-Qwen-7B (AIME25) and R1-Llama-8B (LiveCodeBench v5). All methods use per-channel asymmetric quantization (

G=128

); 1-bit groups use the correction of Eq. 7 . Each H3(50%)L1 entry reports the value with 1-bit correction, with the value without correction shown as a parenthesized subscript. Mixed-precision rows differ only in the channel(-pair) importance score used to allocate the high/low bits, with importance computed over either the full sequence or prefill tokens only.

(a) Method configurations
Method	$b$	$b_{H}$	$b_{L}$	$r$ / $r_{h}$	$r_{\mathrm{eff}}$	$G$	$N$	$N_{s}$
KIVI	2	–	–	–	–	128	128	0
PolarQuant	2	–	–	–	–	128	128	0
ChanMix	–	4	3, 2, 1	0.1	–	128	128	0
Kitty	–	4	2	0.03125	$\tfrac{\lfloor dr\rfloor}{d}$	128	128	32
PN-Mix (1)	–	3	1	0.5	$\tfrac{2\lfloor rd/2\rfloor}{d}$	128	128	32

Appendix

Table 10: Method configurations and average key-cache bit-widths. Average bit-widths are computed with

T_{q}=32{,}768

, where

T_{q}

denotes the number of quantized tokens; the FP16 residual buffer is excluded from the calculation. For Kitty and PN-Mix,

r_{\mathrm{eff}}

denotes the actual fraction of channels assigned to high precision after converting the nominal ratio

r

to an integer number of selected channels or RoPE channel pairs, respectively. Effective K-storage bits exclude saliency metadata, which is reported separately and added to obtain the total average K bit-width.

(a) Long generation: R1-Qwen-7B / AIME25, 96 prefill tokens + 32,672 decoding steps
Reference score	Selection criterion	0–8K	8–16K	16–24K	24K–32.7K	Max–Min
$\\|\mathbf{q}_{i}^{t}\\|_{2}\\|\mathbf{k}_{i}^{s}\\|_{2}$	$\tilde{K}$ range	$0.61/0.35$	$0.62/0.35$	$0.61/0.35$	$0.61/0.35$	$0.009/0.029$
	$\tilde{K}$ magnitude	$0.68/0.49$	$0.69/0.50$	$0.69/0.50$	$0.69/0.51$	$0.008/0.034$
	$\tilde{K}$ pair norm	$0.70/0.54$	$0.70/0.56$	$0.70/0.56$	$0.70/0.56$	$0.010/0.040$
	$\tilde{Q}\tilde{K}$ pair norm	$\mathbf{0.72/0.62}$	$\mathbf{0.72/0.61}$	$\mathbf{0.72/0.61}$	$\mathbf{0.72/0.61}$	$\mathbf{0.007/0.027}$
$\|\langle\mathbf{q}_{i}^{t},\mathbf{k}_{i}^{s}\rangle\|$	$\tilde{K}$ range	$0.59/0.33$	$0.60/0.33$	$0.60/0.33$	$0.60/0.33$	$0.008/0.024$

Appendix

Table 11: Stability of prefill-selected channel pairs during decoding. We measure the overlap between the fixed top-

r\%

channel pairs selected from prefill and the top-

r\%

channel pairs identified by a step-wise reference during decoding. The reference scores are

\|\mathbf{q}_{i}^{t}\|_{2}\|\mathbf{k}_{i}^{s}\|_{2}

and

|\langle\mathbf{q}_{i}^{t},\mathbf{k}_{i}^{s}\rangle|

. For each decode query at step

t

, the reference considers only decode-generated causal keys (

P\leq s\leq t

), where

P

is the prefill length. Each cell reports the overlap ratio aggregated over all eligible query–key interactions, heads, and layers within the bin, in the order

r=50\%/r=10\%

. Overlap ranges from 0 to 1, where 1 indicates identical selections. Max–Min is computed over 1K-step bins for the long-generation setting and 992-step bins for the multi-topic setting.

Method	K bits	KV-Retr.	LongDial.-QA	LongBook-Choice	LongBook-QA	LongBook-Sum	Avg.
		(acc)	(acc)	(acc)	(F1)	(ROUGE-L-Sum)
FP16	16	$\mathrm{59.00}$	$\mathrm{11.00}$	$\mathrm{62.00}$	$\mathrm{9.66}$	$\mathrm{29.07}$	$\mathrm{34.15}$
KIVI	2.25	$\mathrm{32.00}$	$\mathrm{9.00}$	$\mathrm{58.00}$	$\mathbf{10.03}$	$\mathrm{25.55}$	$\mathrm{26.92}$
ChanMix	2.32	$\mathrm{53.00}$	$\mathrm{9.00}$	$\mathrm{58.00}$	$\mathrm{10.01}$	$\mathrm{26.74}$	$\mathrm{31.35}$
Kitty	2.39	$\mathrm{49.00}$	$\mathbf{10.00}$	$\mathrm{61.00}$	$\mathrm{8.82}$	$\mathbf{29.09}$	$\mathrm{31.58}$
PN-Mix (1)	2.26	$\mathbf{59.00}$	$\mathrm{7.00}$	$\mathrm{62.00}$	$\mathrm{8.67}$	$\mathrm{28.76}$	$\mathrm{33.09}$

Appendix

Table 12: Accuracy comparison on InfiniteBench with Llama-3.1-8B-Instruct. We evaluate the first 100 examples of each of five English tasks. Prompts are built with the official task templates and middle-truncated to 32,000 tokens using the official truncation routine; no chat template is applied, and decoding is greedy with the official per-task generation budgets (6–1,200 tokens). All compressed configurations (i.e., every method except FP16) are matched to an average key-cache precision of 2.25–2.39 bits, while the value cache is kept in full precision. Bold indicates the best non-FP16 result in each column.

Len.	Method	S1	S2	S3	MK1	MK2	MK3	MV	MQ	FWE	CWE	QA1	QA2	VT	Avg.
16K	FP16	100.0	100.0	100.0	99.5	100.0	100.0	100.0	99.4	95.2	86.6	70.8	57.5	99.8	92.98
	KIVI	100.0	99.5	93.5	99.5	96.5	69.0	95.2	95.4	93.0	84.7	71.5	54.5	95.2	88.26
	ChanMix	100.0	99.5	99.0	99.0	99.0	97.5	97.2	97.5	92.7	87.7	70.8	56.5	98.7	91.93
	Kitty	100.0	100.0	99.5	99.5	98.5	95.0	99.4	98.5	94.3	87.1	72.8	54.5	98.1	92.09
	PN-Mix (1)	100.0	100.0	100.0	100.0	100.0	99.0	100.0	99.4	95.7	85.0	72.6	56.5	99.4	92.89
	PN-Mix (2)	100.0	100.0	100.0	99.5	99.5	100.0	99.9	99.1	95.0	82.2	72.1	56.5	99.4	92.55

Appendix

Table 13: Accuracy on RULER across context lengths from 16K to 64K with Llama-3.1-8B-Instruct. Tasks are the 13 official RULER categories, where NIAH denotes needle-in-a-haystack: single NIAH (S1–S3), multi-key NIAH (MK1–MK3), multi-value NIAH (MV), multi-query NIAH (MQ), frequent-word extraction (FWE), common-word extraction (CWE), question answering (QA1–QA2), and variable tracking (VT). All compressed configurations (i.e., every method except FP16) are matched to an average key-cache precision of 2.25–2.39 bits, while the value cache is kept in full precision. Bold indicates the best non-FP16 result in each column.

Configuration	Avg key bit	AIME24
Configuration	Avg key bit	Qwen3-8B	R1-Llama-8B
Full (FP16)	16.00	$\mathrm{76.33}_{\pm 4.82}$	$\mathrm{48.33}_{\pm 6.19}$
KIVI
$b{=}2,\,G{=}32$	3.00	$\mathrm{72.33}_{\pm 3.96}$	$\mathrm{42.67}_{\pm 8.14}$
$b{=}2,\,G{=}64$	2.50	$\mathrm{68.00}_{\pm 4.99}$	$\mathrm{35.00}_{\pm 9.80}$
$b{=}2,\,G{=}128$	2.25	$\mathrm{64.00}_{\pm 5.54}$	$\mathrm{28.67}_{\pm 7.02}$

Appendix

Table 14: Configurations for the bit-rate Pareto frontier in Figure 3 . For each operating point, we report the selected hyperparameters and the corresponding AIME24 accuracy (%) with

b_{V}=16

fixed. Kitty and PN-Mix both preserve

N_{s}=32

sink tokens in FP16. Full denotes the FP16 KV-cache baseline.

KV-Cache Quantization

Key-value cache quantization compresses transformer attention state to reduce memory use and bandwidth during long-context inference.

See all 102 papers

SemKV: Semantic Mixed-Precision KV Cache Quantization Guided by the Quality Cliff for Long-Context LLM Inference
Aug 28, 2026Daeha Lee, Do-Hyung Kim, Jae-Hong KimKV-Cache Quantization Mixed-Precision Quantization
Electronics and Telecommunications Research Institute (ETRI), Daejeon, Republic of Korea
RateQuant: Optimal Mixed-Precision KV Cache Quantization via Rate-Distortion Theory
Apr 22, 2026Fei Zuo, Zikang Zhou, Hao Cong +2LLM Inference Efficiency Mixed-Precision Quantization
1BA TechWorks (BMW Group) · 2National University of Singapore · 3Tsinghua University
MixQuant: Adaptive Mixed-Precision Quantization for Large Language Models
Jul 25, 2026Ashitabh Misra, Madhav Agrawal, Arham Jain +1Mixed-Precision Quantization LLM Quantization
University of Illinois at Urbana-Champaign

Aug 28, 2026J/K move · Enter open · S save

SemKV: Semantic Mixed-Precision KV Cache Quantization Guided by the Quality Cliff for Long-Context LLM Inference

Daeha Lee, Do-Hyung Kim, Jae-Hong Kim

Electronics and Telecommunications Research Institute (ETRI), Daejeon, Republic of Korea

KV-Cache Quantization Mixed-Precision Quantization Long-Context Language Model Inference

Open paper AskPDF arXiv

The key-value (KV) cache is the dominant memory bottleneck of long-context large language model (LLM) inference, growing linearly with context length. We show that uniform KV quantization on a fractional-bit grid does not degrade gracefully: under a prespecified multi-seed statistical protocol, Llama-3.1-8B-Instruct with an affine quantizer is statistically indistinguishable from FP16 KV down to 2.322 code bits/value and collapses at 2.0 bits - a quality cliff in (2.0, 2.322] that reappears in generation-time quantization and multi-turn dialogue and transfers to Mistral-7B. The cliff reframes importance-aware mixed precision: above it, eight model-internal importance indicators are statistically interchangeable, so the benefit of mixing is grid interpolation, reaching average precisions uniform quantization cannot realize. SemKV preserves every token, ranks tokens by a model-internal score, and assigns two adjacent above-cliff precisions, achieving a measured 6.0x storage reduction with no statistically detectable quality difference from full KV (n=900, three seeds), and outperforming FP16 token pruning granted a 1.5x larger memory budget. Replacing the affine base with a distortion-optimized quantizer (TurboQuant-MSE) lowers the cliff in every protocol tested, raising the no-detectable-loss operating point to 7.9x. The recipe: measure the cliff for the target deployment setting, then interpolate above it.

Abstract

KV-Cache Quantization

Explore similar work

SemKV: Semantic Mixed-Precision KV Cache Quantization Guided by the Quality Cliff for Long-Context LLM Inference

RateQuant: Optimal Mixed-Precision KV Cache Quantization via Rate-Distortion Theory

MixQuant: Adaptive Mixed-Precision Quantization for Large Language Models