SemKV: Semantic Mixed-Precision KV Cache Quantization Guided by the Quality Cliff for Long-Context LLM Inference

Daeha Lee; Do-Hyung Kim; Jae-Hong Kim

Method family	Deletes tokens	Per-token bits	Indicator	Cliff-mapped frac. grid	Gen.-time
Uniform quantization	No	No	—	No	Partial
Eviction / pruning	Yes	No	Attn.	No	Yes
Importance-aware quant.	Varies	Yes	Attn.	No	Partial
SKVQ	No	Window	Recency	No	Yes
SemKV (this work)	No	Yes	Model-internal (8 tested)	Yes	Yes (block-exact)

Table 1: Positioning against representative KV compression families. Examples: uniform quantization — KIVI, KVQuant; eviction — H2O, SnapKV; importance-aware quantization — MiKV, QAQ, ZipCache. SemKV is distinguished by the combination of all-token preservation, token-wise fractional-grid precision mapped from a measured cliff, and generation-time support. Entries summarize common characteristics of representative methods and do not imply that every method in a family has all listed properties.

Fig. 1: Overall SemKV architecture. SemKV first generates the full-precision KV cache, estimates token-level semantic importance, ranks tokens, allocates higher precision to important tokens, and performs mixed-precision KV quantization before decoding. The two precision levels shown inside the panel (

b_{\mathrm{high}}

/

b_{\mathrm{low}}

) are set in our experiments to fractional-level codes such as 2.585/2.322 bits (Sec. 4.1).

Fig. 2: Semantic importance scoring pipeline. SemKV computes a token-importance score from one selected model-internal indicator and ranks tokens within each prompt; because allocation depends only on the ranking, score normalization is not required (the 0–1 score scale in the panel is illustrative).

Fig. 3: The quality-cliff concept and the offline mapping procedure. Uniform quantization on the level grid remains in a relatively flat regime until degradation increases sharply between two adjacent grid points; small residual deficits may exist above this collapse bracket, particularly in full-cache multi-turn evaluation (Sec. 4.6). In the single-turn setting illustrated here, the above-cliff grid points are statistically indistinguishable from full precision. The boundary is an empirically measured property of the model, base quantizer, and target evaluation setting; improving the quantizer shifts it downward in every protocol we test (the generation-side boundary may sit lower than the prefill and multi-turn boundaries). The curve is schematic; only the grid points are empirically evaluated. SemKV sets

(b_{high},b_{low})

to grid points above the measured cliff and interpolates between them.

Fig. 4: Adaptive bit allocation based on token importance. The top-ranked tokens are assigned high precision, while the remaining tokens are assigned low precision. The dashed box indicates that the bit-precision pair

(b_{high},b_{low})

is supplied by the offline cliff mapping of Fig. 3 and revalidated when the target model, quantizer, protocol, or context-budget composition changes materially. The allocation is realized purely by the top-

r

ranking; no score threshold is estimated. Token subscripts in the two groups denote membership in

\mathcal{H}

and

\mathcal{I}\setminus\mathcal{H}

respectively; the low-precision group is generally not contiguous in token position.

Fig. 5: Mixed-precision KV quantization process. The token-level bit allocation is applied to key and value tensors, resulting in a mixed-precision quantized KV cache used for decoding. The token-wise precision assignment is broadcast across all layers and KV heads (

H_{\mathrm{KV}}

); bits are never allocated along the head dimension itself. The

b_{\mathrm{high}}

/

b_{\mathrm{low}}

labels inside the panel denote the two precision levels; experiments use fractional-level codes such as 2.585/2.322 bits (Sec. 4.1).

Fig. 6: Generation-token mixed-precision quantization. Scores of newly generated tokens are accumulated and quantized block-exactly in deferred fashion, so the prescribed high-precision fraction is met constructively without threshold estimation, and the most recent partial block remains in FP16 during decoding; at the end of each generation turn, the residual block is quantized using the same block-wise top-

r

rule.

Configuration	Value
Backbone models	Llama-3.1-8B-Instruct / Mistral-7B-Instruct-v0.3
Scoring (default)	Hidden-state delta norm, last 4 layers
Indicator ablation	8 model-internal indicators (4 families)
Precision pair	$b_{\mathrm{high}}{=}2.585$ , $b_{\mathrm{low}}{=}2.322$ (above-cliff grid points)
High-precision fraction	25% (single-turn) / 35% (multi-turn incl. generation)
Quantizer	Integer-level scalar quantization + Hadamard rotation

Table 2: SemKV experimental configuration.

Fig. 7: The quality cliff (Llama-3.1-8B, LongBench protocol, 3 seeds; error bars: 95% CI of the paired difference). (a) All uniform settings in

[2.322,3.0]

bits are statistically indistinguishable from full KV under this protocol; 2.0 bits collapses. (b) Interior probes mixing 2.322/2.0 bits: recovery is monotone but even 75% protection remains significantly degraded, and importance-concentrated selection underperforms spread-out selection inside the cliff.

Fig. 8: Grid interpolation at 2.39 average bits on our LongBench protocol (Llama-3.1-8B, 10 subsets

\times

30 items

\times

3 seeds). Paired difference from full KV with 95% confidence intervals for all eight indicator variants of SemKV at the 2.39-bit operating point (

n{=}900

, 3 seeds). Every interval contains zero, and all 28 pairwise indicator comparisons are n.s. (Bonferroni-corrected) — interpolation quality at this operating point is indicator-independent.

Method	Nominal avg. code bits	Compression	Score	$\Delta$ vs. full ( $p$ )
Full KV	16.0	$1.0\times$	0.4406	—
Uniform 2.585-bit	2.585	$5.6\times$	0.4312	$-0.0094$ (0.09)
SemKV hidden-norm	2.39	$6.0\times$	0.4383	$-0.0023$ (0.70)
SemKV variance	2.39	$6.0\times$	0.4377	$-0.0029$ (0.64)
SemKV cos-delta	2.39	$6.0\times$	0.4317	$-0.0089$ (0.10)
SemKV key-norm	2.39	$6.0\times$	0.4328	$-0.0078$ (0.17)

Table 3: Grid interpolation at 2.39 bits: paired difference vs. full KV (Llama-3.1-8B,

n{=}900

). No variant differs significantly from full KV; all pairwise indicator differences are n.s. (Bonferroni-corrected). Compression is measured KV storage vs. FP16, metadata included (Sec. 4.1).

Fig. 9: Generation-token quantization (Llama-3.1-8B, 3 seeds). (a) 2.322-bit generation is statistically indistinguishable from FP16 generation; every configuration containing a 2.0-bit generation component degrades significantly, regardless of the paired high precision. (b) The max-length termination rate is reduced at 2.322 bits and elevated only at 2.0 bits. Max-length termination rate: the fraction of outputs that reach the configured generation limit without emitting EOS. We use this rate as an operational proxy for runaway-generation behavior.

Fig. 10: Scoring cost (Llama-3.1-8B). (a) Per-step scoring cost versus context length: hidden-state scoring is flat (

O(1)

), attention recomputation grows with

T

. (b) End-to-end scoring share of generation time on LongBench. Complexity labels are with respect to context length

T

: hidden-state scoring is

O(1)

per decode step in

T

, while attention recomputation grows with

T

.

Fig. 11: Measured KV storage (packed-buffer microbenchmark; both backbones share this KV shape). (a) KV storage versus context length: SemKV at 2.39 bits tracks

6.0\times

below full FP16 KV at every length. (b) Measured compression at 128k tokens; the pruning baseline that SemKV outperforms in Sec. 4.7 occupies a

1.5\times

larger footprint.

Fig. 12: Multi-turn indicator ablation (MT-Eval follow-up,

n{=}90

paired conversations, 3 seeds): paired difference vs. full KV with 95% CIs. Hidden-state indicators are mutually indistinguishable, and after correction none differs significantly from full KV. These hidden-state indicators lie near the small multi-turn floor of this protocol (Secs. 4.6, 4.9). Attention scoring exhibits a larger deficit that remains statistically detectable after correction.

Fig. 13: Deletion versus low-precision preservation under the same indicator (Llama-3.1-8B, 3 seeds). (a) SemKV at 2.39 bits shows no statistically detectable deficit from full KV while FP16 keep-25% pruning (a strictly larger, 4-bit-equivalent memory budget) collapses; random pruning is statistically indistinguishable from indicator-based pruning. (b) Per-task collapse depth tracks evidence sparsity. The pruning condition is a controlled structural baseline using the same indicator and budget accounting, not an implementation of an engineered eviction system such as H2O or SnapKV.

Method	Nominal / mem.-equiv. bits	Compr.	Score	$\Delta$ vs. full
Full KV	16.0	$1.0\times$	0.4406	—
SemKV hidden-norm	2.39	$6.0\times$	0.4383	$-0.0023$ (0.70)
Prune keep-25% (hidden-norm)	4.0	$4.0\times$	0.0650	$-0.3756$ ( $p{<}10^{-6}$ )
Prune keep-25% (variance)	4.0	$4.0\times$	0.0676	$-0.3730$ ( $p{<}10^{-6}$ )
Prune keep-25% (cos-delta)	4.0	$4.0\times$	0.1493	$-0.2913$ ( $p{<}10^{-6}$ )
Prune keep-25% (random)	4.0	$4.0\times$	0.0632	$-0.3774$ ( $p{<}10^{-6}$ )

Table 4: Structural comparison between SemKV and FP16 pruning (LongBench composite,

n{=}900

). The primary matched-indicator comparison uses hidden-norm; additional pruning indicators test robustness to the selection rule. Compression is measured KV storage vs. FP16; SemKV occupies a

1.5\times

smaller footprint than the pruning baselines it outperforms. For SemKV rows the bit figure is the nominal average code rate on the level grid; for FP16 keep-

k

pruning it is the memory-equivalent rate

16k

— both are converted to measured storage for the compression column.

Fig. 14: Below-cliff stress (MT-Eval expansion, 10 conversations

\times

3 seeds). (a) At the operating point SemKV significantly outperforms both uniform anchors in this stress setting. (b) Runaway generation is suppressed by SemKV relative to the uniform anchors.

Fig. 15: Widened-gap indicator spectrum under stress (3.0/2.0 bits, average 2.30 bits, MT-Eval expansion). In this Llama stress setting, indicator quality separates when the low-bit side lies below the cliff: informative selection reduces catastrophic degradation when most tokens receive a below-cliff precision; indicators below the random line are worse than random.

Fig. 16: Transfer to Mistral-7B-Instruct-v0.3 (10 subsets

\times

30

\times

3 seeds). (a) The cliff transfers; SemKV at 2.39 bits remains statistically indistinguishable from full KV, whereas random selection at the same budget does not. (b) Per-indicator deltas vs. full KV. On Mistral, informative indicators remain mutually interchangeable, but random allocation is not safe — indicator freedom is an operating-point-specific, not universal, result.

Method	Compr.	Llama-3.1-8B $\Delta$	Mistral-7B $\Delta$
Uniform 3.0-bit	$4.9\times$	$-0.0069$ (0.11)	$-0.0044$ (0.24)
Uniform 2.585-bit	$5.6\times$	$-0.0094$ (0.09)	$-0.0009$ (0.86)
Uniform 2.322-bit	$6.2\times$	$-0.0068$ (0.24)	$-0.0067$ (0.19)
Uniform 2.0-bit	$7.1\times$	$-0.045$ ( $p{<}10^{-6}$ )	$-0.0249$ ( $p{<}10^{-3}$ )
SemKV 2.39-bit (delta-norm)	$6.0\times$	$+0.0011$ (0.84)	$-0.0025$ (0.62)

Table 5: Cross-model transfer (paired

\Delta

vs. full KV,

n{=}900

per model). Absolute scores are not comparable across models. Compression is measured KV storage vs. FP16; the two backbones share the same KV shape, so the ratios apply to both.

Fig. 17: Context-budget sweep on Mistral (3 seeds). Within each budget, SemKV stays closest to full KV among the quantized settings; absolute levels include truncation effects. The measured bracket can depend on context-budget composition even when the primary 7.5k protocol is stable.

Method	Code bits	Eff. bits	Score	$\Delta$ vs. full ( $p$ )
Full KV	16.0	16.0	0.4406	—
Affine uniform 2.322-bit	2.333	2.583	0.4338	$-0.0068$ (0.25)
Affine uniform 2.0-bit	2.000	2.250	0.3958	$-0.0448$ ( $p{<}10^{-7}$ )
TQ uniform 3.0-bit	3.000	3.125	0.4296	$-0.0110$ (0.02 † )
TQ uniform 2.585-bit	2.600	2.725	0.4347	$-0.0059$ (0.23)
TQ uniform 2.322-bit	2.333	2.458	0.4380	$-0.0026$ (0.63)

Table 6: Base-quantizer transfer (Llama-3.1-8B, single-turn,

n{=}900

). Effective bits include packing and per-row metadata (Sec. 4.1: affine and channel-split store two scalars per row, TurboQuant-MSE one). Affine anchors reproduce Sec. 4.2 in the same run. The sub-2-bit rows (separate 3-seed run, shared protocol) locate the TurboQuant cliff at

(1.585,2.0]

; the

r{=}75\%

mix sits inside the affine-inaccessible gap between the 4-level and 3-level grids.

Protocol	Method	Eff. bits	$\Delta$ ( $p$ )	Note
Gen.	Affine gen 2.0b	2.250	$-0.0157$ ( $p{<}10^{-5}$ )	anchor (Sec. 4.4)
Gen.	TQ gen 2.322b	2.458	$-0.0031$ (0.13)	safe
Gen.	TQ gen 2.0b	2.125	$+0.0004$ (0.88)	safe
Gen.	TQ gen 1.585b	1.725	$-0.0031$ (0.34)	safe
Gen.	TQ gen 1.0b	1.125	$-0.0329$ ( $p{<}10^{-18}$ )	gen cliff: $(1.0,1.585]$
MT	Affine uniform 2.0b	2.250	$-0.0700$ ( $p{<}10^{-48}$ )	anchor (Sec. 4.6)

Table 7: Generation-time and multi-turn transfer of the TurboQuant base (3 seeds; generation: LongBench vs. FP16-generation baseline,

n{=}900

; multi-turn: MT-Eval five task types vs. full KV in a dedicated transfer run,

n{=}324

paired conversation evaluations: 108 conversations

\times

3 seeds). Affine anchors reproduce Secs. 4.4/4.6 in the same runs. Multi-turn effective bits include per-row metadata; for mixed rows the reported rate uses the realized token-weighted high-precision fraction over the evaluated conversations rather than the nominal

r

alone. Rows at generation 1.0-bit and multi-turn affine-2.322/TQ-2.585 come from a dedicated add-on run whose continuity anchors reproduce the main run exactly.

Fig. 18: Base-quantizer transfer: single-turn uniform sweep from 3.0 down to 1.0 code bits (3 seeds; error bars: 95% CI of the paired difference). The affine cliff at

(2.0,2.322]

moves one grid step down to

(1.585,2.0]

under TurboQuant-MSE; the SemKV

r{=}75\%

mix occupies the affine-inaccessible gap at code 1.9 bits, statistically indistinguishable from full KV.

Fig. 19: Base-quantizer transfer across protocols (3 seeds; error bars: 95% CI). The downward shift transfers across all three protocols: TurboQuant collapses at 1.585 bits in prefill and multi-turn and at 1.0 bit in generation (n/m: multi-turn not measured at 1.0 bit). Dotted blue markers give same-protocol affine anchors — the 2.0-bit collapse anchors of each protocol, and the multi-turn affine 2.322-bit floor, which TurboQuant shrinks

2.5\times

.

SemKV: Semantic Mixed-Precision KV Cache Quantization Guided by the Quality Cliff for Long-Context LLM Inference

Abstract

Figures & tables

KV-Cache Quantization

ARKV: Adaptive and Resource-Efficient KV Cache Management under Limited Memory Budget for Long-Context Inference in LLMs

CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference

SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference

Abstract

KV-Cache Quantization

Explore similar work

ARKV: Adaptive and Resource-Efficient KV Cache Management under Limited Memory Budget for Long-Context Inference in LLMs

CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference

SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference