One Distribution, Two Channels: Coverage-Calibrated KV Retention and Compensation

Shuxin Liu; Qing Liu; Yi Du; Ou Wu

cs.ARSep 28, 2026

One Distribution, Two Channels: Coverage-Calibrated KV Retention and Compensation

Authors: Shuxin Liu, Qing Liu, Yi Du, Ou Wu

Organizations: Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences

Abstract

Long-context decoding is increasingly constrained by key--value (KV) cache memory and bandwidth. Existing fixed-budget compression methods typically separate retention from compensation, while a retention ranking specifies neither discarded attention mass nor the direction of induced output error. We start from an exact factorization: eviction error equals evicted attention mass times the directional gap between the evicted centroid and retained output, highlighting the importance of set-level coverage in retention and mass-preserving memory writing. We introduce CORE COverage Calibration and Evicted-Mass REdistribution for KV Cache, which distills an offline allocation combining query utility and log-determinant coverage into a lightweight cache-aware indexer. At inference, one calibrated distribution drives both channels: its Top- $B$ ordering retains complementary KV states, while its excluded allocation mass and conditional weights parameterize latent-memory writes without a separate write-weight predictor or online log-determinant evaluation. Our analysis provides a four-term pre-compensation error certificate, characterizes non-additive coverage interactions, and establishes mass-independent write stability with hierarchical bounds through recurrent and query-adaptive normalization. Across three backbones, CORE exceeds the strongest RULER baseline by up to 3.78 points at 90% compression; LongBench and repeated-eviction evaluations further demonstrate strong effectiveness and decoding efficiency.

Figures & tables

Figure 1: From separate retention and compensation to a unified eviction error perspective

Figure 2: CORE couples KV retention and compensation via a coverage-calibrated distribution.

Model	Method	RULER-4K Avg.					RULER-16K Avg.
Model	Method	0%	25%	50%	75%	90%	0%	25%	50%	75%	90%
Mistral-7B	H 2 O \corevenuebox NeurIPS’23	93.29	68.65	49.13	34.36	21.15	80.95	52.49	37.11	24.55	15.74
	SnapKV \corevenuebox NeurIPS’24	93.29	58.64	42.71	33.16	20.91	80.95	44.61	32.06	24.57	19.04
	PyramidKV \corevenuebox COLM’25	93.29	62.14	41.68	30.92	20.92	80.95	49.04	34.41	23.50	17.82
	Expected Attention \corevenuebox arXiv’25	93.29	72.60	48.57	28.59	13.74	80.95	59.06	43.28	28.94	15.79
	KeyDiff \corevenuebox NeurIPS’25	93.29	0.00 87.09	81.58	0.00 70.49	0.00 52.17	80.95	75.26	0.00 70.79	65.80	50.40

Table 1: RULER results across KV compression ratios. 0% denotes full KV cache. Within each model and compressed setting, red/bold and orange/underlined indicate best and second-best results.

Figure 3: LongBench performance across six representative tasks and four compression ratios.

Method	AIME25				Math500
	64	128	256	512	64	128	256	512
CAPKV	26.67	40.00	43.33	40.00	32.20	30.00	43.40	39.60
IndexMem	23.33	33.33	43.33	46.67	35.20	40.60	47.40	50.20
MOMENTKV	20.00	30.00	36.67	43.33	28.40	37.60	45.20	49.00
CORE	33.33	43.33	50.00	53.33	37.60	43.00	49.80	52.60

Table 2: Decode-time compression results.

Figure 4: Position-sensitive needle retrieval across context lengths and evidence depths.

Figure 5: Mechanistic analysis of CORE’s retention, allocation fidelity, and write stability.

Figure 6: CORE Evaluation across Compression Performance, Component Analysis, and Efficiency.

Write allocation	RULER-4K		RULER-16K
Write allocation	75%	90%	75%	90%
CORE	80.60	62.44	74.98	68.48
$\text{CORE}_{\text{ind}}$	79.37	61.82	73.02	67.43

Table 3: Write allocation comparison.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Retention principle	Evicted-state representation	Compensation	Retention–compensation link
CAPKV	Information-capacity selection	None	None	Retention only
MOMENTKV	Moment-aware eviction	Low-order moments	First-order approximation	Shared moment statistics
IndexMem	Learned importance indexer	Latent memory	Gated residual readout	Separate modules
CORE	Coverage-calibrated allocation	Allocation-weighted latent memory	Gated residual reconstruction	Shared allocation

Appendix

Table 4: Conceptual positioning of closely related fixed-budget KV-cache methods.

Steps	Median $e_{R}$	P90 $e_{R}$	Overlap vs. 16
$1$	$7.643\times 10^{-3}$	$2.407\times 10^{-2}$	92.80%
$R_{0}=8$	$2.176\times 10^{-7}$	$5.564\times 10^{-5}$	99.74%
$2R_{0}=16$	$4.025\times 10^{-12}$	$1.496\times 10^{-7}$	ref.

Appendix

Table 5: Teacher residuals and Top-

B

selection agreement (relative to

R=16

Comp.	$\mu_{E}$	$\bar{p}(E)$	$M$ , Bound	RetErr	Tight.
25%	0.120	0.3251	0.370	0.2471	0.668
50%	0.300	0.5427	0.550	0.4354	0.792
75%	0.520	0.7436	0.770	0.5289	0.687
90%	0.680	0.8718	0.930	0.5392	0.580

Appendix

Table 6: Empirical normalized eviction-certificate audit. All columns use common state weights.

M

and Bound agree at the displayed precision. Tightness

=\mathrm{RetErr}/\mathrm{Bound}

Compression	Selector	$U_{B}$	$C_{B}$	$J_{B}$
75%	Utility-only	1.3689	0.9883	2.3572
	Student Top- $B$	1.1548	1.3262	2.4810
	Teacher Top- $B$	1.1503	1.3334	2.4837
	Greedy	1.1140	1.3911	2.5051
90%	Utility-only	1.4352	0.9351	2.3703
	Student Top- $B$	1.1522	1.3616	2.5138

Appendix

Table 7: Fixed-budget utility–coverage set quality.

Figure 7: Repeated-write compensation at 75% initial compression: (a) compensation gain, (b) residual–correction alignment

\rho_{g}

, and (c) relative correction magnitude

r_{g}

Writing rule	$G^{(0)}$ (%)	$G^{(\delta)}$ (%)	$\Delta G$
Mass-weighted	50.38	49.72	0.66
Equal-event	34.38	33.94	0.44
Uniform-token	35.73	35.16	0.57

Appendix

Table 8: Fourth-event compensation gain at

\delta=0.05

. Gains are percentages;

\Delta G=G^{(0)}-G^{(\delta)}

is the percentage-point decrease.

Component	Setting
Indexer	$14\!\rightarrow\!128\!\rightarrow\!128\!\rightarrow\!1$ , GELU
Projected key space	$d_{z}=64$
Coverage prototypes	$M_{p}=64$ , fixed
Memory map	$\phi_{\varphi}:\mathbb{R}^{D}\!\rightarrow\!\mathbb{R}^{512}$ , no bias
Memory state	$M_{t}\!\in\!\mathbb{R}^{512\times D}$ , $b_{t}\!\in\!\mathbb{R}^{512}$ , $Z_{t}\!\in\!\mathbb{R}$
Memory readout	$m(q)\!\in\!\mathbb{R}^{D}$ (pre- $W_{O}$ space)

Appendix

Table 9: Implementation details.

Method	NQA	Qasper	MF	HPQA	2WQA	MQue	GR	QMS	MN	TREC	TQA	SSum	PC	PR	LCC	RBP	Avg.
	SD-QA			MD-QA			Summ.			FSL			Synth.		Code
Full KV	30.99	47.39	56.31	59.80	51.33	33.51	35.19	25.34	26.99	29.00	91.71	40.77	10.70	100.00	53.31	47.23	46.22
SnapKV	29.00	30.97	38.87	53.67	41.53	29.11	29.23	22.69	23.25	34.50	91.16	41.64	9.05	95.00	54.76	47.70	42.01
PyramidKV	27.66	30.01	36.23	55.69	42.48	25.76	28.29	22.42	23.49	32.50	91.73	42.10	9.05	97.50	54.73	47.40	41.69
ExpectedAttention	29.77	38.94	44.37	55.93	43.02	34.13	28.85	23.01	24.94	54.75	91.18	37.09	8.50	96.00	40.40	51.12	43.88
KeyDiff	29.75	32.87	42.45	53.05	39.11	25.40	29.49	23.66	22.78	51.50	91.61	40.85	11.06	99.50	37.43	47.43	42.37

Appendix

Table 10: Per-task LongBench results for Llama-3.1-8B at 75% compression. Tasks follow the official LongBench taxonomy; Avg. is the unweighted mean over all 16 tasks.

Method	S1	S2	S3	MK1	MK2	MK3	MV	MQ	VT	CWE	FWE	QA1	QA2	Avg.
Full KV	100.00	100.00	100.00	99.60	100.00	98.80	99.10	99.10	99.72	89.32	90.27	81.00	57.00	93.38
SnapKV	79.20	92.20	2.60	71.20	15.60	2.60	50.90	57.05	69.44	35.58	84.93	32.80	37.20	48.56
PyramidKV	85.80	95.20	2.40	80.00	18.60	2.80	56.20	59.30	76.84	11.26	81.33	33.40	33.80	48.99
ExpectedAttention	100.00	99.60	10.20	96.20	64.40	10.80	80.25	97.50	96.04	28.14	83.67	51.80	43.40	66.31
KeyDiff	100.00	100.00	100.00	99.60	26.20	0.00	99.10	99.45	98.64	71.52	87.80	44.80	44.60	74.75
CAPKV	100.00	100.00	78.20	97.80	14.20	0.40	94.95	96.45	99.48	37.50	54.67	62.20	44.00	67.68

Appendix

Table 11: Per-subtask RULER-16K results for Llama-3.1-8B at 75% compression; Avg. is the unweighted mean over all 13 official subtasks.

Method	KV budget
Method	2K	4K	8K
Full KV	32.02
SnapKV	25.12	26.60	27.09
EA	24.63	25.12	26.11
KeyDiff	25.62	26.11	25.62
CAPKV	25.62	26.60	27.09

Appendix

Table 12: Ultra-long-context accuracy (%).

Figure 8: One-at-a-time hyperparameter sensitivity at 75% compression.

Variant	RULER-4K		RULER-16K
Variant	75%	90%	75%	90%
CORE	80.60	62.44	74.98	68.48
$\text{CORE}_{\text{EA}}$	78.22	50.39	70.65	56.87
$\text{CORE}_{\text{KeyDiff}}$	76.78	61.53	74.87	67.16

Appendix

Table 13: CORE allocation replacement.

Training Source	RULER-4K		RULER-16K		LongBench	Code Avg.
Training Source	75%	90%	75%	90%	75%	75%
LongAlpaca-12k	80.60	62.44	74.98	68.48	46.18	49.42
OpenMathReasoning	80.18	62.69	74.63	68.16	45.61	47.38
SWE-smith-trajectories	79.72	61.83	74.41	67.54	45.83	50.12

Appendix

Table 14: Training-source sensitivity on Llama-3.1-8B. Code Avg. averages LCC and RepoBench-P. Higher scores are better; column-wise best results are bold.

Model	Method	RULER-4K		RULER-16K
Model	Method	75%	90%	75%	90%
Mistral-7B	Locret	$63.72_{\pm{0.26}}$	$37.03_{\pm{0.09}}$	$49.85_{\pm{0.35}}$	$35.53_{\pm{0.03}}$
	TRIM-KV	$68.24_{\pm{0.35}}$	$42.89_{\pm{0.07}}$	$65.95_{\pm{0.05}}$	$53.39_{\pm{0.07}}$
	KVP	$66.54_{\pm{0.32}}$	$42.80_{\pm{0.29}}$	$61.80_{\pm{0.06}}$	$48.68_{\pm{0.35}}$
	IndexMem	$68.95_{\pm{0.16}}$	$43.19_{\pm{0.12}}$	$66.41_{\pm{0.09}}$	$53.89_{\pm{0.16}}$
	CORE	$\mathbf{73.52}_{\pm{0.27}}$	$\mathbf{54.73}_{\pm{0.13}}$	$\mathbf{69.02}_{\pm{0.21}}$	$\mathbf{56.43}_{\pm{0.19}}$

Appendix

Table 15: Training-seed sensitivity on RULER.

Table 24

Method	Add. Params.	Aux. State	Online Overhead
CAPKV	0.00M	$\approx 0$ GiB	$+12.8\%$
IndexMem	$\approx 40$ M	0.27 GiB	$+24.1\%$
CORE	$\approx 40$ M	0.22 GiB	$+7.7\%$

Appendix

Table 18: Additional parameter and runtime costs.

Metric	Method	8K	16K	32K	64K
Prefill latency	CAPKV	1.257	2.914	7.233	21.127
	IndexMem	1.274	3.139	8.062	23.031
	CORE	1.388	3.597	9.402	27.562
TTFT	CAPKV	1.260	2.917	7.236	21.130
	IndexMem	1.277	3.144	8.068	23.036
	CORE	1.393	3.602	9.406	27.567

Appendix

Table 19: Prefill latency and GPU-resident TTFT (s) for Qwen3-14B at 75% compression.

KV-Cache Eviction

Key-value (KV) cache eviction reduces language-model memory use by removing or compressing cached tokens while preserving useful context.

See all 99 papers

RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cache
May 8, 2026Junkai Zhang, Hang Guo, Luca Benini +1KV Caching Long-Context Language Model Inference
ETH Zurich · Tsinghua University
ResKV: Reconstructing Omitted Attention Contributions for Fixed-Budget KV Cache Compression
Jul 31, 2026Yuhang Zhan, Lisi Chen, Shuo ShangSoftmax Attention KV Caching
University of Electronic Science and Technology of China
Read What Matters: Query-Adaptive Quantization for KV Caches
Oct 8, 2026Siddharth Bhandari, Lucas Gretta, Krishna Balasubramanian +1KV-Cache Quantization KV-Cache Compression
Amazon · University of California, Berkeley

May 8, 2026J/K move · Enter open · S save

RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cache

Junkai Zhang, Hang Guo, Luca Benini, Yawei Li

ETH Zurich · Tsinghua University

KV Caching Long-Context Language Model Inference KV-Cache Eviction Rate-Distortion Optimization KV-Cache Compression KV-Cache Quantization

Open paper AskPDF arXiv

Large language models (LLMs) have shown strong performance across diverse tasks, but their inference with long input contexts is bottlenecked by memory size and bandwidth. The Key-Value (KV) cache size grows linearly with sequence length and needs to be re-read from off-chip high-bandwidth memory (HBM) to on-chip memory at every decoding step, resulting in memory-bound inference. Existing methods reduce the cache by either eviction or quantization, but typically treat the two in isolation. In this paper, we cast KV cache compression as a rate-distortion problem, under which eviction and quantization are two end-points of the same bit allocation scheme. This exposes the need to optimize them jointly, motivating our method, RDKV (Rate-Distortion KV cache compression). RDKV derives the weight of each token or channel from the distortion that compression induces on the attention computation. Based on these weights, it assigns each token or channel a bit-width ranging from full precision down to zero bits guided by reverse water-filling, applied once after the prefilling stage. Experiments on LongBench, RULER, and InfiniteBench show that RDKV outperforms the best evaluated baseline by 9.1% on average. On LongBench it recovers 97.81% of full-cache accuracy with only 2.48% cache retention. Compared with full-cache FlashAttention-2 decoding, it achieves 4.5x decode speedup and 1.9x peak memory reduction with 128K context length, while maintaining comparable performance.

Abstract

KV-Cache Eviction

Explore similar work

RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cache

ResKV: Reconstructing Omitted Attention Contributions for Fixed-Budget KV Cache Compression

Read What Matters: Query-Adaptive Quantization for KV Caches