DCKV: Learned Transform Coding of the Prompt KV Cache in Hybrid Language Models

Egor Maximov; Aleksei Merkelov; Vasiliy Alekseev; Ivan Krivorotov; Valentin Beliaev; Nikita Arsenin; Dmitry Bocharov; Aleksei Goncharov

cs.LGOct 27, 2026

DCKV: Learned Transform Coding of the Prompt KV Cache in Hybrid Language Models

Authors: Egor Maximov, Aleksei Merkelov, Vasiliy Alekseev, Ivan Krivorotov, Valentin Beliaev, Nikita Arsenin, Dmitry Bocharov, Aleksei Goncharov

Organizations: MIL Team · DAIMLD

Abstract

Hybrid language models keep a key-value (KV) cache only in their few full-attention layers, but this cache is still the part of a prompt's state that grows with the prompt and that must be stored or moved to reuse it. We present DCKV, a differentiable codec for the KV cache and the first KV-cache codec that is trained end to end. Its transform, its quantizer and its entropy models, which are conditioned on the prompt tokens and on already decoded neighbouring positions, are optimized together against the frozen language model, both for the code length that the entropy models assign and for the change of the model's output. On Qwen3.5-2B, DCKV stores a compressed token in more than 55 times fewer bytes than bf16 at a loss of at most 1.8 points on three long-context benchmarks, and in more than 100 times fewer bytes at a loss of at most 3.6 points.

Figures & tables

Panel of Figure 1 — Figure 1: DCKV. Top: the cache row $x_{t}$ of a token is transformed, quantized and entropy-coded once after prefill, and restored once before decoding from its dequantized coefficients $\tilde{y}_{t}=q_{t}\odot\Delta$ . Bottom: the conditional models tell the entropy coder the mean and the scale of every code from what the decoder already has: the prompt token and, at odd positions, the decoded codes of the two neighbouring tokens.

	Payload, bytes per token			Score change vs. uncompressed, points
Method	RULER 4K	LongBench	RULER 32K	RULER 4K	LongBench	MMLU	RULER 32K
				$n{=}13$	$n{=}18$	$n{=}57$	$n{=}13$
Uncompressed	12,288	12,288	12,288	94.54	42.66	59.34	94.25
KVTC, $16\times$	627	633	548	-0.53	-0.31	-0.62	-0.02
KVTC, $24\times$	410	415	356	-1.87	-1.97	-0.99	-2.45
KVTC, $32\times$	304	307	263	-3.88	-4.17	-0.93	-6.94

Table 1: Payload (bytes per compressed token) and score change against the uncompressed model. The first row gives absolute scores;

n

is the number of tasks. Every DCKV and linear-codec row is a codec trained for its

\lambda

. Linear codec: no nonlinear correction, static entropy model. Bold: on each long-context benchmark, the runs that no other compressed run beats in both payload and score. Table 12 in Appendix F gives their request memory, and Appendix E the benchmarks with shorter prompts. –: not run.

Figure 2: Score change against the payload (bytes per compressed token; bf16 takes 12,288). Lines connect the runs of one family. Not shown: xKV (3,151 and 3,667 bytes per token) and, in (a), the linear codec with the noise proxy at

\lambda{=}3{\cdot}10^{4}

(79 bytes,

-25.3

points).

	RULER 4K		LongBench
Configuration	B/tok.	Change	B/tok.	Change
KVTC $24\times$	410	-1.87	415	-1.97
Separate bases, step 2.0	392	-0.71	389	-1.04
Joint basis, step 2.0	364	-1.30	353	-1.00
Joint basis, step 2.7	265	-2.13	–	–
Trained, noise, $\lambda{=}3{\cdot}10^{5}$	283	-0.71	274	-1.52

Table 2: From KVTC to DCKV: bytes per token and score change. First block: 260–420 bytes per token; second: one run that trains four codecs at

\lambda{=}3{\cdot}10^{5}

; third: 150–185 bytes. Bold: the fewest bytes and the best score change of each benchmark within a block.

Figure 3: Request memory against prompt length, computed from the payload per token on RULER 4K. Markers show measured values on RULER 4K, LongBench and RULER 32K (mean prompts of 3.7K, 10.8K and 32.2K tokens).

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

$\beta$	$P(Q\neq 0)$	$H(Q)$	$h(Z)$	$h(Z)/H(Q)$
1.0	0.607	2.484	2.492	1.00
0.5	0.368	1.560	1.606	1.03
0.2	0.082	0.496	0.730	1.47
0.1	0.007	0.065	0.368	5.66

Appendix

Table 3: Entropy of the rounded code,

H(Q)

, and its uniform-noise estimate,

h(Z)

, in bits, for a Laplace coefficient with scale

\beta

in units of the step.

	Payload, bytes per token			Score change vs. uncompressed, points
Method	RULER 4K	LongBench	RULER 32K	RULER 4K	LongBench	MMLU	RULER 32K
				$n{=}13$	$n{=}18$	$n{=}57$	$n{=}13$
Uncompressed	12,288	12,288	12,288	94.54	42.66	59.34	94.25
DCKV, $\lambda{=}3{\cdot}10^{5}$	214	205	221	-0.56	-1.75	-1.10	-0.45
steps $\times 0.5$	433	421	441	-0.28	+0.26	-1.34	+0.24
DCKV, range, $\lambda{=}3{\cdot}10^{5}$	211	218	–	-1.33	-3.17	–	–

Appendix

Table 4: Ways to change the rate without a codec per rate, in the form of Table 1 : all steps of a trained codec halved, conditional models fitted after training, and DCKV trained as one codec for

\lambda\in[5{\cdot}10^{4},3{\cdot}10^{5}]

. Indented rows change the first row of their block without retraining. –: not run.

Figure 4: As Figure 2 , for two ways to change the rate without a codec per rate: DCKV with halved steps, and one codec trained for a range of

\lambda

, with the additions (DCKV) and without them (linear codec).

	$\lambda{=}5{\cdot}10^{4}$		$\lambda{=}10^{5}$		$\lambda{=}3{\cdot}10^{5}$
Codec	B/tok.	Change	B/tok.	Change	B/tok.	Change
No additions	105.7	-6.06	139.9	-3.23	209.5	-1.34
+ models fitted after training	87.2	-6.06	117.7	-3.23	207.2	-1.34
Nonlinear correction	90.4	-4.70	124.6	-2.35	215.2	-1.30
Trained conditional models	97.3	-4.22	126.9	-2.50	190.9	-1.02
Both: DCKV	83.4	-4.05	117.6	-1.94	210.7	-1.33

Appendix

Table 5: RULER 4K results (

n{=}13

) of four codecs trained in one run of 1B tokens, each for

\lambda\in[5{\cdot}10^{4},3{\cdot}10^{5}]

;

\lambda

is chosen at evaluation. B/tok.: bytes per compressed token; Change: points against the uncompressed model. Models fitted after training do not change the scores. Last row: the three linear codecs of Table 1 .

Development codec
Entropy model	Rank	B/tok.	B/tok. change	MiB
Static Laplace	–	178.1	–	99.1
Token	2048	166.5	-6.5%	131.9
Token	256	167.1	-6.2%	103.9
Neighbours + token	1024	152.6	-14.3%	163.9
Neighbours + token	256	160.5	-9.9%	139.9

Appendix

Table 6: Conditional models fitted in closed form, on RULER 4K: a development codec and the linear codec of Table 1 ; all rows of a block give identical scores. Long streams: rANS streams of 131,072 symbols. MiB: weights (–: not measured).

Language model	Qwen3.5-2B, bf16, frozen
Codec size	$d=6144$ , $n=2048$
Initialization	joint PCA of up to 200K token positions from 48 calibration documents (32 from FineWeb, 16 from OpenR1-Math; 1,008 to 16,218 tokens, 200,192 in total; also used for KVTC); all steps 1.0
Training data	FineWeb-Edu, a 1M-document subset of its 10B-token sample; 4,096 prompt + 256 continuation tokens per sequence; 100 held-out documents
Multipliers $\lambda$	$3{\cdot}10^{5}$ (DCKV and its three ablations); $10^{5}$ and $5{\cdot}10^{4}$ (DCKV), $10^{5}$ (codec without additions); $[5{\cdot}10^{4},3{\cdot}10^{5}]$ (range codecs); $5{\cdot}10^{4}$ , $10^{5}$ , $3{\cdot}10^{5}$ (linear codecs); $3{\cdot}10^{4}$ , $10^{5}$ , $3{\cdot}10^{5}$ (noise proxy)
Additions	correction: one hidden layer of 256 units, GELU, on encoder and decoder, last layer starts at zero (16 MiB); conditional models of rank 2048 (token) and 1024 (context), 32 MiB each, start from their closed-form fit
Range codecs	three step vectors, interpolated over $\ln\lambda$ ; $\lambda$ of a training sequence uniform in $\ln\lambda$

Appendix

Table 7: Training and evaluation settings.

Comparison (A vs B)	Benchmark	$\Delta$	$n$	W/T/L	$p$	B/tok. A	B/tok. B
DCKV $\lambda{=}3{\cdot}10^{5}$ vs uncompressed	RULER 4K	-0.56	13	3/4/6	0.508	214	–
	LongBench	-1.75	18	0/1/17	$<$ 0.001	205	–
	RULER 32K	-0.45	13	3/5/5	0.727	221	–
	MMLU	-1.10	57	15/3/39	0.001	196	–
KVTC $16\times$ vs DCKV $\lambda{=}3{\cdot}10^{5}$	RULER 4K	+0.03	13	5/5/3	0.727	627	214
	LongBench	+1.44	18	14/0/4	0.031	633	205

Appendix

Table 8: Paired comparisons. B/tok.: bytes per compressed token of A and B on the benchmark (–: uncompressed). Fitted models: conditional models fitted in closed form after training to the codec without additions of the same run.

Method	MMLU	GPQA	IFEval
Uncompressed	59.34	38.38	64.70
KVTC, $16\times$	-0.62	-3.03	+1.10
KVTC, $24\times$	-0.99	-1.51	+1.10
KVTC, $32\times$	-0.93	-1.51	+1.10
xKV, ranks 192/288	-0.42	-2.02	+1.66
DCKV, $\lambda{=}3{\cdot}10^{5}$	-1.10	+1.52	+1.10

Appendix

Table 9: Score change on the benchmarks with short prompts. The first row gives absolute scores. MMLU: 57 subjects; GPQA: 198 questions; IFEval: 541 prompts.

Figure 5: Score change on MMLU, GPQA Diamond and IFEval against the payload of the same codec on RULER 4K. Same families and markers as Figure 2 . On IFEval every run scores above the uncompressed reference.

Method	Store	TTFT	TPOT	Store	TTFT	TPOT
	RULER 4K (3.7K tokens)			RULER 32K (32.2K tokens)
Uncompressed	–	862	45.2	–	4,601	42.8
KVTC, $16\times$	865	121	45.7	6,333	6,849	44.4
DCKV, $\lambda{=}3{\cdot}10^{5}$	858	115	44.7	4,445	287	42.1

Appendix

Table 10: Time (ms) of one run of each method, made one after another on one A100 of a shared node; means over about 1,300 prompts. Store: prefill and compression when the prompt is stored. TTFT: from the start of generation to the first token; for a compressed run it includes the decompression of the stored cache, for the uncompressed run the prefill. TPOT: time per output token.

	RULER 4K ( $n{=}13$ )		LongBench ( $n{=}18$ )		RULER 32K ( $n{=}13$ )
Method	Request, MiB	DCKV $-$ KVTC	Request, MiB	DCKV $-$ KVTC	Request, MiB	DCKV $-$ KVTC
DCKV, $\lambda{=}3{\cdot}10^{5}$	12.1		13.5		18.2
KVTC $16\times$	13.5 (+11.7%) ∗	-0.03 (3/5/5)	17.8 (+32.4%) ∗	-1.44 (4/0/14)	28.1 (+55.0%) ∗	-0.43 (2/5/6)
KVTC $24\times$	12.8 (+5.6%)	+1.31 (8/1/4)	15.6 (+15.9%) ∗	+0.22 (13/1/4)	22.3 (+22.8%) ∗	+2.00 (9/3/1)
KVTC $32\times$	12.4 (+2.5%)	+3.33 (10/0/3)	14.5 (+7.8%)	+2.42 (16/0/2)	19.4 (+7.0%)	+6.49 (12/1/0)

Appendix

Table 11: DCKV (

\lambda{=}3{\cdot}10^{5}

) against KVTC at similar request memory. Request: request memory and its difference from DCKV ( ∗ outside

\pm 10\%

, the margin within which we treat request memory as comparable; taken relative to KVTC, the differences mark the same pairs). DCKV

-

KVTC: paired difference in points, with the tasks where DCKV is higher/equal/lower. Codec weights are not included (Table 12 ).

	Request memory, MiB			Weights	Peak GPU memory, MiB
Method	RULER 4K	LongBench	RULER 32K	MiB	RULER 4K	LongBench	RULER 32K
Uncompressed (bf16)	54.0	137.7	387.2	0.0	4,035	4,823	7,213
KVTC, $16\times$	13.5	17.8	28.1	20.1	4,057	4,846	7,234
KVTC, $24\times$	12.8	15.6	22.3	13.6	4,051	4,839	7,228
KVTC, $32\times$	12.4	14.5	19.4	10.4	4,047	4,836	7,224
xKV, ranks 192/288	22.9	42.4	–	0.0	4,035	4,822	–

Appendix

Table 12: Memory of the runs of Table 1 on one A100 with one request at a time. Request memory (§ 3.6 ) includes the recurrent state (9.84 MiB); for the uncompressed model it also holds the generated tokens (under 1%), and for xKV the bases that it computes for each prompt. Weights: codec weights loaded with the model; Peak: peak GPU memory. LongBench: 18 tasks. Weights of DCKV: fp32 parameters of the final checkpoint; the other weights are as loaded, with coding tables (99.1 MiB for the linear codec, of which 96.0 are parameters, and less at

\lambda{=}5{\cdot}10^{4}

, which keeps 1,754 of 2,048 components).

Figure 6: Validation during training of the linear codecs, with mixed quantization and with the noise proxy. (a) Rate of rounded codes against training tokens. (b) KL divergence against rate; each curve starts at the untrained codec (diamond), and circles mark the end of training (hollow: noise proxy).

Figure 7: Score change per task for DCKV (

\lambda{=}3{\cdot}10^{5}

) with its trained steps and with halved steps and for KVTC at

16\times

and

32\times

(payloads on RULER 4K). Tasks are sorted by the mean change of the four, the hardest at the bottom. UUID: needles with UUID values.

Task	Uncompr.	DCKV	DCKV $\times 0.5$	Linear $3{\cdot}10^{5}$	Linear $10^{5}$	Linear $5{\cdot}10^{4}$	KVTC $16\times$	KVTC $24\times$	KVTC $32\times$
cwe	96.7	-1.3	-1.1	-1.5	-7.5	-10.7	-2.2	-6.7	-10.8
fwe	89.7	+0.7	-0.3	-1.3	-5.0	-6.7	-4.0	-4.7	-3.0
niah_multikey_1	100.0	+0.0	+0.0	-1.0	-3.0	-1.0	+0.0	+0.0	-1.0
niah_multikey_2	100.0	-1.0	+0.0	-1.0	+0.0	-1.0	+0.0	+0.0	+0.0
niah_multikey_3	100.0	-1.0	+0.0	-1.0	-7.0	-19.0	+0.0	-4.0	-12.0
niah_multiquery	100.0	+0.0	+0.0	+0.0	-0.7	-1.5	+0.0	-0.7	-2.2

Appendix

Table 13: RULER 4K per task: score of the uncompressed model and change of each configuration, in points. DCKV:

\lambda{=}3{\cdot}10^{5}

;

\times 0.5

: halved steps. cwe: common-word extraction; fwe: frequent-word extraction; niah: needle in a haystack; qa: question answering; vt: variable tracking. The last row is the paired mean change over the 13 tasks.

Task	Uncompr.	DCKV	DCKV $\times 0.5$	Linear $3{\cdot}10^{5}$	Linear $10^{5}$	Linear $5{\cdot}10^{4}$	KVTC $16\times$	KVTC $24\times$	KVTC $32\times$
2wikimqa	34.1	-1.7	+1.0	+2.1	-1.3	-4.2	-1.2	-1.0	-2.6
dureader	24.4	-1.1	+0.1	-0.4	-1.5	-1.6	+0.0	-1.4	-2.6
gov_report	30.0	-0.7	+0.3	-0.8	-0.9	-1.1	-0.4	-0.9	-1.7
hotpotqa	46.8	-0.3	+0.4	-0.3	-2.1	-6.2	-2.8	-5.9	-5.5
lcc †	2.0	+1.8	+1.5	+1.5	+1.4	+1.2	+1.0	+1.9	+2.3
lsht	31.0	-4.0	+7.5	-0.5	-1.0	-2.0	+6.5	+5.0	-8.9

Appendix

Table 14: LongBench per task: score of the uncompressed model and change of each configuration, in points. † Left out of all means: LCC and RepoBench-P score near zero for every configuration in our harness, and on SAMSum and RepoBench-P every run of our codecs and of KVTC scores above the uncompressed reference (§ 4 ). The last row is the paired mean change over the other 18 tasks.

KV-Cache Compression

Key-value cache compression reduces the memory and bandwidth needed to retain attention states during long-context inference.

See all 269 papers

EntropyCache: Decoded Token Entropy Guided KV Caching for Diffusion Language Models
Mar 19, 2026Minsoo Cheong, Donghyun Son, Woosang Lim +1Diffusion Language Model Inference Diffusion Model Caching
Seoul National University
HeadWiseKV: Budgeted Per-Head Cache Residency for Hybrid Long-Context Language Models
Sep 2, 2026Renjie Xie, Juncheng Yang, Aoting Hu +4KV Caching Long-Context Language Model Inference
Nanjing University of Posts and Telecommunications · Tylogi AI Lab / TAIL · Anhui University of Technology +2
VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization
Mar 17, 2026Yixuan Wang, Qingyu Shi, Jiayu Zhou +3KV-Cache Compression KV-Cache Quantization
LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai Innovation Institute · Shanghai Artificial Intelligence Laboratory +2

Mar 19, 2026J/K move · Enter open · S save

EntropyCache: Decoded Token Entropy Guided KV Caching for Diffusion Language Models

Minsoo Cheong, Donghyun Son, Woosang Lim, Sungjoo Yoo

Seoul National University

Diffusion Language Model Inference Diffusion Model Caching Diffusion Language Models

Open paper AskPDF arXiv

Diffusion-based large language models (dLLMs) rely on bidirectional attention, which prevents lossless KV caching and requires a full forward pass at every denoising step. Existing approximate KV caching methods reduce this cost by selectively updating cached states, but their decision overhead scales with context length or model depth. We propose EntropyCache, a training-free KV caching method that uses the maximum entropy of newly decoded token distributions as a constant-cost signal for deciding when to recompute. Our design is grounded in two empirical observations: (1) decoded token entropy correlates with KV cache drift, providing a cheap proxy for cache staleness, and (2) feature volatility of decoded tokens persists for multiple steps after unmasking, motivating recomputation of the

k

most recently decoded tokens. The skip-or-recompute decision requires only

O(V)

computation per step, independent of context length and model scale. Experiments on LLaDA-8B-Instruct and Dream-7B-Instruct show that EntropyCache achieves

15.2\times

26.4\times

speedup on standard benchmarks and

22.4\times

24.1\times

on chain-of-thought benchmarks, with competitive accuracy and decision overhead accounting for only

0.5\%

of inference time. Code is available at https://github.com/mscheong01/EntropyCache.

Abstract

KV-Cache Compression

Explore similar work

EntropyCache: Decoded Token Entropy Guided KV Caching for Diffusion Language Models

HeadWiseKV: Budgeted Per-Head Cache Residency for Hybrid Long-Context Language Models

VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization