EntropyCache: Decoded Token Entropy Guided KV Caching for Diffusion Language Models

Minsoo Cheong; Donghyun Son; Woosang Lim; Sungjoo Yoo

cs.CLMar 19, 2026

EntropyCache: Decoded Token Entropy Guided KV Caching for Diffusion Language Models

Authors: Minsoo Cheong, Donghyun Son, Woosang Lim, Sungjoo Yoo

Organizations: Seoul National University

Abstract

Diffusion-based large language models (dLLMs) rely on bidirectional attention, which prevents lossless KV caching and requires a full forward pass at every denoising step. Existing approximate KV caching methods reduce this cost by selectively updating cached states, but their decision overhead scales with context length or model depth. We propose EntropyCache, a training-free KV caching method that uses the maximum entropy of newly decoded token distributions as a constant-cost signal for deciding when to recompute. Our design is grounded in two empirical observations: (1) decoded token entropy correlates with KV cache drift, providing a cheap proxy for cache staleness, and (2) feature volatility of decoded tokens persists for multiple steps after unmasking, motivating recomputation of the $k$ most recently decoded tokens. The skip-or-recompute decision requires only $O(V)$ computation per step, independent of context length and model scale. Experiments on LLaDA-8B-Instruct and Dream-7B-Instruct show that EntropyCache achieves $15.2\times$ - $26.4\times$ speedup on standard benchmarks and $22.4\times$ - $24.1\times$ on chain-of-thought benchmarks, with competitive accuracy and decision overhead accounting for only $0.5\%$ of inference time. Code is available at https://github.com/mscheong01/EntropyCache.

Figures & tables

Figure 1: (a) Entropy and cosine distance metrics per decoding step experimented on single gsm8k sample using LLADA-8B-Instruct model. (b) Max decoded token entropy vs. avg. value vector cosine distance, plotted on log–log axes.

Figure 2: (a)–(b) PCA projections (PC 1 vs. PC 2) of the last-layer value vectors for two mask tokens over 256 denoising steps in LLADA-8B-Instruct on a single GSM8K sample. Color encodes step progression (dark $\to$ light); the red star marks the decoding step.

Figure 3: Overview of EntropyCache at a single denoising step $t$ . Phase 1 : a full or partial forward pass is executed depending on the recompute flag; in the partial case, only mask tokens and recently decoded tokens are recomputed while the rest are read from cache. Phase 2 : new tokens are decoded from the model logits and the maximum entropy $E^{t+1}$ of the decoded distributions is computed. Phase 3 : the entropy is compared against threshold $\tau$ to set the recompute flag for the next step, and the $k$ most recently decoded tokens $\mathcal{R}^{t+1}$ are selected for partial recomputation.

	GSM8K 256		MATH500 512		MBPP 256		HumanEval 512		Average
Method	Acc.	T-put	Acc.	T-put	Acc.	T-put	Acc.	T-put	Acc.	T-put
LLADA-8B-Instruct
Baseline	77.86	2.28	42.40	2.47	42.20	2.89	46.34	4.32	52.20	2.99
Baseline+Parallel	78.47	8.56 ( $\times$ 3.8)	42.40	9.50 ( $\times$ 3.9)	42.40	16.37 ( $\times$ 5.7)	48.17	15.40 ( $\times$ 3.6)	52.86	12.46 ( $\times$ 4.2)
Fast-DLLM (Dual)	78.01	32.54 ( $\times$ 14.3)	40.00	32.81 ( $\times$ 13.3)	39.20	45.58 ( $\times$ 15.8)	44.51	37.71 ( $\times$ 8.7)	50.43	37.16 ( $\times$ 12.4)
Elastic-Cache	79.15	23.49 ( $\times$ 10.3)	43.80	24.43 ( $\times$ 9.9)	40.60	32.24 ( $\times$ 11.2)	44.51	27.73 ( $\times$ 6.4)	52.02	26.97 ( $\times$ 9.0)

Table 1: Comprehensive dLLM KV cache method comparison. Speedup (

\times

) relative to Baseline. Underlined scores meet or exceed Baseline accuracy. Acc.: Accuracy (%). T-put: Throughput (tok/s). Superscripts indicate max generation length.

	GSM8K 8-shot		MMLU-Pro		BBH 3-shot		GPQA Zero-shot		Average
Method	Acc.	T-put	Acc.	T-put	Acc.	T-put	Acc.	T-put	Acc.	T-put
LLADA-8B-Instruct
Baseline	81.12	2.07	37.60	1.68	59.58	1.35	27.68	5.76	51.49	2.71
Baseline+Parallel	81.43	12.44 ( $\times$ 6.0)	37.57	4.77 ( $\times$ 2.8)	57.90	21.45 ( $\times$ 15.9)	27.68	11.22 ( $\times$ 1.9)	51.15	12.47 ( $\times$ 4.6)
Fast-DLLM (Dual)	80.44	41.68 ( $\times$ 20.1)	37.15	21.56 ( $\times$ 12.8)	52.48	36.89 ( $\times$ 27.3)	24.78	25.34 ( $\times$ 4.4)	48.71	31.37 ( $\times$ 11.6)
Elastic-Cache	81.20	32.89 ( $\times$ 15.9)	36.69	14.78 ( $\times$ 8.8)	52.57	42.39 ( $\times$ 31.4)	24.78	18.08 ( $\times$ 3.1)	48.81	27.04 ( $\times$ 10.0)

Table 2: Chain-of-Thought (CoT) benchmark comparison. Speedup (

\times

) relative to Baseline. Acc.: Accuracy (%). T-put: Throughput (tok/s).

Figure 4: Accuracy–throughput tradeoff on GSM8K (LLaDA-8B-Instruct) for three candidate skipping metrics across varying thresholds

\tau

Figure 5: Grid search over entropy threshold $\tau$ and recent-token budget $k_{\text{recent}}$ on GSM8K. (a, d) Accuracy. (b, e) Throughput. (c, f) Accuracy vs. throughput. Top: LLaDA; bottom: Dream.

Method	Computation	Memory	Bottleneck
$\text{d}^{2}\text{Cache}$ Jiang et al. (2025)	$\mathcal{O}(\ell\cdot L\cdot H)$	$\mathcal{O}(H\cdot L^{2})$	Quadratic in $L$
Elastic-Cache Nguyen-Tri et al. (2025)	$\mathcal{O}(\ell\cdot L\cdot d)$	$\mathcal{O}(\ell\cdot L\cdot d)$	Scales with $\ell$ , $L$ , $d$
EntropyCache (Ours)	$\mathcal{O}(V)$	$\mathcal{O}(k)$	Constant

Table 3: Complexity comparison of dynamic KV cache methods. EntropyCache achieves constant overhead, decoupling decision overhead from both context length and model scale. Notation:

\ell

= number of layers;

L

= context length;

H

= number of attention heads;

d

= hidden dimension;

V

= vocabulary size.

\text{d}^{2}\text{Cache}

and Elastic-Cache overhead grows with

L

and model scale (

\ell

d

), respectively, whereas EntropyCache remains constant regardless of both.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Per-step decoded token entropy vs. average value-vector cosine distance for each benchmark (LLaDA-8B-Instruct, 64 samples per dataset). Gray points denote EOS-filling steps; colored points denote non-EOS content steps. Spearman $\rho$ is computed on non-EOS steps only.

Figure 7: Distribution of per-sample Spearman

\rho

(log–log, non-EOS only) across 64 samples for each benchmark (LLaDA-8B-Instruct). Diamonds indicate means; horizontal bars indicate medians; shaded regions show kernel density estimates.


	$w=8$		$w=16$		$\mathbf{w=32}$		$w=64$		$w=128$
Benchmark	Acc.	T-put	Acc.	T-put	Acc.	T-put	Acc.	T-put	Acc.	T-put
GSM8K	79.08	31.66	78.92	36.68	78.62	38.62	77.33	35.82	68.84	28.99
MATH500	41.60	31.34	40.80	37.45	42.80	39.52	40.60	37.51	38.20	29.77
MBPP	42.00	40.42	40.00	55.52	38.40	64.53	39.20	65.71	37.80	63.93
HumanEval	47.56	31.64	43.90	35.32	47.56	37.86	47.56	36.25	50.00	30.72

Appendix

Table 4: Window size ablation for EntropyCache (

\tau=1.5

k=64

). Acc.: Accuracy (%). T-put: Throughput (tok/s). Bold indicates the selected configuration (

w=32

	$\tau=0.5$	$\tau=1.0$	$\tau=1.5$	$\tau=2.0$	$\tau=2.5$
$k=16$	79.76 / 16.12	79.15 / 30.87	78.47 / 39.44	75.44 / 44.34	74.30 / 46.26
$k=32$	79.83 / 16.10	79.45 / 30.14	79.68 / 38.84	75.28 / 43.69	74.98 / 45.08
$k=64$	79.76 / 16.13	79.38 / 30.16	78.77 / 38.56	76.12 / 42.04	74.30 / 44.07
$k=128$	79.83 / 16.06	78.77 / 29.56	78.32 / 37.32	76.12 / 39.76	75.59 / 40.14
$k=256$	79.83 / 16.04	78.85 / 29.31	78.77 / 35.88	75.74 / 37.44	74.91 / 37.09

Appendix

Table 5: EntropyCache ablation on GSM8K (LLaDA-Inst,

w=32

). Each cell: Accuracy (%) / Throughput (tok/s).

	$\tau=0.5$	$\tau=1.0$	$\tau=1.5$	$\tau=2.0$	$\tau=2.5$
$k=16$	41.20 / 17.66	41.40 / 30.81	41.00 / 42.03	39.60 / 47.40	39.20 / 50.49
$k=32$	40.80 / 17.65	41.40 / 30.44	42.20 / 40.40	43.00 / 46.13	40.60 / 47.90
$k=64$	41.40 / 17.65	41.20 / 30.64	42.80 / 39.83	43.40 / 45.04	39.80 / 46.96
$k=128$	41.20 / 17.48	41.20 / 30.18	43.00 / 37.78	43.20 / 40.82	41.00 / 42.44
$k=256$	41.20 / 17.26	41.00 / 29.61	42.80 / 35.45	43.60 / 36.18	40.20 / 35.06

Appendix

Table 6: EntropyCache ablation on MATH500 (LLaDA-Inst,

w=32

). Each cell: Accuracy (%) / Throughput (tok/s).

	$\tau=0.5$	$\tau=1.0$	$\tau=1.5$	$\tau=2.0$	$\tau=2.5$
$k=16$	42.40 / 33.16	41.80 / 53.91	38.80 / 65.32	40.60 / 70.36	38.60 / 72.36
$k=32$	42.60 / 32.79	41.80 / 53.75	38.80 / 65.19	41.20 / 68.88	40.60 / 71.97
$k=64$	42.60 / 32.80	41.40 / 53.03	38.80 / 63.78	40.00 / 68.68	40.00 / 69.34
$k=128$	42.60 / 32.20	41.60 / 50.95	38.80 / 60.85	40.60 / 65.73	40.80 / 65.63
$k=256$	42.60 / 31.75	41.60 / 50.00	38.60 / 59.14	40.40 / 61.82	41.20 / 61.60

Appendix

Table 7: EntropyCache ablation on MBPP (LLaDA-Inst,

w=32

). Each cell: Accuracy (%) / Throughput (tok/s).

	$\tau=0.5$	$\tau=1.0$	$\tau=1.5$	$\tau=2.0$	$\tau=2.5$
$k=16$	45.73 / 24.55	45.73 / 33.87	47.56 / 38.97	46.34 / 41.73	42.07 / 43.91
$k=32$	45.73 / 24.32	48.78 / 33.13	46.34 / 37.91	44.51 / 40.64	41.46 / 41.00
$k=64$	45.73 / 24.34	48.17 / 33.54	48.78 / 38.10	49.39 / 39.45	42.07 / 39.46
$k=128$	45.73 / 24.30	46.95 / 32.95	48.78 / 36.85	45.12 / 37.41	43.90 / 36.29
$k=256$	45.73 / 24.02	47.56 / 31.85	49.39 / 35.73	47.56 / 33.45	46.34 / 29.19

Appendix

Table 8: EntropyCache ablation on HumanEval (LLaDA-Inst,

w=32

). Each cell: Accuracy (%) / Throughput (tok/s).


	Elastic-Cache		$\text{d}^{2}\text{Cache}$		Ours
Benchmark	RR	T-put	RR	T-put	RR	T-put
GSM8K	0.140	23.49	0.147	29.96	0.114	38.49
MATH500	0.148	24.43	0.146	32.18	0.134	39.64
MBPP	0.243	32.24	0.164	52.77	0.178	64.91
HumanEval	0.168	27.73	0.160	41.36	0.172	38.41

Appendix

Table 9: Recompute ratio (RR) and throughput (T-put, tok/s) for dynamic KV caching methods. Lower RR indicates more cache reuse. Bold: best per benchmark.

Figure 8: Wall-clock time breakdown on a single GSM8K sample (LLaDA-Inst,

w=32

, generation length

=256

). Left: stacked bar chart of inference components. Right: decision overhead decomposition per method. EntropyCache’s decision cost is 71 ms (0.5%), roughly

20\times

and

38\times

cheaper than

\text{d}^{2}\text{Cache}

and Elastic-Cache respectively.

Figure 9: GPU memory usage over time on a single GSM8K sample (LLaDA-Inst,

w=32

, generation length

=256

). Elastic-Cache requires

{\sim}

1 GB more due to extra query and hidden-state caching;

\text{d}^{2}\text{Cache}

and EntropyCache are nearly identical.

Figure 10: Sample responses on GSM8K (ID: 546, ground truth: 130). Highlighted tokens in the EntropyCache response denote decoded tokens with entropy

\geq\tau

that triggered full KV recomputation at the next denoising step.

Figure 11: Sample responses on HumanEval (ID: 127). Highlighted tokens in the EntropyCache response denote decoded tokens with entropy

\geq\tau

that triggered full KV recomputation at the next denoising step.

Diffusion Language Model Inference

Diffusion language model inference studies how to generate, guide, and serve text efficiently through iterative denoising and parallel token updates.

See all 174 papers

Affix Cache for Diffusion Large Language Models
Jun 26, 2026Kaihua Liang, An Zhong, Xin Tan +4KV Caching Memory-Efficient Inference
KAUST · CUHK · LUMS
Fixed State, Long Reach: What a Constant-Size Cache Buys Block Diffusion at Scale
Sep 9, 2026Vaibhav Singh, Pierre-André Noël, Torsten Scholak +2Blockwise Diffusion Language Models Diffusion Language Model Inference
Mila · Concordia University · ServiceNow Research
KV Packet: Recomputation-Free Context-Independent KV Caching for LLMs
Apr 14, 2026Chuangtao Chen, Grace Li Zhang, Xunzhao Yin +3KV Caching KV-Cache Reuse
Technical University of Munich Munich, Germany · Technical University of Darmstadt Darmstadt, Germany · Zhejiang University Hangzhou, China +1

Jun 26, 2026J/K move · Enter open · S save

Affix Cache for Diffusion Large Language Models

Kaihua Liang, An Zhong, Xin Tan, Zafar Ayyub Qazi+3

KAUST · CUHK · LUMS

KV Caching Memory-Efficient Inference Diffusion Language Model Inference Diffusion Language Model Decoding Diffusion Model Caching KV-Cache Reuse

Open paper AskPDF arXiv

Diffusion Large Language Models (DLLMs) enable non-autoregressive decoding, but efficient inference support remains immature: unlike autoregressive models, whose requests reuse a shared prefix key-value (KV) cache, DLLMs use bidirectional attention, so a shared context's KV states depend on the tokens still being decoded, leaving directly reused caches stale and full recomputation necessary. We present ACache, a cross-request cache reuse mechanism for shared spans, or affixes, at any position: prefix, infix, or suffix. ACache measures the influence of affix tokens on the masked generation region to identify a small request-specific subset as Anchor Tokens, and recomputes only their KV states while reusing the remaining affix cache. Built on state-of-the-art intra-request caching mechanisms, ACache recovers most of the accuracy lost to direct affix-cache reuse on average when recomputing around 20% of affix tokens, and at that budget preserves more accuracy than selection criteria adapted from prior cross-request cache-reuse systems. We co-design ACache with a modern inference engine, whose attention reads each request's recomputed Anchor KV states alongside one affix cache shared across concurrent requests. Against the same system with only intra-request caching, ACache cuts recompute latency by up to 56.7%, translating to as much as 1.71

\times

end-to-end throughput, while reducing peak KV cache memory by up to 45.8%.

Abstract

Diffusion Language Model Inference

Explore similar work

Affix Cache for Diffusion Large Language Models

Fixed State, Long Reach: What a Constant-Size Cache Buys Block Diffusion at Scale

KV Packet: Recomputation-Free Context-Independent KV Caching for LLMs