Affix Cache for Diffusion Large Language Models

Kaihua Liang; An Zhong; Xin Tan; Zafar Ayyub Qazi; Hong Xu; Jian Weng; Marco Canini

cs.CLJun 26, 2026

Affix Cache for Diffusion Large Language Models

Authors: Kaihua Liang, An Zhong, Xin Tan, Zafar Ayyub Qazi, Hong Xu, Jian Weng, Marco Canini

Organizations: KAUST · CUHK · LUMS

Abstract

Diffusion Large Language Models (DLLMs) enable non-autoregressive decoding, but efficient inference support remains immature: unlike autoregressive models, whose requests reuse a shared prefix key-value (KV) cache, DLLMs use bidirectional attention, so a shared context's KV states depend on the tokens still being decoded, leaving directly reused caches stale and full recomputation necessary. We present ACache, a cross-request cache reuse mechanism for shared spans, or affixes, at any position: prefix, infix, or suffix. ACache measures the influence of affix tokens on the masked generation region to identify a small request-specific subset as Anchor Tokens, and recomputes only their KV states while reusing the remaining affix cache. Built on state-of-the-art intra-request caching mechanisms, ACache recovers most of the accuracy lost to direct affix-cache reuse on average when recomputing around 20% of affix tokens, and at that budget preserves more accuracy than selection criteria adapted from prior cross-request cache-reuse systems. We co-design ACache with a modern inference engine, whose attention reads each request's recomputed Anchor KV states alongside one affix cache shared across concurrent requests. Against the same system with only intra-request caching, ACache cuts recompute latency by up to 56.7%, translating to as much as 1.71 $\times$ end-to-end throughput, while reducing peak KV cache memory by up to 45.8%.

Figures & tables

Figure 1: Cache reuse in AR LLMs and DLLMs. Left : AR LLMs reuse a shared prefix cache directly. Middle : in DLLMs, bidirectional attention makes that cache depend on the tokens still being decoded, leaving it stale. Right : ACache recomputes only the Anchor Tokens’ KV states.

Figure 2: Quality–throughput tradeoff. ACache’s Fast-dLLM-based system on LLaDA/GSM8K, 2-shot prefix, batch 16. Labels: Recompute ratios.

Figure 3: Anchor Token importance. Each panel shows a randomly selected GSM8K sample (Section 5 setup, 2-shot prompts). Color encodes mask-to-affix attention importance over affix tokens (vertical) and decoding steps (horizontal), highlighting a small subset at stable positions across steps.

Figure 4: System architecture of ACache. Left : requests reuse one precomputed affix cache as an infix (A), prefix (B), or suffix (C), selecting request-specific Anchor Tokens. Right : one request’s read-slot map, resolving each logical position to a private or shared KV slot.

Figure 5: Accuracy with ACache. Accuracy increases with the Anchor ratio (0: direct reuse; 1: full recomputation), except BABILong suffix, which declines overall.

Dataset	Batch	1-shot		2-shot		4-shot
		Fast-dLLM	ACache	Fast-dLLM	ACache	Fast-dLLM	ACache
GSM8K	1	56.3	38.9 ( $30.8\%\downarrow$ )	91.2	46.1 ( $49.5\%\downarrow$ )	124.7	60.0 ( $51.9\%\downarrow$ )
	4	62.8	43.5 ( $30.7\%\downarrow$ )	100.6	52.3 ( $48.0\%\downarrow$ )	144.1	65.5 ( $54.5\%\downarrow$ )
	16	87.9	61.9 ( $29.5\%\downarrow$ )	147.2	76.9 ( $47.8\%\downarrow$ )	213.9	92.6 ( $56.7\%\downarrow$ )
MBPP	1	58.2	44.3 ( $23.8\%\downarrow$ )	91.7	50.6 ( $44.8\%\downarrow$ )	133.0	65.7 ( $50.6\%\downarrow$ )
	4	61.7	47.1 ( $23.6\%\downarrow$ )	96.8	54.4 ( $43.8\%\downarrow$ )	145.9	69.9 ( $52.1\%\downarrow$ )

Table 2: LLaDA recompute latency (ms) under prefix reuse. Dream: Table 9 .

Model	Dataset	Batch	1-shot		2-shot		4-shot
			Fast-dLLM	ACache	Fast-dLLM	ACache	Fast-dLLM	ACache
LLaDA	GSM8K	1	104	99 ( $0.96\times$ )	94	98 ( $1.04\times$ )	85	92 ( $1.09\times$ )
		4	263	274 ( $1.04\times$ )	206	253 ( $1.23\times$ )	166	229 ( $1.37\times$ )
		16	394	433 ( $1.10\times$ )	278	378 ( $1.36\times$ )	204	329 ( $1.61\times$ )
	MBPP	1	209	213 ( $1.02\times$ )	168	194 ( $1.16\times$ )	123	155 ( $1.26\times$ )
		4	380	401 ( $1.06\times$ )	272	366 ( $1.34\times$ )	189	289 ( $1.53\times$ )

Table 3: End-to-end throughput under prefix reuse. Measured in generated tokens per second.

Dataset	Batch	1-shot		2-shot		4-shot
		Fast-dLLM	ACache	Fast-dLLM	ACache	Fast-dLLM	ACache
GSM8K	1	0.38	0.38 ( $0.0\%$ )	0.50	0.75 ( $50.0\%\uparrow$ )	0.75	0.88 ( $16.7\%\uparrow$ )
	4	1.50	1.12 ( $25.0\%\downarrow$ )	2.00	1.50 ( $25.0\%\downarrow$ )	3.00	2.00 ( $33.3\%\downarrow$ )
	16	6.00	4.12 ( $31.2\%\downarrow$ )	8.00	4.62 ( $42.2\%\downarrow$ )	12.00	6.50 ( $45.8\%\downarrow$ )
MBPP	1	2.12	2.12 ( $0.0\%$ )	2.25	2.25 ( $0.0\%$ )	2.50	2.62 ( $5.0\%\uparrow$ )
	4	3.25	3.00 ( $7.7\%\downarrow$ )	3.75	3.25 ( $13.3\%\downarrow$ )	4.75	3.75 ( $21.1\%\downarrow$ )

Table 4: LLaDA peak KV cache memory usage (GB) under prefix reuse. Dream: Table 10 .

Model	Ratio	Criterion	GSM8K			MBPP			BABILong			Avg.
			Pfx.	Inf.	Suf.	Pfx.	Inf.	Suf.	Pfx.	Inf.	Suf.
LLaDA	0.2	CacheBlend	71.6	69.6	69.8	35.5	7.8	22.0	99.5	82.8	63.6	57.3
		ProphetKV	75.6	69.8	70.8	33.5	13.1	25.8	99.7	98.6	43.4	60.9
		BiCache	71.4	69.1	68.2	33.2	3.4	8.4	99.4	73.9	98.4	53.4
		ACache	73.6	69.9	70.3	34.9	15.4	25.1	99.6	98.9	76.1	61.0
	0.3	CacheBlend	71.9	70.4	70.9	35.6	12.3	15.2	99.4	94.7	27.8	58.8

Table 5: ACache’s criterion compared to adapted baselines, 1-shot. Accuracy (%) on Fast-dLLM. Bold marks the best value per column; underline, the second-best. BiCache † is the official implementation; Ratio is the mean equivalent Anchor ratio. Avg. omits BABILong suffix (Section 5.1 ).

Model	Dataset	1-shot		2-shot		4-shot
		BiCache	ACache	BiCache	ACache	BiCache	ACache
LLaDA	GSM8K	96	99 ( $1.03\times$ )	84	98 ( $1.18\times$ )	71	92 ( $1.30\times$ )
	MBPP	192	213 ( $1.11\times$ )	165	194 ( $1.18\times$ )	118	155 ( $1.31\times$ )
Dream	GSM8K	87	125 ( $1.44\times$ )	93	120 ( $1.30\times$ )	83	117 ( $1.41\times$ )
	MBPP	145	190 ( $1.31\times$ )	136	183 ( $1.35\times$ )	119	171 ( $1.44\times$ )

Table 6: End-to-end throughput at batch size 1 under prefix reuse (tokens/s). Both run on Fast-dLLM: BiCache from its official implementation (Appendix C.3 ), ACache at Anchor ratio 0.2.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Affix	Shots	Model	Share	Jaccard similarity				Mass recall
					Random	Fresh		ACache	Fresh		ACache
						2nd	Last		2nd	Last
GSM8K	Prefix	1	LLaDA	62.7	11.3	71.6	56.8	38.9	98.5	95.8	76.7
			Dream	71.8	11.3	78.8	57.4	54.4	97.8	89.5	89.3
		2	LLaDA	62.6	11.2	73.4	52.5	35.2	98.5	95.1	91.4
			Dream	73.4	11.2	84.9	58.2	62.8	99.4	93.6	94.3

Appendix

Table 7: Anchor stability at Anchor ratio 0.2. Fast-dLLM, full recomputation; all values in %, means over each test set. At each block boundary, a set is compared with the current top-

K

set: Random, Fresh (the first’s), or ACache’s Anchors (averaged). Share: the first top-

K

set’s importance.

Model	Dataset	Shot	Affix	Preparation	Forward	Top- $K$	Total	Forward share
LLaDA	GSM8K	1	213	0.49	50.05	0.16	50.69	98.7%
		2	549	0.52	56.46	0.16	57.14	98.8%
		4	981	0.57	66.02	0.18	66.76	98.9%
	MBPP	1	178	0.55	54.70	0.16	55.40	98.7%
		2	455	0.55	61.40	0.19	62.14	98.8%
		4	1016	0.60	71.57	0.21	72.38	98.9%

Appendix

Table 8: Anchor selection cost per request (ms) under prefix reuse, batch size 1. Profiled on the runs of Table 2 (LLaDA) and Table 9 (Dream). Affix reports the shared span’s length in tokens.

Dataset	Batch	1-shot		2-shot		4-shot
		Fast-dLLM	ACache	Fast-dLLM	ACache	Fast-dLLM	ACache
GSM8K	1	52.8	41.6 ( $21.3\%\downarrow$ )	72.5	46.4 ( $35.9\%\downarrow$ )	111.7	54.8 ( $50.9\%\downarrow$ )
	4	60.7	46.6 ( $23.3\%\downarrow$ )	87.0	52.3 ( $39.9\%\downarrow$ )	132.4	63.5 ( $52.0\%\downarrow$ )
	16	96.5	70.2 ( $27.3\%\downarrow$ )	147.0	80.4 ( $45.3\%\downarrow$ )	233.5	102.3 ( $56.2\%\downarrow$ )
MBPP	1	54.4	45.2 ( $17.0\%\downarrow$ )	72.7	49.2 ( $32.4\%\downarrow$ )	119.0	60.3 ( $49.4\%\downarrow$ )
	4	70.9	56.0 ( $21.0\%\downarrow$ )	96.7	62.2 ( $35.7\%\downarrow$ )	162.5	76.7 ( $52.8\%\downarrow$ )

Appendix

Table 9: Dream recompute latency (ms) under prefix reuse. LLaDA: Table 2 .

Dataset	Batch	1-shot		2-shot		4-shot
		Fast-dLLM	ACache	Fast-dLLM	ACache	Fast-dLLM	ACache
GSM8K	1	0.04	0.04 ( $0.0\%$ )	0.05	0.08 ( $50.0\%\uparrow$ )	0.08	0.10 ( $16.7\%\uparrow$ )
	4	0.16	0.12 ( $25.0\%\downarrow$ )	0.22	0.16 ( $25.0\%\downarrow$ )	0.33	0.22 ( $33.3\%\downarrow$ )
	16	0.66	0.45 ( $31.2\%\downarrow$ )	0.88	0.51 ( $42.2\%\downarrow$ )	1.31	0.71 ( $45.8\%\downarrow$ )
MBPP	1	0.23	0.23 ( $0.0\%$ )	0.25	0.25 ( $0.0\%$ )	0.27	0.29 ( $5.0\%\uparrow$ )
	4	0.36	0.33 ( $7.7\%\downarrow$ )	0.41	0.36 ( $13.3\%\downarrow$ )	0.52	0.41 ( $21.1\%\downarrow$ )

Appendix

Table 10: Dream peak KV cache memory usage (GB) under prefix reuse. LLaDA: Table 4 .

Model	Dataset	Batch	1-shot		2-shot		4-shot
			dKV-Cache	ACache	dKV-Cache	ACache	dKV-Cache	ACache
LLaDA	GSM8K	1	54.1	38.1 ( $29.6\%\downarrow$ )	88.4	44.9 ( $49.2\%\downarrow$ )	120.4	58.3 ( $51.6\%\downarrow$ )
		4	166.2	117.4 ( $29.4\%\downarrow$ )	285.7	152.1 ( $46.8\%\downarrow$ )	439.5	181.9 ( $58.6\%\downarrow$ )
		16	627.4	435.2 ( $30.6\%\downarrow$ )	1053.7	533.4 ( $49.4\%\downarrow$ )	1674.0	672.7 ( $59.8\%\downarrow$ )
	MBPP	1	55.8	43.1 ( $22.8\%\downarrow$ )	88.2	48.8 ( $44.6\%\downarrow$ )	129.2	63.9 ( $50.6\%\downarrow$ )
		4	175.3	138.5 ( $21.0\%\downarrow$ )	263.9	162.3 ( $38.5\%\downarrow$ )	477.1	205.1 ( $57.0\%\downarrow$ )

Appendix

Table 11: dKV-Cache periodic full-cache recompute latency (ms) under prefix reuse. OOM marks a run that exceeded GPU memory.

Model	Dataset	Batch	1-shot		2-shot		4-shot
			dKV-Cache	ACache	dKV-Cache	ACache	dKV-Cache	ACache
LLaDA	GSM8K	1	33	35 ( $1.08\times$ )	28	33 ( $1.21\times$ )	24	31 ( $1.30\times$ )
		4	56	62 ( $1.11\times$ )	43	56 ( $1.29\times$ )	33	50 ( $1.51\times$ )
		16	68	77 ( $1.14\times$ )	51	69 ( $1.35\times$ )	38	60 ( $1.60\times$ )
	MBPP	1	32	34 ( $1.05\times$ )	28	33 ( $1.18\times$ )	23	30 ( $1.29\times$ )
		4	55	59 ( $1.08\times$ )	45	55 ( $1.22\times$ )	32	48 ( $1.51\times$ )

Appendix

Table 12: End-to-end throughput under prefix reuse on dKV-Cache. Measured in generated tokens per second. OOM marks a run that exceeded GPU memory.

Model	Dataset	Batch	1-shot		2-shot		4-shot
			dKV-Cache	ACache	dKV-Cache	ACache	dKV-Cache	ACache
LLaDA	GSM8K	1	0.38	0.38 ( $0.0\%$ )	0.50	0.75 ( $50.0\%\uparrow$ )	0.75	0.88 ( $16.7\%\uparrow$ )
		4	1.50	1.12 ( $25.0\%\downarrow$ )	2.00	1.50 ( $25.0\%\downarrow$ )	3.00	2.00 ( $33.3\%\downarrow$ )
		16	6.00	4.12 ( $31.2\%\downarrow$ )	8.00	4.50 ( $43.8\%\downarrow$ )	12.00	6.50 ( $45.8\%\downarrow$ )
	MBPP	1	2.12	2.12 ( $0.0\%$ )	2.25	2.25 ( $0.0\%$ )	2.50	2.62 ( $5.0\%\uparrow$ )
		4	3.25	2.88 ( $11.5\%\downarrow$ )	3.75	3.00 ( $20.0\%\downarrow$ )	4.75	3.75 ( $21.1\%\downarrow$ )

Appendix

Table 13: Peak KV cache memory usage (GB) under prefix reuse on dKV-Cache. OOM marks a run that exceeded GPU memory.

Model	Dataset	Batch	1-shot		2-shot		4-shot
			Fast-dLLM	ACache	Fast-dLLM	ACache	Fast-dLLM	ACache
LLaDA	GSM8K	1	56.6	42.7 ( $24.5\%\downarrow$ )	91.6	47.3 ( $48.4\%\downarrow$ )	123.6	61.9 ( $49.9\%\downarrow$ )
		4	62.9	50.5 ( $19.7\%\downarrow$ )	101.7	55.5 ( $45.5\%\downarrow$ )	140.7	69.6 ( $50.6\%\downarrow$ )
		16	90.7	72.0 ( $20.7\%\downarrow$ )	148.7	80.3 ( $46.0\%\downarrow$ )	215.0	97.9 ( $54.5\%\downarrow$ )
	MBPP	1	61.1	49.1 ( $19.6\%\downarrow$ )	94.2	55.8 ( $40.7\%\downarrow$ )	138.0	69.7 ( $49.4\%\downarrow$ )
		4	67.6	56.0 ( $17.3\%\downarrow$ )	106.3	63.5 ( $40.3\%\downarrow$ )	152.6	77.1 ( $49.5\%\downarrow$ )

Appendix

Table 14: Recompute latency (ms) under infix reuse. The shared affix sits between the query header and answer prompt rather than before the query.

Model	Dataset	Batch	1-shot		2-shot		4-shot
			Fast-dLLM	ACache	Fast-dLLM	ACache	Fast-dLLM	ACache
LLaDA	GSM8K	1	105	98 ( $0.93\times$ )	92	95 ( $1.03\times$ )	82	88 ( $1.08\times$ )
		4	262	250 ( $0.95\times$ )	202	234 ( $1.16\times$ )	162	207 ( $1.28\times$ )
		16	388	392 ( $1.01\times$ )	273	349 ( $1.28\times$ )	193	297 ( $1.54\times$ )
	MBPP	1	198	186 ( $0.94\times$ )	162	178 ( $1.10\times$ )	119	146 ( $1.22\times$ )
		4	368	349 ( $0.95\times$ )	265	326 ( $1.23\times$ )	183	266 ( $1.45\times$ )

Appendix

Table 15: End-to-end throughput under infix reuse. Measured in generated tokens per second.

Model	Dataset	Batch	1-shot		2-shot		4-shot
			Fast-dLLM	ACache	Fast-dLLM	ACache	Fast-dLLM	ACache
LLaDA	GSM8K	1	0.38	0.62 ( $66.7\%\uparrow$ )	0.62	1.12 ( $80.0\%\uparrow$ )	0.75	1.38 ( $83.3\%\uparrow$ )
		4	1.50	1.38 ( $8.3\%\downarrow$ )	2.12	2.00 ( $5.9\%\downarrow$ )	3.00	2.50 ( $16.7\%\downarrow$ )
		16	6.00	4.38 ( $27.1\%\downarrow$ )	8.12	5.50 ( $32.3\%\downarrow$ )	12.00	7.00 ( $41.7\%\downarrow$ )
	MBPP	1	2.12	2.25 ( $5.9\%\uparrow$ )	2.25	2.62 ( $16.7\%\uparrow$ )	2.50	3.12 ( $25.0\%\uparrow$ )
		4	3.25	3.25 ( $0.0\%$ )	3.75	3.75 ( $0.0\%$ )	4.88	4.25 ( $12.8\%\downarrow$ )

Appendix

Table 16: Peak KV cache memory usage (GB) under infix reuse.

Model	Dataset	Batch	1-shot		2-shot		4-shot
			Fast-dLLM	ACache	Fast-dLLM	ACache	Fast-dLLM	ACache
LLaDA	GSM8K	1	56.4	39.7 ( $29.6\%\downarrow$ )	91.3	46.4 ( $49.2\%\downarrow$ )	123.6	61.1 ( $50.6\%\downarrow$ )
		4	62.5	46.0 ( $26.5\%\downarrow$ )	101.7	54.1 ( $46.8\%\downarrow$ )	141.9	68.9 ( $51.5\%\downarrow$ )
		16	90.6	66.6 ( $26.5\%\downarrow$ )	150.4	80.4 ( $46.6\%\downarrow$ )	220.0	96.8 ( $56.0\%\downarrow$ )
	MBPP	1	59.3	45.4 ( $23.4\%\downarrow$ )	91.7	50.8 ( $44.6\%\downarrow$ )	134.4	66.2 ( $50.7\%\downarrow$ )
		4	63.3	51.6 ( $18.5\%\downarrow$ )	97.8	56.8 ( $41.9\%\downarrow$ )	148.4	73.5 ( $50.4\%\downarrow$ )

Appendix

Table 17: Recompute latency (ms) under suffix reuse. The shared affix sits after the masked generation region rather than before the query.

Model	Dataset	Batch	1-shot		2-shot		4-shot
			Fast-dLLM	ACache	Fast-dLLM	ACache	Fast-dLLM	ACache
LLaDA	GSM8K	1	104	99 ( $0.95\times$ )	90	95 ( $1.06\times$ )	83	88 ( $1.06\times$ )
		4	263	253 ( $0.96\times$ )	200	236 ( $1.18\times$ )	163	212 ( $1.30\times$ )
		16	393	410 ( $1.04\times$ )	273	358 ( $1.31\times$ )	200	307 ( $1.54\times$ )
	MBPP	1	127	111 ( $0.87\times$ )	113	138 ( $1.23\times$ )	87	100 ( $1.15\times$ )
		4	290	260 ( $0.90\times$ )	225	292 ( $1.30\times$ )	161	223 ( $1.38\times$ )

Appendix

Table 18: End-to-end throughput under suffix reuse. Measured in generated tokens per second.

Model	Dataset	Batch	1-shot		2-shot		4-shot
			Fast-dLLM	ACache	Fast-dLLM	ACache	Fast-dLLM	ACache
LLaDA	GSM8K	1	0.38	0.62 ( $66.7\%\uparrow$ )	0.50	1.12 ( $125.0\%\uparrow$ )	0.75	1.38 ( $83.3\%\uparrow$ )
		4	1.50	1.38 ( $8.3\%\downarrow$ )	2.00	2.00 ( $0.0\%$ )	3.00	2.50 ( $16.7\%\downarrow$ )
		16	6.00	4.38 ( $27.1\%\downarrow$ )	8.00	5.12 ( $35.9\%\downarrow$ )	12.00	7.00 ( $41.7\%\downarrow$ )
	MBPP	1	2.12	2.25 ( $5.9\%\uparrow$ )	2.25	2.50 ( $11.1\%\uparrow$ )	2.50	3.12 ( $25.0\%\uparrow$ )
		4	3.25	3.00 ( $7.7\%\downarrow$ )	3.75	3.38 ( $10.0\%\downarrow$ )	4.75	4.25 ( $10.5\%\downarrow$ )

Appendix

Table 19: Peak KV cache memory usage (GB) under suffix reuse.

Model	Ratio	Variant	GSM8K			MBPP			BABILong			Avg.
			Pfx.	Inf.	Suf.	Pfx.	Inf.	Suf.	Pfx.	Inf.	Suf.
LLaDA	0.2	ACache	73.6	69.9	70.3	34.9	15.4	25.1	99.6	98.9	76.1	61.0
		Anchors only	71.3	69.9	69.1	2.9	6.3	4.9	99.8	99.9	15.4	53.0
	0.3	ACache	74.1	71.0	70.8	34.8	24.7	26.8	99.7	98.9	78.6	62.6
		Anchors only	73.1	70.2	70.8	5.3	16.2	7.7	99.8	99.7	56.2	55.3
Dream	0.2	ACache	70.7	74.0	70.7	50.0	36.7	35.9	95.2	86.4	84.5	64.9

Appendix

Table 20: Keeping versus discarding the non-anchor affix cache, 1-shot. Accuracy (%) on Fast-dLLM. Anchors only selects the Anchor Tokens but discards the rest of the affix cache. Bold marks the better value per column. ACache rows repeat Table 5 ; as there, Avg. omits BABILong suffix.

Ratio	Selection	GSM8K			MBPP			BABILong			Avg.
		Pfx.	Inf.	Suf.	Pfx.	Inf.	Suf.	Pfx.	Inf.	Suf.
0.2	Top	73.6	69.9	70.3	34.9	15.4	25.1	99.6	98.9	76.1	61.0
	Random	73.2	69.0	69.0	34.1	11.1	18.6	99.6	80.4	84.5	56.9
	Bottom	71.6	69.2	68.4	34.4	4.1	8.5	99.1	74.2	98.1	53.7
0.3	Top	74.1	71.0	70.8	34.8	24.7	26.8	99.7	98.9	78.6	62.6
	Random	72.0	68.2	69.4	33.8	17.3	23.3	99.6	84.6	79.5	58.5

Appendix

Table 21: Top, random, and bottom Anchor selection, LLaDA, 1-shot. Accuracy (%) on Fast-dLLM; Top is ACache. Bold marks the best per column and ratio; Avg. omits BABILong suffix.

Model	Method	Ratio	Acc.
LLaDA	Direct reuse	0	19.8
	ACache	0.2	59.1
		0.3	65.4
	BiCache †	0.50	66.9
	Full recomputation	1	69.0
Dream	Direct reuse	0	78.5

Appendix

Table 22: System-prompt reuse on GSM8K: strict accuracy (%) on Fast-dLLM. As in BiCache, an answer counts only if the number after “####” matches. BiCache † is the official implementation; Ratio is its mean equivalent Anchor ratio. Bold marks the best reuse method per model; underline, the second-best.

Model	Batch 1			Batch 4		Batch 16
	BiCache †	Fast-dLLM	ACache	Fast-dLLM	ACache	Fast-dLLM	ACache
LLaDA	62	76	91 ( $1.19\times$ )	138	214 ( $1.55\times$ )	164	290 ( $1.77\times$ )
Dream	92	101	126 ( $1.26\times$ )	167	273 ( $1.64\times$ )	194	357 ( $1.84\times$ )

Appendix

Table 23: System-prompt reuse on GSM8K: end-to-end throughput (tokens/s). Fast-dLLM is the full recomputation of Table 22 ; ACache uses Anchor ratio 0.2; BiCache † supports only batch size 1.

Explore similar work

Enabling KV Caching of Shared Prefix for Diffusion Language Models
May 26, 2026Younghun Go, Jaehoon Han, Changyong Shin +2Cache Diffusion Language Models
Korea University, South Korea
Fixed State, Long Reach: What a Constant-Size Cache Buys Block Diffusion at Scale
Sep 14, 2026Vaibhav Singh, Pierre-André Noël, Torsten Scholak +2Diffusion Language Models Cache
Mila · Concordia University · ServiceNow Research
Bifocal Diffusion Language Models: Asymmetric Bidirectional Context for Parallel Generation
Jun 26, 2026Yuhang Chen, Xianfeng Wu, Jinhao Duan +11Diffusion Language Models Bidirectional
University of North Carolina at Chapel Hill · Meta AI

May 26, 2026, cs.LGJ/K move · Enter open · S save

Enabling KV Caching of Shared Prefix for Diffusion Language Models

Younghun Go, Jaehoon Han, Changyong Shin, Chuck Yoo+1

Korea University, South Korea

Cache Diffusion Language Models Large Language Model Serving

Open paper AskPDF arXiv

Key-value (KV) caching for shared prefixes is essential for high-throughput large language model (LLM) serving, but it faces critical challenges in emerging diffusion language models (DLMs). In DLMs, bidirectional attention means that updating any token dynamically alters the entire context and its corresponding KVs. Thus, existing caching techniques developed for LLMs, which assume that KVs remain invariant once computed, corrupt the shared prefix KVs. Our experiments show that applying these techniques to DLMs causes model accuracy to collapse to near zero. To unlock high-throughput DLM serving, we propose bidirectional prefix caching, BiCache, the first KV caching technique for shared prefixes in DLMs. BiCache is designed based on key observations from our comprehensive analysis: shared prefix KVs remain stable and reusable in shallow layers, while the depth of shallow layers depends on the fraction of shared prefix tokens in each request. Thus, BiCache dynamically identifies a safe layer depth for reusing shared prefix KVs and eliminates redundant computation. Evaluations demonstrate that BiCache significantly improves serving throughput by 36.3%-98.3% compared to existing techniques without accuracy collapse (only 0-1.8% difference).

Abstract

Explore similar work

Enabling KV Caching of Shared Prefix for Diffusion Language Models

Fixed State, Long Reach: What a Constant-Size Cache Buys Block Diffusion at Scale

Bifocal Diffusion Language Models: Asymmetric Bidirectional Context for Parallel Generation