RIFT: KV Cache Compression for Draft Models with Complementary Compressibility Training

Dor Tsur; Maor Ashkenazi; Maximilian Golub; Peter Hicks; Yaniv Galron; Talor Abramovich; Yoav Miron; Tiyasa Mitra

cs.CLOct 22, 2026

RIFT: KV Cache Compression for Draft Models with Complementary Compressibility Training

Authors: Dor Tsur, Maor Ashkenazi, Maximilian Golub, Peter Hicks, Yaniv Galron, Talor Abramovich, Yoav Miron, Tiyasa Mitra,

Organizations: NVIDIA

Abstract

Speculative decoding is a popular decode acceleration method. As such, it is particularly relevant to long-output generation, which requires an increasingly large KV cache. KV cache compression offers a way to constrain costs in long-context drafting. However, compression methods that focus on prefill information can lose relevance as the output length grows. We term this phenomenon decode staleness and show that it degrades draft model accuracy under selection-based KV cache compression. We introduce RIFT, a family of selection-based KV cache compression strategies that balance prefill information and recent generated tokens. We show that refreshing the KV cache selection during token generation significantly improves the drafter's quality at the cost of additional decode latency. To alleviate the refresh costs, we propose an adaptive KV refresh strategy that reduces the number of score calculations while retaining the drafter's quality. We further demonstrate how a compressibility-driven drafter training phase, based on sparse attention masks, can improve the drafter's quality under a compressed KV cache. Combined with compressibility training, RIFT enables Qwen3.5-4B, drafting for Qwen3.5-122B, to retain over 98% of its full-cache acceptance length on 32K-token inputs, 4K-token outputs, and an 8K-token KV cache budget.

Figures & tables

Panel of Figure 1 — Figure 1: AL improvement of RIFT and compressibility training on a Qwen3.5-4B draft (Qwen3.5-122B target), SPEED-Bench 32K ISL low-entropy, draft length 11. (a) AL vs. generation interval at 8K cache budget. (b) AL vs. KV cache budget. AL is measured vs. the full-KV draft AL. Diagonal pattern denotes compression-aware training.

Panel of Figure 2 — Figure 2: Fraction of full-KV attention mass retained by SW and SnapKV (with 64 token SW) in the Qwen3.5-4B draft on 32K-ISL data, with an 8K budget. SnapKV retains less attention as generation proceeds, while SW retains more.

Panel of Figure 3 — Figure 3: Effect of the sliding-window allocation ratio. $4K$ OSL and $8K$ KV budget. Qwen3.5-4B drafts Qwen3.5-122B on SPEED-Bench 32K ISL data. $\rho=0$ denotes SnapKV with a fixed 64-token SW, and $\rho=1$ corresponds to pure SWA.

Panel of Figure 4 — Figure 4: Effect of RIFT-dyn’s refresh interval on AL (blue) and projected rescoring overhead relative to measured draft decode time (orange). Qwen3.5-4B, 100K ISL, 8K cache budget, and 4K OSL. Appendix D.2 describes the latency projection.

Panel of Figure 5 — Figure 5: KV rescore patterns comparison on a candidate sample from SPEED-Bench. RIFT-ada attains a higher AL and requires less KV refreshes. It can be seen that KV refreshes are not periodic and depend on generation location.

Draft (ISL)	$T$	SW	SnapKV	RIFT
				stat	dyn	ada
Qwen3.5 (32K)	0	5.23	4.07	5.62	6.12	6.19
Qwen3.5 (32K)	1	4.70	3.69	4.99	5.48	5.55
Qwen3.5 (100K)	0	6.12	4.46	6.40	7.14	7.17
Qwen3.5 (100K)	1	5.40	3.82	5.55	6.21	6.36
Llama3.1 (32K)	0	2.24	5.19	6.07	6.91	$7.48$

Table 1: AL on low-entropy SPEED-Bench 32K/100K ISL, with an 8K budget, 4K OSL, and draft length 11. Qwen3.5: 4B

\rightarrow

122B; Llama3.1: 8B

\rightarrow

70B; Nemotron3.5-Lightning in self-speculation and as an external draft to Nemotron3-Ultra target.

Budget	Selection score	Baseline	+ RIFT-stat
4K	CriticalKV	3.57	5.02 (+40.6%)
	Expected Attention	3.59	5.03 (+40.2%)
	CurDKV	3.29	4.84 (+47.1%)
8K	CriticalKV	4.13	5.62 (+36.2%)
	Expected Attention	4.06	5.56 (+36.9%)
	CurDKV	3.76	5.35 (+42.2%)

Table 2: Absolute AL at

T=0

(relative gains in parentheses): fixed 64-token SW versus RIFT-stat (

\rho=0.5

). Qwen3.5-4B drafts Qwen3.5-122B on SPEED-Bench 32K ISL, 4K OSL, draft length 11.

Draft	Target	SnapKV	+ RIFT-stat	+ RIFT-dyn
Qwen3.5-4B	DSV4-Flash	3.50	4.26	5.35
Qwen3.5-4B	DSV4-Pro	3.43	4.26	5.07
Llama3.2-1B	DSV4-Flash	2.50	3.24	3.50
Llama3.2-1B	DSV4-Pro	2.37	3.07	3.26

Table 3: Absolute AL for adapted drafters of DeepSeek-V4-Flash/Pro on low-entropy SPEED-Bench: 32K ISL, 8K budget, 4K OSL,

T=0

, draft length 11.

Panel of Figure 6 — Figure 6: RIFT-ada AL-refresh tradeoff at $T=1$ for Qwen3.5-4B drafting Qwen3.5-122B on SPEED-Bench 32K low-entropy data, with an 8K budget and 4K OSL.

	32K ISL		100K ISL
Method	$T=0$	$T=1$	$T=0$	$T=1$
StreamingLLM	5.34	4.73	5.60	4.98
H 2 O	5.73	5.03	6.65	5.77
TriAttention	5.43	4.77	6.10	5.11
RIFT-stat	5.62	4.99	6.40	5.55
RIFT-dyn	6.12	5.48	7.14	6.21

Table 4: Absolute AL against decode-time baselines. Qwen3.5-4B draft, low-entropy SPEED-Bench, 8K budget, 4K OSL, and draft length 11.

		RIFT-stat			RIFT-dyn
Draft $\rightarrow$ target	$T$	2K	4K	8K	2K	4K	8K
Qwen3.5 (4B $\rightarrow$ 122B)	0	$+7.70\%$	$+6.87\%$	$+5.70\%$	$+7.48\%$	$+5.50\%$	$+3.72\%$
Qwen3.5 (4B $\rightarrow$ 122B)	1	$+16.19\%$	$+14.88\%$	$+12.28\%$	$+15.51\%$	$+12.25\%$	$+9.21\%$
Lightning $\rightarrow$ Nemotron3-Ultra	0	$+7.51\%$	$+7.48\%$	$+7.59\%$	$+7.88\%$	$+7.86\%$	$+5.59\%$
Lightning $\rightarrow$ Nemotron3-Ultra	1	$+15.69\%$	$+15.63\%$	$+15.08\%$	$+16.42\%$	$+15.84\%$	$+13.19\%$
Lightning self-speculation	0	$+5.3\%$	$+3.4\%$	$+2.5\%$	$+6.5\%$	$+2.8\%$	$+1.0\%$

Table 5: Relative AL improvement from compression-aware KD over the base model under the same KV compression policy and budget, evaluated on SPEED-Bench 32k ISL with draft length 11.

Panel of Figure 7 — Figure 7: Compression-aware distillation. The snowflake and fire symbols denote frozen and trainable passes, respectively. The mask combines recent positions (purple) with importance-selected context (green).

Method	$T$	2K	4K	8K
RIFT-stat	0	$+5.44\%$	$+4.68\%$	$+3.71\%$
RIFT-stat	1	$+7.31\%$	$+6.37\%$	$+4.66\%$
RIFT-dyn	0	$+5.34\%$	$+3.51\%$	$+1.82\%$
RIFT-dyn	1	$+7.25\%$	$+4.73\%$	$+2.37\%$

Table 6: Relative AL improvement from compression-aware training over a similar training recipe with no compression-driven masks. Qwen3.5-4B

\to

Qwen3.5-122B, SPEED-Bench 32k ISL, draft length 11.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure 8: Attention KV cache share of total memory in two hybrid architectures, evaluated at the end of prefill.

Panel of Figure 9 — Figure 9: Full-attention mass shifts from prompt tokens to the generated prefix in the Qwen3.5-4B draft on 32K-ISL data.

Panel of Figure 10 — Figure 10: Interval AL across model families, with full-KV references. Target/draft pairs (left to right): GPT-OSS-120B/20B, Nemotron-Super-120B/Nano-30B, Qwen3-235B/8B, and Llama-3.1-70B/8B. Each panel uses its own AL scale. The corresponding Qwen3.5 comparison appears in Figure 1 (a).

Panel of Figure 11 — Figure 11: Interval AL versus generated-token position for Qwen3.5-4B drafting Qwen3.5-122B responses, with thinking enabled and disabled. Reasoning magnifies the late staleness gap, but the SnapKV-SW crossover persists even without it.

Panel of Figure 12 — Figure 12: Additional sliding-window ratio ablations with the same base draft model, requests, and evaluation settings as Figure 3 . Panel titles specify the output-length cap and KV budget. Stars mark $\rho=0.5$ in the first four panels and the optimum $\rho=0.6$ in the 1K-output, 1K-budget panel (AL $3.648$ , versus $3.597$ at $\rho=0.5$ ); the zero-labeled endpoint denotes fixed-64 SnapKV, and $\rho=1$ denotes pure SWA. Broken diagonals indicate omitted AL range.

Panel of Figure 13 — Figure 13: Effect of draft length for Qwen3.5-4B drafting for Qwen3.5-122B-A10B on low-entropy SPEED-Bench 32K prompts, with a 4K output cap. Compressed methods use an 8K KV budget and block size 128; Full-KV retains the full context. Each point is mean per-request AL.

Panel of Figure 14 — Figure 14: Mean per-request acceptance length versus forced output sequence length for Qwen3.5-4B drafting for Qwen3.5-122B-A10B on SPEED-Bench 32K low-entropy prompts. All compressed methods use an 8K KV budget and draft length 11.

ISL	Entropy	$T$	Budget	SW	SnapKV	RIFT-stat	RIFT-dyn	RIFT-ada
32K	Low	0	2K	64.8%	49.7%	70.3%	79.5%	81.5%
			4K	71.3%	54.5%	77.7%	88.5%	90.2%
			8K	80.9%	63.0%	86.9%	94.6%	95.8%
	Low	1	2K	61.8%	47.1%	65.9%	74.7%	77.0%
			4K	68.0%	52.0%	72.5%	84.1%	86.3%
			8K	78.6%	61.8%	83.4%	91.7%	92.8%

Appendix

Table 7: retained AL (%) of full-KV, Qwen3.5-4B draft with, Qwen3.5-122B target across SPEED-Bench workloads (ISL and entropy category) and cache budgets. The absolute full-KV ALs for 32K low

T=0/1

, 32K high

T=0

, 100K low

T=0

, and 100K high

T=0

are

6.466/5.978

4.617

7.472

, and

4.506

, respectively. All settings use draft length 11 and maximum output length 4K.

ISL	Entropy	$T$	SW	SnapKV	RIFT-stat	RIFT-dyn	RIFT-ada
32K	Low	0	$85.5\%$	$67.3\%$	$89.0\%$	$95.9\%$	$98.1\%$
	Low	1	$85.5\%$	$68.2\%$	$86.9\%$	$94.2\%$	$96.4\%$
	High	0	$91.5\%$	$75.0\%$	$95.6\%$	$97.9\%$	$98.6\%$
	High	1	$92.6\%$	$77.7\%$	$94.9\%$	$97.1\%$	$97.9\%$
100K	Low	0	$82.6\%$	$61.0\%$	$87.5\%$	$94.6\%$	$97.4\%$
	Low	1	$84.6\%$	$62.0\%$	$86.9\%$	$92.7\%$	$97.1\%$

Appendix

Table 8: Base Qwen3.5-0.8B draft performance at an 8K cache budget, expressed as a percentage of the matching Full KV AL for Qwen3.5-122B target responses. The absolute full-KV ALs for 32K low

T=0/1

and high

T=0/1

are

4.807/4.148

and

3.272/2.836

; for 100K low

T=0/1

and high

T=0

, they are

5.763/4.748

and

3.225

Draft	Target	$T$	Budget	SW	SnapKV	RIFT-stat	RIFT-dyn
Nano	Super	0	2K	53.0%	41.8%	52.6%	62.1%
			4K	57.1%	45.1%	57.2%	73.2%
			8K	63.4%	52.8%	63.9%	83.4%
Nano	Ultra	0	2K	69.1%	56.0%	69.9%	76.6%
			4K	74.4%	60.5%	75.2%	85.9%
			8K	81.3%	68.4%	82.8%	93.2%

Appendix

Table 9: Full-KV AL retention in Nemotron3-family, low-entropy SPEED-Bench 32K split. The absolute full-KV ALs are

6.944

(Nano

\rightarrow

Super,

T=0

4.994

(Nano

\rightarrow

Ultra,

T=0

6.647/5.520

(Lightning

\rightarrow

Super,

T=0/1

5.394/4.859

(Lightning

\rightarrow

Ultra,

T=0/1

), and

10.009/11.222

(Lightning self-speculation,

T=0/1

Draft $\rightarrow$ target	Budget	SW	SnapKV	RIFT-stat	RIFT-dyn
Llama-3.1-8B $\rightarrow$ Llama-3.1-70B	2K	26.6%	53.7%	64.3%	70.5%
	4K	27.6%	58.1%	70.1%	79.7%
	8K	28.6%	66.1%	77.4%	88.0%
GPT-OSS-20B $\rightarrow$ GPT-OSS-120B	2K	62.3%	32.0%	56.4%	67.7%
	4K	65.5%	32.6%	64.9%	76.8%
	8K	69.6%	34.0%	71.4%	87.8%

Appendix

Table 10: Llama3.1 and GPT-OSS cache-budget sweeps on the

T=0

, low-entropy SPEED-Bench 32K workload. Values are AL as a percentage of the matching full-KV AL. The absolute full-KV AL is

7.849

for Llama and

5.746

for GPT-OSS. All settings use draft length 11 and maximum output length 4K.

Table 11: DeepSeek V4 Pro with trained external drafters on the

T=0

, low-entropy SPEED-Bench workloads. For Qwen3.5-0.8B, Qwen3.5-4B, and Llama3.2-1B, respectively, the absolute full-KV ALs are

4.343/5.389/3.969

at 32K and

4.263/5.427/3.674

at 100K.

Table 12: DeepSeek V4 Flash with trained external drafters on the

T=0

, low-entropy SPEED-Bench workloads. For Qwen3.5-0.8B, Qwen3.5-4B, and Llama3.2-1B, respectively, the absolute full-KV ALs are

4.830/5.714/4.401

at 32K and

5.308/6.338/4.896

at 100K.

	RIFT-stat			RIFT-dyn			RIFT-ada
$T$	2K	4K	8K	2K	4K	8K	2K	4K	8K
0	60.5%	66.0%	72.7%	84.3%	93.4%	97.1%	88.4%	94.8%	98.3%
1	56.6%	62.5%	69.3%	81.5%	91.6%	96.1%	87.2%	94.7%	98.2%

Appendix

Table 13: Post-training AL as a percentage of the matching full-KV AL for cross-tokenizer Qwen3.5-4B

\rightarrow

Kimi-K3 drafting. The absolute full-KV AL is

4.834/5.138

T=0/1

Panel of Figure 15 — Figure 15: Effect of SnapKV block size for Qwen3.5-4B drafting for Qwen3.5-122B with a 50/50 local–remote cache split. The 32K experiments use an 8K cache budget; the 100K experiments use a 16K budget. The top and bottom rows report $T=0$ and $T=1$ , respectively. Moderate block sizes are broadly robust, whereas very coarse blocks reduce selection resolution and lower AL.

Budget	Selector	$T=0$	$T=1$
4K	CriticalKV	$3.57\rightarrow 5.02$ $(+40.6\%)$	$3.15\rightarrow 4.33$ $(+37.3\%)$
	Expected Attention	$3.59\rightarrow 5.03$ $(+40.2\%)$	$3.09\rightarrow 4.30$ $(+39.0\%)$
	CurDKV	$3.29\rightarrow 4.84$ $(+47.1\%)$	$2.84\rightarrow 4.14$ $(+45.5\%)$
8K	CriticalKV	$4.13\rightarrow 5.62$ $(+36.2\%)$	$3.74\rightarrow 4.98$ $(+33.1\%)$
	Expected Attention	$4.06\rightarrow 5.56$ $(+36.9\%)$	$3.56\rightarrow 4.83$ $(+35.9\%)$
	CurDKV	$3.76\rightarrow 5.35$ $(+42.2\%)$	$3.26\rightarrow 4.63$ $(+42.1\%)$

Appendix

Table 14: RIFT-stat extends beyond SnapKV: Qwen3.5-4B draft (Qwen3.5-122B target), 32K ISL, 4K OSL with draft length of 11. Each entry shows baseline AL

\rightarrow

AL with RIFT-stat, followed by the relative improvement.

Panel of Figure 16 — Figure 16: RIFT-ada AL-refresh tradeoffs for Qwen3.5-4B drafting Qwen3.5-122B on SPEED-Bench low-entropy data, with a 4K maximum OSL. Titles specify ISL, temperature, and KV cache budget.

Panel of Figure 17 — Figure 17: Nemotron3.5-Lightning self-speculation RIFT-ada refresh - AL tradeoff. SPEED-Bench 32k ISL, low entropy category.

Benchmark	Budget (Avg. compression)	SnapKV	RIFT-stat
LongBench-v1	1K (9.95 $\times$ )	60.7	76.2 (+15.5)
	2K (4.98 $\times$ )	61.7	81.8 (+20.1)
	4K (2.49 $\times$ )	63.4	89.5 (+26.1)
	8K (1.24 $\times$ )	67.2	96.4 (+29.2)
LongBench-v2	1K (119.41 $\times$ )	60.3	61.9 (+1.6)
	2K (59.70 $\times$ )	67.7	70.4 (+2.6)

Appendix

Table 15: Full-KV score retention (%) for Qwen3.5-4B under the modified LongBench protocol described above. Absolute full-KV scores are

47.607

for LongBench-v1 and

38.569

for LongBench-v2.

Task	SnapKV	SWA	RIFT-stat	RIFT-dyn	RIFT-ada
$T=0$
Sorting	3.65	3.96	4.84	5.35	5.61
Creative writing	3.90	4.86	5.79	6.29	6.28
NIAH	4.30	4.86	5.50	6.16	6.18
Code debugging	3.76	5.02	5.58	6.67	6.80
Creative writing	3.03	3.91	4.18	4.39	4.42

Appendix

Table 16: Mean AL using SPEED-Bench data of ISLs 16K, 32K and 100K, at

T=0

and

T=1

. Draft model is Qwen3.5-4B drafting for Qwen3.5-122B. Draft length is 11 and max OSL is 4K.

Model	ISL	$T$	StreamingLLM	RIFT-stat	RIFT-dyn	RIFT-ada
Qwen3.5	32K	0	5.34	5.58	6.08	6.17
	32K	1	4.73	4.92	5.40	5.49
	100K	0	5.60	5.77	6.61	6.73
	100K	1	4.98	5.07	5.76	5.92
Llama3.1	32K	0	5.87	5.87	6.58	7.08
	32K	1	4.20	4.41	4.89	5.41

Appendix

Table 17: Mean AL on SPEED-Bench low-entropy data. Qwen3.5-4B drafts for Qwen3.5-122B and Llama3.1-8B for Llama3.1-70B. All rows use an 8K budget, draft length 11, and a 4K maximum OSL. StreamingLLM retains an 8K window plus four sinks.

$T$	Budget	StreamingLLM	H 2 O	TriAttention	RIFT-stat	RIFT-dyn	RIFT-ada
ISL 32K
0	2K	4.52	4.68	4.54	4.54	5.14	5.28
	4K	4.79	5.25	4.97	5.03	5.73	5.82
	8K	5.34	5.73	5.43	5.62	6.12	6.17
1	2K	3.90	4.06	3.86	3.94	4.47	4.54
	4K	4.17	4.53	4.27	4.34	5.03	5.10

Appendix

Table 18: Mean AL on SPEED-Bench low entropy for Qwen3.5-4B drafting Qwen3.5-122B, with draft length 11 and a 4K maximum OSL. StreamingLLM uses the stated window plus four sinks.

	$T=0$			$T=1$
Training configuration	2K	4K	8K	2K	4K	8K
Random, dynamic $p$	4.67	5.15	5.70	4.32	4.71	5.34
Random, $p=0.1$	4.31	4.76	5.20	3.99	4.37	4.87
Random, $p=0.3$	4.56	5.04	5.56	4.22	4.62	5.21
Random, $p=0.5$	4.59	5.09	5.67	4.24	4.66	5.29
Random, $p=0.7$	4.63	5.12	5.70	4.25	4.66	5.31

Appendix

Table 19: AL of the trained draft model (Qwen3.5-4B) under various masking strategies. ’dynamic

p

’ refers to the same

p

allocation used in the main draft-compressibility results.

	RIFT-stat			RIFT-dyn
Training configuration	2K	4K	8K	2K	4K	8K
(a) Sliding-window ratio $\rho$ during training
$\rho=0.10$	4.67	5.19	5.80	5.36	5.92	6.28
$\rho=0.25$	4.69	5.21	5.82	5.39	5.92	6.29
$\rho=0.50$	4.72	5.22	5.81	5.40	5.93	6.29
$\rho=0.75$	4.72	5.21	5.79	5.38	5.93	6.30

Appendix

Table 20: Training-mask hyperparameter ablations for Qwen3.5-4B drafting Qwen3.5-122B on SPEED-Bench 32K low-entropy. All checkpoints are evaluated with

\rho=0.5

with

T=0

	$T=0$		$T=1$
Training configuration	RIFT-stat	RIFT-dyn	RIFT-stat	RIFT-dyn
LoRA-32, constant $10^{-5}$	$\mathbf{5.30\pm 0.13}$	$\mathbf{5.85\pm 0.13}$	$4.93\pm 0.11$	$\mathbf{5.46\pm 0.12}$
LoRA-128, decay from $10^{-5}$	$5.29\pm 0.13$	$\mathbf{5.85\pm 0.13}$	$4.93\pm 0.12$	$5.45\pm 0.13$
Full FT, constant $5{\times}10^{-6}$	$5.29\pm 0.13$	$5.82\pm 0.13$	$4.91\pm 0.12$	$5.43\pm 0.13$
Full FT, decay from $10^{-5}$	$\mathbf{5.30\pm 0.13}$	$\mathbf{5.85\pm 0.13}$	$\mathbf{4.94\pm 0.12}$	$\mathbf{5.46\pm 0.13}$

Appendix

Table 21: Training hyperparameter comparison. Results are reported in average ALs with 95% confidence intervals.

	RIFT-stat			RIFT-dyn			RIFT-ada
$T$	2K	4K	8K	2K	4K	8K	2K	4K	8K
0	$+4.05\%$	$+2.24\%$	$+0.97\%$	$+4.90\%$	$+2.54\%$	$+0.97\%$	$+2.57\%$	$+0.37\%$	$-0.34\%$
1	$+12.50\%$	$+9.87\%$	$+7.03\%$	$+12.04\%$	$+7.95\%$	$+5.12\%$	$+9.80\%$	$+5.49\%$	$+3.79\%$

Appendix

Table 22: Relative AL improvement from compressibility training for Qwen3.5-4B drafting Qwen3.5-122B on SPEED-Bench 100K.

$T$	2K	4K	8K
0	$+15.57\%$	$+13.50\%$	$+11.52\%$
1	$+26.54\%$	$+23.80\%$	$+21.07\%$

Appendix

Table 23: Relative AL improvement from draft compressibility training for SnapKV: Qwen3.5-4B draft (Qwen3.5-122B target), 100K low-entropy split.

ISL	Entropy	$T$	Full KV	SW	RIFT-stat	RIFT-dyn
32K	Low	0	$+0.55\%$	$+0.90\%$	$+0.84\%$	$+0.81\%$
	Low	1	$+7.77\%$	$+6.43\%$	$+9.71\%$	$+8.87\%$
	High	0	$+0.82\%$	$+0.80\%$	$+0.98\%$	$+0.84\%$
	High	1	$+8.95\%$	$+7.30\%$	$+9.33\%$	$+9.32\%$
100K	Low	0	$+0.00\%$	$+1.73\%$	$+0.90\%$	$+0.77\%$
	Low	1	$+5.90\%$	$+5.70\%$	$+8.24\%$	$+7.68\%$

Appendix

Table 24: Relative AL improvement from training the Qwen3.5-0.8B draft, measured on SPEED-Bench at an 8K cache budget against the corresponding untrained draft.

		$T=0$		$T=1$
Inference method	Budget	Base	Trained	Base	Trained
Full KV	–	5.39	5.75	4.86	5.49
Sliding window	2K	3.62	3.87	3.21	3.73
	4K	3.92	4.19	3.49	4.02
	8K	4.31	4.60	3.86	4.43
RIFT-stat	2K	3.75	4.04	3.39	3.92

Appendix

Table 25: Average accepted length before and after compressibility training of Nemotron3.5-Lightning drafting Nemotron-3-Ultra on SPEED-Bench 32K low-entropy. The trained checkpoint uses skewed KL with

\alpha=0.5

. Static and Dynamic SnapKV use a sliding-window ratio of

\rho=0.5

Draft $\rightarrow$ target	ISL	Entropy	$T$	2K	4K	8K
Qwen3.5-4B $\rightarrow$ Qwen3.5-122B	32K	Low	0	$+3.88\%$	$+2.38\%$	$+1.69\%$
Qwen3.5-4B $\rightarrow$ Qwen3.5-122B	32K	Low	1	$+10.03\%$	$+7.88\%$	$+6.70\%$

Appendix

Table 26: Relative AL improvement from compressibility training on RIFT-ada.

KV-Cache Compression

Key-value cache compression reduces the memory and bandwidth needed to retain attention states during long-context inference.

See all 265 papers

DeferKV: Rethinking Eviction Timing for One-Shot KV Cache Compression
Oct 5, 2026Zhe Wang, Jiakai Li, Yujia Sun +2Long-Context Language Model Inference KV-Cache Eviction
University of Electronic Science and Technology of China, Chengdu, China · Ubiquitous Intelligence and Trusted Services Key Laboratory of Sichuan Province, Chengdu, China
When Hidden States Drift: Can KV Caches Rescue Long-Range Speculative Decoding?
Apr 29, 2026Tianyu Liu, Yuhao Shen, Xinyi Hu +8Speculative Decoding LLM Inference Acceleration
1Qwen Applications Business Group of Alibaba · University of Science and Technology of China · 3Zhejiang University
KVFetch: Temporal Prefetching for the Missing Half of KV Cache Compression
Sep 23, 2026Linfeng DongKV-Cache Management KV-Cache Compression

Oct 5, 2026J/K move · Enter open · S save

DeferKV: Rethinking Eviction Timing for One-Shot KV Cache Compression

Zhe Wang, Jiakai Li, Yujia Sun, Rongzheng Wang+1

University of Electronic Science and Technology of China, Chengdu, China · Ubiquitous Intelligence and Trusted Services Key Laboratory of Sichuan Province, Chengdu, China

Long-Context Language Model Inference KV-Cache Eviction KV-Cache Compression

Open paper AskPDF arXiv

Long-context large language models (LLMs) have demonstrated strong capabilities across a wide range of tasks, but the growing KV cache introduces substantial memory and inference overhead. Existing one-shot KV cache compression methods typically commit to irreversible eviction immediately after prefill, before any signal from actual generation becomes available. Our quantitative analysis shows that early queries from the actual generation stage provide attention signals that are more consistent with subsequent decode attention, with the largest single-step gain occurring at the prefill-decode boundary. Based on this observation, we propose DeferKV, which moves the eviction decision from the end of prefill to the first real decoding step and temporally combines prompt-side and decode-side observations, thereby better aligning KV importance estimation with subsequent generation requirements. DeferKV requires no additional training, draft model, or future-query prediction module, making it simple and easy to deploy. Experiments on LongBench, RULER, and Needle-in-a-Haystack demonstrate that DeferKV consistently improves model performance under KV cache compression while maintaining low inference latency.

Abstract

KV-Cache Compression

Explore similar work

DeferKV: Rethinking Eviction Timing for One-Shot KV Cache Compression

When Hidden States Drift: Can KV Caches Rescue Long-Range Speculative Decoding?

KVFetch: Temporal Prefetching for the Missing Half of KV Cache Compression