DeferKV: Rethinking Eviction Timing for One-Shot KV Cache Compression

Zhe Wang; Jiakai Li; Yujia Sun; Rongzheng Wang; Shuang Liang

cs.CLOct 5, 2026

DeferKV: Rethinking Eviction Timing for One-Shot KV Cache Compression

Authors: Zhe Wang, Jiakai Li, Yujia Sun, Rongzheng Wang, Shuang Liang

Organizations: University of Electronic Science and Technology of China, Chengdu, China · Ubiquitous Intelligence and Trusted Services Key Laboratory of Sichuan Province, Chengdu, China

Abstract

Long-context large language models (LLMs) have demonstrated strong capabilities across a wide range of tasks, but the growing KV cache introduces substantial memory and inference overhead. Existing one-shot KV cache compression methods typically commit to irreversible eviction immediately after prefill, before any signal from actual generation becomes available. Our quantitative analysis shows that early queries from the actual generation stage provide attention signals that are more consistent with subsequent decode attention, with the largest single-step gain occurring at the prefill-decode boundary. Based on this observation, we propose DeferKV, which moves the eviction decision from the end of prefill to the first real decoding step and temporally combines prompt-side and decode-side observations, thereby better aligning KV importance estimation with subsequent generation requirements. DeferKV requires no additional training, draft model, or future-query prediction module, making it simple and easy to deploy. Experiments on LongBench, RULER, and Needle-in-a-Haystack demonstrate that DeferKV consistently improves model performance under KV cache compression while maintaining low inference latency.

Figures & tables

Figure 1: Consistency of prompt-end and first-decode attention with subsequent decode attention on GovReport using Qwen3-8B. Each point denotes a historical KV token. The horizontal axis shows its average attention from $D_{2}$ to the end of generation, while the vertical axis shows the average over the last 32 prefill queries and the attention from the first decode query. The dashed diagonal marks exact agreement, and the shaded region indicates values within a factor of two.

Figure 2: JSD-Based Similarity and Top-

B

KV Selection Recall at different query positions.

Figure 3: Overview of DeferKV. The first real decode query is combined with recent prompt queries and temporally weighted to estimate KV importance for subsequent cache compression.

Qwen3-8B , $B_{\mathrm{total}}=64L$
	Single-Doc QA			Multi-Doc QA			Summ.			Few-Shot			Synth.		Code
Method	NrtvQA	Qasper	MF-en	HotpotQA	2WikiMQA	MuSiQue	GovReport	QMSum	MultiNews	TREC	TriviaQA	SAMSum	PCount	PR-en	LCC	RB-P	Avg.
SnapKV	17.41	30.06	38.05	47.77	37.71	20.65	14.00	19.37	12.93	43.50	81.92	36.53	5.00	87.50	59.30	53.58	37.83
LAQ	18.99	37.88	43.31	53.88	41.35	23.24	16.18	19.40	14.78	66.00	88.40	37.64	5.50	94.50	62.31	55.91	42.45
SpecKV	19.61	29.97	49.79	51.03	36.97	22.14	13.27	19.72	14.82	37.50	73.63	36.71	6.00	95.00	45.15	46.62	37.37
LookaheadKV	18.22	32.27	40.12	47.66	38.01	23.58	14.61	18.36	13.18	46.95	83.29	36.69	5.00	90.40	59.06	54.28	38.86

Table 1: Performance comparison on 16 LongBench datasets. The best and second-best results among the five cache-compression methods are highlighted in bold and underlined, respectively. Full KV is included only as a reference.

Method	4K	8K	16K	32K	64K	128K	Avg.
Full KV	99.34	98.83	98.55	94.89	89.85	79.32	93.46
SnapKV	83.63	75.51	71.09	66.12	56.26	45.67	66.38
LAQ	87.42	81.78	78.27	74.09	68.23	62.06	75.31
SpecKV	86.14	79.43	75.60	70.53	62.83	54.17	71.45
LookaheadKV	86.66	80.69	76.98	72.43	65.73	59.26	73.63
DeferKV	92.72	86.29	83.89	80.31	75.89	69.04	81.36

Table 2: Performance comparison on the RULER benchmark across different context lengths.

Figure 4: Peak GPU memory and end-to-end inference latency of different methods on Qwen3-8B.

Delay $d$	Score $\uparrow$	Peak Mem. $\downarrow$	Latency $\downarrow$
$\mathbf{1}$	27.06	22.84	11.562
$2$	27.11	22.85	11.618
$3$	27.15	22.85	11.671
$4$	27.21	22.86	11.751

Table 3: Effect of compression delay steps on 32K-token GovReport samples using Qwen3-8B. Peak memory is reported in GiB, and latency is reported in seconds.

Method	Avg. Score $\uparrow$
DeferKV	46.12
w/o Decode Feedback	44.38 ( $-$ 1.74)
w/o Temporal Weighting	45.45 ( $-$ 0.67)

Table 4: Ablation results of DeferKV on LongBench using Qwen3-8B.

Figure 5: Effect of the temporal discount factor

\eta

on model performance.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Qwen3-8B , $B_{\mathrm{total}}=\mathrm{FullKV}$
	Single-Document QA			Multi-Document QA			Summarization			Few-Shot Learning			Synthetic		Code
Method	NrtvQA	Qasper	MF-en	HotpotQA	2WikiMQA	MuSiQue	GovReport	QMSum	MultiNews	TREC	TriviaQA	SAMSum	PCount	PR-en	LCC	RB-P	Avg.
Full KV	24.73	48.05	54.39	58.16	43.79	34.27	33.45	24.12	24.84	71.50	90.21	44.56	9.50	100.00	69.34	65.48	49.77
Qwen3-8B , $B_{\mathrm{total}}=64L$
SnapKV	17.41	30.06	38.05	47.77	37.71	20.65	14.00	19.37	12.93	43.50	81.92	36.53	5.00	87.50	59.30	53.58	37.83
LAQ	18.99	37.88	43.31	53.88	41.35	23.24	16.18	19.40	14.78	66.00	88.40	37.64	5.50	94.50	62.31	55.91	42.45

Appendix

Table 5: Detailed LongBench results of Qwen3-8B under different prompt-KV budgets. The best and second-best results within each budget are shown in bold and underlined, respectively.

Llama-3.1-8B-Instruct , $B_{\mathrm{total}}=\mathrm{FullKV}$
	Single-Document QA			Multi-Document QA			Summarization			Few-Shot Learning			Synthetic		Code
Method	NrtvQA	Qasper	MF-en	HotpotQA	2WikiMQA	MuSiQue	GovReport	QMSum	MultiNews	TREC	TriviaQA	SAMSum	PCount	PR-en	LCC	RB-P	Avg.
Full KV	31.20	46.66	56.64	58.10	48.51	31.57	34.37	25.18	27.14	73.00	91.65	43.80	7.64	99.50	65.18	58.90	49.94
Llama-3.1-8B-Instruct , $B_{\mathrm{total}}=64L$
SnapKV	27.32	25.25	44.30	53.98	42.48	27.10	17.88	22.20	17.45	41.00	85.37	37.72	8.50	100.00	55.35	48.69	40.91
LAQ	27.66	28.03	47.05	58.12	45.06	30.60	18.85	22.51	19.01	71.50	90.02	40.16	8.50	99.50	56.39	51.29	44.64

Appendix

Table 6: Detailed LongBench results of Llama-3.1-8B-Instruct under different prompt-KV budgets. The best and second-best results within each budget are shown in bold and underlined, respectively.

Mistral-7B-Instruct-v0.3 , $B_{\mathrm{total}}=\mathrm{FullKV}$
	Single-Document QA			Multi-Document QA			Summarization			Few-Shot Learning			Synthetic		Code
Method	NrtvQA	Qasper	MF-en	HotpotQA	2WikiMQA	MuSiQue	GovReport	QMSum	MultiNews	TREC	TriviaQA	SAMSum	PCount	PR-en	LCC	RB-P	Avg.
Full KV	27.01	37.97	50.14	51.69	39.17	28.02	34.27	25.66	26.62	76.00	88.59	47.50	7.00	98.00	61.53	62.59	47.61
Mistral-7B-Instruct-v0.3 , $B_{\mathrm{total}}=64L$
SnapKV	17.65	22.08	36.24	41.95	33.98	20.89	18.83	21.14	16.34	38.50	86.96	39.81	4.50	77.00	51.57	49.59	36.06
LAQ	19.72	28.68	40.14	48.46	35.96	23.44	20.82	21.47	18.93	70.58	89.15	42.32	4.75	79.80	53.08	51.19	40.53

Appendix

Table 7: Detailed LongBench results of Mistral-7B-Instruct-v0.3 under different prompt-KV budgets. The best and second-best results within each budget are shown in bold and underlined, respectively.

Method	Prompt-KV Budget $B_{\mathrm{total}}$					Avg.
	$64L$	$128L$	$256L$	$512L$	$1024L$
PyramidKV	38.30	43.25	46.40	48.00	48.88	44.97
DeferKV w. PyramidKV	44.18	46.30	47.66	48.52	49.18	47.17 ( $\uparrow 4.89\%$ )
AdaKV	38.55	43.50	46.58	48.08	48.92	45.13
DeferKV w. AdaKV	44.30	46.38	47.72	48.57	49.22	47.24 ( $\uparrow 4.68\%$ )
ChunkKV	37.60	42.80	46.05	47.78	48.68	44.58

Appendix

Table 8: Compatibility results on LongBench using Qwen3-8B. Each integrated variant retains the allocation and selection mechanisms of the corresponding compression method while replacing its prompt-only importance scores with DeferKV scores.

Llama-3.1-8B-Instruct , $B_{\mathrm{total}}=1024L$ , context length $=4K$
Method	S-NIAH-1	S-NIAH-2	S-NIAH-3	MK-NIAH-1	MK-NIAH-2	MK-NIAH-3	MQ-NIAH	MV-NIAH	CWE	FWE	VT	Avg. Acc.
Full KV	100.00	100.00	99.60	100.00	100.00	99.60	99.90	96.25	99.78	97.67	99.96	99.34
SnapKV	100.00	99.00	20.60	99.60	93.00	31.00	99.20	90.75	91.56	95.31	99.96	83.63
LAQ	100.00	99.10	53.90	99.60	96.20	36.80	99.60	93.37	88.25	94.90	99.90	87.42
SpecKV	100.00	99.00	42.60	99.50	95.20	34.60	99.58	92.51	89.65	95.00	99.90	86.14
LookaheadKV	100.00	99.00	47.30	99.60	95.60	35.50	99.60	92.82	88.94	95.00	99.90	86.66

Appendix

Table 9: Detailed RULER results across context lengths from 4K to 128K using Llama-3.1-8B-Instruct with

B_{\mathrm{total}}=1024L

Figure 6: Needle-in-a-Haystack retrieval performance of Full KV, SnapKV, LookaheadKV, and DeferKV on Mistral-7B-Instruct-v0.3 with $B_{\mathrm{total}}=128L$ .

Figure 7: Needle-in-a-Haystack retrieval performance of Full KV, SnapKV, LookaheadKV, and DeferKV on Mistral-7B-Instruct-v0.3 with $B_{\mathrm{total}}=1024L$ .

Decoding	Full KV	SnapKV	LAQ	SpecKV	LookaheadKV	DeferKV
Greedy	49.77	42.95	45.07	43.32	44.63	46.12
$T=0.2$	49.64	42.07	44.58	40.94	43.02	46.01
$T=0.8$	48.91	41.34	43.82	40.16	42.28	45.32

Appendix

Table 10: Average LongBench performance under greedy and stochastic decoding using Qwen3-8B with

B_{\mathrm{total}}=128L

Explore similar work

Moment-KV: Momentum-Based Decode-Time KV Cache Compression for Long Generation
May 28, 2026Soumyadeep Jana, Sagar Nishad, Sanasam Ranbir SinghKey-Value Cache Compression Large Language Model Decoding
Indian Institute of Technology Guwahati, India
CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference
Jun 23, 2026Xiaolin Lin, Jingcun Wang, Olga Kondrateva +3Key-Value Cache Compression LLM Inference Optimization
Technical University of Darmstadt, Darmstadt, Germany · University of Notre Dame, Notre Dame, IN, USA · Technical University of Ilmenau, Ilmenau, Germany
Behavior-Preserving KV Cache Compression
Oct 5, 2026Doo Hwan Hwang, Junyoung Jang, Junho Na +2Key-Value Cache Compression
Kim Jaechul Graduate School of AI, KAIST, Daejeon, Republic of Korea · KT Corporation, Seoul, Republic of Korea

May 28, 2026, cs.AIJ/K move · Enter open · S save

Moment-KV: Momentum-Based Decode-Time KV Cache Compression for Long Generation

Soumyadeep Jana, Sagar Nishad, Sanasam Ranbir Singh

Indian Institute of Technology Guwahati, India

Key-Value Cache Compression Large Language Model Decoding Cache Data Compression Methods Temporal Attention Data Generation

Open paper AskPDF arXiv

Key-Value (KV) cache remains a major bottleneck for deploying Large Language Models (LLMs) in long-generation tasks. Prior work often applies uniform compression across both prefill and decoding caches, but compressing the prefill cache degrades performance by corrupting critical context. While preserving the prefill cache is essential, decoding-phase compression remains underexplored, with existing methods relying on rigid recency windows or instantaneous attention. Our analysis of attention dynamics reveals strong temporal patterns: critical tokens receive sustained attention over long horizons, while local reasoning involves short-lived bursts. Static heuristics fail to capture this behavior, leading to premature eviction of important tokens or retention of stale ones. We propose Moment-KV, a decoding-time KV cache compression method based on momentum-driven temporal attention aggregation. Our method models token importance as a continuously evolving state, where attention is aggregated with decay, capturing both long-term influence and recent relevance. Experiments show that Moment-KV significantly improves generation fidelity in long-generation tasks (2.3-3.2 %) while maintaining decoding latency.

Abstract

Explore similar work

Moment-KV: Momentum-Based Decode-Time KV Cache Compression for Long Generation

CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference

Behavior-Preserving KV Cache Compression