LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation

Jinwoo Ahn; Ingyu Seong; Akhil Kedia; Junhan Kim; Hyemi Jang; Kangwook Lee; Yongkweon Jeon

cs.LGMar 11, 2026

LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation

Authors: Jinwoo Ahn, Ingyu Seong, Akhil Kedia, Junhan Kim, Hyemi Jang, Kangwook Lee, Yongkweon Jeon

Organizations: Samsung Research

Abstract

Transformer-based large language models (LLMs) rely on key-value (KV) caching to avoid redundant computation during autoregressive inference. While this mechanism greatly improves efficiency, the cache size grows linearly with the input sequence length, quickly becoming a bottleneck for long-context tasks. Existing solutions mitigate this problem by evicting prompt KV that are deemed unimportant, guided by estimated importance scores. Notably, a recent line of work proposes to improve eviction quality by "glimpsing into the future", in which a draft generator produces a surrogate future response approximating the target model's true response, and this surrogate is subsequently used to estimate the importance of cached KV more accurately. However, these approaches rely on computationally expensive draft generation, which introduces substantial prefilling overhead and limits their practicality in real-world deployment. To address this challenge, we propose LookaheadKV, a lightweight eviction framework that leverages the strength of surrogate future response without requiring explicit draft generation. LookaheadKV augments transformer layers with parameter-efficient modules trained to predict true importance scores with high accuracy. Our design ensures negligible runtime overhead comparable to existing inexpensive heuristics, while achieving accuracy superior to more costly approximation methods. Extensive experiments on long-context understanding benchmarks, across a wide range of models, demonstrate that our method not only outperforms recent competitive baselines in various long-context understanding tasks, but also reduces the eviction cost by up to 14.5x, leading to significantly faster time-to-first-token. Our code is available at https://github.com/SamsungLabs/LookaheadKV.

Figures & tables

Figure 1: An overview of LookaheadKV (a) Training. During training, lookahead tokens and lookahead LoRA are trained to predict the ground-truth importance scores obtained with pre-generated model response via a KL divergence loss. (b) Inference. During prefill, LookaheadKV utilizes the learned modules to identify essential tokens and compress the KV cache, facilitating memory-efficient decoding.

Figure 2: Accuracy-overhead trade-off across KV cache eviction methods.

Panel of Figure 3 — Figure 3: Time-to-First-Token (TTFT) latency overhead ratio across context lengths. Similar to SnapKV, LookaheadKV introduces negligible TTFT overhead across all tested context lengths; draft-based methods (LAQ, SpecKV) incur substantial latency, especially for shorter contexts.

Model	Budget	PyramidKV	SnapKV	StreamingLLM	SpecKV	LAQ	LookaheadKV
	FullKV score: 5.72
LLaMA3.2-1B	64	4.64	4.70	4.54	N/A	5.03	5.21
LLaMA3.2-1B	128	5.10	5.39	4.94	N/A	5.45	5.60
	256	5.49	5.67	5.37	N/A	5.64	5.62
	FullKV score: 7.35
LLaMA3.2-3B	64	6.30	6.28	5.96	6.52	6.48	6.87

Table 2: MT‑Bench evaluation results. Bold and underlined values indicate best and second best scores, respectively.

Figure 5: HTML-to-TSV evaluation results using LLaMA3.1-8B.

		Theoretical Cost				Empirical Cost
Context Length	Method	Compute (TFLOPs)	Memory Traffic (GB)	TTFT (ms)	TTFT Overhead (ms)	TTFT (ms)	TTFT Overhead (ms)
$8$ K	Forward Pass Only	$136.00$	$13.00$	$256.99$	N/A	$290.81$	N/A
	LookaheadKV	$136.53$	$13.05$	$258.03$	$1.0345$	$301.69$	$10.8800$
	SnapKV	$136.00$	$13.02$	$257.00$	$0.0085$	$310.98$	$20.1700$
	SpecKV	$159.12$	$81.07$	$336.52$	$79.5264$	$411.32$	$120.5100$
	LAQ	$137.06$	$444.52$	$491.58$	$234.5877$	$800.19$	$509.3800$

Table 3: Theoretical and empirical cost analysis of LLaMA3.1-8B at

C=128

Method	FullKV	SnapKV	SpecKV	LAQ	LookaheadKV
Greedy	49.88	43.50	45.45	46.61	47.72
$T=0.2$	49.58 (-0.60%)	43.29 (-0.48%)	44.99 (-1.01%)	46.73 (+0.26%)	47.75 (+0.06%)
$T=0.8$	47.82 (-4.13%)	41.39 (-4.85%)	43.43 (-4.44%)	45.27 (-2.87%)	45.81 (-4.00%)

Table 4: Average LongBench performance at different temperature settings on LLaMA3.1-8B, with

C=128

. LookaheadKV outperforms baselines across all tested temperature settings.

	$n_{\text{lookahead}}=4$		$n_{\text{lookahead}}=8$		$n_{\text{lookahead}}=16$		$n_{\text{lookahead}}=32$		$n_{\text{lookahead}}=64$		$n_{\text{lookahead}}=128$
Module	score	overhead(%)	score	overhead(%)	score	overhead(%)	score	overhead(%)	score	overhead(%)	score	overhead(%)
emb-only	$25.53$	$3.36$	$25.67$	$3.77$	$26.36$	$3.42$	$26.40$	$4.18$	$25.8$	$7.3$	$26.2$	$10.7$
QV	$26.45$	$3.65$	$26.40$	$4.08$	$26.87$	$3.99$	$26.85$	$4.42$	$26.7$	$7.7$	$27.1$	$10.7$
all	$26.60$	$4.20$	$27.02$	$4.20$	$27.01$	$4.67$	$27.12$	$4.96$	$27.1$	$8.5$	$27.0$	$11.0$

Table 5: 2D ablation across lookahead sizes and trainable modules, on LLaMA3.2-1B. Average LongBench scores with cache budget of 64 and TTFT overhead are reported.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Context Length	FullKV	LookaheadKV	SnapKV	SpecKV	LAQ
64K	$84.15$	69.45	$39.64$	$64.02$	$64.10$
128K	$73.72$	54.83	$30.56$	$52.62$	$50.67$

Appendix

Table 6: RULER evaluation results on longer context lengths using Llama3.1-8B-Instruct at

C=128

FullKV	LookaheadKV	LookaheadKV
		+ Suffix Window
32.01	29.10	28.52 (-1.99%)

Appendix

Table 7: Average LongBench scores using LookaheadKV window only and LookaheadKV + SnapKV-style suffix window, evaluated using LLaMA3.2-1B-Instruct with

C=64

Generation Method	$T=0.2$	$T=0.4$	$T=0.6$	$T=0.8$	LLaMA3.2-1B
Recall@512 (%)	$95.06$	$93.73$	$91.40$	$91.37$	$88.66$
Kendall’s Tau	$91.44$	$88.63$	$84.61$	$84.79$	$80.05$

Appendix

Table 8: Importance score similarity with stochastic response using various temperatures vs. greedy response on LLaMA3.1-8B. LLaMA3.2-1B presents the similarity of importance scores using greedy response generated with LLaMA3.2-1B vs. LLaMA3.1-8B.

	Single-Document QA			Multi-Document QA			Summarization			Few-shot Learning			Synthetic		Code		Avg.
	NrtQA	Qasper	MF-en	HotpotQA	2WikiMQA	Musique	GovReport	QMSum	MultiNews	TREC	TriviaQA	SAMSum	PCount	Pre	Lcc	RB-P	Avg.
FullKV	19.24	15.96	42.47	35.53	29.42	19.87	28.34	22.18	25.64	64.00	80.85	38.83	3.00	4.78	38.63	43.35	32.01
KV Cache Size = 64
StreamingLLM	14.42	12.30	24.02	24.04	24.05	9.35	13.56	19.23	13.61	35.50	68.94	29.09	4.00	3.71	37.61	37.33	23.17
SnapKV	14.24	12.18	30.53	27.30	25.66	12.44	14.27	19.27	12.37	35.00	73.57	29.29	2.00	4.49	38.01	36.84	24.22
PyramidKV	13.33	11.36	26.05	25.34	24.01	10.57	13.64	19.49	11.96	35.00	69.95	27.75	1.50	4.33	35.25	36.01	22.85

Appendix

Table 9: LongBench evaluation results for Llama3.2-1B

	Single-Document QA			Multi-Document QA			Summarization			Few-shot Learning			Synthetic		Code		Avg.
	NrtQA	Qasper	MF-en	HotpotQA	2WikiMQA	Musique	GovReport	QMSum	MultiNews	TREC	TriviaQA	SAMSum	PCount	Pre	Lcc	RB-P	Avg.
FullKV	18.94	24.78	46.15	39.15	33.11	19.03	30.33	23.06	25.18	74.00	85.21	42.87	0.00	94.50	46.82	40.24	40.21
KV Cache Size = 64
StreamingLLM	12.25	16.83	20.38	22.62	24.61	7.26	11.43	18.96	12.12	39.00	66.48	35.14	0.00	12.50	42.95	36.90	23.71
SnapKV	13.01	17.50	30.47	28.14	25.68	8.63	11.89	19.77	11.03	37.50	78.59	37.62	0.00	54.50	44.42	36.92	28.48
PyramidKV	13.20	16.31	27.19	23.83	25.80	6.95	11.23	19.12	10.72	37.00	73.99	35.99	0.00	43.00	43.01	35.97	26.46

Appendix

Table 10: LongBench evaluation results for Qwen3-1.7B

	Single-Document QA			Multi-Document QA			Summarization			Few-shot Learning			Synthetic		Code		Avg.
	NrtQA	Qasper	MF-en	HotpotQA	2WikiMQA	Musique	GovReport	QMSum	MultiNews	TREC	TriviaQA	SAMSum	PCount	Pre	Lcc	RB-P	Avg.
FullKV	25.35	40.77	50.31	53.79	40.05	25.62	33.11	24.38	25.95	72.00	88.89	43.25	4.00	97.50	54.65	56.83	46.03
KV Cache Size = 64
StreamingLLM	19.79	20.68	24.68	42.63	34.09	16.14	16.53	20.22	14.78	38.00	75.84	33.19	4.50	10.50	46.11	43.76	28.84
SnapKV	21.88	20.56	39.12	48.33	35.39	22.39	16.37	20.30	14.41	39.50	82.17	35.05	4.50	21.50	47.96	44.30	32.11
PyramidKV	23.06	19.20	36.26	46.34	34.81	20.17	16.17	20.96	14.02	41.00	81.93	35.08	4.00	23.00	46.82	43.94	31.67

Appendix

Table 11: LongBench evaluation results for Llama3.2-3B

	Single-Document QA			Multi-Document QA			Summarization			Few-shot Learning			Synthetic		Code		Avg.
	NrtQA	Qasper	MF-en	HotpotQA	2WikiMQA	Musique	GovReport	QMSum	MultiNews	TREC	TriviaQA	SAMSum	PCount	Pre	Lcc	RB-P	Avg.
FullKV	27.45	43.30	54.45	55.63	43.43	31.61	32.24	24.61	25.00	73.00	88.76	43.65	0.75	96.50	64.29	61.39	47.88
KV Cache Size = 64
StreamingLLM	12.46	23.96	25.93	38.56	33.40	19.47	13.74	19.71	13.04	39.50	75.48	34.33	0.50	64.50	51.42	48.46	32.15
SnapKV	15.28	25.03	31.61	40.00	34.95	18.83	12.88	19.78	12.49	40.50	75.62	33.69	1.00	69.00	51.48	47.38	33.10
PyramidKV	15.50	24.84	34.33	40.70	35.07	19.39	13.48	19.85	13.03	41.50	76.69	33.95	1.50	73.00	52.98	47.77	33.97

Appendix

Table 12: LongBench evaluation results for Qwen3-4B

	Single-Document QA			Multi-Document QA			Summarization			Few-shot Learning			Synthetic		Code		Avg.
	NrtQA	Qasper	MF-en	HotpotQA	2WikiMQA	Musique	GovReport	QMSum	MultiNews	TREC	TriviaQA	SAMSum	PCount	Pre	Lcc	RB-P	Avg.
FullKV	31.63	46.66	56.93	58.10	48.50	31.57	34.46	25.28	26.98	72.50	91.65	43.79	6.64	99.50	65.12	58.78	49.88
KV Cache Size = 64
StreamingLLM	25.75	21.75	31.22	49.09	42.11	23.98	17.29	20.99	16.04	38.50	82.81	34.50	7.50	99.50	54.27	48.14	38.34
SnapKV	27.37	24.99	41.77	54.27	45.27	27.52	16.75	21.73	16.32	39.00	86.32	36.58	7.50	98.50	55.57	48.07	40.47
PyramidKV	24.25	22.87	41.03	53.07	43.55	26.36	16.46	21.52	15.61	38.50	81.95	36.68	7.50	99.50	54.40	47.20	39.40

Appendix

Table 13: LongBench evaluation results for Llama3.1-8B

	Single-Document QA			Multi-Document QA			Summarization			Few-shot Learning			Synthetic		Code		Avg.
	NrtQA	Qasper	MF-en	HotpotQA	2WikiMQA	Musique	GovReport	QMSum	MultiNews	TREC	TriviaQA	SAMSum	PCount	Pre	Lcc	RB-P	Avg.
FullKV	26.04	47.76	53.33	59.23	43.37	36.05	33.66	24.05	24.79	71.50	90.21	44.43	2.00	100.00	69.39	65.57	49.46
KV Cache Size = 64
StreamingLLM	16.62	25.37	25.56	40.57	33.95	18.48	13.97	18.98	12.40	39.50	80.65	35.10	1.50	69.00	59.25	53.13	34.00
SnapKV	15.87	28.01	35.93	41.97	33.93	21.23	14.24	19.01	12.20	42.00	80.85	32.86	3.50	69.00	58.23	51.93	35.05
PyramidKV	15.45	27.42	36.67	42.79	34.00	19.69	14.67	18.95	12.89	43.00	80.62	33.89	2.00	71.50	58.34	52.26	35.26

Appendix

Table 14: LongBench evaluation results for Qwen3-8B

		Theoretical Cost				Empirical Cost
Context Length	Method	Compute (TFLOPs)	Memory Traffic (GB)	TTFT (ms)	TTFT Overhead (ms)	TTFT (ms)	TTFT Overhead (ms)
$4$ K	Forward Pass Only	$60.00$	$13.00$	$113.37$	N/A	$130.08$	N/A
	LookaheadKV	$60.47$	$13.05$	$114.30$	$0.9164$	$141.46$	$11.3800$
	SnapKV	$60.00$	$13.02$	$113.38$	$0.0085$	$143.22$	$13.1400$
	SpecKV	$69.79$	$76.94$	$165.48$	$52.1031$	$222.50$	$92.4200$
	LAQ	$60.94$	$443.52$	$347.19$	$233.8089$	$636.66$	$506.5800$

Appendix

Table 15: Theoretical and Practical Analysis across various context lengths and methods.

Parameters	Values
Optimizer	Adam
$\beta_{1},\beta_{2}$	$0.9,0.95$
Effective Batch Size	$32$
Drop-out ( $p$ )	$0.0$
Max Sequence Length	$16384$ (prompt length) + $512$ (response length)
Train Iters	$7600$

Appendix

Table 16: Training hyperparameters.

KV-Cache Eviction

Key-value (KV) cache eviction reduces language-model memory use by removing or compressing cached tokens while preserving useful context.

See all 98 papers

May 25, 2026

IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference

Large Language Models (LLMs) are increasingly expected to operate over long contexts, yet standard softmax attention incurs a KV cache that grows linearly with sequence length, quickly becoming the bottleneck for long context inference. A practical remedy is to evict less important KV entries; however, existing eviction policies are largely heuristic and struggle to capture the rich, input-dependent distribution of token importance. In this work, we introduce a learnable indexer that predicts KV importance, enabling more accurate retention of critical tokens. Meanwhile, naively evicting tokens permanently discards their information, leading to irreversible forgetting and degraded retrieval over long ranges. To address this, we propose a lightweight latent memory module that compresses evicted tokens into a compact, online-updated state and provides residual readouts to compensate for the attention contributions lost through KV eviction. Collectively, our method enables accurate long-context inference under a bounded KV budget, delivering consistent improvements on RULER (4K/16K) across Qwen, Mistral, and Llama models (up to 25 points under aggressive eviction), markedly more stable Needle-in-a-Haystack retrieval, and superior LongBench scores and compression curves compared to existing eviction policies.

Xintong Yang, Hao Gu, Binxing Xu +6

The Hong Kong University of Science and Technology · Zhejiang University

Memory-Augmented Language Models Self-Attention KV Caching Long-Context Language Model Inference KV-Cache Eviction LLM Inference Acceleration

Ask PDF

Aug 9, 2026

DistillCache: KL-Guided Adaptive KV-Cache Eviction for Memory-Efficient LLM Inference

Transformer-based large language models (LLMs) achieve strong performance across many tasks, but their Key-Value (KV) cache grows linearly with sequence length, creating a severe memory bottleneck for long-context inference. Existing heuristic eviction methods (e.g., H

_2

O and SnapKV) rely on static attention or positional signals that often fail to capture a token's future predictive influence. We propose DistillCache, a reinforcement learning framework that formulates KV-cache eviction as a sequential decision problem. DistillCache learns a lightweight policy network using rich internal model signals (attention statistics, value norms, entropy, and position) and trains it with REINFORCE via a per-step KL-divergence reward to preserve the full-cache output distribution. On a 7B-parameter instruction-tuned Transformer (Mistral-7B-Instruct-v0.3), DistillCache retains 94.2% of full-cache accuracy on LongBench at a 25% cache budget, outperforming both strong heuristic baselines (H

_2

O, SnapKV) by up to 2.7 absolute points and, under our re-implementations, concurrent RL-based methods (ForesightKV, RLKV) by up to 1.4 points on long-context tasks. On reasoning benchmarks, DistillCache is competitive with the best concurrent method and surpasses it under aggressive compression. It also delivers up to 2.1x full-cache throughput while maintaining competitive practical efficiency. These results highlight the effectiveness of learned, distribution-aware policies for memory-efficient long-context LLM inference.

Asaad Althoubi

Oklahoma State University Stillwater, OK USA

KV Caching Long-Context Language Model Inference KV-Cache Eviction KV-Cache Management Memory-Efficient Inference LLM Inference Acceleration

Ask PDF

May 8, 2026

Reformulating KV Cache Eviction Problem for Long-Context LLM Inference

Large language models (LLMs) support long-context inference but suffer from substantial memory and runtime overhead due to Key-Value (KV) Cache growth. Existing KV Cache eviction methods primarily rely on local attention weights, neglecting the influence of value representations, output projection, and inter-head interactions. In this work, we reformulate KV Cache eviction from a conventional head-wise, weight-averaging approach into an output-aware, layer-wise matrix multiplication approximation problem. We introduce LaProx, a novel eviction strategy that explicitly models the multiplicative interaction between attention maps and projected value states to accurately quantify token contributions while accounting for inter-head dependencies. Building on this metric, we propose the first unified eviction strategy that assigns globally comparable importance scores to tokens, enabling model-wide selection instead of local, head-wise decisions. Experimental results across 19 datasets on long-context benchmarks LongBench and Needle-In-A-Haystack demonstrate that our approach maintains model performance with only 5% of the KV cache and consistently outperforms prior works across all configurations. Notably, our method achieves up to 2

\times

accuracy loss reduction under extreme compression scenarios compared to existing state-of-the-art baselines with minimal overhead.

Tho Mai, Joo-Young Kim

KAIST Daejeon, South Korea

KV Caching Long-Context Language Model Inference KV-Cache Eviction KV-Cache Compression

Ask PDF

Abstract

KV-Cache Eviction

Explore similar work

IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference

DistillCache: KL-Guided Adaptive KV-Cache Eviction for Memory-Efficient LLM Inference

Reformulating KV Cache Eviction Problem for Long-Context LLM Inference