LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation

Jinwoo Ahn; Ingyu Seong; Akhil Kedia; Junhan Kim; Hyemi Jang; Kangwook Lee; Yongkweon Jeon

cs.LGMar 11, 2026

LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation

Authors: Jinwoo Ahn, Ingyu Seong, Akhil Kedia, Junhan Kim, Hyemi Jang, Kangwook Lee, Yongkweon Jeon

Organizations: Samsung Research

Abstract

Transformer-based large language models (LLMs) rely on key-value (KV) caching to avoid redundant computation during autoregressive inference. While this mechanism greatly improves efficiency, the cache size grows linearly with the input sequence length, quickly becoming a bottleneck for long-context tasks. Existing solutions mitigate this problem by evicting prompt KV that are deemed unimportant, guided by estimated importance scores. Notably, a recent line of work proposes to improve eviction quality by "glimpsing into the future", in which a draft generator produces a surrogate future response approximating the target model's true response, and this surrogate is subsequently used to estimate the importance of cached KV more accurately. However, these approaches rely on computationally expensive draft generation, which introduces substantial prefilling overhead and limits their practicality in real-world deployment. To address this challenge, we propose LookaheadKV, a lightweight eviction framework that leverages the strength of surrogate future response without requiring explicit draft generation. LookaheadKV augments transformer layers with parameter-efficient modules trained to predict true importance scores with high accuracy. Our design ensures negligible runtime overhead comparable to existing inexpensive heuristics, while achieving accuracy superior to more costly approximation methods. Extensive experiments on long-context understanding benchmarks, across a wide range of models, demonstrate that our method not only outperforms recent competitive baselines in various long-context understanding tasks, but also reduces the eviction cost by up to 14.5x, leading to significantly faster time-to-first-token. Our code is available at https://github.com/SamsungLabs/LookaheadKV.

Figures & tables

Figure 1: An overview of LookaheadKV (a) Training. During training, lookahead tokens and lookahead LoRA are trained to predict the ground-truth importance scores obtained with pre-generated model response via a KL divergence loss. (b) Inference. During prefill, LookaheadKV utilizes the learned modules to identify essential tokens and compress the KV cache, facilitating memory-efficient decoding.

Figure 2: Accuracy-overhead trade-off across KV cache eviction methods.

Panel of Figure 3 — Figure 3: Time-to-First-Token (TTFT) latency overhead ratio across context lengths. Similar to SnapKV, LookaheadKV introduces negligible TTFT overhead across all tested context lengths; draft-based methods (LAQ, SpecKV) incur substantial latency, especially for shorter contexts.

Model	Budget	PyramidKV	SnapKV	StreamingLLM	SpecKV	LAQ	LookaheadKV
	FullKV score: 5.72
LLaMA3.2-1B	64	4.64	4.70	4.54	N/A	5.03	5.21
LLaMA3.2-1B	128	5.10	5.39	4.94	N/A	5.45	5.60
	256	5.49	5.67	5.37	N/A	5.64	5.62
	FullKV score: 7.35
LLaMA3.2-3B	64	6.30	6.28	5.96	6.52	6.48	6.87

Table 2: MT‑Bench evaluation results. Bold and underlined values indicate best and second best scores, respectively.

Figure 5: HTML-to-TSV evaluation results using LLaMA3.1-8B.

		Theoretical Cost				Empirical Cost
Context Length	Method	Compute (TFLOPs)	Memory Traffic (GB)	TTFT (ms)	TTFT Overhead (ms)	TTFT (ms)	TTFT Overhead (ms)
$8$ K	Forward Pass Only	$136.00$	$13.00$	$256.99$	N/A	$290.81$	N/A
	LookaheadKV	$136.53$	$13.05$	$258.03$	$1.0345$	$301.69$	$10.8800$
	SnapKV	$136.00$	$13.02$	$257.00$	$0.0085$	$310.98$	$20.1700$
	SpecKV	$159.12$	$81.07$	$336.52$	$79.5264$	$411.32$	$120.5100$
	LAQ	$137.06$	$444.52$	$491.58$	$234.5877$	$800.19$	$509.3800$

Table 3: Theoretical and empirical cost analysis of LLaMA3.1-8B at

C=128

Method	FullKV	SnapKV	SpecKV	LAQ	LookaheadKV
Greedy	49.88	43.50	45.45	46.61	47.72
$T=0.2$	49.58 (-0.60%)	43.29 (-0.48%)	44.99 (-1.01%)	46.73 (+0.26%)	47.75 (+0.06%)
$T=0.8$	47.82 (-4.13%)	41.39 (-4.85%)	43.43 (-4.44%)	45.27 (-2.87%)	45.81 (-4.00%)

Table 4: Average LongBench performance at different temperature settings on LLaMA3.1-8B, with

C=128

. LookaheadKV outperforms baselines across all tested temperature settings.

	$n_{\text{lookahead}}=4$		$n_{\text{lookahead}}=8$		$n_{\text{lookahead}}=16$		$n_{\text{lookahead}}=32$		$n_{\text{lookahead}}=64$		$n_{\text{lookahead}}=128$
Module	score	overhead(%)	score	overhead(%)	score	overhead(%)	score	overhead(%)	score	overhead(%)	score	overhead(%)
emb-only	$25.53$	$3.36$	$25.67$	$3.77$	$26.36$	$3.42$	$26.40$	$4.18$	$25.8$	$7.3$	$26.2$	$10.7$
QV	$26.45$	$3.65$	$26.40$	$4.08$	$26.87$	$3.99$	$26.85$	$4.42$	$26.7$	$7.7$	$27.1$	$10.7$
all	$26.60$	$4.20$	$27.02$	$4.20$	$27.01$	$4.67$	$27.12$	$4.96$	$27.1$	$8.5$	$27.0$	$11.0$

Table 5: 2D ablation across lookahead sizes and trainable modules, on LLaMA3.2-1B. Average LongBench scores with cache budget of 64 and TTFT overhead are reported.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Context Length	FullKV	LookaheadKV	SnapKV	SpecKV	LAQ
64K	$84.15$	69.45	$39.64$	$64.02$	$64.10$
128K	$73.72$	54.83	$30.56$	$52.62$	$50.67$

Appendix

Table 6: RULER evaluation results on longer context lengths using Llama3.1-8B-Instruct at

C=128

FullKV	LookaheadKV	LookaheadKV
		+ Suffix Window
32.01	29.10	28.52 (-1.99%)

Appendix

Table 7: Average LongBench scores using LookaheadKV window only and LookaheadKV + SnapKV-style suffix window, evaluated using LLaMA3.2-1B-Instruct with

C=64

Generation Method	$T=0.2$	$T=0.4$	$T=0.6$	$T=0.8$	LLaMA3.2-1B
Recall@512 (%)	$95.06$	$93.73$	$91.40$	$91.37$	$88.66$
Kendall’s Tau	$91.44$	$88.63$	$84.61$	$84.79$	$80.05$

Appendix

Table 8: Importance score similarity with stochastic response using various temperatures vs. greedy response on LLaMA3.1-8B. LLaMA3.2-1B presents the similarity of importance scores using greedy response generated with LLaMA3.2-1B vs. LLaMA3.1-8B.

	Single-Document QA			Multi-Document QA			Summarization			Few-shot Learning			Synthetic		Code		Avg.
	NrtQA	Qasper	MF-en	HotpotQA	2WikiMQA	Musique	GovReport	QMSum	MultiNews	TREC	TriviaQA	SAMSum	PCount	Pre	Lcc	RB-P	Avg.
FullKV	19.24	15.96	42.47	35.53	29.42	19.87	28.34	22.18	25.64	64.00	80.85	38.83	3.00	4.78	38.63	43.35	32.01
KV Cache Size = 64
StreamingLLM	14.42	12.30	24.02	24.04	24.05	9.35	13.56	19.23	13.61	35.50	68.94	29.09	4.00	3.71	37.61	37.33	23.17
SnapKV	14.24	12.18	30.53	27.30	25.66	12.44	14.27	19.27	12.37	35.00	73.57	29.29	2.00	4.49	38.01	36.84	24.22
PyramidKV	13.33	11.36	26.05	25.34	24.01	10.57	13.64	19.49	11.96	35.00	69.95	27.75	1.50	4.33	35.25	36.01	22.85

Appendix

Table 9: LongBench evaluation results for Llama3.2-1B

	Single-Document QA			Multi-Document QA			Summarization			Few-shot Learning			Synthetic		Code		Avg.
	NrtQA	Qasper	MF-en	HotpotQA	2WikiMQA	Musique	GovReport	QMSum	MultiNews	TREC	TriviaQA	SAMSum	PCount	Pre	Lcc	RB-P	Avg.
FullKV	18.94	24.78	46.15	39.15	33.11	19.03	30.33	23.06	25.18	74.00	85.21	42.87	0.00	94.50	46.82	40.24	40.21
KV Cache Size = 64
StreamingLLM	12.25	16.83	20.38	22.62	24.61	7.26	11.43	18.96	12.12	39.00	66.48	35.14	0.00	12.50	42.95	36.90	23.71
SnapKV	13.01	17.50	30.47	28.14	25.68	8.63	11.89	19.77	11.03	37.50	78.59	37.62	0.00	54.50	44.42	36.92	28.48
PyramidKV	13.20	16.31	27.19	23.83	25.80	6.95	11.23	19.12	10.72	37.00	73.99	35.99	0.00	43.00	43.01	35.97	26.46

Appendix

Table 10: LongBench evaluation results for Qwen3-1.7B

	Single-Document QA			Multi-Document QA			Summarization			Few-shot Learning			Synthetic		Code		Avg.
	NrtQA	Qasper	MF-en	HotpotQA	2WikiMQA	Musique	GovReport	QMSum	MultiNews	TREC	TriviaQA	SAMSum	PCount	Pre	Lcc	RB-P	Avg.
FullKV	25.35	40.77	50.31	53.79	40.05	25.62	33.11	24.38	25.95	72.00	88.89	43.25	4.00	97.50	54.65	56.83	46.03
KV Cache Size = 64
StreamingLLM	19.79	20.68	24.68	42.63	34.09	16.14	16.53	20.22	14.78	38.00	75.84	33.19	4.50	10.50	46.11	43.76	28.84
SnapKV	21.88	20.56	39.12	48.33	35.39	22.39	16.37	20.30	14.41	39.50	82.17	35.05	4.50	21.50	47.96	44.30	32.11
PyramidKV	23.06	19.20	36.26	46.34	34.81	20.17	16.17	20.96	14.02	41.00	81.93	35.08	4.00	23.00	46.82	43.94	31.67

Appendix

Table 11: LongBench evaluation results for Llama3.2-3B

	Single-Document QA			Multi-Document QA			Summarization			Few-shot Learning			Synthetic		Code		Avg.
	NrtQA	Qasper	MF-en	HotpotQA	2WikiMQA	Musique	GovReport	QMSum	MultiNews	TREC	TriviaQA	SAMSum	PCount	Pre	Lcc	RB-P	Avg.
FullKV	27.45	43.30	54.45	55.63	43.43	31.61	32.24	24.61	25.00	73.00	88.76	43.65	0.75	96.50	64.29	61.39	47.88
KV Cache Size = 64
StreamingLLM	12.46	23.96	25.93	38.56	33.40	19.47	13.74	19.71	13.04	39.50	75.48	34.33	0.50	64.50	51.42	48.46	32.15
SnapKV	15.28	25.03	31.61	40.00	34.95	18.83	12.88	19.78	12.49	40.50	75.62	33.69	1.00	69.00	51.48	47.38	33.10
PyramidKV	15.50	24.84	34.33	40.70	35.07	19.39	13.48	19.85	13.03	41.50	76.69	33.95	1.50	73.00	52.98	47.77	33.97

Appendix

Table 12: LongBench evaluation results for Qwen3-4B

	Single-Document QA			Multi-Document QA			Summarization			Few-shot Learning			Synthetic		Code		Avg.
	NrtQA	Qasper	MF-en	HotpotQA	2WikiMQA	Musique	GovReport	QMSum	MultiNews	TREC	TriviaQA	SAMSum	PCount	Pre	Lcc	RB-P	Avg.
FullKV	31.63	46.66	56.93	58.10	48.50	31.57	34.46	25.28	26.98	72.50	91.65	43.79	6.64	99.50	65.12	58.78	49.88
KV Cache Size = 64
StreamingLLM	25.75	21.75	31.22	49.09	42.11	23.98	17.29	20.99	16.04	38.50	82.81	34.50	7.50	99.50	54.27	48.14	38.34
SnapKV	27.37	24.99	41.77	54.27	45.27	27.52	16.75	21.73	16.32	39.00	86.32	36.58	7.50	98.50	55.57	48.07	40.47
PyramidKV	24.25	22.87	41.03	53.07	43.55	26.36	16.46	21.52	15.61	38.50	81.95	36.68	7.50	99.50	54.40	47.20	39.40

Appendix

Table 13: LongBench evaluation results for Llama3.1-8B

	Single-Document QA			Multi-Document QA			Summarization			Few-shot Learning			Synthetic		Code		Avg.
	NrtQA	Qasper	MF-en	HotpotQA	2WikiMQA	Musique	GovReport	QMSum	MultiNews	TREC	TriviaQA	SAMSum	PCount	Pre	Lcc	RB-P	Avg.
FullKV	26.04	47.76	53.33	59.23	43.37	36.05	33.66	24.05	24.79	71.50	90.21	44.43	2.00	100.00	69.39	65.57	49.46
KV Cache Size = 64
StreamingLLM	16.62	25.37	25.56	40.57	33.95	18.48	13.97	18.98	12.40	39.50	80.65	35.10	1.50	69.00	59.25	53.13	34.00
SnapKV	15.87	28.01	35.93	41.97	33.93	21.23	14.24	19.01	12.20	42.00	80.85	32.86	3.50	69.00	58.23	51.93	35.05
PyramidKV	15.45	27.42	36.67	42.79	34.00	19.69	14.67	18.95	12.89	43.00	80.62	33.89	2.00	71.50	58.34	52.26	35.26

Appendix

Table 14: LongBench evaluation results for Qwen3-8B

		Theoretical Cost				Empirical Cost
Context Length	Method	Compute (TFLOPs)	Memory Traffic (GB)	TTFT (ms)	TTFT Overhead (ms)	TTFT (ms)	TTFT Overhead (ms)
$4$ K	Forward Pass Only	$60.00$	$13.00$	$113.37$	N/A	$130.08$	N/A
	LookaheadKV	$60.47$	$13.05$	$114.30$	$0.9164$	$141.46$	$11.3800$
	SnapKV	$60.00$	$13.02$	$113.38$	$0.0085$	$143.22$	$13.1400$
	SpecKV	$69.79$	$76.94$	$165.48$	$52.1031$	$222.50$	$92.4200$
	LAQ	$60.94$	$443.52$	$347.19$	$233.8089$	$636.66$	$506.5800$

Appendix

Table 15: Theoretical and Practical Analysis across various context lengths and methods.

Parameters	Values
Optimizer	Adam
$\beta_{1},\beta_{2}$	$0.9,0.95$
Effective Batch Size	$32$
Drop-out ( $p$ )	$0.0$
Max Sequence Length	$16384$ (prompt length) + $512$ (response length)
Train Iters	$7600$

Appendix

Table 16: Training hyperparameters.

KV-Cache Eviction

Key-value (KV) cache eviction reduces language-model memory use by removing or compressing cached tokens while preserving useful context.

See all 98 papers

IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference
May 25, 2026Xintong Yang, Hao Gu, Binxing Xu +6Memory-Augmented Language Models Self-Attention
The Hong Kong University of Science and Technology · Zhejiang University
DistillCache: KL-Guided Adaptive KV-Cache Eviction for Memory-Efficient LLM Inference
Aug 9, 2026Asaad AlthoubiKV Caching Long-Context Language Model Inference
Oklahoma State University Stillwater, OK USA
Reformulating KV Cache Eviction Problem for Long-Context LLM Inference
May 8, 2026Tho Mai, Joo-Young KimKV Caching Long-Context Language Model Inference
KAIST Daejeon, South Korea

May 25, 2026J/K move · Enter open · S save

IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference

Xintong Yang, Hao Gu, Binxing Xu, Lujun Li+5

The Hong Kong University of Science and Technology · Zhejiang University

Memory-Augmented Language Models Self-Attention KV Caching Long-Context Language Model Inference KV-Cache Eviction LLM Inference Acceleration

Open paper AskPDF arXiv

Large Language Models (LLMs) are increasingly expected to operate over long contexts, yet standard softmax attention incurs a KV cache that grows linearly with sequence length, quickly becoming the bottleneck for long context inference. A practical remedy is to evict less important KV entries; however, existing eviction policies are largely heuristic and struggle to capture the rich, input-dependent distribution of token importance. In this work, we introduce a learnable indexer that predicts KV importance, enabling more accurate retention of critical tokens. Meanwhile, naively evicting tokens permanently discards their information, leading to irreversible forgetting and degraded retrieval over long ranges. To address this, we propose a lightweight latent memory module that compresses evicted tokens into a compact, online-updated state and provides residual readouts to compensate for the attention contributions lost through KV eviction. Collectively, our method enables accurate long-context inference under a bounded KV budget, delivering consistent improvements on RULER (4K/16K) across Qwen, Mistral, and Llama models (up to 25 points under aggressive eviction), markedly more stable Needle-in-a-Haystack retrieval, and superior LongBench scores and compression curves compared to existing eviction policies.

Abstract

KV-Cache Eviction

Explore similar work

IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference

DistillCache: KL-Guided Adaptive KV-Cache Eviction for Memory-Efficient LLM Inference

Reformulating KV Cache Eviction Problem for Long-Context LLM Inference