EchoPress: Query-Agnostic KV Cache Pruning via Virtual Context Reconstruction

Jiawei Lin; Saibo Geng; Thomas Bourgeat

cs.LGSep 30, 2026

EchoPress: Query-Agnostic KV Cache Pruning via Virtual Context Reconstruction

Authors: Jiawei Lin, Saibo Geng, Thomas Bourgeat

Organizations: EPFL

Abstract

KV cache pruning reduces long-context inference memory usage by evicting less important key-value pairs. KVzip estimates importance through context reconstruction: prompting a model to repeat the context chunk by chunk. This achieves strong compression quality at the cost of additional forward passes. Learned approximations reduce this cost but require model-specific training. We analyze how KVzip identifies important cached information and show how to approximate its reconstruction scores using information already computed during prefill. These findings motivate EchoPress, a training-free method that approximates reconstruction attention using queries and keys from standard prefill. For each request, it reconstructs only the first chunk to calibrate importance scores for the remaining context. Experiments on LongBench and RULER with Qwen3-8B and Llama-3.1-8B-Instruct show that EchoPress matches KVzip in task accuracy across eviction ratios from 50% to 90%, while reducing compression overhead by a factor of 1.7-19.6 and total prefill time by a factor of up to 2.9. Code is available at https://github.com/ljwljwljwljw/kvpress/tree/echo-press.

Figures & tables

Figure 1: (a) Qwen3-8B RULER-4K accuracy (13-task mean) for query-agnostic methods from the KVPress leaderboard ( Hsieh et al., 2024 ; Jégou et al., 2024 ) , with our EchoPress results overlaid. RestoreKV(+), KVzap, and Fast KVzip use model-specific training ( Baek et al., 2026 ; Jégou and Jeblick, 2026 ; Kim et al., 2026a ) . EchoPress requires neither training nor external calibration data. (b) Qwen3-8B compression overhead (additional time beyond plain prefill, including cache compaction) on one 80 GB A100 GPU at 50% eviction. The KVzip baseline uses its official CUDA implementation ( Kim et al., 2025 ) .

Figure 2: EchoPress scoring pipeline and virtual reconstruction. (a) KVzip reconstructs every chunk. EchoPress reconstructs only the first, reuses prefill states to score later chunks, and calibrates their scores using the first chunk for each layer and KV head before global pruning (dashed cells). (b) Virtual scoring combines \scriptsize1⃝ full visibility of teal cached keys in the target chunk, \scriptsize2⃝ RoPE-shifted queries, and \scriptsize3⃝ a causal purple virtual copy. Shared rotations cancel in the copy, allowing reuse of

QK^{\top}

. Prompt R determines the offset but is omitted from scoring. For each query, a joint softmax normalizes attention over sink, cached, and causally visible copy keys. Each cached key’s raw score is the largest attention probability it receives across query positions and query heads sharing its KV head. The probabilities shown are illustrative; the full attention matrix and repeated cache are never materialized.

Figure 3: Virtual-score scale distortion across layers and KV heads. For each context and cell, we divide the exact first-chunk score quantile at $q=0.9$ by the corresponding virtual-score quantile. Each heatmap shows the median ratio over eight LongBench contexts.

Figure 4: Prefill-to-reconstruction query differences using eight LongBench contexts per model. Thin lines show individual contexts; thick lines show their mean. Left: the first chunk versus the average of later full chunks. Right: fixed second-chunk tokens and positions with varying preceding context, isolating the effect of full-context conditioning. The vertical axis is the mean of one minus the cosine similarity between matched prefill and reconstruction queries, so lower values indicate closer states.

Figure 5: Benchmark averages versus KV eviction ratio

r

(higher is better). The top row shows LongBench and RULER-4K; the bottom row shows RULER-32K/64K with 20 examples per task. Dashed lines show uncompressed scores. Additional results are in Appendix A .

Figure 6: Total prefill time for Qwen3-8B at 128K tokens and a target of 50% eviction on one 80 GB A100 GPU. KVzip and EchoPress use KVzip’s official codebase with FlashAttention-2; KVzap-MLP and Fast KVzip use KVPress/SDPA. Each bar stacks its plain-prefill median and compression overhead, including physical compaction. KVzap-MLP uses a fixed retained budget.

Figure 7: Ablation of the virtual scorer, adding one component at a time. Each bar reports within-head rank agreement (Spearman correlation) with exact reconstruction scores, computed as the mean over eight contexts of the within-context median across layer–KV-head correlations.

Scoring and calibration	RULER Avg.	NIAH (8)
Exact first chunk, no calibration	71.0	72.4
Per-request, per-layer/head exact $\rightarrow$ virtual	50.8	43.8
Per-request, global virtual $\rightarrow$ exact	77.3	83.5
EchoPress: per-request, per-layer/head virtual $\rightarrow$ exact	87.6	95.7
KVzip	87.5	92.5

Table 1: First-chunk calibration ablation on Qwen3-8B using a fixed 25% RULER-4K subset at

r=0.9

. Avg. weights all 13 tasks equally. NIAH weights the eight needle-retrieval tasks equally.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Qwen3-8B
Method	$r$	Single QA	Multi QA	Summ.	Few- shot	Retr.	Code	Avg.
Uncompressed	0	42.4	49.2	29.1	40.7	91.4	64.5	47.5
KVzip	0.5	42.4	48.4	29.2	46.4	94.0	64.7	48.5
EchoPress	0.5	42.4	48.7	29.1	44.0	95.0	63.4	48.2
KVzip	0.75	41.7	49.0	28.7	56.5	92.0	63.1	49.8
EchoPress	0.75	42.1	50.0	29.0	56.6	93.6	54.0	49.6

Appendix

Table 2: LongBench category means. Avg. weights all 12 tasks equally. Retr. denotes passage retrieval and Code denotes LCC. Bold marks the better compressed method at each

r

and displayed precision unless tied.

Qwen3-8B
Method	$r$	NIAH (8)	VT	CWE	FWE	QA (2)	Avg.
Uncompressed	0	100.0	100.0	98.9	95.4	72.9	95.4
KVzip	0.5	100.0	100.0	99.0	95.9	71.4	95.2
EchoPress	0.5	100.0	100.0	99.1	95.7	71.7	95.2
KVzip	0.75	99.9	100.0	98.9	96.4	70.8	95.1
EchoPress	0.75	99.8	100.0	98.0	95.9	70.5	94.9

Appendix

Table 3: RULER-4K group means. NIAH denotes eight needle-retrieval tasks, VT variable tracking, CWE/FWE common/frequent-word extraction, and QA the two question-answering tasks. Avg. weights all 13 tasks equally.

Question answering
Method	$r$	Qasper	MultiFQA	NrtvQA	HotpotQA	2WikiMQA	MuSiQue
Uncompressed	0	44.8	55.6	26.7	62.9	49.2	35.4
KVzip	0.5	44.5	55.8	26.8	62.8	47.8	34.7
EchoPress	0.5	44.6	56.4	26.1	62.9	48.8	34.5
KVzip	0.75	42.6	55.0	27.6	63.8	47.8	35.3
EchoPress	0.75	43.4	54.4	28.4	63.7	48.9	37.3

Appendix

Table 4: LongBench per-task scores on Qwen3-8B.

Question answering
Method	$r$	Qasper	MultiFQA	NrtvQA	HotpotQA	2WikiMQA	MuSiQue
Uncompressed	0	47.1	55.3	28.9	59.5	51.8	32.7
KVzip	0.5	47.3	56.8	29.1	55.8	48.2	28.6
EchoPress	0.5	47.3	57.0	28.7	57.4	49.7	29.9
KVzip	0.75	44.6	54.8	27.9	56.1	46.4	27.4
EchoPress	0.75	46.0	55.6	29.7	54.2	46.3	28.0

Appendix

Table 5: LongBench per-task scores on Llama-3.1-8B-Instruct.

Needle retrieval
Method	$r$	S1	S2	S3	MK1	MK2	MK3	MQ	MV
Uncompressed	0	100.0	100.0	100.0	100.0	100.0	100.0	99.9	100.0
KVzip	0.5	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0
EchoPress	0.5	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0
KVzip	0.75	100.0	100.0	100.0	99.8	100.0	99.8	100.0	99.9
EchoPress	0.75	100.0	100.0	100.0	99.6	100.0	99.2	100.0	100.0

Appendix

Table 6: RULER-4K per-task accuracy on Qwen3-8B. S1–S3 and MK1–MK3 are single-needle and multi-key variants. MQ/MV are multi-query/value retrieval.

Needle retrieval
Method	$r$	S1	S2	S3	MK1	MK2	MK3	MQ	MV
Uncompressed	0	100.0	100.0	99.8	99.8	100.0	99.8	99.9	99.9
KVzip	0.5	99.8	100.0	100.0	100.0	100.0	99.8	100.0	100.0
EchoPress	0.5	99.8	100.0	100.0	100.0	100.0	99.8	100.0	100.0
KVzip	0.75	100.0	100.0	99.8	100.0	100.0	100.0	99.9	99.9
EchoPress	0.75	100.0	100.0	100.0	100.0	100.0	99.8	100.0	100.0

Appendix

Table 7: RULER-4K per-task accuracy on Llama-3.1-8B-Instruct.

Budget	$r$	Full	KVzip	EchoPress	$\Delta$	Full	KVzip	EchoPress	$\Delta$
		Qwen3-8B				Llama-3.1-8B
32K	0.75	87.88	86.98	86.19	$-0.79$	86.38	87.47	86.31	$-1.17$
32K	0.9	87.88	73.99	71.83	$-2.15$	86.38	74.12	76.65	$+2.53$
64K	0.75	77.72	77.75	78.08	$+0.33$	85.22	81.13	83.85	$+2.72$
64K	0.9	77.72	69.11	69.87	$+0.76$	85.22	65.79	65.94	$+0.15$

Appendix

Table 8: RULER accuracy at 32K and 64K, using 20 examples per task. Full denotes the uncompressed baseline.

\Delta

is EchoPress minus KVzip in percentage points.

Setting	Configuration
Data	NVIDIA/RULER v1 ( c3f5e3b4f87f ), base template, seed 42, model-specific tokenizers; no truncation.
Lengths	Budgets of 32,768 / 65,536 tokens include output allowances; actual prefill prefixes span 28,057–32,718 / 60,224–65,478 tokens.
Runtime	bf16, SDPA, batch size one; PyTorch 2.8.0 (CUDA 12.8), transformers 5.2.0, Triton 3.4.0.
Positions	Qwen3: fixed YaRN factor four [ Peng et al., 2024 ] , original context 32,768, checkpoint RoPE base. Llama: released RoPE configuration. Both use a maximum position limit of 131,072.
Decoding	Greedy with official per-task output limits; Qwen thinking disabled.
Compression	The 2,048-token accuracy configuration, global KV-pair allocation, and compression before the question. Question and generated tokens remain uncompressed. EchoPress uses exact first-chunk scores and per-request, per-layer/KV-head calibration, without an offline map.

Appendix

Table 9: Settings for the RULER-32K/64K evaluation.

Setting	Accuracy	Timing
Engine / attention	KVPress / SDPA	KVzip / FlashAttention-2
Chunk budget	First chunk: 2,048 tokens including prompt; later chunks: up to 2,048	2,000 context tokens plus repeat prompt
Prompts / prefix	KVPress chat/prefix settings	KVzip prompts and prefix/sinks
Context coverage	RULER 4K/32K/64K; LongBench $\leq$ 32K	4K–128K, excluding protected prefix
Output	Task scores	Compacted cache; no generation

Appendix

Table 10: Accuracy and timing settings for KVzip and EchoPress. Both use bf16 and batch size one; timing uses one 80 GB A100 at 50% eviction.

Qwen3-8B
Context	Plain 16K-chunk	Plain Full K	Plain Full E	KVzip 16K-chunk	KVzip full prefill	EchoPress	Speedup
4K	0.36	0.36	0.36	0.77	0.78	0.46	1.7 $\times$
8K	0.73	0.73	0.72	1.63	1.62	0.57	2.9 $\times$
16K	1.67	1.63	1.61	3.68	3.68	0.79	4.7 $\times$
32K	4.11	3.96	3.91	9.03	9.04	1.23	7.3 $\times$
64K	11.23	10.90	10.78	24.53	24.47	2.14	11.5 $\times$

Appendix

Table 11: Plain-prefill baselines and compression overhead in seconds. Speedup is KVzip 16K-chunk overhead divided by EchoPress overhead. “Full K” and “Full E” are the respective sweeps’ full-prefill baselines.

	Qwen3-8B			Llama-3.1-8B
Context	KVzip 16K-chunk	KVzip full	EchoPress	KVzip 16K-chunk	KVzip full	EchoPress
4K	1.13	1.14	0.81	1.01	1.01	0.74
8K	2.36	2.36	1.29	2.09	2.09	1.17
16K	5.34	5.31	2.39	4.73	4.69	2.17
32K	13.14	12.99	5.15	11.60	11.47	4.64
64K	35.76	35.37	12.92	31.61	31.35	11.57

Appendix

Table 12: Median total prefill time (seconds), including compression and physical compaction, across five repetitions.

Figure 8: Compression-time breakdown at 128K on one 80 GB A100. For EchoPress, scoring includes exact first-chunk reconstruction, virtual scoring, and calibration. Overhead subtracts the matching plain-prefill baseline.

Setting	Configuration
Input / prefill	Saved concatenated LongBench gov_report contexts; three-token chat prefix; 131,075 total tokens; one full-context prefill.
Software	PyTorch 2.8.0 (CUDA 12.8), transformers 5.2.0, Triton 3.4.0; SDPA.
RoPE / precision	Fixed YaRN factor four, original size 32,768, original RoPE base. bf16 model/KVzap weights; released mixed bf16/fp32 Fast KVzip gates.
KVzap-MLP	nvidia/KVzap-mlp-Qwen3-8B , revision bd5c59178466 . Replace fixed threshold with global fixed-budget selection; protect the final 128 tokens.
Fast KVzip	Jang-Hyun/Fast-KVzip , revision 67ce1265f510 , q4_dim16_sink16.pt gates. Global allocation; promote first four and last 4,096 token scores. KVPress full prefill replaces the released implementation’s eviction between prefill chunks.
Repetitions	Five seeded, interleaved repetitions after one warm-up per method; plain-prefill baseline in each repetition; CUDA synchronized at both boundaries.

Appendix

Table 13: Settings for the KVPress implementations of the learned methods.

Measurement	Qwen3-8B	Llama-3.1-8B
Query difference, first $\to$ later	$0.154{}\to 0.141{}$	$0.114{}\to 0.099{}$
Query difference, prefix $0\to$ Full	$0.158{}\to 0.143{}$	$0.117{}\to 0.105{}$
Mean within-chunk Spearman $\rho$	$0.678{}\to 0.692{}$	$0.735{}\to 0.745{}$
Median within-chunk Spearman $\rho$	$0.717{}\to 0.718{}$	$0.772{}\to 0.770{}$

Appendix

Table 14: First-chunk diagnostics, equally weighting contexts after averaging later full chunks within each context; partial final chunks are excluded. Arrows compare first with later chunks, except the prefix control (

0\to

Full). The median row replaces only the within-chunk mean over layers and KV heads.

Explore similar work

Aug 4, 2026cs.AI

TaskPress: Query-Agnostic KV Cache Compression via Task-Guided Pruning

Long-context inference with large language models is constrained by the linear growth of the key-value cache to sequence length. While pruning offers mitigation, prevailing methods determine query-specific token importance that cannot be reused across unseen queries. In contrast, we introduce TaskPress, a framework for task-guided, query-agnostic KV cache eviction. Instead of optimizing the cache for a single query, TaskPress constructs a reusable memory representation conditioned on a high-level task guide. The guide functions as a meta-query during prefill to filter irrelevant tokens before downstream queries are issued. In addition, TaskPress leverages quantization scale factors as a zero-cost signal for detecting influential representation outliers, providing an efficient proxy for token importance. Experiments on conducted on various tasks with long context input demonstrate that TaskPress efficiently creates a compact, reusable cache across diverse queries.

Wonpyo Park, Seung-won Hwang

Seoul National University · Google

Key-Value Cache Compression Key-Value Cache Eviction Cache Efficient Long-Context Inference Token-Level Uncertainty

Ask PDF

Oct 5, 2026cs.CL

Behavior-Preserving KV Cache Compression

KV caches are a major bottleneck in long-context inference and long-form generation with large language models. Existing training-free eviction policies largely rely on proxy importance signals, such as attention mass, to decide which past tokens to retain. We argue that cache compression should instead preserve the predictive behavior of the full-cache model, retaining entries whose removal would substantially change the model's output distribution. We propose Behavior-Preserving KV Cache Compression, a training-free framework that scores candidate evictions by estimating the compressed-cache logits induced by their removal and evaluating the resulting KL to the full-cache next-token distribution. Using pre-eviction forward statistics, the method avoids running separate masked forward passes for each candidate. Across diverse architectures and both prefill-time and generation-time compression, our method delivers substantial gains in downstream task quality over lightweight attention-based heuristics at matched retained-KV budgets, with the largest gains under aggressive compression. It achieves these gains with additional compression-time computation while retaining an end-to-end speedup over full-cache inference in our evaluated settings.

Doo Hwan Hwang, Junyoung Jang, Junho Na +2

Kim Jaechul Graduate School of AI, KAIST, Daejeon, Republic of Korea · KT Corporation, Seoul, Republic of Korea

Key-Value Cache Compression

Ask PDF

Aug 3, 2026cs.LG

AnchorKV: Anchor-Residual KV Cache Compression

The key-value (KV) cache is the primary memory bottleneck in long-context LLM inference. Existing approaches attack it from opposite ends: eviction methods permanently discard tokens, degrading performance whenever a discarded token later proves essential, while quantization methods retain all tokens at low precision but offer limited compression. We propose AnchorKV, a compression scheme that shrinks the cache by

20\times

without discarding a single token. AnchorKV represents the cache using a small set of anchors stored exactly, expresses every other token through its most similar anchor, and refines only those whose approximation most affects the model's output. AnchorKV consistently preserves accuracy across models and datasets, retaining 99% of the full-cache score at the 70B scale, while keeping the entire context at a fraction of its cost.

Malik Khalaf, Yara Shamshoum, Nitzan Hodos +2

Department of Computer Science, Technion - Israel Institute of Technology

Key-Value Cache Compression Key-Value Cache Cache Large Language Model Compression LLM Inference Optimization Data Compression Methods+1

Ask PDF