Where Matters More Than What: Decoding-aligned KV Cache Compression via Position-aware Pseudo Queries

Zhenxu Tian; Yi Su; Juntao Li; Min Zhang

cs.CLMar 12, 2026

Where Matters More Than What: Decoding-aligned KV Cache Compression via Position-aware Pseudo Queries

Authors: Zhenxu Tian, Yi Su, Juntao Li, Min Zhang

Organizations: School of Computer Science and Technology, Soochow University, China

Abstract

The Key-Value (KV) cache is crucial for efficient Large Language Models (LLMs) inference, but excessively long contexts drastically increase KV cache memory footprint. Existing KV cache compression methods typically rely on input-side attention patterns within a prompt observation window to estimate token importance during the prefill stage. They fail to preserve critical tokens for future generation since these assessments are not derived from the decoding process. Intuitively, an effective observation window should mirror the decoding-stage queries to accurately reflect which tokens the generation process will attend to. However, ground-truth decoding queries are inherently unavailable during inference. For constructing pseudo queries to approximate them, we find that positional information plays a more critical role than semantic content. Motivated by this insight, we propose decoding-aligned KV cache compression via position-aware pseudo queries (DapQ), a novel and lightweight eviction framework that leverages position-aware pseudo queries to simulate the output tokens, thereby establishing an effective observation window for importance assessment. It aligns closely with the actual generation context and enables precise token eviction. Extensive evaluations across multiple benchmarks and LLMs demonstrate that DapQ achieves superior performance, particularly under strict memory constraints (e.g., up to nearly lossless performance 99.5% on NIAH with 3% KV cache budgets).

Figures & tables

Figure 1: An overview of DapQ. A synthetic pseudo context (length $N$ ) is appended to the original context (length $L_{p}$ ), forming an extended sequence of length $L_{p}{+}N$ . The model processes this sequence during the prefill phase and then obtains pseudo queries for the synthetic tokens, which are endowed with the correct positional encodings of the first $N$ decoding steps. These pseudo queries compute attention scores with all keys from the original prompt, establishing the token importance distribution. The $topK$ tokens are retained in the compressed KV cache, while the others, along with all the synthetic tokens are evicted. Autoregressive decoding then begins from position $L_{p}$ .

Experiment	Content Similarity	Positional Similarity	Post ROPE	Pre ROPE
SC & SP	Same ( “The report discusses the Federal……Airport Improvement Program (AIP). The program” )	Same (4424,4425,4426……4453,4454,4455)	1.0000	1.0000
DC & SP	Different ( “Sorry, I don’t know. Sorry, I don’t know. Sorry, I don’t know. Sorry, I don’t know. Sorry, I” )	Same (4424,4425,4426……4453,4454,4455)	0.7238	0.7238
SC & DP	Same ( “The report discusses the Federal……Airport Improvement Program (AIP). The program” )	Different (0,1,2……29,30,31)	0.3522	0.7913
DC & DP	Different ( “Sorry, I don’t know. Sorry, I don’t know. Sorry, I don’t know. Sorry, I don’t know. Sorry, I” )	Different (0,1,2……29,30,31)	0.3267	0.7434

Table 1: Query similarity comparison under different content and position conditions. Post ROPE denotes similarity after ROPE has been applied to query vectors. Pre ROPE indicates similarity measured before ROPE application.

Panel of Figure 2 — Figure 2: Analysis of Positional Dominance and Offset Sensitivity in Query Similarity. We set the pseudo queries of fixed length 32. (a) Boxplot of query similarity distributions for a 4k context under different content and position conditions, each aggregated from 100 independent trials. DC: pseudo-queries content is constructed by randomly sampling 32 tokens from the model’s vocabulary; DP: pseudo-queries positions are assigned by randomly sampling a consecutive span of 32 index positions from the context length range [0, m] (e.g., [0, 4000]). (b) Query similarity curves over offset positions for contexts of lengths 2k, 4k, 6k, and 8k. The x-axis denotes the starting position assigned to pseudo queries (e.g., an x-axis value of 3500 corresponds to position IDs $3500\sim 3531$ ).

Figure 3: Recall Performance of different methods across various Window Sizes.

		Single-Document QA		Multi-Document QA		Summarization		Few-shot Learning			Synthetic		Code
	Methods	Qasper	MF-en	HotpotQA	2WikiMQA	GovReport	MultiNews	TREC	TriviaQA	SAMSum	PCount	PRe	Lcc	RB-P	Avg.
	FullKV	37.72	40.64	50.17	34.88	31.03	25.64	70.00	89.85	40.55	13.30	83.67	58.96	52.71	48.39
	KV Cache Size = 256
Llama3-8B-Instruct	H2O	28.11	36.63	48.62	31.50	21.87	21.44	45.67	89.49	38.28	12.11	83.67	61.49	53.36	44.02
	PyramidKV	30.88	38.11	50.20	33.88	22.54	21.84	60.00	89.26	37.07	12.78	83.67	61.34	52.51	45.70
	SnapKV	30.84	38.39	49.75	33.80	22.18	21.53	57.00	89.65	36.97	12.11	84.00	61.78	54.92	45.61

Table 2: Main Results on LongBench: performance comparison of different KV cache compression methods

Method	Batch Size
Method	1	10	20	30	40	50
FullKV	11.59	26.43	25.60	26.59	OOM	OOM
LaCache	11.44	35.77	40.64	42.49	OOM	OOM
SLM	10.81	34.49	38.21	39.25	39.98	40.46
H2O	10.56	34.34	37.71	39.02	39.62	40.01
PyramidKV	10.54	34.12	38.00	39.03	39.86	40.11

Table 3: Comparison of throughput (tokens/s) with different batch sizes.

Method	Context Length
Method	8K	16K	32K	64K	128K
FullKV	1.1106	2.5607	6.5718	18.9441	60.8399
LaCache	1.1283	2.5921	6.6356	19.0519	61.5180
SLM	1.1236	2.5801	6.6008	18.9853	61.0339
H2O	1.1348	2.6046	6.6352	19.0379	61.5029
PyramidKV	1.1253	2.5958	6.6337	19.0462	61.5345

Table 4: Comparison of Time-to-First-Token (TTFT (s)) with different context lengths.

Panel of Figure 4 — Figure 4: Ablation analysis of pseudo queries with respect to quality, length and insert positions. (a) Performance under a fixed $Q_{\text{pseudo}}$ length $32$ , with varying semantic content of $Q_{\text{pseudo}}$ : Fm_Ln is the concatenation of the first m and the last n tokens from the input context; RS_C is constructed by concatenating 32 randomly sampled individual tokens from the context; RC_C is a randomly sampled consecutive span of 32 tokens from the context; and Fix_C is a fixed, repetitive nonsensical sequence (e.g., “Sorry, I don’t know. Sorry, I don’t know…” ). (b) Performance under varying observation window sizes $N$ . (c) Performance under varying insert positions of $Q_{\text{pseudo}}$ . Left Parts: we divide the original context interval $[0,L_{p})$ equally into three segments: part1 (beginning), part2 (middle), and part3 (end). For each segment, we randomly select a continuous sequence of 32 positions and map the position IDs of $Q_{\text{pseudo}}$ to it. Right Parts: we only move backwards the position IDs of $Q_{\text{pseudo}}$ to continuous 32 positions at different offsets after the context $L_{p}$ (e.g., $[L_{p}+0,L_{p}+32)$ ), $[L_{p}+32,L_{p}+64)$ ).

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Experiment	Content Sim.	Positional Sim.	Post ROPE	Pre ROPE
SC & SP	Same	Same	1.0000	1.0000
DC & SP	Different	Same	0.7142	0.7142
SC & DP	Same	Different	0.4270	0.7341
DC & DP	Different	Different	0.4113	0.7224

Appendix

Table 5: Query cosine similarity comparison on the LCC dataset using Qwen2.5-7B-Instruct.

Experiment	Content Sim.	Positional Sim.	Post ROPE	Pre ROPE
SC & SP	Same	Same	1.0000	1.0000
DC & SP	Different	Same	0.7236	0.7236
SC & DP	Same	Different	0.4671	0.7359
DC & DP	Different	Different	0.4428	0.7192

Appendix

Table 6: Query cosine similarity comparison on the 2WikiMQA dataset using Qwen2.5-7B-Instruct.

Method	Batch Size
Method	1	10	20	30	40	50
FullKV	16.87	33.74	52.49	71.24	OOM	OOM
LaCache	16.87	33.74	52.49	71.24	OOM	OOM
SLM	16.01	25.17	35.36	45.55	55.73	65.92
H2O	16.01	25.17	35.36	45.55	55.73	65.92
PyramidKV	16.01	25.23	35.47	45.72	55.96	66.20

Appendix

Table 7: Comparison of memory usage (GB) with different batch sizes.

Window Size	128	64	32	16	8	4	2	1
DapQ	0.9785	0.9747	0.9726	0.9713	0.9705	0.9644	0.9615	0.8768
SnapKV	0.9463	0.9429	0.9508	0.9466	0.9416	0.9385	0.9332	0.8426

Appendix

Table 8: Cosine similarity of softmax-normalized attention weights for DapQ vs. SnapKV.

		Single-Document QA		Multi-Document QA		Summarization		Few-shot Learning			Synthetic		Code
	Methods	Qasper	MF-en	HotpotQA	2WikiMQA	GovReport	MultiNews	TREC	TriviaQA	SAMSum	PCount	PRe	Lcc	RB-P	Avg.
	FullKV	37.72	40.64	50.17	34.88	31.03	25.64	70.00	89.85	40.55	13.30	83.67	58.96	52.71	48.39
	KV Cache Size = 256
Llama3-8B-Instruct	H2O	28.11	36.63	48.62	31.50	21.87	21.44	45.67	89.49	38.28	12.11	83.67	61.49	53.36	44.02
	PyramidKV	30.88	38.11	50.20	33.88	22.54	21.84	60.00	89.26	37.07	12.78	83.67	61.34	52.51	45.70
	SnapKV	30.84	38.39	49.75	33.80	22.18	21.53	57.00	89.65	36.97	12.11	84.00	61.78	54.92	45.61

Appendix

Table 9: Performance comparison of different methods across various LLMs on LongBench.

		Difficulty		Length
LLMs	Methods	Easy	Hard	Short	Medium	Long	Overall
Llama3-8B Instruct	FullKV	28.65	26.37	32.22	24.19	25.00	27.24
	KV Cache Size = 128
	H2O	32.29	25.40	32.22	26.98	23.15	28.03
	PyramidKV	30.21	24.44	31.67	26.05	19.44	26.64
	SnapKV	30.73	25.72	32.22	26.05	23.15	27.63

Appendix

Table 10: Performance comparison of different methods across various LLMs on LongBenchv2. For DapQ, the pseudo queries are constructed via prefix-suffix concatenation: using the first 8 and last 24 tokens for LLaMA series models, and the first 2 and last 30 tokens for Qwen models. Notably, several compressed methods surpass the FullKV baseline. We attribute this phenomenon to the noise reduction mechanism of cache eviction. By selectively retaining critical tokens, these methods effectively reduce noise and sparsify the context, potentially leading to more focused and effective model reasoning. This effect is particularly pronounced in long-context benchmarks.

		Single NIAH			Multi-key NIAH
LLM	Methods	S-NIAH-1	S-NIAH-2	S-NIAH-3	MK-NIAH-1	MK-NIAH-2	MK-NIAH-3	MQ-NIAH	MV-NIAH	CWE	FWE	VT	AVG
Llama3-8B-Instruct	FullKV	100	100	100	99.2	91.8	95.8	99.75	97.5	97.82	82.93	98.32	96.65
	KV Cache Size = 2048
	LaCache	21	27.4	1.8	29.6	29	3.2	12.2	6.45	86.08	86.07	11.04	28.53
	SLM	26.6	25.4	25.4	22.6	24.2	17.8	24.1	24.3	4.9	78.6	21.16	26.82
	H2O	100	90	14	78	47	12.6	74.7	45	90.04	79.33	97.68	66.21

Appendix

Table 11: Performance comparison of different methods across various kv cache size on Ruler for llama3-8B-Instruct.

		Single NIAH			Multi-key NIAH
LLM	Methods	S-NIAH-1	S-NIAH-2	S-NIAH-3	MK-NIAH-1	MK-NIAH-2	MK-NIAH-3	MQ-NIAH	MV-NIAH	CWE	FWE	VT	AVG
Qwen2.5-7B-Instruct	FullKV	100	99.8	99.8	99.8	98	93.2	99.8	93.9	77.38	87.67	95.36	94.97
	KV Cache Size = 4096
	LaCache	3	1.8	2.4	4	4.8	3	3.3	2.25	62.78	87.73	6.32	16.49
	SLM	26.4	28	27	24.4	19.6	11.6	26.1	27.1	36.96	91.53	22.84	31.05
	H2O	100	98.4	24	96.8	19.4	9.4	85.75	68.15	67.56	92.33	93.92	68.70

Appendix

Table 12: Performance comparison of different methods across various kv cache size on Ruler for Qwen2.5-7B-Instruct.

		Single NIAH			Multi-key NIAH
LLM	Methods	S-NIAH-1	S-NIAH-2	S-NIAH-3	MK-NIAH-1	MK-NIAH-2	MK-NIAH-3	MQ-NIAH	MV-NIAH	CWE	FWE	VT	AVG
Qwen3-8B	FullKV	100	100	100	99.6	99.6	99.6	99.9	99.75	83.98	90.67	100	97.55
	KV Cache Size = 4096
	LaCache	17.6	14.6	13.2	16.8	20.2	7.4	14.75	6.2	62.46	68.47	12.48	23.11
	SLM	26.6	28	27	24.4	19.6	18.4	26.15	27.1	36.62	93	22.96	31.80
	H2O	100	99.8	22	99.8	84.8	32.4	99.5	89.9	46.74	93	99.92	78.90

Appendix

Table 13: Performance comparison of different methods across various kv cache size on Ruler for Qwen3-8B.

		ICL					LONGQA		RAG
		exact_match					f1	rougeL_f1	substring_exact_match
	Methods	icl_banking77	icl_clinic150	icl_nlu	icl_trec_coarse	icl_trec_fine	narrativeqa	infbench_qa_eng	kilt_hotpotqa	kilt_nq	kilt_popqa_3	kilt_triviaqa	Avg.
Llama3-8B-Instruct	FullKV	38.60	73.60	76.20	39.40	25.00	12.00	16.56	52.00	42.17	47.67	80.00	45.75
	KV Cache Size = 1024
	H2O	32.00	64.00	70.60	36.00	21.20	11.30	15.28	47.67	44.17	47.67	81.50	42.85
	PyramidKV	24.80	55.40	68.60	29.60	16.00	11.61	16.00	52.33	44.17	48.17	81.33	40.73

Appendix

Table 14: Performance comparison of different methods across various LLMs on sub-task categories of the HELMET benchmark.

LLM	KV Cache Size	Method	Acc
Llama3-8B-Instruct		FullKV	100.00
	256	H2O	66.81
		PyramidKV	93.94
		SnapKV	90.97
		DapQ	99.46
	128	H2O	50.92

Appendix

Table 15: Performance comparison of different methods across various LLMs on Needle-in-a-Haystack.

Panel of Figure 5 — Figure 5: Visualization of Needle-in-a-Haystack results. We take LLaMA-3-8B-Instruct (8k context, 256 KV size) as a representative example to demonstrate the performance differences. The vertical axis represents the depth percentage, and the horizontal axis represents the token length.

KV-Cache Compression

Key-value cache compression reduces the memory and bandwidth needed to retain attention states during long-context inference.

See all 267 papers

DeferKV: Rethinking Eviction Timing for One-Shot KV Cache Compression
Oct 5, 2026Zhe Wang, Jiakai Li, Yujia Sun +2Long-Context Language Model Inference KV-Cache Eviction
University of Electronic Science and Technology of China, Chengdu, China · Ubiquitous Intelligence and Trusted Services Key Laboratory of Sichuan Province, Chengdu, China
Moment-KV: Momentum-Based Decode-Time KV Cache Compression for Long Generation
May 28, 2026Soumyadeep Jana, Sagar Nishad, Sanasam Ranbir SinghKV Caching Long-Context Language Modeling
Indian Institute of Technology Guwahati, India
Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction
Jan 25, 2026Jang-Hyun Kim, Dongyoon Han, Sangdoo YunKV-Cache Eviction KV-Cache Compression
NAVER AI Lab.

Oct 5, 2026J/K move · Enter open · S save

DeferKV: Rethinking Eviction Timing for One-Shot KV Cache Compression

Zhe Wang, Jiakai Li, Yujia Sun, Rongzheng Wang+1

University of Electronic Science and Technology of China, Chengdu, China · Ubiquitous Intelligence and Trusted Services Key Laboratory of Sichuan Province, Chengdu, China

Long-Context Language Model Inference KV-Cache Eviction KV-Cache Compression

Open paper AskPDF arXiv

Long-context large language models (LLMs) have demonstrated strong capabilities across a wide range of tasks, but the growing KV cache introduces substantial memory and inference overhead. Existing one-shot KV cache compression methods typically commit to irreversible eviction immediately after prefill, before any signal from actual generation becomes available. Our quantitative analysis shows that early queries from the actual generation stage provide attention signals that are more consistent with subsequent decode attention, with the largest single-step gain occurring at the prefill-decode boundary. Based on this observation, we propose DeferKV, which moves the eviction decision from the end of prefill to the first real decoding step and temporally combines prompt-side and decode-side observations, thereby better aligning KV importance estimation with subsequent generation requirements. DeferKV requires no additional training, draft model, or future-query prediction module, making it simple and easy to deploy. Experiments on LongBench, RULER, and Needle-in-a-Haystack demonstrate that DeferKV consistently improves model performance under KV cache compression while maintaining low inference latency.

Abstract

KV-Cache Compression

Explore similar work

DeferKV: Rethinking Eviction Timing for One-Shot KV Cache Compression

Moment-KV: Momentum-Based Decode-Time KV Cache Compression for Long Generation

Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction