The Key-Value (KV) cache is crucial for efficient Large Language Models (LLMs) inference, but excessively long contexts drastically increase KV cache memory footprint. Existing KV cache compression methods typically rely on input-side attention patterns within a prompt observation window to estimate token importance during the prefill stage. They fail to preserve critical tokens for future generation since these assessments are not derived from the decoding process. Intuitively, an effective observation window should mirror the decoding-stage queries to accurately reflect which tokens the generation process will attend to. However, ground-truth decoding queries are inherently unavailable during inference. For constructing pseudo queries to approximate them, we find that positional information plays a more critical role than semantic content. Motivated by this insight, we propose decoding-aligned KV cache compression via position-aware pseudo queries (DapQ), a novel and lightweight eviction framework that leverages position-aware pseudo queries to simulate the output tokens, thereby establishing an effective observation window for importance assessment. It aligns closely with the actual generation context and enables precise token eviction. Extensive evaluations across multiple benchmarks and LLMs demonstrate that DapQ achieves superior performance, particularly under strict memory constraints (e.g., up to nearly lossless performance 99.5% on NIAH with 3% KV cache budgets).
Figures & tables
Figure 1: An overview of DapQ. A synthetic pseudo context (length N ) is appended to the original context (length Lp ), forming an extended sequence of length Lp+N . The model processes this sequence during the prefill phase and then obtains pseudo queries for the synthetic tokens, which are endowed with the correct positional encodings of the first N decoding steps. These pseudo queries compute attention scores with all keys from the original prompt, establishing the token importance distribution. The topK tokens are retained in the compressed KV cache, while the others, along with all the synthetic tokens are evicted. Autoregressive decoding then begins from position Lp .
Experiment
Content Similarity
Positional Similarity
Post ROPE
Pre ROPE
SC & SP
Same ( “The report discusses the Federal……Airport Improvement Program (AIP). The program” )
Same (4424,4425,4426……4453,4454,4455)
1.0000
1.0000
DC & SP
Different ( “Sorry, I don’t know. Sorry, I don’t know. Sorry, I don’t know. Sorry, I don’t know. Sorry, I” )
Same (4424,4425,4426……4453,4454,4455)
0.7238
0.7238
SC & DP
Same ( “The report discusses the Federal……Airport Improvement Program (AIP). The program” )
Different (0,1,2……29,30,31)
0.3522
0.7913
DC & DP
Different ( “Sorry, I don’t know. Sorry, I don’t know. Sorry, I don’t know. Sorry, I don’t know. Sorry, I” )
Different (0,1,2……29,30,31)
0.3267
0.7434
Table 1: Query similarity comparison under different content and position conditions. Post ROPE denotes similarity after ROPE has been applied to query vectors. Pre ROPE indicates similarity measured before ROPE application.
Figure 2: Analysis of Positional Dominance and Offset Sensitivity in Query Similarity. We set the pseudo queries of fixed length 32. (a) Boxplot of query similarity distributions for a 4k context under different content and position conditions, each aggregated from 100 independent trials. DC: pseudo-queries content is constructed by randomly sampling 32 tokens from the model’s vocabulary; DP: pseudo-queries positions are assigned by randomly sampling a consecutive span of 32 index positions from the context length range [0, m] (e.g., [0, 4000]). (b) Query similarity curves over offset positions for contexts of lengths 2k, 4k, 6k, and 8k. The x-axis denotes the starting position assigned to pseudo queries (e.g., an x-axis value of 3500 corresponds to position IDs 3500∼3531 ).
Figure 3: Recall Performance of different methods across various Window Sizes.
Single-Document QA
Multi-Document QA
Summarization
Few-shot Learning
Synthetic
Code
Methods
Qasper
MF-en
HotpotQA
2WikiMQA
GovReport
MultiNews
TREC
TriviaQA
SAMSum
PCount
PRe
Lcc
RB-P
Avg.
FullKV
37.72
40.64
50.17
34.88
31.03
25.64
70.00
89.85
40.55
13.30
83.67
58.96
52.71
48.39
KV Cache Size = 256
Llama3-8B-Instruct
H2O
28.11
36.63
48.62
31.50
21.87
21.44
45.67
89.49
38.28
12.11
83.67
61.49
53.36
44.02
PyramidKV
30.88
38.11
50.20
33.88
22.54
21.84
60.00
89.26
37.07
12.78
83.67
61.34
52.51
45.70
SnapKV
30.84
38.39
49.75
33.80
22.18
21.53
57.00
89.65
36.97
12.11
84.00
61.78
54.92
45.61
Table 2: Main Results on LongBench: performance comparison of different KV cache compression methods
Method
Batch Size
1
10
20
30
40
50
FullKV
11.59
26.43
25.60
26.59
OOM
OOM
LaCache
11.44
35.77
40.64
42.49
OOM
OOM
SLM
10.81
34.49
38.21
39.25
39.98
40.46
H2O
10.56
34.34
37.71
39.02
39.62
40.01
PyramidKV
10.54
34.12
38.00
39.03
39.86
40.11
Table 3: Comparison of throughput (tokens/s) with different batch sizes.
Method
Context Length
8K
16K
32K
64K
128K
FullKV
1.1106
2.5607
6.5718
18.9441
60.8399
LaCache
1.1283
2.5921
6.6356
19.0519
61.5180
SLM
1.1236
2.5801
6.6008
18.9853
61.0339
H2O
1.1348
2.6046
6.6352
19.0379
61.5029
PyramidKV
1.1253
2.5958
6.6337
19.0462
61.5345
Table 4: Comparison of Time-to-First-Token (TTFT (s)) with different context lengths.
Figure 4: Ablation analysis of pseudo queries with respect to quality, length and insert positions. (a) Performance under a fixed Qpseudo length 32 , with varying semantic content of Qpseudo : Fm_Ln is the concatenation of the first m and the last n tokens from the input context; RS_C is constructed by concatenating 32 randomly sampled individual tokens from the context; RC_C is a randomly sampled consecutive span of 32 tokens from the context; and Fix_C is a fixed, repetitive nonsensical sequence (e.g., “Sorry, I don’t know. Sorry, I don’t know…” ). (b) Performance under varying observation window sizes N . (c) Performance under varying insert positions of Qpseudo . Left Parts: we divide the original context interval [0,Lp) equally into three segments: part1 (beginning), part2 (middle), and part3 (end). For each segment, we randomly select a continuous sequence of 32 positions and map the position IDs of Qpseudo to it. Right Parts: we only move backwards the position IDs of Qpseudo to continuous 32 positions at different offsets after the context Lp (e.g., [Lp+0,Lp+32) ), [Lp+32,Lp+64) ).
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Experiment
Content Sim.
Positional Sim.
Post ROPE
Pre ROPE
SC & SP
Same
Same
1.0000
1.0000
DC & SP
Different
Same
0.7142
0.7142
SC & DP
Same
Different
0.4270
0.7341
DC & DP
Different
Different
0.4113
0.7224
Appendix
Table 5: Query cosine similarity comparison on the LCC dataset using Qwen2.5-7B-Instruct.
Experiment
Content Sim.
Positional Sim.
Post ROPE
Pre ROPE
SC & SP
Same
Same
1.0000
1.0000
DC & SP
Different
Same
0.7236
0.7236
SC & DP
Same
Different
0.4671
0.7359
DC & DP
Different
Different
0.4428
0.7192
Appendix
Table 6: Query cosine similarity comparison on the 2WikiMQA dataset using Qwen2.5-7B-Instruct.
Method
Batch Size
1
10
20
30
40
50
FullKV
16.87
33.74
52.49
71.24
OOM
OOM
LaCache
16.87
33.74
52.49
71.24
OOM
OOM
SLM
16.01
25.17
35.36
45.55
55.73
65.92
H2O
16.01
25.17
35.36
45.55
55.73
65.92
PyramidKV
16.01
25.23
35.47
45.72
55.96
66.20
Appendix
Table 7: Comparison of memory usage (GB) with different batch sizes.
Window Size
128
64
32
16
8
4
2
1
DapQ
0.9785
0.9747
0.9726
0.9713
0.9705
0.9644
0.9615
0.8768
SnapKV
0.9463
0.9429
0.9508
0.9466
0.9416
0.9385
0.9332
0.8426
Appendix
Table 8: Cosine similarity of softmax-normalized attention weights for DapQ vs. SnapKV.
Single-Document QA
Multi-Document QA
Summarization
Few-shot Learning
Synthetic
Code
Methods
Qasper
MF-en
HotpotQA
2WikiMQA
GovReport
MultiNews
TREC
TriviaQA
SAMSum
PCount
PRe
Lcc
RB-P
Avg.
FullKV
37.72
40.64
50.17
34.88
31.03
25.64
70.00
89.85
40.55
13.30
83.67
58.96
52.71
48.39
KV Cache Size = 256
Llama3-8B-Instruct
H2O
28.11
36.63
48.62
31.50
21.87
21.44
45.67
89.49
38.28
12.11
83.67
61.49
53.36
44.02
PyramidKV
30.88
38.11
50.20
33.88
22.54
21.84
60.00
89.26
37.07
12.78
83.67
61.34
52.51
45.70
SnapKV
30.84
38.39
49.75
33.80
22.18
21.53
57.00
89.65
36.97
12.11
84.00
61.78
54.92
45.61
Appendix
Table 9: Performance comparison of different methods across various LLMs on LongBench.
Difficulty
Length
LLMs
Methods
Easy
Hard
Short
Medium
Long
Overall
Llama3-8B Instruct
FullKV
28.65
26.37
32.22
24.19
25.00
27.24
KV Cache Size = 128
H2O
32.29
25.40
32.22
26.98
23.15
28.03
PyramidKV
30.21
24.44
31.67
26.05
19.44
26.64
SnapKV
30.73
25.72
32.22
26.05
23.15
27.63
Appendix
Table 10: Performance comparison of different methods across various LLMs on LongBenchv2. For DapQ, the pseudo queries are constructed via prefix-suffix concatenation: using the first 8 and last 24 tokens for LLaMA series models, and the first 2 and last 30 tokens for Qwen models. Notably, several compressed methods surpass the FullKV baseline. We attribute this phenomenon to the noise reduction mechanism of cache eviction. By selectively retaining critical tokens, these methods effectively reduce noise and sparsify the context, potentially leading to more focused and effective model reasoning. This effect is particularly pronounced in long-context benchmarks.
Single NIAH
Multi-key NIAH
LLM
Methods
S-NIAH-1
S-NIAH-2
S-NIAH-3
MK-NIAH-1
MK-NIAH-2
MK-NIAH-3
MQ-NIAH
MV-NIAH
CWE
FWE
VT
AVG
Llama3-8B-Instruct
FullKV
100
100
100
99.2
91.8
95.8
99.75
97.5
97.82
82.93
98.32
96.65
KV Cache Size = 2048
LaCache
21
27.4
1.8
29.6
29
3.2
12.2
6.45
86.08
86.07
11.04
28.53
SLM
26.6
25.4
25.4
22.6
24.2
17.8
24.1
24.3
4.9
78.6
21.16
26.82
H2O
100
90
14
78
47
12.6
74.7
45
90.04
79.33
97.68
66.21
Appendix
Table 11: Performance comparison of different methods across various kv cache size on Ruler for llama3-8B-Instruct.
Single NIAH
Multi-key NIAH
LLM
Methods
S-NIAH-1
S-NIAH-2
S-NIAH-3
MK-NIAH-1
MK-NIAH-2
MK-NIAH-3
MQ-NIAH
MV-NIAH
CWE
FWE
VT
AVG
Qwen2.5-7B-Instruct
FullKV
100
99.8
99.8
99.8
98
93.2
99.8
93.9
77.38
87.67
95.36
94.97
KV Cache Size = 4096
LaCache
3
1.8
2.4
4
4.8
3
3.3
2.25
62.78
87.73
6.32
16.49
SLM
26.4
28
27
24.4
19.6
11.6
26.1
27.1
36.96
91.53
22.84
31.05
H2O
100
98.4
24
96.8
19.4
9.4
85.75
68.15
67.56
92.33
93.92
68.70
Appendix
Table 12: Performance comparison of different methods across various kv cache size on Ruler for Qwen2.5-7B-Instruct.
Single NIAH
Multi-key NIAH
LLM
Methods
S-NIAH-1
S-NIAH-2
S-NIAH-3
MK-NIAH-1
MK-NIAH-2
MK-NIAH-3
MQ-NIAH
MV-NIAH
CWE
FWE
VT
AVG
Qwen3-8B
FullKV
100
100
100
99.6
99.6
99.6
99.9
99.75
83.98
90.67
100
97.55
KV Cache Size = 4096
LaCache
17.6
14.6
13.2
16.8
20.2
7.4
14.75
6.2
62.46
68.47
12.48
23.11
SLM
26.6
28
27
24.4
19.6
18.4
26.15
27.1
36.62
93
22.96
31.80
H2O
100
99.8
22
99.8
84.8
32.4
99.5
89.9
46.74
93
99.92
78.90
Appendix
Table 13: Performance comparison of different methods across various kv cache size on Ruler for Qwen3-8B.
ICL
LONGQA
RAG
exact_match
f1
rougeL_f1
substring_exact_match
Methods
icl_banking77
icl_clinic150
icl_nlu
icl_trec_coarse
icl_trec_fine
narrativeqa
infbench_qa_eng
kilt_hotpotqa
kilt_nq
kilt_popqa_3
kilt_triviaqa
Avg.
Llama3-8B-Instruct
FullKV
38.60
73.60
76.20
39.40
25.00
12.00
16.56
52.00
42.17
47.67
80.00
45.75
KV Cache Size = 1024
H2O
32.00
64.00
70.60
36.00
21.20
11.30
15.28
47.67
44.17
47.67
81.50
42.85
PyramidKV
24.80
55.40
68.60
29.60
16.00
11.61
16.00
52.33
44.17
48.17
81.33
40.73
Appendix
Table 14: Performance comparison of different methods across various LLMs on sub-task categories of the HELMET benchmark.
LLM
KV Cache Size
Method
Acc
Llama3-8B-Instruct
FullKV
100.00
256
H2O
66.81
PyramidKV
93.94
SnapKV
90.97
DapQ
99.46
128
H2O
50.92
Appendix
Table 15: Performance comparison of different methods across various LLMs on Needle-in-a-Haystack.
Figure 5: Visualization of Needle-in-a-Haystack results. We take LLaMA-3-8B-Instruct (8k context, 256 KV size) as a representative example to demonstrate the performance differences. The vertical axis represents the depth percentage, and the horizontal axis represents the token length.
Long-context large language models (LLMs) have demonstrated strong capabilities across a wide range of tasks, but the growing KV cache introduces substantial memory and inference overhead. Existing one-shot KV cache compression methods typically commit to irreversible eviction immediately after prefill, before any signal from actual generation becomes available. Our quantitative analysis shows that early queries from the actual generation stage provide attention signals that are more consistent with subsequent decode attention, with the largest single-step gain occurring at the prefill-decode boundary. Based on this observation, we propose DeferKV, which moves the eviction decision from the end of prefill to the first real decoding step and temporally combines prompt-side and decode-side observations, thereby better aligning KV importance estimation with subsequent generation requirements. DeferKV requires no additional training, draft model, or future-query prediction module, making it simple and easy to deploy. Experiments on LongBench, RULER, and Needle-in-a-Haystack demonstrate that DeferKV consistently improves model performance under KV cache compression while maintaining low inference latency.
Zhe Wang, Jiakai Li, Yujia Sun +2
University of Electronic Science and Technology of China, Chengdu, China · Ubiquitous Intelligence and Trusted Services Key Laboratory of Sichuan Province, Chengdu, China
Key-Value (KV) cache remains a major bottleneck for deploying Large Language Models (LLMs) in long-generation tasks. Prior work often applies uniform compression across both prefill and decoding caches, but compressing the prefill cache degrades performance by corrupting critical context. While preserving the prefill cache is essential, decoding-phase compression remains underexplored, with existing methods relying on rigid recency windows or instantaneous attention. Our analysis of attention dynamics reveals strong temporal patterns: critical tokens receive sustained attention over long horizons, while local reasoning involves short-lived bursts. Static heuristics fail to capture this behavior, leading to premature eviction of important tokens or retention of stale ones. We propose Moment-KV, a decoding-time KV cache compression method based on momentum-driven temporal attention aggregation. Our method models token importance as a continuously evolving state, where attention is aggregated with decay, capturing both long-term influence and recent relevance. Experiments show that Moment-KV significantly improves generation fidelity in long-generation tasks (2.3-3.2 %) while maintaining decoding latency.
Efficient key-value (KV) cache management is crucial for the practical deployment of large language models (LLMs), yet existing compression techniques often incur a trade-off between performance degradation and computational overhead. We propose a novel gating-based KV cache eviction method for frozen-weight LLMs that achieves high compression ratios with negligible computational cost. Our approach introduces lightweight sink-attention gating modules to identify and retain critical KV pairs, and integrates seamlessly into both the prefill and decoding stages. The proposed gate training algorithm relies on forward passes of an LLM, avoiding expensive backpropagation, while achieving strong task generalization through a task-agnostic reconstruction objective. Extensive experiments across the Qwen2.5-1M, Qwen3, and Gemma3 families show that our method maintains near-lossless performance while evicting up to 70% of the KV cache. The results are consistent across a wide range of tasks, including long-context understanding, code comprehension, and mathematical reasoning, demonstrating the generality of our approach.