Organizations: Institute of Automation, Chinese Academy of Sciences, Beijing, China · University of Electronic Science and Technology of China, Chengdu, China · The Chinese University of Hong Kong, Shenzhen, China · Tianjin university, Tianjin, China · University of Science and Technology Beijing, Beijing, China
Prefill-only KV compression freezes a token subset at the end of prefill and decodes from it without further eviction. The retention decision is therefore irreversible, yet existing methods estimate the corrective signals it relies on, per-head reliability and prompt-level compression sensitivity, online from a single noisy prompt. We argue this is the wrong statistical unit: these signals exhibit far higher cross-prompt regularity than within-prompt signal-to-noise. We introduce CompilerKV, a KV-retention policy whose corrective tables are compiled offline from a calibration corpus, reducing online correction after the standard observation-window scan to O(1) lookups plus a budget clamp. We find that compiled retention tables behave as portable architectural priors: rankings transfer across disjoint corpora on four backbones (mean Spearman ρˉ=0.90), and direct model-to-model table transfer costs only 0.4--0.8 LongBench points on average. At a 512-token budget, CompilerKV attains compressed-SOTA on all four backbones, improving over the strongest prefill-only baseline by +1.67 points on average (task-bootstrap 95% CI [+1.08,+2.37]). Pressure regimes amplify the gap: under a fixed 512/32k cache ratio, CompilerKV remains the strongest compressed method through 128k RULER (∼73 vs.\ FullKV ∼79, SnapKV ∼38); on 32k NIAH it reaches 0.89 vs.\ SnapKV 0.42; and at 32k input, retaining only 1.56% of the prefill KV, batch-16 serving remains feasible where FullKV is OOM.
Figures & tables
Figure 1: Empirical evidence that E1–E3 are real, distinct failure modes on calibration data. (a) The same token’s attention score has very different distributions across prompts (low-noise vs. high-noise prompt) and across heads (a scale-biased head spans a much wider range), confirming E1. (b) The aggregated distribution of compiled head-reliability weights is bimodal: a substantial cluster of heads is consistently down-weighted (noise emitters), while a smaller cluster receives weights >1.1 (retrieval-critical), confirming E2. (c) The compression damage of a fixed Top- k rule (color: NLL gap to FullKV) varies systematically with attention entropy and local perplexity, confirming E3 as a prompt-level effect that no global k can absorb.
Figure 2: Overview of the CompilerKV framework. The pipeline has three stages: (1) a stabilized utility score that suppresses transient noise; (2) a compiled Head Heterogeneity Table that governs functional differences among attention heads; and (3) a compiled Risk-Adaptive Gating Table that calibrates the retention threshold to each prompt’s complexity.
Single-Doc QA
Multi-Doc QA
Summarization
Few-Shot Learning
Synthetic & Code
Model
Method
NarrativeQA
Qasper
MF-en
HotpotQA
2WikiMQA
MuSiQue
GovReport
QMSum
MultiNews
TREC
TriviaQA
SAMSum
Lcc
RB-P
Pcount
PRe
Avg.
InternLM-2.5 7B-Chat-1M
FullKV
22.47
27.58
39.98
40.96
33.52
26.61
33.01
25.18
26.28
72.5
86.76
39.84
55.86
57.90
2.92
100.00
43.21
SnapKV
16.86
23.28
36.24
32.14
19.89
23.21
17.81
23.18
22.44
71.0
84.05
34.34
50.32
53.34
1.00
96.50
37.85
PyramidKV
17.62
21.08
37.52
32.21
21.31
22.03
19.37
24.06
22.22
73.0
83.94
34.61
50.45
49.72
1.05
95.50
37.86
DynamicKV
17.77
23.87
37.74
32.98
21.13
23.21
19.13
23.49
22.48
75.0
84.89
36.70
50.70
51.08
0.91
95.50
38.54
AdaKV
18.23
24.64
38.30
33.64
21.66
23.55
19.74
23.96
22.92
75.3
85.28
37.23
51.31
51.91
1.30
95.78
39.05
Table 1: LongBench results (512-token budget). The best compressed result is bold and the second best is underlined ; FullKV is shown as the lossless reference and is not used for compressed-method ranking. AdaKV and CAKE are included as recent head- and layer-adaptive baselines.
Figure 3: Performance vs. KV cache budget. Left: mean LongBench accuracy on LLaMA-3-8B. Right: cross-task tail score (Task-P10). All budget curves are monotone with cache size; CompilerKV is the strongest compressed method across all budgets and slightly exceeds FullKV at larger budgets.
Setting
Mistral-7B
LLaMA-3-8B
Qwen2-7B
InternLM-2.5
Ablation under E1–E3 framing
CompilerKV (Full)
42.94
42.55
41.13
42.61
w/o Stabilized Utility
40.20
39.84
38.61
38.09
w/o Head Reliability
40.76
40.31
39.58
39.07
w/o Risk-Adaptive τ
38.97
38.52
37.73
37.24
Compiler objective
Table 2: Ablation and compiler probe (LongBench avg., 512-token budget). Top: component removals cause the predicted E1–E3 regressions. Middle: offline reward regression ( α=0 ) isolates the conservative CQL gain. Bottom: joint head–risk compilation outperforms independent compilation.
Table 6
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Model
Benchmark / setup
Score
External reported results
Full-KV
LLaMA-3.1-8B-Instruct
LongBench Avg. / -
52.2
DuoAttention
LLaMA-3.1-8B-Instruct
LongBench Avg. / -512 KV
41.2
SnapKV
LLaMA-3.1-8B-Instruct
LongBench Avg. / -512 KV
45.2
Quest
LLaMA-3.1-8B-Instruct
LongBench Avg. / -512 KV
35.1
SparQ
LLaMA-3.1-8B-Instruct
LongBench Avg. / -512 KV
43.8
Appendix
Table A1 : Additional KV-compression baselines. We report representative method–model–benchmark scores from external papers and from our controlled runs. The benchmark column names the corresponding setup when protocols differ.
Figure A1 : Budget scaling across benchmark families (LLaMA-3-8B-Instruct). Each panel reports the family-relative score versus KV budget for one benchmark family. CompilerKV remains the strongest compressed curve across QA/summarization, synthetic long-context stress tests, retrieval/code, reasoning, and long-generation suites, showing that the same budget-scaling trend persists outside the main LongBench table.
Figure A2 : RULER length scaling at a fixed cache ratio (LLaMA-3-8B-Instruct). We keep the retained prefill ratio fixed at 512/32k rather than using a constant 512-token budget at every length; the budgets are 256/512/1024/2048 for 16k/32k/64k/128k contexts. CompilerKV remains the strongest compressed method across lengths, while staying below FullKV at long contexts as expected under aggressive compression.
Figure A3 : Math and reasoning robustness (LLaMA-3-8B-Instruct). Each panel reports score versus KV budget for one reasoning benchmark. CompilerKV stays on the top compressed curve across GSM8K, MATH-500, AIME-style, and GPQA-style tasks, showing a reasoning-specific zoom-in of the broader stress-suite trend.
Figure A4 : Quality-memory and speed-memory Pareto view (LLaMA-3-8B-Instruct, 32k context). Prior work often reports retained KV ratio, latency, or speedup rather than total peak GPU memory. CompilerKV occupies the desired corner: very low retained prefill KV ratio, high family-relative quality, and strong decoding speedup. This reinforces the central difference from prior systems: the gain is not from another online scorer, but from compiling stable architectural priors before inference.
Figure A5 : Needle-in-a-Haystack stress heatmaps (Mistral-7B-Instruct-v0.2). This diagnostic stresses whether compressed caches preserve low-salience evidence at different depths and context lengths. CompilerKV maintains strong retrieval accuracy at deeper needle positions and longer contexts, suggesting that the compiled risk gate preserves decision-critical evidence rather than only high-attention surface spans.
Figure A6 : Retrieval, QA, and code-comprehension stress suite (LLaMA-3-8B-Instruct). We show how each method scales with KV budget on SQuAD-long, NIAH, SCBench-QA, SCBench-Code, RepoBench-P, and LongGen. CompilerKV remains the strongest compressed method throughout the budget range and is especially robust in low-budget retrieval/code settings, which is consistent with the paper’s view that offline compilation is a different decision paradigm rather than another online token scorer.
Table A3 : Pressure-regime summary. Stress settings where the gain is no longer a small LongBench-margin effect. RULER-128k uses the fixed 512/32k retained-cache ratio from Fig. A2 ; NIAH is from Fig. A7 ; batching is from Table A7 .
Backbone
Spearman ρ
Layer-bin mean
Layer-bin min
LLaMA-3-8B
0.91
0.91
0.88
Mistral-7B
0.90
0.90
0.87
Qwen2-7B
0.90
0.89
0.86
InternLM2.5-7B
0.89
0.89
0.85
Mean
0.90
0.90
0.87
Appendix
Table A4 : Cross-corpus stability of compiled head reliability across backbones. For each backbone, we compile two independent Head Heterogeneity Tables using disjoint calibration corpora (arXiv + PubMed vs. ShareGPT + UltraChat) and report Spearman ρ between the resulting per-head reliability rankings. The stable correlations across all four architectures support the claim that the compiled table captures model-level head functionality rather than a corpus artifact.
Source table
Target model
Target-compiled
Transferred
Drop
Same-family transfer
LLaMA-3-8B
LLaMA-3.1-8B
52.60
52.30
−0.30
LLaMA-3.1-8B
LLaMA-3-8B
42.55
42.28
−0.27
LLaMA-3.1-8B
LLaMA-3.2-family
50.10
49.70
−0.40
Cross-family transfer
Mistral-7B
LLaMA-3-8B
42.55
41.85
−0.70
Appendix
Table A5 : Model-to-model transfer of compiled retention tables. We compile on a source backbone, deploy the resulting table on a target backbone without target recompilation, and compare with target-compiled CompilerKV. Same-family transfer costs 0.3 – 0.4 points; broader cross-family transfer costs 0.4 – 0.8 points on average, showing that the compiled table is a portable retention prior rather than a per-run tuning artifact.
Context
Method
Prefill KV Compression
Decoding Efficiency
Input
Output
Budget
Retained
Saving
TTFT(s) ↓
Decode rate(tok/s) ↑
Latency(s) ↓
8k
2k
FullKV
8k
100.0%
0.0%
0.66
27.63
74.79
CompilerKV
512
6.25%
93.75%
0.72
36.20 ( 1.31× )
58.90 ( 1.27× )
16k
4k
FullKV
16k
100.0%
0.0%
1.45
19.55
209.56
CompilerKV
512
3.12%
96.88%
1.52
34.80 ( 1.78× )
118.40 ( 1.77× )
32k
8k
FullKV
32k
100.0%
0.0%
3.52
11.65
706.56
Appendix
Table A6 : Main efficiency profile. Retained prefill KV ratio and decoding efficiency on a single A100-80GB. CompilerKV fixes the prefill budget to 512 tokens; generated KV states are appended normally. Speedup multipliers in parentheses are relative to FullKV at the matching context length.
Figure A7 : Needle-in-a-Haystack Pressure Test on Mistral-7B. Visual comparison of retrieval accuracy (Green=100%, Red=0%) across varying context lengths and needle depths. While baselines like StreamingLLM and SnapKV struggle with long-range dependencies and AdaKV shows fragmentation at extreme lengths, CompilerKV maintains a robust retrieval pattern comparable to the FullKV oracle.
Figure A8 : Universality across model architectures. Average performance trends on four different LLMs (LLaMA-3, Qwen2, InternLM2.5, and Mistral) under varying KV budgets. The consistent degradation patterns support that CompilerKV’s risk-adaptive mechanism generalizes across architectures.
Figure A9 : Risk-Adaptive Threshold Gating Policy. The plots show the learned retention threshold ( τ ) across layers for samples with varying risk levels (binned by Perplexity and Attention Entropy). For high-risk samples (High Perplexity, rightmost plots), the policy automatically lowers thresholds to preserve more information, validating our risk-adaptive mechanism.
Model
Method
Single-Document QA
Multi-Document QA
Summarization
Few-shot Learning
Synthetic
Code
Avg.
NrtvQA
Qasper
MF-en
HotpotQA
2WikiMQA
Musique
GovReport
QMSum
MultiNews
TREC
TriviaQA
SAMSum
PCount
PRe
Lcc
RB-P
18409
3619
4559
9151
4887
11214
8734
10614
2113
5177
8209
6258
11141
9289
1235
4206
–
LLaMA-3-8B -Instruct
FullKV
25.16
31.81
39.59
43.09
36.15
21.77
28.62
23.34
26.33
75.00
90.50
42.36
5.20
69.25
59.04
53.93
41.95
StreamingLLM
17.85
9.50
23.09
37.84
29.02
16.77
17.91
20.42
20.16
44.00
73.00
30.00
5.80
69.50
48.38
49.31
32.03
H2O
21.58
12.54
28.49
37.13
32.36
18.88
20.23
22.16
21.14
39.00
86.62
39.19
5.50
69.50
57.39
54.46
35.39
SnapKV
21.71
12.37
32.38
37.44
30.48
19.50
19.06
21.36
20.07
45.5
87.74
38.15
5.50
68.85
57.42
54.61
35.76
Appendix
Table A8 : LongBench results at the extreme 128-token budget. CompilerKV remains the strongest compressed method on all four backbones, but the severe information bottleneck keeps it below the FullKV reference, as expected. Bold marks the best compressed average; FullKV is shown only as the lossless reference.
KV cache eviction methods typically use a single retention-rule family throughout a model, making eviction-method identity a model-level design choice. Yet Transformer layers differ substantially in their attention behavior, representations, and sensitivity to compression, suggesting that a uniform rule may overlook useful layer-wise structure. This raises a basic question: should eviction methods themselves vary across layers? We investigate this question by composing existing eviction methods across Transformer layers and systematically exploring the resulting layer-wise design space. Using simple offline profiles, we construct fixed routes and study how their quality varies with method placement and cache budget. On LongBench, heterogeneous routing improves performance on a majority of tasks over homogeneous policies at the same cache budget. Even when method counts are held fixed, the profile-guided placement ranks second among 100 evaluated assignments, demonstrating that routing quality depends strongly on where methods are placed. Moreover, the same fixed route outperforms the best of nine standalone baselines across all five tested cache budgets. Together, these results establish layer-wise method composition as an exploitable, placement-sensitive design dimension for KV cache compression.
Chao Fei, Kaihua Liang, Hanzhi Hu +4
King Abdullah University of Science and Technology · Fudan University
As context windows scale to tens or hundreds of thousands of tokens, KV cache compression has become essential for efficient LLM inference. Existing methods fall into three families: score-based eviction, summary compensation, and offload-and-recall. Yet all three decide what to keep or recall by content relevance to the current query. We show this shared design is structurally incomplete. A cache supports two access modes: associative lookup by content and sequential traversal by position; current compressors implement only the first. The gap matters in practice: retrieval-augmented generation, code completion, and structured-data extraction all require the model to reproduce identifiers, field values, or code tokens verbatim from the context. Under compression, content-based eviction retains the head of such a sequence but discards its continuation, causing verbatim copying to break irreversibly midway, a failure we call sequential forgetting. This failure resists better scoring, larger budgets, summary compensation, and dynamic re-scoring; it is the dominant source of remaining quality loss under compression. We propose KVFetch, a training-free, drop-in framework that opens a temporal recall channel for any score-based compressor. It demotes evicted candidates to a quantized cold tier, detects active copying through a monotone read pointer, and prefetches positional successors into fixed-size hot-tier slots without increasing attention cost. On RULER-16K under an iso-budget control, KVFetch recovers verbatim copying from 0.8 to 78.4 and raises the 13-task average by +8.4, with gains concentrating on tasks that require sequential access. On LongBench, where no task requires sequential access, the channel remains dormant and imposes no cost.
KV caches are a major bottleneck in long-context inference and long-form generation with large language models. Existing training-free eviction policies largely rely on proxy importance signals, such as attention mass, to decide which past tokens to retain. We argue that cache compression should instead preserve the predictive behavior of the full-cache model, retaining entries whose removal would substantially change the model's output distribution. We propose Behavior-Preserving KV Cache Compression, a training-free framework that scores candidate evictions by estimating the compressed-cache logits induced by their removal and evaluating the resulting KL to the full-cache next-token distribution. Using pre-eviction forward statistics, the method avoids running separate masked forward passes for each candidate. Across diverse architectures and both prefill-time and generation-time compression, our method delivers substantial gains in downstream task quality over lightweight attention-based heuristics at matched retained-KV budgets, with the largest gains under aggressive compression. It achieves these gains with additional compression-time computation while retaining an end-to-end speedup over full-cache inference in our evaluated settings.
Doo Hwan Hwang, Junyoung Jang, Junho Na +2
Kim Jaechul Graduate School of AI, KAIST, Daejeon, Republic of Korea · KT Corporation, Seoul, Republic of Korea