Organizations: Institute of Automation, Chinese Academy of Sciences, Beijing, China · University of Electronic Science and Technology of China, Chengdu, China · The Chinese University of Hong Kong, Shenzhen, China · Tianjin university, Tianjin, China · University of Science and Technology Beijing, Beijing, China
Prefill-only KV compression freezes a token subset at the end of prefill and decodes from it without further eviction. The retention decision is therefore irreversible, yet existing methods estimate the corrective signals it relies on, per-head reliability and prompt-level compression sensitivity, online from a single noisy prompt. We argue this is the wrong statistical unit: these signals exhibit far higher cross-prompt regularity than within-prompt signal-to-noise. We introduce CompilerKV, a KV-retention policy whose corrective tables are compiled offline from a calibration corpus, reducing online correction after the standard observation-window scan to O(1) lookups plus a budget clamp. We find that compiled retention tables behave as portable architectural priors: rankings transfer across disjoint corpora on four backbones (mean Spearman ρˉ=0.90), and direct model-to-model table transfer costs only 0.4--0.8 LongBench points on average. At a 512-token budget, CompilerKV attains compressed-SOTA on all four backbones, improving over the strongest prefill-only baseline by +1.67 points on average (task-bootstrap 95% CI [+1.08,+2.37]). Pressure regimes amplify the gap: under a fixed 512/32k cache ratio, CompilerKV remains the strongest compressed method through 128k RULER (∼73 vs.\ FullKV ∼79, SnapKV ∼38); on 32k NIAH it reaches 0.89 vs.\ SnapKV 0.42; and at 32k input, retaining only 1.56% of the prefill KV, batch-16 serving remains feasible where FullKV is OOM.
Figures & tables
Figure 1: Empirical evidence that E1–E3 are real, distinct failure modes on calibration data. (a) The same token’s attention score has very different distributions across prompts (low-noise vs. high-noise prompt) and across heads (a scale-biased head spans a much wider range), confirming E1. (b) The aggregated distribution of compiled head-reliability weights is bimodal: a substantial cluster of heads is consistently down-weighted (noise emitters), while a smaller cluster receives weights >1.1 (retrieval-critical), confirming E2. (c) The compression damage of a fixed Top- k rule (color: NLL gap to FullKV) varies systematically with attention entropy and local perplexity, confirming E3 as a prompt-level effect that no global k can absorb.
Figure 2: Overview of the CompilerKV framework. The pipeline has three stages: (1) a stabilized utility score that suppresses transient noise; (2) a compiled Head Heterogeneity Table that governs functional differences among attention heads; and (3) a compiled Risk-Adaptive Gating Table that calibrates the retention threshold to each prompt’s complexity.
Single-Doc QA
Multi-Doc QA
Summarization
Few-Shot Learning
Synthetic & Code
Model
Method
NarrativeQA
Qasper
MF-en
HotpotQA
2WikiMQA
MuSiQue
GovReport
QMSum
MultiNews
TREC
TriviaQA
SAMSum
Lcc
RB-P
Pcount
PRe
Avg.
InternLM-2.5 7B-Chat-1M
FullKV
22.47
27.58
39.98
40.96
33.52
26.61
33.01
25.18
26.28
72.5
86.76
39.84
55.86
57.90
2.92
100.00
43.21
SnapKV
16.86
23.28
36.24
32.14
19.89
23.21
17.81
23.18
22.44
71.0
84.05
34.34
50.32
53.34
1.00
96.50
37.85
PyramidKV
17.62
21.08
37.52
32.21
21.31
22.03
19.37
24.06
22.22
73.0
83.94
34.61
50.45
49.72
1.05
95.50
37.86
DynamicKV
17.77
23.87
37.74
32.98
21.13
23.21
19.13
23.49
22.48
75.0
84.89
36.70
50.70
51.08
0.91
95.50
38.54
AdaKV
18.23
24.64
38.30
33.64
21.66
23.55
19.74
23.96
22.92
75.3
85.28
37.23
51.31
51.91
1.30
95.78
39.05
Table 1: LongBench results (512-token budget). The best compressed result is bold and the second best is underlined ; FullKV is shown as the lossless reference and is not used for compressed-method ranking. AdaKV and CAKE are included as recent head- and layer-adaptive baselines.
Figure 3: Performance vs. KV cache budget. Left: mean LongBench accuracy on LLaMA-3-8B. Right: cross-task tail score (Task-P10). All budget curves are monotone with cache size; CompilerKV is the strongest compressed method across all budgets and slightly exceeds FullKV at larger budgets.
Setting
Mistral-7B
LLaMA-3-8B
Qwen2-7B
InternLM-2.5
Ablation under E1–E3 framing
CompilerKV (Full)
42.94
42.55
41.13
42.61
w/o Stabilized Utility
40.20
39.84
38.61
38.09
w/o Head Reliability
40.76
40.31
39.58
39.07
w/o Risk-Adaptive τ
38.97
38.52
37.73
37.24
Compiler objective
Table 2: Ablation and compiler probe (LongBench avg., 512-token budget). Top: component removals cause the predicted E1–E3 regressions. Middle: offline reward regression ( α=0 ) isolates the conservative CQL gain. Bottom: joint head–risk compilation outperforms independent compilation.
Table 6
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Model
Benchmark / setup
Score
External reported results
Full-KV
LLaMA-3.1-8B-Instruct
LongBench Avg. / -
52.2
DuoAttention
LLaMA-3.1-8B-Instruct
LongBench Avg. / -512 KV
41.2
SnapKV
LLaMA-3.1-8B-Instruct
LongBench Avg. / -512 KV
45.2
Quest
LLaMA-3.1-8B-Instruct
LongBench Avg. / -512 KV
35.1
SparQ
LLaMA-3.1-8B-Instruct
LongBench Avg. / -512 KV
43.8
Appendix
Table A1 : Additional KV-compression baselines. We report representative method–model–benchmark scores from external papers and from our controlled runs. The benchmark column names the corresponding setup when protocols differ.
Figure A1 : Budget scaling across benchmark families (LLaMA-3-8B-Instruct). Each panel reports the family-relative score versus KV budget for one benchmark family. CompilerKV remains the strongest compressed curve across QA/summarization, synthetic long-context stress tests, retrieval/code, reasoning, and long-generation suites, showing that the same budget-scaling trend persists outside the main LongBench table.
Figure A2 : RULER length scaling at a fixed cache ratio (LLaMA-3-8B-Instruct). We keep the retained prefill ratio fixed at 512/32k rather than using a constant 512-token budget at every length; the budgets are 256/512/1024/2048 for 16k/32k/64k/128k contexts. CompilerKV remains the strongest compressed method across lengths, while staying below FullKV at long contexts as expected under aggressive compression.
Figure A3 : Math and reasoning robustness (LLaMA-3-8B-Instruct). Each panel reports score versus KV budget for one reasoning benchmark. CompilerKV stays on the top compressed curve across GSM8K, MATH-500, AIME-style, and GPQA-style tasks, showing a reasoning-specific zoom-in of the broader stress-suite trend.
Figure A4 : Quality-memory and speed-memory Pareto view (LLaMA-3-8B-Instruct, 32k context). Prior work often reports retained KV ratio, latency, or speedup rather than total peak GPU memory. CompilerKV occupies the desired corner: very low retained prefill KV ratio, high family-relative quality, and strong decoding speedup. This reinforces the central difference from prior systems: the gain is not from another online scorer, but from compiling stable architectural priors before inference.
Figure A5 : Needle-in-a-Haystack stress heatmaps (Mistral-7B-Instruct-v0.2). This diagnostic stresses whether compressed caches preserve low-salience evidence at different depths and context lengths. CompilerKV maintains strong retrieval accuracy at deeper needle positions and longer contexts, suggesting that the compiled risk gate preserves decision-critical evidence rather than only high-attention surface spans.
Figure A6 : Retrieval, QA, and code-comprehension stress suite (LLaMA-3-8B-Instruct). We show how each method scales with KV budget on SQuAD-long, NIAH, SCBench-QA, SCBench-Code, RepoBench-P, and LongGen. CompilerKV remains the strongest compressed method throughout the budget range and is especially robust in low-budget retrieval/code settings, which is consistent with the paper’s view that offline compilation is a different decision paradigm rather than another online token scorer.
Table A3 : Pressure-regime summary. Stress settings where the gain is no longer a small LongBench-margin effect. RULER-128k uses the fixed 512/32k retained-cache ratio from Fig. A2 ; NIAH is from Fig. A7 ; batching is from Table A7 .
Backbone
Spearman ρ
Layer-bin mean
Layer-bin min
LLaMA-3-8B
0.91
0.91
0.88
Mistral-7B
0.90
0.90
0.87
Qwen2-7B
0.90
0.89
0.86
InternLM2.5-7B
0.89
0.89
0.85
Mean
0.90
0.90
0.87
Appendix
Table A4 : Cross-corpus stability of compiled head reliability across backbones. For each backbone, we compile two independent Head Heterogeneity Tables using disjoint calibration corpora (arXiv + PubMed vs. ShareGPT + UltraChat) and report Spearman ρ between the resulting per-head reliability rankings. The stable correlations across all four architectures support the claim that the compiled table captures model-level head functionality rather than a corpus artifact.
Source table
Target model
Target-compiled
Transferred
Drop
Same-family transfer
LLaMA-3-8B
LLaMA-3.1-8B
52.60
52.30
−0.30
LLaMA-3.1-8B
LLaMA-3-8B
42.55
42.28
−0.27
LLaMA-3.1-8B
LLaMA-3.2-family
50.10
49.70
−0.40
Cross-family transfer
Mistral-7B
LLaMA-3-8B
42.55
41.85
−0.70
Appendix
Table A5 : Model-to-model transfer of compiled retention tables. We compile on a source backbone, deploy the resulting table on a target backbone without target recompilation, and compare with target-compiled CompilerKV. Same-family transfer costs 0.3 – 0.4 points; broader cross-family transfer costs 0.4 – 0.8 points on average, showing that the compiled table is a portable retention prior rather than a per-run tuning artifact.
Context
Method
Prefill KV Compression
Decoding Efficiency
Input
Output
Budget
Retained
Saving
TTFT(s) ↓
Decode rate(tok/s) ↑
Latency(s) ↓
8k
2k
FullKV
8k
100.0%
0.0%
0.66
27.63
74.79
CompilerKV
512
6.25%
93.75%
0.72
36.20 ( 1.31× )
58.90 ( 1.27× )
16k
4k
FullKV
16k
100.0%
0.0%
1.45
19.55
209.56
CompilerKV
512
3.12%
96.88%
1.52
34.80 ( 1.78× )
118.40 ( 1.77× )
32k
8k
FullKV
32k
100.0%
0.0%
3.52
11.65
706.56
Appendix
Table A6 : Main efficiency profile. Retained prefill KV ratio and decoding efficiency on a single A100-80GB. CompilerKV fixes the prefill budget to 512 tokens; generated KV states are appended normally. Speedup multipliers in parentheses are relative to FullKV at the matching context length.
Figure A7 : Needle-in-a-Haystack Pressure Test on Mistral-7B. Visual comparison of retrieval accuracy (Green=100%, Red=0%) across varying context lengths and needle depths. While baselines like StreamingLLM and SnapKV struggle with long-range dependencies and AdaKV shows fragmentation at extreme lengths, CompilerKV maintains a robust retrieval pattern comparable to the FullKV oracle.
Figure A8 : Universality across model architectures. Average performance trends on four different LLMs (LLaMA-3, Qwen2, InternLM2.5, and Mistral) under varying KV budgets. The consistent degradation patterns support that CompilerKV’s risk-adaptive mechanism generalizes across architectures.
Figure A9 : Risk-Adaptive Threshold Gating Policy. The plots show the learned retention threshold ( τ ) across layers for samples with varying risk levels (binned by Perplexity and Attention Entropy). For high-risk samples (High Perplexity, rightmost plots), the policy automatically lowers thresholds to preserve more information, validating our risk-adaptive mechanism.
Model
Method
Single-Document QA
Multi-Document QA
Summarization
Few-shot Learning
Synthetic
Code
Avg.
NrtvQA
Qasper
MF-en
HotpotQA
2WikiMQA
Musique
GovReport
QMSum
MultiNews
TREC
TriviaQA
SAMSum
PCount
PRe
Lcc
RB-P
18409
3619
4559
9151
4887
11214
8734
10614
2113
5177
8209
6258
11141
9289
1235
4206
–
LLaMA-3-8B -Instruct
FullKV
25.16
31.81
39.59
43.09
36.15
21.77
28.62
23.34
26.33
75.00
90.50
42.36
5.20
69.25
59.04
53.93
41.95
StreamingLLM
17.85
9.50
23.09
37.84
29.02
16.77
17.91
20.42
20.16
44.00
73.00
30.00
5.80
69.50
48.38
49.31
32.03
H2O
21.58
12.54
28.49
37.13
32.36
18.88
20.23
22.16
21.14
39.00
86.62
39.19
5.50
69.50
57.39
54.46
35.39
SnapKV
21.71
12.37
32.38
37.44
30.48
19.50
19.06
21.36
20.07
45.5
87.74
38.15
5.50
68.85
57.42
54.61
35.76
Appendix
Table A8 : LongBench results at the extreme 128-token budget. CompilerKV remains the strongest compressed method on all four backbones, but the severe information bottleneck keeps it below the FullKV reference, as expected. Bold marks the best compressed average; FullKV is shown only as the lossless reference.