Who Should Score the Cache? Mid-Depth Layers Are the Best Judges of KV-Cache Eviction
Organizations: Colby College
Abstract
Key-value (KV) cache eviction methods rank tokens by attention scores, and each layer scores its own cache. To our knowledge, no eviction method asks which layer should score. We find, across twenty-one models, that the depth of the scoring layer governs eviction quality. On the eighteen models with per-layer caches, spanning 0.5B to 109B, long-context retrieval accuracy collapses at shallow layers, peaks between 0.4 and 0.75 of depth, and degrades at the last layer. The curve arises because a layer judges tokens well only once it has used the question to find the answer, which takes several layers. Modeling eviction as a top- cut, we compute answer survival probability from prefill attention and reproduce the measured curves. This approach selects the scoring layer using calibration prompts with known answers, and the selected layer is the best layer measured on most models and never significantly below it. It improves on the own-layer default on sixteen of the eighteen models, with significant gains on seven, all at 14B or below, while on larger models the default is already close to the best mid-depth layer. Scoring every cache from the selected layer while keeping one set of tokens for all heads also beats per-head SnapKV by 18 to 43 points on five models up to 14B under tight budgets. This gain carries over to financial filings at 32K context, all at the same memory budget, with no extra forward pass. The same curve explains a penalty in shared-cache models, where several layers read one cache and the layer that writes it scores by default. On Gemma-4-E2B the writer is shallow and loses 28 points to a mid-depth reader at a 10% budget. We also find that in open-book question answering (LongBench) no mid-depth layer beats the own-layer default.
Figures & tables
| Scoring method | 1.5B @15% | 1.5B @10% | SmolLM2 @15% | 7B @10% | 14B @5% |
|---|---|---|---|---|---|
| No eviction (ceiling) | 91.1 | 90.8 | 76.4 | 93.0 | 91.7 |
| H2O (observed attention) | 18.4 | 16.6 | 8.8 | 21.6 | 21.7 |
| SnapKV (per-head, own layer) | 53.0 | 39.2 | 28.4 | 57.4 | 57.6 |
| AdaKV (SnapKV scores) | 57.8 | 42.8 | 30.9 | 53.9 | 53.6 |
| PyramidKV (pyramid budgets) | 33.1 | 22.6 | 20.2 | 28.0 | 34.8 |
| Own layer, single set | 63.1 | 49.5 | 48.5 | 77.1 | 56.0 |
| Model | Cache | Budget | Best scorer (depth) | Best own | Worst own |
|---|---|---|---|---|---|
| Qwen2.5-14B | per-layer | 5% | L30 (0.64) | +17.7 [+5.6,+30.9] | -50.7 [-71.9,-31.5] |
| Qwen2.5-32B | per-layer | 5% | L41 (0.65) | +3.9 [-4.2,+11.4] | -44.6 [-60.2,-29.4] |
| Qwen3-0.6B | per-layer | 15% | L16 (0.59) | +20.4 [+5.1,+37.4] | -45.4 [-65.4,-26.9] |
| Qwen3-1.7B | per-layer | 10% | L17 (0.63) | +8.3 [-4.6,+21.6] | -55.4 [-72.8,-38.5] |
| Qwen3-14B | per-layer | 5% | L28 (0.72) | -2.1 [-6.9,+2.4] | -42.0 [-63.6,-22.4] |
| Qwen3.5-9B | hybrid 8/32 | 5% | L15 (0.48) | +9.8 [-6.2,+26.2] | -29.8 [-43.1,-18.7] |
| Scorer of the shared cache | Accuracy (%) | vs. writer |
|---|---|---|
| No eviction (ceiling) | 90.3 | – |
| Writer, L14 (default) | 23.1 | – |
| Reader L19 | 51.2 | |
| Reader L24 | 48.3 | |
| Reader L29 | 42.5 | |
| Reader L34 (terminal) | 37.0 |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Default damage at each candidate budget (% of context kept) | |||||||||||
| Model | 0.25 | 0.5 | 1 | 2 | 3 | 5 | 10 | 15 | 20 | 25 | Binding |
| Qwen2.5-0.5B | – | – | – | – | – | 63.2 f | 48.0 f | 44.0 f | 42.6 f | 38.8 | 25% |
| Qwen2.5-1.5B | – | – | – | – | – | 54.9 f | 34.1 f | 28.4 | 24.4 | – | 15% |
| SmolLM2-1.7B | – | – | – | – | – | 46.8 f | 38.2 f | 33.2 f | 27.4 | – | 20% |
| Qwen2.5-7B | – | – | – | – | – | 36.6 f | 13.8 | – | – | – | 10% |
| Llama-3.2-3B | – | – | – | – | – | 19.0 | 5.2 | – | – | – | 5% |
| Model | Budget | Pick | Best other | Pick own | Pick best other |
|---|---|---|---|---|---|
| Qwen2.5-0.5B | 15% | L14 | L15 | +31.4 [+9.9,+53.8] | +10.5 [-0.5,+22.9] |
| Qwen2.5-1.5B | 15% | L20 | L19 | +26.6 [+8.4,+46.5] | +7.4 [+1.8,+14.1] |
| SmolLM2-1.7B | 15% | L15 | L16 | +11.8 [+3.3,+21.5] | +0.1 [-6.0,+6.4] |
| Llama-3.2-3B | 5% | L14 | L16 | +7.2 [-5.8,+22.4] | -1.7 [-13.8,+8.0] |
| Qwen2.5-7B | 10% | L18 | L19 | +5.5 [-12.5,+26.9] | +1.5 [-3.2,+6.1] |
| Llama-3.1-8B | 5% | L15 | L13 | +6.0 [-2.9,+16.1] | -1.4 [-5.6,+2.2] |
| Qwen2.5-1.5B @10% | Qwen2.5-1.5B @15% | SmolLM2-1.7B @15% | ||||
| Excluded task | none | ours/SnapKV | none | ours/SnapKV | none | ours/SnapKV |
| cwe | 58 | 5/6 | 58 | 12/7 | 4 | 4/10 |
| fwe | 80 | 73/50 | 80 | 67/57 | 70 | 47/40 |
| niah_multikey_2 | 100 | 60/20 | 100 | 90/20 | 90 | 10/10 |
| niah_multikey_3 | 90 | 10/0 | 90 | 20/10 | 60 | 10/0 |
| niah_single_3 | 100 | 0/0 | 100 | 0/0 | 100 | 0/0 |
| Scoring method | 1.5B @5% | 1.5B @20% | 0.5B @15% | 3B @5% | 8B @5% |
|---|---|---|---|---|---|
| No eviction (ceiling) | 90.8 | 90.8 | 84.6 | 91.3 | 93.9 |
| H2O (observed attention) | 13.7 | 19.9 | 15.5 | 17.4 | 21.1 |
| SnapKV (per-head, own layer) | 17.9 | 60.5 | 35.1 | 74.7 | 78.6 |
| AdaKV (SnapKV scores) | 20.1 | 66.8 | 31.4 | 74.0 | 75.1 |
| PyramidKV (pyramid budgets) | 10.4 | 44.8 | 22.7 | 62.2 | 72.3 |
| Own layer, single set | 23.1 | 73.4 | 31.9 | 74.6 | 78.2 |
| vs. SnapKV | 1.5B @15% | 1.5B @10% | SmolLM2 @15% | 7B @10% | 14B @5% |
|---|---|---|---|---|---|
| single set (own layer) | |||||
| 95% CI | |||||
| mid layer (per-head) | |||||
| 95% CI | |||||
| AdaKV | |||||
| 95% CI |
| Scorer | F1 | vs. default | 95% CI |
|---|---|---|---|
| Qwen2.5-1.5B, 5% (binding); 8 of 9 datasets informative | |||
| none (no eviction) | 36.6 | – | – |
| own (per-layer default) | 30.2 | – | – |
| L2 | 24.4 | -5.8 | [-12.0,-0.3] |
| L19 | 30.0 | -0.2 | [-3.7,+3.1] |
| L20 | 29.4 | -0.9 | [-4.7,+2.5] |
| Replication | reader writer |
|---|---|
| reader L19 writer @ budget 5% | |
| reader L19 writer @ budget 15% | |
| reader L19 writer @ budget 25% | |
| reader L19 writer @ 8K context | |
| reader L19 writer @ 16K context | |
| Gemma-3n: reader L24 writer L19 |
| Model | Budget | Heads own | Layer | Heads layer |
|---|---|---|---|---|
| Qwen2.5-1.5B | 15% | +26.0 [+6.4,+47.8] | L19 (grid) | +6.8 [+0.2,+15.0] |
| L20 (pick) | -0.6 [-3.7,+2.0] | |||
| Llama-3.1-8B | 5% | +9.0 [+1.1,+17.8] | L13 | +1.4 [-1.6,+5.0] |
| Mistral-Small-24B | 5% | +18.7 [+2.0,+38.5] | L17 | +11.5 [+3.5,+20.1] |
| Scorer | Depth | Recall (%) | vs. default |
| Llama-3.1-8B, per-layer caches (1%, ) | |||
| none (no eviction) | – | 45.9 | – |
| own (per-layer default) | – | 25.2 | – |
| L2 | 0.06 | 1.4 | -23.8 [-28.8,-18.9] |
| L8 | 0.26 | 3.2 | -22.0 [-26.9,-17.5] |
| L12 | 0.39 | 1.9 | -23.3 [-28.4,-18.5] |
KV-Cache Eviction
Key-value (KV) cache eviction reduces language-model memory use by removing or compressing cached tokens while preserving useful context.
See all 104 papers