The Gating Mechanism of KV Cache Eviction
Organizations: The Hong Kong University of Science and Technology · New York University · The Hong Kong University of Science and Technology (Guangzhou) · Tencent HunYuan ·
Abstract
KV cache eviction must decide which entries carry information that the rest of the cache can no longer supply. A natural way to estimate this is a learnable gate: give each cached entry a gate on its attention weight and train the gate on the model's own objective. The gate belongs inside the normalizing softmax, where suppressing one entry hands its mass to the others rather than discarding it. That same normalizer absorbs any per-query constant, so a constant gate cancels and reproduces the frozen model exactly. Trained on the language-modeling loss the gate stays near a constant score while its loss falls throughout: optimization succeeds, but a frozen backbone was fit for dense attention, so its loss prefers the no-op, and a near-constant score ranks nothing. The no-op is not a floor the gate improves from but what its objective asks for. A second obstacle survives once the gate moves, since the same gradient charges the gate for errors the frozen model already makes, which no reweighting of positions separates out. Much of the machinery in learnable eviction, from sparsity penalties to relaxations of the discrete decision, works around the first obstacle rather than removing it; both are removed by putting dense attention beyond the gate's reach and making the dense model's predictions its target. Under these two conditions capacity is never the missing ingredient: neither a larger scorer, nor explicit access to the past, nor a trainable backbone helps. The same holds for what eviction unavoidably discards, which is recovered by parameter-free centroids read inside the same softmax far better than by any memory read outside it, since the mass they must return is a share of that normalizer and available nowhere else. At 75% KV cache compression, this gate reaches 90.14 on RULER at 8K and 46.03 on LongBench with Qwen3-8B, 3.72 and 1.36 points below full-cache inference.
Figures & tables
| Component | RULER 8K | ||||
|---|---|---|---|---|---|
| Configuration | Always-attended | Loss | Head budget allocation | Avg. | NIAH |
| Direct instantiation | none | LM | uniform | 4.33 | 2.00 |
| always-attended entries | sinks window | LM | uniform | 44.86 | 53.73 |
| distillation | sinks window | KL | uniform | 58.86 | 72.78 |
| head budget | sinks window | KL | mass | 67.60 | 86.17 |
| Dense backbone (no eviction) | — | — | — | 70.01 | 86.95 |
| Component | Choice | Avg. | NIAH | |
| Always-attended | none | 4.72 | 2.00 | |
| sinks only | 0.57 | 0.00 | ||
| window only | 57.81 | 72.71 | ||
| sinks window † | 67.60 | 86.17 | — | |
| Loss | LM loss (CE) | 51.50 | 64.65 | |
| LongCE | 53.01 | 61.99 |
| RULER 8K | |||||
|---|---|---|---|---|---|
| Loss | Trained | Tokens | Evict 75% | Full KV | Compression gap |
| reverse KL | gate only † | 7M | 67.60 | 70.01 | |
| reverse KL | gate backbone | 7M | 62.83 | 71.65 | |
| LM loss | gate backbone | 7M | 60.45 | 69.07 | |
| LM loss | gate backbone | 79M | 59.68 | 69.26 | |
| LM loss | gate backbone | 197M | 57.42 | 69.24 | |
| Model & Benchmark | No slots | slots | |
|---|---|---|---|
| Qwen3-0.6B Math500 | 512 | 48.00 | 52.00 |
| Qwen3-8B RULER 8K | 2048 | 76.65 | 77.15 |
| Qwen3-4B (8K) | Qwen3-4B (16K) | Qwen3-8B (8K) | Qwen3-8B (16K) | |||||
| Method | 50% | 75% | 50% | 75% | 50% | 75% | 50% | 75% |
| Full-KV | 92.41 | 92.41 | 91.64 | 91.64 | 93.86 | 93.86 | 92.96 | 92.96 |
| SnapKV | 54.24 | 30.85 | 61.61 | 35.80 | 54.32 | 35.13 | 62.74 | 41.54 |
| KeyDiff | 73.72 | 62.31 | 70.00 | 61.99 | 77.35 | 67.14 | 74.44 | 66.87 |
| DMA | 68.18 | 36.34 | 64.69 | 33.30 | 75.22 | 41.76 | 71.78 | 36.19 |
| TrimKV | 86.38 | 75.42 | 86.40 | 76.29 | 87.23 | 80.26 | 85.91 | 79.09 |
| Model | Method | RULER Tasks (8K) | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| S1 | S2 | S3 | MK1 | MK2 | MK3 | MV | MQ | VT | CWE | FWE | QA1 | QA2 | Avg. | ||
| Qwen3-4B | Full-KV | 100.00 | 100.00 | 100.00 | 99.80 | 99.80 | 99.80 | 99.85 | 99.70 | 99.96 | 84.56 | 86.27 | 75.20 | 56.40 | 92.41 |
| SnapKV | 70.20 | 22.20 | 2.40 | 15.40 | 8.00 | 3.40 | 14.05 | 13.85 | 38.36 | 62.18 | 89.00 | 29.60 | 32.40 | 30.85 | |
| KeyDiff | 100.00 | 88.60 | 92.00 | 74.60 | 4.60 | 0.00 | 89.90 | 89.50 | 100.00 | 69.14 | 74.07 | 15.00 | 12.60 | 62.31 | |
| DMA | 99.00 | 24.20 | 2.40 | 27.40 | 5.40 | 0.00 | 14.30 | 20.70 | 79.60 | 48.16 | 71.47 | 47.80 | 32.00 | 36.34 | |
| TrimKV | 100.00 | 98.80 | 92.60 | 95.00 | 48.40 | 10.80 | 91.55 | 95.60 | 99.96 | 77.00 | 71.60 | 49.60 | 49.60 | 75.42 | |
| Method | Compression ratio | Single QA | Multi QA | Summ. | Few-shot | Synth. | Code | Avg. |
|---|---|---|---|---|---|---|---|---|
| Full-KV | 41.96 | 48.58 | 27.61 | 65.40 | 50.30 | 62.47 | 47.39 | |
| SnapKV | 50% | 35.60 | 46.52 | 25.93 | 65.26 | 50.00 | 63.10 | 45.39 |
| KeyDiff | 50% | 28.87 | 28.91 | 24.46 | 62.27 | 45.61 | 45.61 | 36.91 |
| DMA | 50% | 38.87 | 42.72 | 26.53 | 64.33 | 35.25 | 55.13 | 42.25 |
| TrimKV | 50% | 41.93 | 46.29 | 27.20 | 67.81 | 51.75 | 56.35 | 46.54 |
| IndexMem++ | 50% | 40.82 | 49.06 | 27.64 | 65.27 | 52.61 | 62.32 | 47.53 |
| Math500 | AIME25 | |||
|---|---|---|---|---|
| Method | ||||
| Full-KV | 0.832 (5,258) | 0.700 (17,845) | ||
| KVzap | 0.792 † (6,326) | 0.548 † (9,850) | 0.446 (22,019) | 0.183 (29,013) |
| Fast-KVzip | 0.736 † (6,974) | 0.480 † (11,146) | 0.375 (23,904) | 0.154 (29,462) |
| R-KV | 0.811 (9,741) | 0.605 (12,757) | 0.229 (27,044) | 0.121 (29,625) |
| TriAttention | 0.863 (8,563) | 0.758 (12,056) | 0.383 (23,109) | 0.192 (28,507) |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Gate location | Form | Physical meaning | Redistributes mass? |
|---|---|---|---|
| Query | retrieval mode | Yes | |
| Key | address / retrievability | Yes, non-monotonically | |
| Value | returned content | No | |
| SDPA output | whether retrieval is used | No | |
| Logit, multiplicative | competition sharpness | Non-monotonically | |
| Denominator only | mass leaves the row | No |
| Bias family | Canonical form | Frozen gate ranking? | Fold width | |
|---|---|---|---|---|
| Per-key scalar | Yes | |||
| Separable lifetime | Yes | |||
| Key-specific decay | depends on | No | ||
| Pairwise cumulative | depends on | No | high | |
| Query-conditioned | depends on | No | – |
| Method | Exact support | Compressed history and write policy | Readout | Joint |
|---|---|---|---|---|
| Infini-attention | Current segment | Associative state over completed segments | Learned convex gate between two outputs | No |
| RAttention | Recent SWA window | Linear state read only for out-of-window tokens | Independently RMS-normalized sum | No |
| Tensor Cache | Recent SWA window | Fast-weight state updated on FIFO eviction | Scalar-gated additive residual | No |
| NHA | Recent SWA window | Fixed recurrent KV slots | Concatenate slots and local KV | Yes |
| Key-Value Means | Recent block window | Overflow tokens merged into compressed KV rows | Single softmax with regional temperatures | Yes |
| CentroidKV | Sink, recent, and unmerged KV | Similar KV entries merged into degree-weighted centroids | Single softmax with bias | Yes |
| Benchmark | Budget | Allocator | Floor | Sink / local | CMP |
|---|---|---|---|---|---|
| RULER | mass | 4 / 128 | 64 | ||
| LongBench | mass | 4 / 128 | 64 | ||
| Math500 | 512, 1,024 | static | 4 / 128 | 0 | |
| AIME25 | 1,024, 2,048 | static | 4 / 128 | 0 |
| Method | S1 | S2 | S3 | MK1 | MK2 | MK3 | MV | MQ | VT | CWE | FWE | QA1 | QA2 | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Full-KV | 100.00 | 100.00 | 100.00 | 99.80 | 99.80 | 99.80 | 99.85 | 99.70 | 99.96 | 84.56 | 86.27 | 75.20 | 56.40 | 92.41 |
| Compression ratio: 50% | ||||||||||||||
| SnapKV | 85.20 | 89.00 | 5.60 | 43.60 | 31.60 | 28.80 | 43.00 | 41.05 | 78.12 | 77.10 | 93.27 | 45.80 | 43.00 | 54.24 |
| KeyDiff | 100.00 | 100.00 | 98.80 | 91.80 | 43.00 | 11.00 | 98.85 | 95.80 | 99.96 | 80.98 | 82.80 | 34.00 | 21.40 | 73.72 |
| DMA | 99.80 | 84.80 | 29.80 | 80.60 | 61.60 | 0.60 | 81.85 | 88.60 | 96.20 | 56.44 | 85.80 | 70.80 | 49.40 | 68.18 |
| TrimKV | 100.00 | 99.60 | 99.80 | 99.80 | 99.00 | 42.40 | 99.65 | 99.95 | 99.96 | 80.88 | 77.73 | 67.20 | 57.00 | 86.38 |
| Method | S1 | S2 | S3 | MK1 | MK2 | MK3 | MV | MQ | VT | CWE | FWE | QA1 | QA2 | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Full-KV | 100.00 | 100.00 | 100.00 | 99.40 | 99.00 | 98.80 | 99.95 | 99.95 | 99.80 | 69.82 | 97.00 | 72.80 | 54.80 | 91.64 |
| Compression ratio: 50% | ||||||||||||||
| SnapKV | 92.40 | 93.80 | 13.40 | 69.20 | 31.40 | 21.40 | 60.10 | 74.35 | 94.36 | 66.28 | 98.27 | 43.00 | 43.00 | 61.61 |
| KeyDiff | 100.00 | 99.80 | 99.40 | 75.00 | 27.80 | 3.80 | 93.60 | 95.80 | 99.88 | 70.16 | 90.73 | 29.00 | 25.00 | 70.00 |
| DMA | 100.00 | 83.40 | 32.60 | 77.20 | 51.60 | 0.20 | 79.35 | 83.75 | 92.56 | 33.70 | 94.07 | 65.60 | 47.00 | 64.69 |
| TrimKV | 100.00 | 100.00 | 100.00 | 98.60 | 97.40 | 40.40 | 99.15 | 99.90 | 99.80 | 63.98 | 99.13 | 71.60 | 53.20 | 86.40 |
| Method | S1 | S2 | S3 | MK1 | MK2 | MK3 | MV | MQ | VT | CWE | FWE | QA1 | QA2 | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Full-KV | 100.00 | 100.00 | 100.00 | 99.80 | 99.60 | 99.60 | 100.00 | 100.00 | 100.00 | 94.24 | 91.00 | 75.80 | 60.20 | 93.86 |
| Compression ratio: 50% | ||||||||||||||
| SnapKV | 88.40 | 73.20 | 3.00 | 42.40 | 29.80 | 24.40 | 37.95 | 37.70 | 88.80 | 91.74 | 92.33 | 49.00 | 47.40 | 54.32 |
| KeyDiff | 100.00 | 99.80 | 99.00 | 95.20 | 53.40 | 17.00 | 99.70 | 99.55 | 100.00 | 93.20 | 87.67 | 34.60 | 26.40 | 77.35 |
| DMA | 99.00 | 82.40 | 80.80 | 83.60 | 75.60 | 1.60 | 75.60 | 89.25 | 99.16 | 77.28 | 88.93 | 74.00 | 50.60 | 75.22 |
| TrimKV | 100.00 | 100.00 | 99.80 | 99.40 | 99.00 | 21.00 | 99.55 | 99.90 | 100.00 | 94.30 | 90.40 | 72.00 | 58.60 | 87.23 |
| Method | S1 | S2 | S3 | MK1 | MK2 | MK3 | MV | MQ | VT | CWE | FWE | QA1 | QA2 | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Full-KV | 100.00 | 100.00 | 99.80 | 99.60 | 100.00 | 99.60 | 99.60 | 100.00 | 100.00 | 82.54 | 93.80 | 75.20 | 58.40 | 92.96 |
| Compression ratio: 50% | ||||||||||||||
| SnapKV | 98.20 | 94.80 | 9.20 | 67.60 | 33.40 | 18.40 | 63.80 | 67.95 | 99.00 | 77.86 | 95.60 | 44.60 | 45.20 | 62.74 |
| KeyDiff | 100.00 | 100.00 | 99.00 | 82.20 | 31.60 | 13.60 | 98.70 | 98.00 | 100.00 | 86.40 | 94.67 | 33.00 | 30.60 | 74.44 |
| DMA | 99.60 | 80.40 | 79.80 | 79.40 | 61.80 | 0.80 | 73.95 | 90.25 | 98.56 | 55.06 | 94.47 | 68.40 | 50.60 | 71.78 |
| TrimKV | 100.00 | 100.00 | 99.60 | 97.80 | 98.80 | 17.80 | 97.65 | 99.70 | 100.00 | 82.04 | 97.07 | 70.40 | 56.00 | 85.91 |
| Method | NQA | Qasper | MFQA | Hotpot | 2Wiki | MuSiQue | GovR | QMSum | MNews |
|---|---|---|---|---|---|---|---|---|---|
| Full-KV | 26.31 | 43.98 | 55.59 | 62.30 | 48.22 | 35.23 | 33.55 | 24.50 | 24.77 |
| Compression ratio: 50% | |||||||||
| SnapKV | 23.50 | 37.36 | 45.94 | 59.48 | 47.17 | 32.91 | 31.91 | 22.63 | 23.26 |
| KeyDiff | 16.81 | 29.46 | 40.33 | 36.23 | 34.46 | 16.03 | 29.34 | 22.37 | 21.66 |
| DMA | 22.40 | 42.19 | 52.01 | 55.56 | 39.37 | 33.24 | 32.21 | 23.22 | 24.16 |
| TrimKV | 26.25 | 47.66 | 51.89 | 59.33 | 43.08 | 36.45 | 33.19 | 24.09 | 24.31 |
| Method | Retention score | Protected region (within budget) |
|---|---|---|
| SnapKV | Observed-attention mean with width-5 average pooling | Final 64 context positions |
| KeyDiff | Cosine-based post-RoPE key scoring | Final context position |
| DMA | Query-independent value-derived gate score | Sink 4, local 128 |
| TrimKV | Released gate deltas, magnitude and lifetime | Sink 4, local 128 |
| KVpop | Recurrent scorer state | Sink 4, local 256 |
| IndexMem++ | Learned gate on ; mass allocation across heads | Sink 4, local 128, CMP 64 |
| NLL | KL from Full-KV | ||||
|---|---|---|---|---|---|
| Full-KV | IndexMem++ | Recency | IndexMem++ | Recency | |
| AIME25 | 0.197 | 0.207 | 0.279 | 0.021 | 0.082 |
| SWE | 1.068 | 1.221 | 1.551 | 0.120 | 0.630 |
KV-Cache Eviction
Key-value (KV) cache eviction reduces language-model memory use by removing or compressing cached tokens while preserving useful context.
See all 104 papers