Authors: Hao Gu, Xintong Yang, Zhiwei Li, Lei Zhu, Jian Yang, Qiyuan Zhu, Dezhi Li, Jingyuan Zhang, Zhizhuo Kou, Hao Wang, Bei Liu, Jiacheng Liu, Sirui Han
Organizations: The Hong Kong University of Science and Technology · New York University · The Hong Kong University of Science and Technology (Guangzhou) · Tencent HunYuan · Georgia Institute of Technology · City University of Hong Kong · Shanghai Jiaotong University
KV cache eviction must decide which entries carry information that the rest of the cache can no longer supply. A natural way to estimate this is a learnable gate: give each cached entry a gate on its attention weight and train the gate on the model's own objective. The gate belongs inside the normalizing softmax, where suppressing one entry hands its mass to the others rather than discarding it. That same normalizer absorbs any per-query constant, so a constant gate cancels and reproduces the frozen model exactly. Trained on the language-modeling loss the gate stays near a constant score while its loss falls throughout: optimization succeeds, but a frozen backbone was fit for dense attention, so its loss prefers the no-op, and a near-constant score ranks nothing. The no-op is not a floor the gate improves from but what its objective asks for. A second obstacle survives once the gate moves, since the same gradient charges the gate for errors the frozen model already makes, which no reweighting of positions separates out. Much of the machinery in learnable eviction, from sparsity penalties to relaxations of the discrete decision, works around the first obstacle rather than removing it; both are removed by putting dense attention beyond the gate's reach and making the dense model's predictions its target. Under these two conditions capacity is never the missing ingredient: neither a larger scorer, nor explicit access to the past, nor a trainable backbone helps. The same holds for what eviction unavoidably discards, which is recovered by parameter-free centroids read inside the same softmax far better than by any memory read outside it, since the mass they must return is a share of that normalizer and available nowhere else. At 75% KV cache compression, this gate reaches 90.14 on RULER at 8K and 46.03 on LongBench with Qwen3-8B, 3.72 and 1.36 points below full-cache inference.
Figures & tables
Figure 1: The two branches of IndexMem++: evict and recovery both inside one softmax.
Figure 2: Differentiable gates can act at different points in attention. (a) Gates on queries, keys, values, or outputs; (b) an additive logit bias; (c) a gate on the value branch; (d) a multiplicative logit scale; and (e) a modification of the denominator alone. All keep the gate in the computation graph, but they have different effects on attention mass. A value gate suppresses a contribution without renormalizing the other weights, whereas a per-key logit bias loggj redistributes mass and recovers attention over the retained set in the binary limit.
Component
RULER 8K
Configuration
Always-attended
Loss
Head budget allocation
Avg.
NIAH
Direct instantiation
none
LM
uniform
4.33
2.00
+ always-attended entries
sinks + window
LM
uniform
44.86 (+40.53)
53.73
+ distillation
sinks + window
KL
uniform
58.86 (+14.00)
72.78
+ head budget
sinks + window
KL
mass
67.60 (+8.74)
86.17
Dense backbone (no eviction)
—
—
—
70.01
86.95
Table 1: Each component of the paradigm, added one at a time. Qwen3-0.6B, RULER 8K, cache budget 2048 of 8192 entries ( 75% compress ratio). Every arm trains for 100 steps and differs from the row above in the single field marked in bold.
Component
Choice
Avg.
NIAH
Δ
Always-attended
none
4.72
2.00
−62.87
sinks only
0.57
0.00
−67.03
window only
57.81
72.71
−9.79
sinks + window †
67.60
86.17
—
Loss
LM loss (CE)
51.50
64.65
−16.09
LongCE
53.01
61.99
−14.59
Table 2: Which choice, within each component. Every arm is the reference recipe ( † ) with a single field changed, and Δ is its Avg. against that reference. Two seeds of the reference itself differ by 0.28 , so a Δ of that size is not a difference. Setup and per-task scores are in Appendix F .
RULER 8K
Loss
Trained
Tokens
Evict 75%
Full KV
Compression gap
reverse KL
gate only †
7M
67.60
70.01
−2.41
reverse KL
gate + backbone
7M
62.83 (−4.77)
71.65
−8.82
LM loss
gate + backbone
7M
60.45 (−7.15)
69.07
−8.62
LM loss
gate + backbone
79M
59.68 (−7.92)
69.26
−9.57
LM loss
gate + backbone
197M
57.42 (−10.18)
69.24
−11.82
Table 3: Train the backbone with gate decalibrates the gate rather than damage the model.
Figure 3: Scorer architectures. All variants map xj=Norm(hj) to per-head retention scores and differ only in what they read besides xj . (a) A linear readout and (b) the token-local MLP used as the reference scorer see the current hidden state alone. The three history variants add one branch that is summed into the MLP’s pre-activation: (c) a depthwise causal convolution over the previous eight tokens, (d) a gated recurrence carrying a fixed-size state Sj−1 , and (e) a single-head causal attention over the prefix up to j . The added branch is initialized to zero, so each variant starts exactly from (b).
Figure 4: Recovery architectures. Linear-state and TTT memories add recovery after attention; centroid KV competes with retained entries within the same softmax.
Model & Benchmark
K
No slots
64 slots
Qwen3-0.6B Math500
512
48.00
52.00 (+4.00)
Qwen3-8B RULER 8K
2048
76.65
77.15 (+0.49)
Table 4: Centroid recovery at matched budgets. The total cache budget K includes recovery slots.
Qwen3-4B (8K)
Qwen3-4B (16K)
Qwen3-8B (8K)
Qwen3-8B (16K)
Method
50%
75%
50%
75%
50%
75%
50%
75%
Full-KV
92.41
92.41
91.64
91.64
93.86
93.86
92.96
92.96
SnapKV
54.24
30.85
61.61
35.80
54.32
35.13
62.74
41.54
KeyDiff
73.72
62.31
70.00
61.99
77.35
67.14
74.44
66.87
DMA
68.18
36.34
64.69
33.30
75.22
41.76
71.78
36.19
TrimKV
86.38
75.42
86.40
76.29
87.23
80.26
85.91
79.09
Table 5: RULER overall results. Results for Qwen3-4B and Qwen3-8B with 8K and 16K contexts at compression ratios of 50% and 75%. Full-KV uses the uncompressed cache, so its score is repeated in both compression columns. The Qwen3-4B checkpoint of KVpop is not publicly available.
Model
Method
RULER Tasks (8K)
S1
S2
S3
MK1
MK2
MK3
MV
MQ
VT
CWE
FWE
QA1
QA2
Avg.
Qwen3-4B
Full-KV
100.00
100.00
100.00
99.80
99.80
99.80
99.85
99.70
99.96
84.56
86.27
75.20
56.40
92.41
SnapKV
70.20
22.20
2.40
15.40
8.00
3.40
14.05
13.85
38.36
62.18
89.00
29.60
32.40
30.85
KeyDiff
100.00
88.60
92.00
74.60
4.60
0.00
89.90
89.50
100.00
69.14
74.07
15.00
12.60
62.31
DMA
99.00
24.20
2.40
27.40
5.40
0.00
14.30
20.70
79.60
48.16
71.47
47.80
32.00
36.34
TrimKV
100.00
98.80
92.60
95.00
48.40
10.80
91.55
95.60
99.96
77.00
71.60
49.60
49.60
75.42
Table 6: RULER task-level results at 8K and a compression ratio of 75%. Qwen3-4B and Qwen3-8B on all 13 RULER tasks.
Method
Compression ratio
Single QA
Multi QA
Summ.
Few-shot
Synth.
Code
Avg.
Full-KV
41.96
48.58
27.61
65.40
50.30
62.47
47.39
SnapKV
50%
35.60
46.52
25.93
65.26
50.00
63.10
45.39
KeyDiff
50%
28.87
28.91
24.46
62.27
45.61
45.61
36.91
DMA
50%
38.87
42.72
26.53
64.33
35.25
55.13
42.25
TrimKV
50%
41.93
46.29
27.20
67.81
51.75
56.35
46.54
IndexMem++
50%
40.82
49.06
27.64
65.27
52.61
62.32
47.53
Table 7: LongBench results on Qwen3-8B. Category means and the average over 15 tasks, at 50% and 75% compression.
Math500
AIME25
Method
K=1024
K=512
K=2048
K=1024
Full-KV
0.832 (5,258)
0.700 (17,845)
KVzap
0.792 † (6,326)
0.548 † (9,850)
0.446 (22,019)
0.183 (29,013)
Fast-KVzip
0.736 † (6,974)
0.480 † (11,146)
0.375 (23,904)
0.154 (29,462)
R-KV
0.811 (9,741)
0.605 (12,757)
0.229 (27,044)
0.121 (29,625)
TriAttention
0.863 (8,563)
0.758 (12,056)
0.383 (23,109)
0.192 (28,507)
Table 8: Mathematical reasoning with Qwen3-8B. Each cell reports pass@1, with the mean number of generated tokens shown in smaller gray text.
Figure 5: Decode throughput versus input length for Qwen3-4B and Qwen3-8B. Full-KV* is the Full-KV reference measured under the protocol of Appendix D .
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Gate location
Form
Physical meaning
Redistributes mass?
Query
qi←giQ⊙qi
retrieval mode
Yes
Key
kj←gjK⊙kj
address / retrievability
Yes, non-monotonically
Value
vj←gjV⊙vj
returned content
No
SDPA output
oi←giO⊙oi
whether retrieval is used
No
Logit, multiplicative
sij←γjsij
competition sharpness
Non-monotonically
Denominator only
Zi←Zi+ci
mass leaves the row
No
Appendix
Table 9: Gates at different locations in attention have different semantics. Among per-entry retention mechanisms, an additive logit gate provides monotonic control over normalized attention mass: lowering one entry’s bias lowers its weight and transfers mass to the others. A scalar key gate is the multiplicative logit gate, since gjKqi⊤kj rescales the logit, and its effect on an entry’s weight depends on the sign of that logit. A denominator gate removes mass from the row without assigning it to any entry.
Bias family
bij
Canonical form
Frozen gate ranking?
Fold width
Per-key scalar
ϕ(j)
ϕ(j)
Yes
1
Separable lifetime
ci−cj
−cj
Yes
1
Key-specific decay
sj+refi−jlogβj
depends on i
No
2
Pairwise cumulative
∑j<k<iSkj
depends on i
No
high
Query-conditioned
f(qi,kj)
depends on i
No
–
Appendix
Table 10: Additive in-softmax biases viewed through their canonical form. The second row includes forms such as the linear part of ALiBi and cumulative forgetting that can be written as a difference of prefix potentials; the third captures token-specific lifetime decay as used by methods such as TrimKV and by the per-key retention rate of Equation 6 ; the fourth and fifth capture richer pairwise or query-conditioned biases. Only biases whose canonical form is key-only induce a persistent ordering; the remaining rows can change the ordering of memories over time.
Method
Exact support
Compressed history and write policy
Readout
Joint
Infini-attention
Current segment
Associative state over completed segments
Learned convex gate between two outputs
No
RAttention
Recent SWA window
Linear state read only for out-of-window tokens
Independently RMS-normalized sum
No
Tensor Cache
Recent SWA window
Fast-weight state updated on FIFO eviction
Scalar-gated additive residual
No
NHA
Recent SWA window
Fixed recurrent KV slots
Concatenate slots and local KV
Yes
Key-Value Means
Recent block window
Overflow tokens merged into compressed KV rows
Single softmax with regional temperatures
Yes
CentroidKV
Sink, recent, and unmerged KV
Similar KV entries merged into degree-weighted centroids
Single softmax with lognr bias
Yes
Appendix
Table 11: Representative mechanisms for retaining information that is no longer stored as individual exact KV entries. “Joint” denotes a single softmax over the exact and compressed supports.
Benchmark
Budget K
Allocator
Floor
Sink / local
CMP
RULER
⌊(1−c)N⌋
mass
K/4
4 / 128
64
LongBench
⌊(1−c)N⌋
mass
K/4
4 / 128
64
Math500
512, 1,024
static
K/4
4 / 128
0
AIME25
1,024, 2,048
static
K/4
4 / 128
0
Appendix
Table 12: IndexMem++ inference configuration per benchmark. One trained gate is used throughout; the budget K is the average per-KV-head total support and is inclusive of every protected entry and centroid slot listed here.
Table 17: Complete LongBench results on Qwen3-8B. 15 tasks (3,550 examples), at compression ratios 50% and 75%.
Method
Retention score
Protected region (within budget)
SnapKV
Observed-attention mean with width-5 average pooling
Final 64 context positions
KeyDiff
Cosine-based post-RoPE key scoring
Final context position
DMA
Query-independent value-derived gate score
Sink 4, local 128
TrimKV
Released gate deltas, magnitude and lifetime
Sink 4, local 128
KVpop
Recurrent scorer state
Sink 4, local 256
IndexMem++
Learned gate on hj ; mass allocation across heads
Sink 4, local 128, CMP 64
Appendix
Table 18: RULER retention criteria and protected regions. Budgets follow the accounting above; the protected region of each method is funded within its budget rather than added on top.
NLL
KL from Full-KV
Full-KV
IndexMem++
Recency
IndexMem++
Recency
AIME25
0.197
0.207
0.279
0.021
0.082
SWE
1.068
1.221
1.551
0.120
0.630
Appendix
Table 19: Prediction on fixed continuations with Qwen3-8B at K=2,048 , averaged over eight traces per domain. Lower is better.
Learned KV-cache eviction often faces a soft-to-hard mismatch: during training, differentiable gates typically attenuate token contributions, whereas inference saves memory only when KV entries are physically removed. We ask whether the attention substrate affects this soft-to-hard transition. Using GPT-2-scale Transformers trained on OpenWebText, we run a controlled 2×2×2 comparison over attention type, learned gating, and positional encoding. Although sigmoid attention is worse as a dense language model, learned hard eviction changes the useful operating points: sigmoid-gated models delete KV entries with negligible PPL change relative to their own no-eviction references. Under a matched live-cache protocol on the same dense backbones, learned sigmoid gates obtain lower PPL than our H2O and KeyDiff implementations, whereas softmax gates do not uniformly beat these post-hoc methods. The results suggest that attention normalization can substantially affect whether a training-time soft gate transfers cleanly to hard KV deletion.
Isaac, Li
Rucheng · University of Pittsburgh, Pittsburgh, PA, USA.
Training-free KV-cache compression methods such as SnapKV, H2O, and PyramidKV evict tokens at the end of prefill, aiming to preserve the attention mass that future queries are expected to use -optimizing what to keep. We show that this objective fails in two distinct ways. (1) Compensation: restoring the evicted attention mass can recover the attention-level target without recovering task quality. (2) Selection: covering more of the true decode-query mass can hurt quality when the recovered mass is fragmented rather than concentrated in coherent spans. These failures share a common cause: eviction occurs before the queries that determine the answer trajectory exist. We propose Draft-Guided Eviction (DGE), which defers eviction until after drafting the first k=2 answer tokens using the full cache - just one decode step beyond prefill. Because the draft is generated from the answer's own prefix, no cache entries are discarded before this trajectory signal becomes available. The per-head cache budget remains unchanged, and DGE can be applied directly to SnapKV, PyramidKV, H2O, and StreamingLLM without modifying their eviction scores. Unlike extra-pass methods, DGE changes when eviction occurs rather than what cache entries are selected. Extensive experiments demonstrate that DGE outperforms prior methods at every evaluated budget on five of six instruct-tuned backbones, achieving 44.2 on LongBench, nearly matching FullKV at 44.3. The timing-only control DGE-W achieves the same score, demonstrating that the gain comes from when eviction occurs rather than what is selected - an effect we term trajectory anchoring.
KV-cache eviction can do more than compress. In long-context LLMs, keeping only some cached tokens sometimes matches or exceeds full-cache accuracy, because many redundant prefill tokens otherwise dilute attention away from the tokens that carry the answer. This benefit is not uniform, and evicting the wrong tokens can drop accuracy to zero on tasks that require precise retrieval, so the useful question is not only which tokens to keep but also whether to evict this input at all. We show that one label-free number computed from the prefill attention, the drop between early and late layers in how much attention heads agree on which tokens to read, predicts per input, before any decoding, which of the two cases an input falls under. We build this into PAGE (Partition-Aware Gated Eviction), a wrapper that runs any SnapKV-style evictor when the drop is large and keeps the full cache when it is small, with no training, labels, or fine-tuning. PAGE is a safety mechanism rather than a compressor, so we measure it by the failures it prevents. It cuts the harm rate on capacity-bound inputs from 0.75 to 0.026, and on multi-key retrieval with Mistral-7B plain SnapKV falls from 99% to 0% as the budget shrinks, while PAGE holds it at 89%. Elsewhere, it passes the base evictor through unchanged, which is the intended behaviour and is what we observe in 8 of 16 cells. Code is available at https://anonymous.4open.science/r/PAGE-018239.
Pankaj Kumar, Subhankar Mishra
School of Computer Sciences National Institute of Science Education and Research An OCC of Homi Bhabha National Institute, India