GateSight: Recurrent Memory Anticipates Attention Memory for Query-Agnostic KV Eviction in Hybrid LLMs

Junjie Li; Jiong Lou; Jie Li

cs.AIOct 10, 2026

GateSight: Recurrent Memory Anticipates Attention Memory for Query-Agnostic KV Eviction in Hybrid LLMs

Authors: Junjie Li, Jiong Lou, Jie Li

Organizations: Shanghai Jiao Tong University

Abstract

Hybrid language models interleave Gated DeltaNet (GDN) layers, whose recurrent memory has a fixed size, with full-attention layers, whose key-value (KV) cache grows with the context and dominates serving memory. In this paper, we ask whether the recurrent memory already knows what the attention memory will need. We first show that which GDN heads of Qwen3.8-27B keep long memories is fixed by the weights rather than by the document. Next, we define the future utility of a KV entry by how much removing it changes the attention output over the model's own continuations, and find that the write gate, decay and write magnitude that GDN layers emit during prefill anticipate this utility before any query is seen. Because the relation is spread across many GDN heads and layers, and a linear map recovers its trend but less of its high-utility tail than a nonlinear one, we propose GateSight, which learns it with lightweight per-head multilayer perceptrons that read these signals and no attention score. Their predicted utility decides which entries each attention layer keeps, and how much each layer keeps is set by the model's native attention output gate, whose mean differs by more than an order of magnitude across layers. Finally, at serving time, GateSight scores the cache once after prefill and physically releases discarded KV entries. Experimental results on SCBench and RULER-4096 with Qwen3.8-27B and, with the same recipe, the sparse-attention model Qwen3.8-Flash-Next, up to the 262K-token limit of the models, show that with a tenth of the attention KV cache, GateSight is the most accurate of five query-agnostic methods on three of the four model-benchmark pairs and leads KVzip on all four, by 9.5 to 47.2 points. Notably, it scores a full-length context with negligible overhead relative to prefill.

Figures & tables

Figure 1: GDN memory dynamics of Qwen3.8-27B are structured and stable across documents. (a) Median over heads (line), 10th–90th percentile (band), global median (dashed), slow heads and threshold (orange). (b) Half-life ordering of heads, compared per document pair; dashed: 10th and 90th percentiles. (c) Per-document against global slow set, one dot per document; bars: medians.

Figure 2: Native GDN signals anticipate future KV utility

U

(Qwen3.8-27B). (a) Unqualified signals use the window mean; subscript

\mathrm{s}

: slow heads. (b) Effective write, slow versus all heads. (c)

\beta\|v\|_{\mathrm{s}}

: top/bottom utility-decile probabilities over 10% chance. (a, c) average 16 layers; (a–c) average four KV heads. (d)

n

: documents; bars: 95% bootstrap CIs over documents.

Predictor	$\rho$ ( $\uparrow$ )	Ov@10 ( $\uparrow$ )
Untrained signals
random	$+0.000$	10.0
age	$+0.071$	19.2
slow-head $\beta\\|v\\|$	$+0.229$	18.2
single-head oracle	$+0.232$	22.6
Learned, 867 features

Table 1: Predicting

U

from native GDN signals (mean over layers and heads).

\rho

: Spearman; Ov@10: top-10% overlap (%). Best in bold.

Panel of Figure 3 — Figure 3: GateSight for attention layer $\ell$ and KV head $h$ . (1) One prefill of the context. (2) A per-head MLP, trained offline, predicts the future utility of each entry. (3) Each head keeps its top- $K_{\ell}$ entries under the gate-proportional layer budget. (4) Kept entries are packed into the front pages, and the freed tail returns to the block pool.

Figure 4: Task-averaged accuracy under KV pruning: (a, b) Qwen3.8-27B and (c, d) Qwen3.8-Flash-Next, each on SCBench and RULER.

Method	Context length (tokens)
Method	4K	16K	64K	256K
GateSight	0.010	0.023	0.069	0.246
TriAttention	0.014	0.016	0.034	0.109
KeyDiff	0.004	0.007	0.023	0.084
SnapKV	0.028	0.063	0.131	0.452
KVzip	0.747	2.787	13.540	100.279

Table 2: Wall-clock scoring time (s) per context in the same vLLM pipeline, Qwen3.8-27B, by context length; median of three runs after a warm-up.

Benchmark	Method	$p=0.7$	$0.4$	$0.1$	Mean
Qwen3.8-27B
SCBench	GateSight	63.3	51.7	37.4	54.3
	KVzip	64.5	60.4	27.9	56.2
RULER	GateSight	97.4	91.9	76.6	91.5
	KVzip	97.7	95.5	29.4	83.4
Qwen3.8-Flash-Next

Table 3: Comparison with KVzip: task-averaged accuracy (%); Mean: nine kept fractions. Higher of the two methods in bold.

Allocation	$p=0.7$	$0.4$	$0.1$	Mean
SCBench
gate-proportional	72.4	57.6	39.2	60.6
flat	61.2	48.4	38.6	53.7
PyramidKV	43.8	38.7	37.8	44.3
RULER
gate-proportional	100.0	100.0	98.6	99.8

Table 4: Allocation ablation on Qwen3.8-27B: accuracy (%) over three SCBench and three RULER tasks; Mean: nine kept fractions. Best in bold.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: The example document at attention layer 55; each trace is scaled to its maximum after the first token.

Untrained signals
Predictor	$\rho$ ( $\uparrow$ )	Ov@ $k$ ( $\uparrow$ )			Cov@ $k$ ( $\uparrow$ )
Predictor	$\rho$ ( $\uparrow$ )	5	10	20	5	10	20
random	$+0.000$	5.0	10.0	19.9	14.1	19.0	27.8
age (older first)	$+0.071$	11.6	19.2	29.4	24.4	31.2	40.0
$\\|v\\|$ , preceding GDN layer	$+0.121$	7.9	13.9	24.8	19.5	24.8	34.1
$\beta$ , window mean	$+0.107$	8.9	14.9	25.4	21.2	26.0	34.6

Appendix

Table 5: Complete Table 1 . Ov@

k

and Cov@

k

: overlap with and coverage of the true top-

k

% entries (%). Best in bold.

Figure 6: Per-cell correlation within the window of attention layers 15, 31, 47 and 63 (a to d).

Figure 7: The GDN layer whose effective write correlates most with

U

(attention layers 31 to 51: layer 28; 55 and 59: layer 52).

Figure 8: Per-layer version of Table 1 .

Set	Random		Age		$\beta\\|v\\|$ , window		Slow-head $\beta\\|v\\|$		Estimator
Set	$\rho$	Ov@10	$\rho$	Ov@10	$\rho$	Ov@10	$\rho$	Ov@10	$\rho$	Ov@10
Nemotron, 1K (240 docs)	$+0.00$	9.9	$+0.09$	19.3	$+0.11$	14.5	$+0.23$	18.3	$+0.61$	46.7
Nemotron, 2K (40)	$+0.00$	10.0	$+0.19$	29.8	$+0.18$	19.0	$+0.34$	26.3	$+0.60$	45.3
Nemotron, 4K (40)	$+0.00$	10.0	$+0.10$	24.4	$+0.16$	18.0	$+0.34$	27.2	$+0.56$	41.0
Nemotron, 8K (19)	$+0.00$	9.9	$+0.23$	31.9	$+0.13$	17.9	$+0.33$	27.8	$+0.57$	41.7
RULER-8K (60 contexts)	$-0.00$	10.0	$+0.12$	24.3	$+0.21$	25.3	$+0.30$	33.2	$+0.43$	40.6

Appendix

Table 6: Across context length and task text: Spearman

\rho

with

U

and top-10% overlap (%). The estimator’s 1K row is in-sample.

Figure 9: Per-task SCBench accuracy, Qwen3.8-27B.

Figure 10: Per-task SCBench accuracy, Qwen3.8-Flash-Next.

Figure 11: Per-task RULER accuracy, Qwen3.8-27B.

Figure 12: Per-task RULER accuracy, Qwen3.8-Flash-Next.

Figure 13: Allocation ablation per task, Qwen3.8-27B: SCBench (top) and RULER (bottom).

GDN layers read	inputs	parameters	$\rho$	Ov@5	Ov@10	Cov@10
3	435	1.8M	$+0.534$	32.1	40.2	57.6
6	867	3.4M	$+0.543$	32.9	41.0	58.4
9	1,299	5.0M	$+0.547$	33.2	41.2	58.6
12	1,731	6.4M	$+0.548$	33.2	41.3	58.7

Appendix

Table 7: Estimator window sweep (Qwen3.8-27B): held-out metrics as in Table 5 , 16-layer means, with the input width and the parameter count of the 64 MLPs. Bold: the window GateSight uses.

Method	Context length (tokens)
Method	4K	16K	64K	256K
GateSight	0.004	0.011	0.043	0.175
TriAttention	0.011	0.011	0.013	0.030
KeyDiff	0.004	0.004	0.007	0.020
SnapKV	0.006	0.006	0.012	0.044
KVzip	0.566	1.880	8.384	55.436

Appendix

Table 8: Wall-clock scoring time (s) per context in the same vLLM pipeline, Qwen3.8-Flash-Next, by context length; median of three runs after a warm-up.

KV-Cache Eviction

Key-value (KV) cache eviction reduces language-model memory use by removing or compressing cached tokens while preserving useful context.

See all 99 papers

Aug 24, 2026

Sigmoid Attention as a Better Substrate for Learned KV Cache Eviction

Learned KV-cache eviction often faces a soft-to-hard mismatch: during training, differentiable gates typically attenuate token contributions, whereas inference saves memory only when KV entries are physically removed. We ask whether the attention substrate affects this soft-to-hard transition. Using GPT-2-scale Transformers trained on OpenWebText, we run a controlled

2\times2\times2

comparison over attention type, learned gating, and positional encoding. Although sigmoid attention is worse as a dense language model, learned hard eviction changes the useful operating points: sigmoid-gated models delete KV entries with negligible PPL change relative to their own no-eviction references. Under a matched live-cache protocol on the same dense backbones, learned sigmoid gates obtain lower PPL than our H

_2

O and KeyDiff implementations, whereas softmax gates do not uniformly beat these post-hoc methods. The results suggest that attention normalization can substantially affect whether a training-time soft gate transfers cleanly to hard KV deletion.

Isaac, Li

Rucheng · University of Pittsburgh, Pittsburgh, PA, USA.

KV-Cache Eviction Gated Attention

Ask PDF

Aug 5, 2026

QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding

Autoregressive large language model inference is increasingly constrained by the memory footprint of the Key-Value (KV) cache. A dominant line of work reduces this footprint by evicting tokens that appear unimportant under attention-derived scores. However, such policies make an implicit irreversible decision: once a token is evicted, it cannot become useful again. We show that this assumption is brittle during decoding. Token and window importance drift as generated queries evolve, causing standard eviction policies to permanently discard states that later receive substantial attention under the full-cache model. To characterize this behaviour, we introduce Future Missed Mass and Global LIR, two diagnostics that measure future attention assigned to discarded states and the reactivation of historically inactive regions. We propose QEvict, a three-tier KV-cache management scheme that replaces binary retain-or-delete eviction with recoverable eviction. QEvict maintains high-confidence windows in full precision, stores intermediate windows in a quantized recoverable tier, and deletes only the lowest-confidence windows. During decoding, cumulative attention scores update window importance and when a quantized window becomes important again, it is dequantized and promoted to the full-precision. Under a fixed memory budget, this design preserves broader historical context while retaining exact full precision for the most important regions. Across long-context understanding, retrieval, and reasoning benchmarks, QEvict consistently improves over representative eviction and quantization baselines, reducing missed attention and improving information retention

Ayushman Garg, Akshita Gupta, Shaswata Bhattacharya +3

Mehta Family School of Data Science and Artificial Intelligence, Indian Institute of Technology Roorkee · Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi · Department of Electrical Engineering, Indian Institute of Technology Delhi

KV Caching Long-Context Language Model Inference KV-Cache Eviction KV-Cache Management KV-Cache Quantization KV-Cache Pruning

Ask PDF

Sep 3, 2026

Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning

Large language models achieve superior performance on tasks that require extended reasoning, but long chains of thought make the KV cache a severe memory bottleneck. Existing KV cache compression methods share one paradigm: score each cached token by some estimate of how much it will matter later, and keep the top-scoring ones. We show that the selection signal contributes almost nothing. Random Attention keeps the prompt and evicts uniformly at random within each attention head, computing no score at all; across four models and six reasoning tasks, it matches the strongest baseline in task performance while delivering 32-43% higher throughput than that method when deployed with vLLM. Controlled experiments explain this by showing that 1) the prompt is the fragile part of the cache, and most of the gap between selectors is just whether their selection signal happened to keep it; 2) the reasoning trace protects itself against eviction with redundancy at two levels, in the text (the model restates what it still needs as it works) and across attention heads (each keeps its own copy of the trace), so once the prompt is safe, a random draw retains enough copies of what the model still needs, and no score is required to pick them. Our code is publicly available at https://github.com/SalesforceAIResearch/Random-Attention.

Heng Wang, Jielin Qiu, Wenting Zhao +8

Salesforce AI Research · University of Illinois Urbana-Champaign

KV Caching Memory-Efficient Optimization KV-Cache Eviction Efficient Language Model Reasoning LLM Inference Acceleration KV-Cache Compression

Ask PDF

Abstract

KV-Cache Eviction

Explore similar work

Sigmoid Attention as a Better Substrate for Learned KV Cache Eviction

QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding

Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning