Sigmoid Attention as a Better Substrate for Learned KV Cache Eviction

Isaac; Li

cs.LGAug 24, 2026

Sigmoid Attention as a Better Substrate for Learned KV Cache Eviction

Authors: Isaac, Li

Organizations: Rucheng · University of Pittsburgh, Pittsburgh, PA, USA.

Abstract

Learned KV-cache eviction often faces a soft-to-hard mismatch: during training, differentiable gates typically attenuate token contributions, whereas inference saves memory only when KV entries are physically removed. We ask whether the attention substrate affects this soft-to-hard transition. Using GPT-2-scale Transformers trained on OpenWebText, we run a controlled $2\times2\times2$ comparison over attention type, learned gating, and positional encoding. Although sigmoid attention is worse as a dense language model, learned hard eviction changes the useful operating points: sigmoid-gated models delete KV entries with negligible PPL change relative to their own no-eviction references. Under a matched live-cache protocol on the same dense backbones, learned sigmoid gates obtain lower PPL than our H $_2$ O and KeyDiff implementations, whereas softmax gates do not uniformly beat these post-hoc methods. The results suggest that attention normalization can substantially affect whether a training-time soft gate transfers cleanly to hard KV deletion.

Figures & tables

Backbone	Dense PPL
SM RoPE	22.079
Sig RoPE	22.440
SM NoPE	23.398
Sig NoPE	24.603

Table 1: Dense no-eviction test PPL.

Figure 1: Validation threshold sweep. Each point shows the validation PPL and realized hard-KV compression rate for one threshold

\tau

. Stars mark the selected

\tau^{\star}

for each gated model.

Model	$\tau^{\star}$	$\Delta\text{PPL}$	Compression
SM+G RoPE	0.010	$+0.077$	$18.7\%$
Sig+G RoPE	0.050	$-0.001$	$18.8\%$
SM+G NoPE	0.010	$+0.016$	$\phantom{0}9.7\%$
Sig+G NoPE	0.050	$+0.033$	$31.5\%$

Table 2: Validation-selected operating points. Compression is hard KV-entry deletion during prefill and decode.

Model	$\tau^{\star}$	Comp.	Learned PPL	H 2 O	KeyDiff
SM+G RoPE	0.010	$19.2\%$	22.514	22.145	22.984
Sig+G RoPE	0.050	$19.8\%$	22.424	22.480	22.474
SM+G NoPE	0.010	$10.2\%$	23.576	23.423	23.588
Sig+G NoPE	0.050	$32.2\%$	24.637	25.107	25.786

Table 3: Frozen-threshold test operating points. H 2 O and KeyDiff are run on the corresponding dense backbone at the same final live-cache size as the learned gate.

Figure 2: Random support-removal sensitivity on dense RoPE models. For each drop rate

p

, we randomly remove K/V support and measure the relative perturbation of each layer’s attention output.

Figure 3: Per-layer mechanism evidence. Left: gate binarization. Right: coefficient of variation of attention row sums.

Model	content	stopword	punct.	newline	whitesp.	digit
SM+G RoPE	0.213	0.144	0.224	0.170	0.274	0.196
Sig+G RoPE	0.272	0.184	0.270	0.176	0.328	0.235
SM+G NoPE	0.223	0.144	0.195	0.135	0.334	0.224
Sig+G NoPE	0.281	0.178	0.254	0.273	0.455	0.263

Table 4: Mean gate value by token type on the test split. Buckets are assigned from the GPT-2 decoded surface form.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

$\tau$	SM+G RoPE	Sig+G RoPE	SM+G NoPE	Sig+G NoPE
0.000	19.62 / 0.0%	19.48 / 0.0%	20.24 / 0.0%	21.12 / 0.0%
0.005	19.63 / 13.4%	19.48 / 14.2%	20.24 / 6.5%	21.11 / 27.3%
0.010	19.69 / 18.7%	19.48 / 14.6%	20.26 / 9.7%	21.11 / 28.4%
0.020	19.96 / 25.3%	19.48 / 15.6%	20.36 / 13.7%	21.11 / 29.3%
0.050	21.03 / 32.9%	19.48 / 18.8%	21.57 / 21.8%	21.16 / 31.5%
0.100	23.13 / 40.7%	26.02 / 24.8%	995.48 / 34.6%	22.35 / 36.6%

Appendix

Table 5: Full validation threshold sweep.

Figure 4: Gate heatmap on a narrative prompt.

Figure 5: Gate heatmap on a technical prompt.

Figure 6: Gate heatmap on a dialogue prompt.

Figure 7: Gate heatmap on a factual prompt.

Figure 8: Gate heatmap on a question-answering prompt.

KV-Cache Eviction

Key-value (KV) cache eviction reduces language-model memory use by removing or compressing cached tokens while preserving useful context.

See all 99 papers

GateSight: Recurrent Memory Anticipates Attention Memory for Query-Agnostic KV Eviction in Hybrid LLMs
Oct 10, 2026Junjie Li, Jiong Lou, Jie LiKV-Cache Eviction KV-Cache Compression
Shanghai Jiao Tong University
ValueDiff: Value-Geometric KV Cache Eviction for Sink-Suppressed LLMs
Sep 20, 2026Junyoung Park, Jungwook Choi, Mingu LeeLLM Inference Efficiency KV-Cache Eviction
Qualcomm AI Research · Hanyang University
Jacap: Robust KV Cache Eviction via Jacobian-Based Nonlinear Information Capacity Preservation
Sep 8, 2026Jiaming Yang, Chenwei Tang, Liangli Zhen +2KV Caching KV-Cache Eviction
College of Computer Science, Sichuan University, Chengdu, China · 2Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education, Chengdu, China · Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR), Singapore

Oct 10, 2026J/K move · Enter open · S save

GateSight: Recurrent Memory Anticipates Attention Memory for Query-Agnostic KV Eviction in Hybrid LLMs

Junjie Li, Jiong Lou, Jie Li

Shanghai Jiao Tong University

KV-Cache Eviction KV-Cache Compression Hybrid Attention

Open paper AskPDF arXiv

Hybrid language models interleave Gated DeltaNet (GDN) layers, whose recurrent memory has a fixed size, with full-attention layers, whose key-value (KV) cache grows with the context and dominates serving memory. In this paper, we ask whether the recurrent memory already knows what the attention memory will need. We first show that which GDN heads of Qwen3.8-27B keep long memories is fixed by the weights rather than by the document. Next, we define the future utility of a KV entry by how much removing it changes the attention output over the model's own continuations, and find that the write gate, decay and write magnitude that GDN layers emit during prefill anticipate this utility before any query is seen. Because the relation is spread across many GDN heads and layers, and a linear map recovers its trend but less of its high-utility tail than a nonlinear one, we propose GateSight, which learns it with lightweight per-head multilayer perceptrons that read these signals and no attention score. Their predicted utility decides which entries each attention layer keeps, and how much each layer keeps is set by the model's native attention output gate, whose mean differs by more than an order of magnitude across layers. Finally, at serving time, GateSight scores the cache once after prefill and physically releases discarded KV entries. Experimental results on SCBench and RULER-4096 with Qwen3.8-27B and, with the same recipe, the sparse-attention model Qwen3.8-Flash-Next, up to the 262K-token limit of the models, show that with a tenth of the attention KV cache, GateSight is the most accurate of five query-agnostic methods on three of the four model-benchmark pairs and leads KVzip on all four, by 9.5 to 47.2 points. Notably, it scores a full-length context with negligible overhead relative to prefill.

Abstract

KV-Cache Eviction

Explore similar work

GateSight: Recurrent Memory Anticipates Attention Memory for Query-Agnostic KV Eviction in Hybrid LLMs

ValueDiff: Value-Geometric KV Cache Eviction for Sink-Suppressed LLMs

Jacap: Robust KV Cache Eviction via Jacobian-Based Nonlinear Information Capacity Preservation