Efficient key-value (KV) cache management is crucial for the practical deployment of large language models (LLMs), yet existing compression techniques often incur a trade-off between performance degradation and computational overhead. We propose a novel gating-based KV cache eviction method for frozen-weight LLMs that achieves high compression ratios with negligible computational cost. Our approach introduces lightweight sink-attention gating modules to identify and retain critical KV pairs, and integrates seamlessly into both the prefill and decoding stages. The proposed gate training algorithm relies on forward passes of an LLM, avoiding expensive backpropagation, while achieving strong task generalization through a task-agnostic reconstruction objective. Extensive experiments across the Qwen2.5-1M, Qwen3, and Gemma3 families show that our method maintains near-lossless performance while evicting up to 70% of the KV cache. The results are consistent across a wide range of tasks, including long-context understanding, code comprehension, and mathematical reasoning, demonstrating the generality of our approach.
Figures & tables
Figure 1: Highlighted results. (a) KV cache compression performance on OpenAI MRCR [ 36 ] compared to state-of-the-art methods using Qwen2.5-7b-1M [ 52 ] . (b) Chunked-prefill efficiency for 170K tokens on Qwen2.5-7b-14M (BF16 precision) with 30% KV cache budget.
Method
Attention
Store KV feature
MoD [ 40 ]
Conditional
Conditional
MoBA [ 35 ]
Always
Always
Fast KVzip
Always
Conditional
Table 1: Gating strategies for Transformer attention. The table below indicates operation execution per layer for a given input. Mixture-of-Depths (MoD) uses gating to skip attention computation, while Mixture-of-Block-Attention (MoBA) applies gating to select blocks for sparse attention with stored KV features. Our method bridges these approaches by conducting an attention operation for every input token while improving computational and memory efficiency through gated KV eviction.
Figure 2: Computational flow. (a) Illustration of gating at each attention layer. (b) During each forward pass, we jointly calculate importance scores and evict low-importance KV pairs. For decoding, we maintain a small buffer to cache recent hidden states, performing gating and eviction in parallel once the buffer is full. This parallelized computation reduces latency overhead. We utilize a chunk size of 16K for prefill and a buffer size of 128 tokens for decoding. Empirically, we observe that retaining a local window of recent KV features at each compression step slightly improves performance (see Section B.5 ), though our method remains robust even without the local window.
Figure 3: Gating formulation of existing KV compression methods at each attention layer. Note that we compare the core components of baseline methods, as each method jointly uses other techniques. For instance, FastGen combines its approach with cumulative attention scores [ 54 ] .
Figure 5: Gate Training Design Analysis . We evaluate gate training targets and inputs on Qwen2.5-7B-1M using a retrieval task (SCBench.KV) [ 29 ] and a contextual QA task (SQuAD) [ 39 ] . In (a), we compare different tasks to obtain target scores for gate training. For a fair comparison, we fix the gate architecture and only vary the task used to compute the maximum attention scores in our gate training ( Figure 4 ). We use FineWeb-Edu for training data, except for QA where we use LongAlpaca [ 8 ] following Huang et al. [22] .
Figure 6: Maximum attention scores received by KV features across specific tasks. We visualize layers 7, 14, and 21 of Qwen2.5-7B-1M, with other layers exhibiting similar patterns, using the gate training data from Figure 5 a .
Figure 7: (a) Architecture analysis . We compare our sink-attention, a variant without learnable denominator terms, a two-layer MLP, and a linear model, using Qwen2.5-7B-1M. For a fair comparison, we equalize the number of parameters across all architectures. (b) Prefill and decoding efficiency of the Qwen2.5-7B-1M model using a 30% KV budget ratio with PyTorch and FlashAttention-2 [ 13 ] on a single H100 GPU. Points in the plot correspond to context lengths of 160K, 240K, and 320K. In Section B.3 , we provide an end-to-end latency analysis.
Figure 8: Gate architecture. We propose a sink-attention architecture using a fixed set of keys per layer. Given a hidden state h∈RD , we compute the importance scores of KV as s∈[0,1]H .
Model
Training Time (h)
Storage (GB)
Qwen3-8B
0.59
0.18
Qwen3-30B-A3B
0.70
0.11
Qwen3-14B
0.83
0.30
Table 2: Training efficiency. Gate training time using a H100 GPU and gate sizes across diverse model scales. The 30B-A3B model has a smaller hidden state size than the other models, resulting in smaller gates.
Figure 11: Decoding-intensive tasks . For AIME24, we averaged scores over 16 random seeds.
Figure 12: Retention Rates per Attention Head. We apply a 36% KV cache budget to Qwen2.5-7B-1M, which has 28 layers with 4 KV heads per layer. The left side corresponds to the early layers. Heads in the middle layers tend to retain the majority of token information, whereas gates in the early layers learn to retain only a small subset of KV features.
Head Type
Most Retained (Top 1→8 )
Most Evicted (Top 1→8 )
Sparse
Structural markers (. .\n\n .\n , :\n \n \s)
High-frequency tokens (, the . and to of a I)
Medium
High-frequency tokens (, . and the of to I in)
High-frequency tokens (the , and to of a . I)
Dense
High-frequency tokens (, the . and to of a I)
Whitespace tokens (the \s , a \t \s\s ( 1)
Table 3: Gate behavior analysis. The most frequently retained and evicted tokens for each head type. The \s represents a space.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 13: Gate training and validation loss trajectory per attention layer for Qwen2.5-7B-1M. We plot loss values averaged every four layers in the model.
KV Cache Budget
Fast KVzip (OpenR1 Math)
Fast KVzip (FineWeb-Edu)
SnapKV
R-KV
4096
70.6%
72.7%
63.7%
60.0%
2048
53.3%
63.1%
42.3%
32.3%
512
16.7%
13.3%
0.8%
0.0%
Appendix
Table 4: Effect of gate training data on AIME24 (Qwen3-8B).
Figure 14: Effect of Training Data Size. We evaluate gates trained with different amounts of training data sampled from FineWeb-Edu using Qwen2.5-7B-1M. For the other settings, we uniformly reduce the context lengths of the data.
KV Ratio
FastKVzip Time (s)
KVzip Time (s)
FastKVzip Acc
KVzip Acc
1.00
22.5 (1.0 × )
69.1
68.2
68.2
0.50
16.8 (1.3 × )
68.7
71.6
68.6
0.40
15.0 (1.5 × )
68.4
66.4
66.0
0.30
13.1 (1.7 × )
68.2
65.4
65.0
Appendix
Table 5: End-to-end latency and accuracy on SCBench-KV with Qwen2.5-7B-1M. Acceleration ratios in parentheses are relative to vanilla (ratio 1.0). FastKVzip eliminates reconstruction overhead entirely, achieving up to 5 × speedup over KVzip while maintaining comparable accuracy.
Figure 15: Effect of Gate Configuration. We evaluate gates with different numbers of sink keys S and projection dimensions D′ using Qwen2.5-7B-1M on SCBench.KV.
Figure 16: Effect of Local Window Size. We evaluate gates with a range of local window sizes, as described in Figure 2 , using Qwen2.5-7B-1M. Note that our algorithm performs eviction that all configurations have an identical total KV cache size after eviction. In our main experiment, set the local window hyperparameter as 4K tokens for prefill and 128 tokens for decoding.
Method
Training Data
Task
Target
Gate Architecture
Locret [ 22 ]
Instruction-based QA
Question-answering
Attention scores
MLP
DMS [ 55 ]
Mathematics (or Programming)
Next-token logit distillation
End-to-end loss
Linear
TrimKV [ 5 ]
Mathematics (or Book)
Next-token prediction
End-to-end loss
MLP
Fast KVzip (Ours)
General text data
Reconstruction
Attention scores
Sink attention
Appendix
Table 6: Comparison of gating approaches for KV cache compression. While DMS and TrimKV trains separate gates for reasoning and instruction-following tasks using distinct datasets, we employ a single dataset for gate training across all models and evaluations.
Figure 17: Comparison to TrimKV [ 5 ] with prefill-intensive tasks. We evaluate the official TrimKV model with Qwen3-4B-Instruct-2507 on retrieval and contextual QA tasks in Figure 10 , and report the averaged scores. For each dataset, results are normalized with respect to full-cache performance prior to averaging. For Locret, the released models are outdated, and we replace performance comparisons with our analysis in Figure 5 .
Figure 18: Prediction Comparison of Fast KVzip to KVzip. We use Qwen2.5-7B-1M with the SCBench.En.QA dataset, which were not used during training. For each layer, we plot the averaged score over heads. The dashed line represents the threshold value that results in a 36% KV cache budget ratio.
The increasing memory demand of the Key-Value (KV) cache poses a significant bottleneck for Large Language Models (LLMs) in long-context applications. Existing low-rank KV compression methods reduce this footprint by modifying model projections, limiting the flexibility to switch back to standard full-cache inference when sufficient memory is available. In this paper, we propose EchoKV, a flexible KV cache compression framework that supports on-demand transitions from full KV caching to compressed caching. Unlike traditional compression-decompression paradigms, EchoKV utilizes a lightweight network to reconstruct the discarded KV components from a partial subset, exploiting intrinsic inter-layer and intra-layer similarities among attention heads. We further introduce a lightweight two-stage fine-tuning strategy, requiring only a few minutes on a single A100 GPU for a 7B model. Experimental results on LongBench and RULER demonstrate that EchoKV consistently outperforms existing methods across multiple compression ratios and backbone models while preserving the throughput of full-cache inference in short-context scenarios.
Shiyu Ji, Yixuan Wang, Yijun Liu +2
Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology, China
Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware. Existing KV cache eviction methods typically apply heuristic token scoring over all heads in GQA-based LLMs. These methods ignore the different functionalities of attention heads, leading to the eviction of critical tokens and thus degrading the performance of LLMs. To address this issue, we propose CompressKV, a resource-efficient KV-cache compression framework for GQA-based LLMs. Instead of aggregating attention scores from all heads, CompressKV identifies Semantic Retrieval Heads (SRHs) that capture both the initial and final tokens of a prompt and semantically important mid-context evidence, and uses them to select tokens whose KV pairs should be retained. Furthermore, CompressKV allocates cache budgets across layers according to offline estimates of layer-wise eviction error. Experiments on LongBench and Needle-in-a-Haystack show that CompressKV consistently outperforms existing KV-cache eviction methods across memory budgets. Notably, it preserves over 97% of full-cache performance using only 3% of the KV cache on LongBench question-answering tasks and achieves 90% accuracy with just 0.7% KV storage on Needle-in-a-Haystack. These results demonstrate an improved resource--performance trade-off for long-context LLM inference. Our code is publicly available at: https://github.com/TUDa-HWAI/CompressKV
Xiaolin Lin, Jingcun Wang, Olga Kondrateva +3
Technical University of Darmstadt, Darmstadt, Germany · University of Notre Dame, Notre Dame, IN, USA · Technical University of Ilmenau, Ilmenau, Germany
Long-context large language models (LLMs) have demonstrated strong capabilities across a wide range of tasks, but the growing KV cache introduces substantial memory and inference overhead. Existing one-shot KV cache compression methods typically commit to irreversible eviction immediately after prefill, before any signal from actual generation becomes available. Our quantitative analysis shows that early queries from the actual generation stage provide attention signals that are more consistent with subsequent decode attention, with the largest single-step gain occurring at the prefill-decode boundary. Based on this observation, we propose DeferKV, which moves the eviction decision from the end of prefill to the first real decoding step and temporally combines prompt-side and decode-side observations, thereby better aligning KV importance estimation with subsequent generation requirements. DeferKV requires no additional training, draft model, or future-query prediction module, making it simple and easy to deploy. Experiments on LongBench, RULER, and Needle-in-a-Haystack demonstrate that DeferKV consistently improves model performance under KV cache compression while maintaining low inference latency.
Zhe Wang, Jiakai Li, Yujia Sun +2
University of Electronic Science and Technology of China, Chengdu, China · Ubiquitous Intelligence and Trusted Services Key Laboratory of Sichuan Province, Chengdu, China