cs.LGJan 25, 2026
SaveFast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction
Organizations: NAVER AI Lab.
Abstract
Efficient key-value (KV) cache management is crucial for the practical deployment of large language models (LLMs), yet existing compression techniques often incur a trade-off between performance degradation and computational overhead. We propose a novel gating-based KV cache eviction method for frozen-weight LLMs that achieves high compression ratios with negligible computational cost. Our approach introduces lightweight sink-attention gating modules to identify and retain critical KV pairs, and integrates seamlessly into both the prefill and decoding stages. The proposed gate training algorithm relies on forward passes of an LLM, avoiding expensive backpropagation, while achieving strong task generalization through a task-agnostic reconstruction objective. Extensive experiments across the Qwen2.5-1M, Qwen3, and Gemma3 families show that our method maintains near-lossless performance while evicting up to 70% of the KV cache. The results are consistent across a wide range of tasks, including long-context understanding, code comprehension, and mathematical reasoning, demonstrating the generality of our approach.
Figures & tables
Figure 1: Highlighted results. (a) KV cache compression performance on OpenAI MRCR [ 36 ] compared to state-of-the-art methods using Qwen2.5-7b-1M [ 52 ] . (b) Chunked-prefill efficiency for 170K tokens on Qwen2.5-7b-14M (BF16 precision) with 30% KV cache budget.
| Method | Attention | Store KV feature |
|---|---|---|
| MoD [ 40 ] | Conditional | Conditional |
| MoBA [ 35 ] | Always | Always |
| Fast KVzip | Always | Conditional |
Table 1: Gating strategies for Transformer attention. The table below indicates operation execution per layer for a given input. Mixture-of-Depths (MoD) uses gating to skip attention computation, while Mixture-of-Block-Attention (MoBA) applies gating to select blocks for sparse attention with stored KV features. Our method bridges these approaches by conducting an attention operation for every input token while improving computational and memory efficiency through gated KV eviction.
Figure 2: Computational flow. (a) Illustration of gating at each attention layer. (b) During each forward pass, we jointly calculate importance scores and evict low-importance KV pairs. For decoding, we maintain a small buffer to cache recent hidden states, performing gating and eviction in parallel once the buffer is full. This parallelized computation reduces latency overhead. We utilize a chunk size of 16K for prefill and a buffer size of 128 tokens for decoding. Empirically, we observe that retaining a local window of recent KV features at each compression step slightly improves performance (see Section B.5 ), though our method remains robust even without the local window.
Figure 3: Gating formulation of existing KV compression methods at each attention layer. Note that we compare the core components of baseline methods, as each method jointly uses other techniques. For instance, FastGen combines its approach with cumulative attention scores [ 54 ] .
Figure 5: Gate Training Design Analysis . We evaluate gate training targets and inputs on Qwen2.5-7B-1M using a retrieval task (SCBench.KV) [ 29 ] and a contextual QA task (SQuAD) [ 39 ] . In (a), we compare different tasks to obtain target scores for gate training. For a fair comparison, we fix the gate architecture and only vary the task used to compute the maximum attention scores in our gate training ( Figure 4 ). We use FineWeb-Edu for training data, except for QA where we use LongAlpaca [ 8 ] following Huang et al. [22] .
Figure 6: Maximum attention scores received by KV features across specific tasks. We visualize layers 7, 14, and 21 of Qwen2.5-7B-1M, with other layers exhibiting similar patterns, using the gate training data from Figure 5 a .
Figure 7: (a) Architecture analysis . We compare our sink-attention, a variant without learnable denominator terms, a two-layer MLP, and a linear model, using Qwen2.5-7B-1M. For a fair comparison, we equalize the number of parameters across all architectures. (b) Prefill and decoding efficiency of the Qwen2.5-7B-1M model using a 30% KV budget ratio with PyTorch and FlashAttention-2 [ 13 ] on a single H100 GPU. Points in the plot correspond to context lengths of 160K, 240K, and 320K. In Section B.3 , we provide an end-to-end latency analysis.
Figure 8: Gate architecture. We propose a sink-attention architecture using a fixed set of keys per layer. Given a hidden state , we compute the importance scores of KV as .
| Model | Training Time (h) | Storage (GB) |
|---|---|---|
| Qwen3-8B | 0.59 | 0.18 |
| Qwen3-30B-A3B | 0.70 | 0.11 |
| Qwen3-14B | 0.83 | 0.30 |
Table 2: Training efficiency. Gate training time using a H100 GPU and gate sizes across diverse model scales. The 30B-A3B model has a smaller hidden state size than the other models, resulting in smaller gates.
Figure 11: Decoding-intensive tasks . For AIME24, we averaged scores over 16 random seeds.
Figure 12: Retention Rates per Attention Head. We apply a 36% KV cache budget to Qwen2.5-7B-1M, which has 28 layers with 4 KV heads per layer. The left side corresponds to the early layers. Heads in the middle layers tend to retain the majority of token information, whereas gates in the early layers learn to retain only a small subset of KV features.
| Head Type | Most Retained (Top ) | Most Evicted (Top ) |
|---|---|---|
| Sparse | Structural markers (. .\n\n .\n , :\n \n \s) | High-frequency tokens (, the . and to of a I) |
| Medium | High-frequency tokens (, . and the of to I in) | High-frequency tokens (the , and to of a . I) |
| Dense | High-frequency tokens (, the . and to of a I) | Whitespace tokens (the \s , a \t \s\s ( 1) |
Table 3: Gate behavior analysis. The most frequently retained and evicted tokens for each head type. The \s represents a space.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 13: Gate training and validation loss trajectory per attention layer for Qwen2.5-7B-1M. We plot loss values averaged every four layers in the model.
| KV Cache Budget | Fast KVzip (OpenR1 Math) | Fast KVzip (FineWeb-Edu) | SnapKV | R-KV |
|---|---|---|---|---|
| 4096 | 70.6% | 72.7% | 63.7% | 60.0% |
| 2048 | 53.3% | 63.1% | 42.3% | 32.3% |
| 512 | 16.7% | 13.3% | 0.8% | 0.0% |
Table 4: Effect of gate training data on AIME24 (Qwen3-8B).
Figure 14: Effect of Training Data Size. We evaluate gates trained with different amounts of training data sampled from FineWeb-Edu using Qwen2.5-7B-1M. For the other settings, we uniformly reduce the context lengths of the data.
| KV Ratio | FastKVzip Time (s) | KVzip Time (s) | FastKVzip Acc | KVzip Acc |
|---|---|---|---|---|
| 1.00 | 22.5 (1.0 ) | 69.1 | 68.2 | 68.2 |
| 0.50 | 16.8 (1.3 ) | 68.7 | 71.6 | 68.6 |
| 0.40 | 15.0 (1.5 ) | 68.4 | 66.4 | 66.0 |
| 0.30 | 13.1 (1.7 ) | 68.2 | 65.4 | 65.0 |
Table 5: End-to-end latency and accuracy on SCBench-KV with Qwen2.5-7B-1M. Acceleration ratios in parentheses are relative to vanilla (ratio 1.0). FastKVzip eliminates reconstruction overhead entirely, achieving up to 5 speedup over KVzip while maintaining comparable accuracy.
Figure 15: Effect of Gate Configuration. We evaluate gates with different numbers of sink keys and projection dimensions using Qwen2.5-7B-1M on SCBench.KV.
Figure 16: Effect of Local Window Size. We evaluate gates with a range of local window sizes, as described in Figure 2 , using Qwen2.5-7B-1M. Note that our algorithm performs eviction that all configurations have an identical total KV cache size after eviction. In our main experiment, set the local window hyperparameter as 4K tokens for prefill and 128 tokens for decoding.
| Method | Training Data | Task | Target | Gate Architecture |
|---|---|---|---|---|
| Locret [ 22 ] | Instruction-based QA | Question-answering | Attention scores | MLP |
| DMS [ 55 ] | Mathematics (or Programming) | Next-token logit distillation | End-to-end loss | Linear |
| TrimKV [ 5 ] | Mathematics (or Book) | Next-token prediction | End-to-end loss | MLP |
| Fast KVzip (Ours) | General text data | Reconstruction | Attention scores | Sink attention |
Table 6: Comparison of gating approaches for KV cache compression. While DMS and TrimKV trains separate gates for reasoning and instruction-following tasks using distinct datasets, we employ a single dataset for gate training across all models and evaluations.
Figure 17: Comparison to TrimKV [ 5 ] with prefill-intensive tasks. We evaluate the official TrimKV model with Qwen3-4B-Instruct-2507 on retrieval and contextual QA tasks in Figure 10 , and report the averaged scores. For each dataset, results are normalized with respect to full-cache performance prior to averaging. For Locret, the released models are outdated, and we replace performance comparisons with our analysis in Figure 5 .
Figure 18: Prediction Comparison of Fast KVzip to KVzip. We use Qwen2.5-7B-1M with the SCBench.En.QA dataset, which were not used during training. For each layer, we plot the averaged score over heads. The dashed line represents the threshold value that results in a 36% KV cache budget ratio.
More in this topic
KV-Cache Eviction
Key-value (KV) cache eviction reduces language-model memory use by removing or compressing cached tokens while preserving useful context.
See all 98 papers