KV-Cache Eviction

Key-value cache eviction studies how to shrink the tokens stored during generation so long-context models fit in limited GPU memory and serve requests faster. Common approaches rank tokens using attention, recency, value magnitude, or redundancy; newer methods estimate future or output-level utility, allocate budgets across layers and heads, or combine eviction with quantization and recoverable memory. Methods are commonly evaluated on long-context understanding, retrieval, and reasoning benchmarks such as LongBench, RULER, and Needle-in-a-Haystack, alongside latency and memory measurements. The central tension is that cheap importance signals can miss information needed later, while more accurate prediction or reconstruction may add computation; results also depend on workload, model, hardware, and the rigor of matched-budget evaluation.