KV-Cache Compression

Key-value cache compression reduces the memory and bandwidth cost of storing attention states, enabling longer contexts, larger batches, and inference on constrained hardware. Common approaches include token eviction and merging, quantization, low-rank or latent representations, and adaptive per-layer or per-head budgets; some methods also reconstruct or retrieve compressed states on demand. Work is evaluated on long-context retrieval and question answering benchmarks such as LongBench, RULER, and Needle-in-a-Haystack, as well as reasoning, multimodal tasks, and end-to-end serving. The central tension is that aggressive compression can harm accuracy, evidence use, or safety, while a high compression ratio does not guarantee faster inference or lower peak memory, so workload-aware evaluation remains essential.