KV-Cache Pruning

Key-value cache pruning studies how to shrink the attention states that language and multimodal models retain during generation, enabling longer contexts and more efficient serving. Methods score tokens using signals such as attention and recency, predicted future utility, task or reasoning-step relevance, and uncertainty; others allocate budgets across layers or regions, quantize entries, or combine pruning with specialized hardware. Work spans long-context retrieval and reasoning, multi-turn agents, GUI tasks, and video generation, with both training-free policies and learned pruning mechanisms. A central tension is that useful importance signals and cache budgets vary by task, while aggressive compression must also deliver real gains in latency and memory traffic—not just a smaller cache—without losing answer-critical evidence.