Prefix Caching

Prefix caching stores the key-value (KV) states computed for a prompt prefix and reuses them when later requests share that prefix, reducing repeated prefill work and time-to-first-token. Serving systems use radix-tree matching, cache-aware ordering and routing, and scheduling policies to exploit reuse in settings such as retrieval-augmented generation and multi-agent workloads. Eviction policies and GPU, host-memory, or disk offloading manage limited cache capacity; hybrid recurrent models can use checkpoints or suffix replay to reconstruct reusable state. The main tension is that cache hits improve efficiency only when reused state remains correct and useful, while memory pressure, changing workloads, load imbalance, and privacy risks can limit sharing or affect quality.