cs.AINov 6, 2026

Intent-Aware Caching for Efficient LLM Serving

Authors: Kazi Hasan Ibn Arif, JinYi Yoon, Dimitrios S. Nikolopoulos, Hans Vandierendonck, Deepu John, Bo Ji

Organizations: Virginia Tech Blacksburg, VA, USA · Inha University Incheon, South Korea · Queen’s University Belfast Belfast, UK · University College Dublin Dublin, Ireland

Abstract

Modern LLM serving systems use prefix caching to accelerate multi-turn conversational workloads. They cache the key-value states of previous conversation prefixes and reuse them when subsequent requests share the same prefix, to avoid redundant prefill computation and reduce serving latency. However, achieving a high cache hit rate remains challenging because cache reuse depends on whether a conversation continues and how soon the next turn arrives. Our analysis of real workloads shows that user intent is a useful signal for this behavior, as different intents exhibit different continuation probabilities and inter-turn delays. Based on this key insight, we propose Intent-Aware Caching (IAC), an online eviction policy that classifies requests by intent, learns lightweight reuse statistics from recent serving logs and leverages these statistics to guide an oracle-inspired eviction strategy. Extensive evaluation on real workloads (WildChat and ShareChat) shows that IAC reaches a 34.6% cache hit rate. This substantially outperforms both the previous learning-based approach (18.8%) and traditional methods such as least recently used (LRU, 12.2%), while reducing average serving latency by up to 46.1%.

Figures & tables

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. WAKMan: Workload Aware KVCache Manager

    Oct 26, 2026Suhas Kumar Morisetty, Vaibhavi Lokegaonkar, Sachin Mathew Varghese +1KV-Cache ManagementKV-Cache Eviction

  2. Multi-Turn Semantic Caching for Low-Cost LLM Serving

    Oct 31, 2026Baran Atalar, Selly Beyene, Xutong Liu +1Semantic CachingLLM Serving

  3. Not All Tokens Are Worth Caching: Learning Semantic-Aware Eviction for LLM Prefix Caches

    May 12, 2026Shaoke Fang, Ziang Li, Wenfei Wu +3LLM Inference EfficiencyKV Caching