LLM agents drive inference engines with closed-loop sessions whose context grows every turn, and serving them economically depends on prefix caching: a hit costs an incremental prefill of about a thousand tokens, a miss recomputes a 10--40K-token context. We show that this makes prefix-cached agent serving metastable. With an identical base workload, a few minutes of extra arrivals leaves a vLLM server in a degraded regime--a ~10% hit ratio, about half the throughput, and a backlog that grows for the remaining 70 minutes--while its no-burst twin stays healthy (9 of 12 pairs; none in the reverse direction). The regime appears at loads the cache-warm server sustains, is absent without prefix caching, and also appears with Qwen3-32B on SWE-bench agent trace shapes and on SGLang, which tolerates larger triggers. Slower service keeps more sessions in the system, their working set outgrows the KV pool, and every turn becomes a recomputation; a first-order workload-balance condition, calibrated independently, brackets when the regime can drain. Getting out is the harder problem: throttling new admissions recovered 1 of 7 matched collapsed runs, while suspending admitted sessions against a capacity-based limit recovered all of them within minutes. A CPU offload tier delays the collapse but does not prevent it. A budgeter that bounds the resident KV demand of active sessions to the pool, with thresholds that need no per-workload tuning, prevented the failure in every run we tested.
Figures & tables
Figure 1: A burst that outlasts itself. Same model, same server, same base arrivals; the red run additionally receives a 5-minute injected burst (shaded). Its hit ratio and throughput never recover after the burst ends, while its blue no-burst twin stays healthy.
Experiment
Model / engine
KV pool
Base λ (/s)
Trigger
Control
Runs
Paired history dependence (§ 3 )
8B / vLLM
400K
0.035
+1.5 λ , 5 min @15
–
12 pairs
No-cache control
8B / vLLM, no cache
400K
0.0086
+16 sessions, 5 min @15
–
4 pairs + 4
Flush trigger
8B / vLLM
400K
0.04
cache flush @10
none / budgeter
5 + 3
Trace shapes
32B / vLLM
≈ 365K
0.025
2 × , 4 min @10
none / budgeter / cap
4 each
Second engine
8B / SGLang
400K
0.04
2 × 2 min; +1.5 λ 5–10 min; flush
–
4+6+3, 4 pairs
Recovery (§ 6 )
8B / vLLM
400K
0.04
2 × , 4 min @10
5 arms @20
16 seeds
Table 1: Experiments. Times are minutes from the start of a run; runs last 30–120 minutes. Synthetic sessions: 25 turns, 6K → 37K tokens (session-mix shift: 12 turns, 3K/22K start); trace shapes: mini-SWE-agent on SWE-bench, truncated at 38K tokens.
No burst (twin)
Burst
Seed
Hit
Sessions
Hit
Sessions
501
0.95
8
0.12 ∗
109
502
0.95
8
0.11 ∗
142
503
0.95
8
0.12 ∗
144
504
0.95
9
0.12 ∗
142
505
0.95
6
0.95
5
Table 2: Paired runs with an identical base workload: token-weighted hit ratio over the last 10 minutes and mean number of sessions in the system over the last 5 minutes of the 90-minute run. ∗ persistently degraded from its first degraded window after the burst to the end of the run.
Figure 2: Qwen3-32B, automatically sized KV pool, SWE-bench trace shapes, λ=0.025 (same seed): a 2 × burst from minute 10 to 14, the same burst with the budgeter, and no burst.
Figure 3: SGLang at λ=0.04 absorbs a 2 × , 2-minute burst (from minute 10) but not an extra 1.5λ for 5 or 10 minutes (from minute 15).
Figure 4: The loop in one pair (seed 501): resident demand relative to the KV pool, recomputed prompt tokens, and hit ratio, per minute. The burst (shaded) pushes demand past the pool; the twin never crosses it.
Control (from 20 min)
Recovered
Time to recover (min)
Hit (end)
Cohort done
None
0/8
–
0.12–0.13
0%
Admission-only cap
2/8
21–23
0.07–0.93
0–92%
Latency AIMD + park
5/7
14–25
0.84–0.89
50–100%
Session cap + park
7/7
≤ 6
0.95
100%
budgeter + park
7/7
< 5
0.93–0.95
100%
Table 3: Recovery when control starts at minute 20 (the 4-minute burst ran from minute 10 to 14), over runs that were collapsed at that point; denominators differ because arms share arrivals but not engine state. Time: from when control starts to the start of the first 5-minute window from which every later window has a hit ratio ≥ 0.8; “ < 5” means within the first window. Cohort: fraction of sessions arriving in the next 20 minutes that completed.
Figure 5: Recovery with no control and four controllers enabled at minute 20, one seed (identical arrivals; this seed was collapsed at minute 20 in every arm).
Figure 6: Workload shift from short to long sessions. Each point is a run: loose fixed caps collapse after the shift (bottom), tight ones make first-phase sessions wait (right); only the budgeter and the adaptive cap sit at the top left. Two seeds per configuration; points offset by ≤ 0.4 min for visibility.
Figure 7: Qwen3-32B, full KV pool, SWE-bench trace shapes: recovery when control starts at minute 25 (one seed).
Figure 8: Fraction of each arrival cohort (5-minute cohorts for 8B, 10-minute for 32B) that completed by the end of the run (minute 60 for 8B, 90 for 32B), averaged over the runs collapsed when control began. Later cohorts have less time to complete.
Loop
Same-load
Persistence
Recovery of
observed
history dependence
after trigger
admitted sessions
CONCUR Chen et al. (2026)
✓
–
–
–
KAIROS Yuan et al. (2026)
✓
–
argued
–
MISA-T Hong et al. (2026)
✓
–
–
–
VAMP Kim et al. (2026)
✓
–
–
–
Metronome Meng & Li (2026)
KV-pool exhaustion
–
–
–
Table 4: What prior agent-serving work establishes about cache-induced collapse. “Same-load history dependence”: identical subsequent load, different outcome depending on a past trigger. “Recovery of admitted”: measured recovery once collapsed, with control over already-admitted sessions. “–”: not demonstrated in the cited evaluation; CONCUR has agent pause/resume, but does not measure recovery from an established collapse.
Long-running LLM applications and autonomous agents repeatedly send growing context, making prefix caching critical for reducing prefill cost. This raises a fundamental question: do decades of traditional cache-management techniques transfer directly to agentic and conversational LLM serving? We analyze production traces comprising over 20 billion processed tokens and evaluate 14 eviction algorithms across capacity-constrained HBM and large memory-pool settings. We find that simple Least Recently Used (LRU) is surprisingly robust, offering competitive performance against sophisticated traditional policies, though substantial headroom to Belady's optimum remains. The reason is structural: prefix reuse is dominated by the steady pacing of active sessions, making recency an unusually strong predictor for eviction. Nevertheless, prefix caching diverges from traditional workloads in ways that recency ignores, notably the prevalence of single-turn requests, extreme session footprint skew, and highly non-uniform miss costs because attention computation scales with token depth. To close the gap to optimality, we establish several design principles for prefix-cache management: keeping recency as the baseline while incorporating quick demotion to filter one-hit prompts, compute-aware eviction to prioritize expensive misses, and capacity-adaptive eviction granularity. Our traces and simulator are publicly available.
Large Language Model (LLM) agents can handle increasingly complex tasks, navigating long trajectories of observations and actions. As these trajectories grow, key-value (KV) cache I/O and storage become the dominant cost. Trajectories can also exceed the LLM's maximum context length, requiring control over active cache growth. Prior work uses context engineering to reduce cache size by trimming or summarizing history. However, modifying the trajectory prefix invalidates subsequent cached states and triggers costly re-prefilling of retained context. Task accuracy can also suffer when too little context is retained to limit re-prefill costs. We propose cache engineering as a new scheme. It allows agents to directly control their active KV cache without modifying the trajectory or re-prefilling retained context. Specifically, agents can proactively use drop to exclude selected cache from future attention and repos to adjust retained keys' position encodings. We formalize these cache operations and identify history dependence as a serving challenge: identical token sequences can yield different cache states under different cache operation sequences, making prefix matching insufficient for correct cache reuse. Thus, we co-design the AKV serving engine. It provides a stateless interface compatible with existing serving APIs, while internally managing stateful cache reuse, eviction, and restoration. Experiments show that a tool-response rolling-drop cache policy improves average benchmark scores by 1.7 points and reduces average cost by 27.7%, compared with conventional threshold-triggered summarization. With AKV, it achieves up to 1.2x the output throughput and a 1.5x speedup in P95 request latency. Recursive self-improvement of cache policies further reduces cost by 7.1% over expert design. Our code is at https://github.com/thu-nics/AKV.
Tianyu Fu, Ruoxi Wang, Zihan Min +6
Tsinghua University · The Chinese University of Hong Kong
Prefix caching introduces a fundamental tradeoff in multi-agent large language model (LLM) serving: retaining a long system-prompt key-value (KV) cache for an agent accelerates future calls, yet it reduces the GPU memory available for batching concurrent requests. In multi-stage workflows, existing schedulers tend to prioritize either immediate prefix locality or overall workflow progress. However, under a shared KV cache budget, optimizing either objective in isolation can prolong tasklevel job completion time (JCT) through downstream delays or frequent prefix replacement. To strike a balance, we here propose TOPAS, a Task-Oriented Prefix-Aware Scheduler that jointly decides which agent prefixes to keep in the cache and which requests to schedule for execution. TOPAS scores candidate post-decision states by trading off the expected reduction in each task's longest remaining service path against the near-term benefit of downstream prefix reuse, accounting for the costs of prefix movement and preemption. A task-level aging mechanism is also incorporated to prevent starvation. We implement TOPAS within the SGLang framework and assess its performance on three synthetic DAGs and two MetaGPT software-development workflows. Compared with the best performing baseline for each workload and metric, TOPAS reduces the mean/p99 JCT by up to 39.8%/49.4% on the synthetic workloads, while lowering mean JCT by 9.8% on MetaGPT-SOP and mean/p99 JCT by 22.0%/26.6% on MetaGPT-TL.
Hongqiu Ni, Han Tian, Chi Zhang +2
University of Science and Technology of China, China · Hefei University of Technology, China