Silent and Costly: Characterizing and Detecting Prompt-Caching Mis-Integration in LLM Agents
Authors: Ying Zhang, Ruoqi Guo, Yi Liu, Gelei Deng, Yuekang Li, Yepang Liu, Leo Zhang
Organizations: Wake Forest University Winston-Salem, USA · Griffith University Gold Coast, Australia · Nanyang Technological University Singapore · University of New South Wales Sydney, Australia · Southern University of Science and Technology Shenzhen, China
Large language model (LLM) services process massive volumes of user requests, resulting in substantial inference costs. To reduce both cost and latency, providers have introduced prompt caching. However, correctly integrating prompt caching is non-trivial due to heterogeneous provider-specific APIs and continuously evolving caching semantics. To understand prompt-caching mis-integration in downstream applications, we conduct the first empirical study of 120 prompt-caching-related pull requests from 10 popular open-source agent systems. Our findings show that prompt-caching mis-integrations arise in both request-side misconfiguration and read-side usage miscount. More importantly, most mis-integrations fail silently without causing observable runtime failures. Motivated by these findings, we design CACHELINT, which models a prompt-caching mis-integration as an invocation site together with the preconditions required for correct cache integration. CACHELINT locates caching invocation sites with CodeQL and discharges each precondition either against a dated provider capability oracle or with an LLM guard check. CACHELINT achieves 0.71 precision and 0.78 recall in identifying the 120 labeled mis-integrations across the 10 studied agent systems. Our evaluation further demonstrates the effectiveness of CACHELINT in real-world settings, identifying 32 mis-integrations across 571 agent projects with 0.86 precision. These findings have received 21 developer confirmations and led to 18 upstream fixes, including one in Anthropic's official SDK.
Figures & tables
Fig. 1: The overview of our work.
Fig. 2: A prompt-caching mis-integration in zylon-ai/private-gpt : a sub-second timestamp at the front of the system prompt breaks the cached prefix on every call; the fix coarsens it to day granularity.
Project
#Star
#PR
#PC
OpenClaw [ 23 ]
379,764
53,078
39
Hermes-Agent [ 24 ]
198,679
37,707
21
AutoGPT [ 25 ]
185,051
8,235
6
OpenCode [ 26 ]
176,850
13,884
9
LobeHub [ 27 ]
78,924
8,479
1
OpenHands [ 28 ]
77,888
10,430
11
TABLE I: Top ten agent applications (#PC = merged PRs that fix a prompt-caching defect).
Type
Root cause
#
%
Description
Symptom
Fixing strategy
Request side 97 (81%)
Provider compatibility
47
39
Model-name gating
37
31
A hardcoded model-name check guards caching; a new model fails it.
Silent
Capability lookup
Incompatible marker
10
8
The cache marker’s format is one the provider rejects.
Crash
Marker guard
Prefix stability
41
34
Prefix reassembly
15
13
Each call rebuilds the prefix in a different order, breaking the byte-exact match.
Silent
Cache-boundary fix
Volatile value
13
11
A per-request value (timestamp, UUID) is embedded in the cached prefix.
Silent
Cache-boundary fix
TABLE II: Taxonomy of the 120 prompt-caching defects: two types and eight root causes, each with its count and share, mechanism, most common symptom, and most common fix. Bold rows give group subtotals.
Fig. 3: Root cause vs. fixing strategy for the 120 defects. Each cell is a count, shaded by group and darker for larger counts.
Fig. 4: CacheLint ’s findings on 33 real-world projects, placed by code size (KLOC) and popularity (stars).
Large language model (LLM) agent reliability concerns dependable, safe, and verifiable behavior across tools, multi-step tasks, and changing environments.
System prompts for LLM-based coding agents are software artifacts that govern agent behavior, yet lack the testing infrastructure applied to conventional software. We present Arbiter, a framework combining formal evaluation rules with multi-model LLM scouring to detect interference patterns in system prompts. Applied to three major coding agent system prompts: Claude Code (Anthropic), Codex CLI (OpenAI), and Gemini CLI (Google), we identify 152 findings across the undirected scouring phase and 21 hand-labeled interference patterns in directed analysis of one vendor. We show that prompt architecture (monolithic, flat, modular) strongly correlates with observed failure class but not with severity, and that multi-model evaluation discovers categorically different vulnerability classes than single-model analysis. One scourer finding was structural data loss in Gemini CLI's memory system was consistent with an issue filed and patched by Google, which addressed the symptom without addressing the schema-level root cause identified by the scourer. Total cost of cross-vendor analysis: $0.27 USD.
Tony Mason
the University of British Columbia · the Georgia Institute of Technology
Over the past year, prompt caching in Large Language Models (LLMs) has become increasingly more popular across inference APIs. Prompt caching helps save precious compute resources and speeds up response times by reusing parts of the KV cache of a specific prompt for another request. However, many implementations of prompt caching are not secure against timing attacks or even basic metadata disclosure. Gu et al. (ICML 2025) develop a method to audit prompt caching in LLMs. This paper investigates whether OpenRouter's API gateway architecture introduces prompt caching vulnerabilities that bypass provider-level prompt cache isolation guarantees. Most LLM inference providers implement per-account or per-organization prompt caching to prevent data leaks, but does routing through OpenRouter with shared organizational credentials inadvertently create global cache sharing across all OpenRouter users?
Ryan Fahey
Khoury College of Computer Sciences Northeastern University Boston, MA, USA
Personal AI agents incur substantial cost via repeated LLM calls. We show existing caching methods fail: GPTCache achieves 37.9% accuracy on real benchmarks; APC achieves 0-12%. The root cause is optimizing for the wrong property -- cache effectiveness requires key consistency and precision, not classification accuracy. We observe cache-key evaluation reduces to clustering evaluation and apply V-measure decomposition to separate these on n=8,682 points across MASSIVE, BANKING77, CLINC150, and NyayaBench v2, our new 8,514-entry multilingual agentic dataset (528 intents, 20 W5H2 classes, 63 languages). We introduce W5H2, a structured intent decomposition framework. Using SetFit with 8 examples per class, W5H2 achieves 91.1%+/-1.7% on MASSIVE in ~2ms -- vs 37.9% for GPTCache and 68.8% for a 20B-parameter LLM at 3,447ms. On NyayaBench v2 (20 classes), SetFit achieves 55.3%, with cross-lingual transfer across 30 languages. Our five-tier cascade handles 85% of interactions locally, projecting 97.5% cost reduction. We provide risk-controlled selective prediction guarantees via RCPS with nine bound families.