Why Agent Caching Fails and How to Fix It: Structured Intent Canonicalization with Few-Shot Learning

Abhinaba Basu

cs.CLFeb 21, 2026

Why Agent Caching Fails and How to Fix It: Structured Intent Canonicalization with Few-Shot Learning

Authors: Abhinaba Basu

Abstract

Personal AI agents incur substantial cost via repeated LLM calls. We show existing caching methods fail: GPTCache achieves 37.9% accuracy on real benchmarks; APC achieves 0-12%. The root cause is optimizing for the wrong property -- cache effectiveness requires key consistency and precision, not classification accuracy. We observe cache-key evaluation reduces to clustering evaluation and apply V-measure decomposition to separate these on n=8,682 points across MASSIVE, BANKING77, CLINC150, and NyayaBench v2, our new 8,514-entry multilingual agentic dataset (528 intents, 20 W5H2 classes, 63 languages). We introduce W5H2, a structured intent decomposition framework. Using SetFit with 8 examples per class, W5H2 achieves 91.1%+/-1.7% on MASSIVE in ~2ms -- vs 37.9% for GPTCache and 68.8% for a 20B-parameter LLM at 3,447ms. On NyayaBench v2 (20 classes), SetFit achieves 55.3%, with cross-lingual transfer across 30 languages. Our five-tier cascade handles 85% of interactions locally, projecting 97.5% cost reduction. We provide risk-controlled selective prediction guarantees via RCPS with nine bound families.

Figures & tables

Figure 1: Overview of W5H2 structured intent canonicalization for agent caching. (a) The embedding similarity trap: semantically similar queries (“check email” vs. “send email,” cos $=0.91$ ) require different tool sequences, while paraphrases of the same intent (cos $=0.65$ ) fall below typical thresholds. (b) W5H2 decomposes queries into structured fields; the ( What , Where ) pair forms the cache key. (c) Five-tier cascade architecture with per-tier latency and traffic share. (d) SetFit accuracy on MASSIVE (8-class) across methods: 22M-parameter SetFit (91.1%) outperforms a 20B-parameter LLM (68.8%) at $>$ 700 $\times$ lower latency.

Technique	Domain	Acc. (%)	Latency	Multilingual	Verdict
LSH (SimHash)	IR/Hashing	30–40	${<}1$ ms	No	Reject
Semantic hashing	Deep learning	75–80	${\sim}5$ ms	Partial	Maybe
Frame semantic parsing	Linguistics	65–75	50–100ms	Limited	Promising
Query fingerprinting	Databases	70–85	${<}1$ ms	With NER	Useful
Winnowing	Plagiarism det.	35–45	${<}1$ ms	No	Reject
SetFit (contrastive)	Few-shot ML	91–97	$\mathbf{{\sim}2\text{--}5}$ ms	Yes	Winner

Table 1: Cross-domain canonicalization technique survey. Accuracy is on a preliminary 10-class agent intent pilot (NyayaBench v1); NyayaBench v2 (20-class) results appear in Table 2 . ∗ PDDL requires an LLM call, defeating the purpose of caching.

Method	Accuracy $\star$	Hit Rate	Latency	Training
GPTCache @0.70	83.1% $\dagger$	16.9%	${\sim}5$ ms	None
GPTCache @0.85	84.8% $\dagger$	5.6%	${\sim}5$ ms	None
GPTCache KMeans ( $k{=}20$ )	49.1%	98.3%	${\sim}5$ ms	None
SetFit-EN (22M)	55.3 $\pm$ 1.0%	98.3%	2.4ms	160 examples
SetFit-EN 16-shot	62.6 $\pm$ 1.5%	—	—	320 examples
SetFit-Multi (118M)	55.4% (EN)	—	—	160 examples

Table 2: Main results on NyayaBench v2 (20-class, 528 intents

\to

20 W5H2 classes). GPTCache results use our operationalization of its core mechanism (embedding similarity + KMeans clustering); see footnote in Section 5.4 . Accuracy : classification accuracy over all test examples (SetFit, KMeans) or precision among cache hits (threshold methods). Hit Rate : fraction of queries receiving a cached response. Latency on CPU.

Benchmark	Method	Classes	Shots/class	Accuracy
MASSIVE	LLM 0-shot (GPT-oss-20b)	8	0	68.8%
MASSIVE	LLM 4-shot	8	4	${\sim}$ 79%
MASSIVE	BERT-base (supervised)	8	full	97.3%
MASSIVE	SetFit-EN (22M)	8	8	91.1 $\pm$ 1.7%
MASSIVE	SetFit-Multi (118M)	8	8	84.4%
CLINC150	SetFit-EN	150	8	85.9%

Table 3: Accuracy on established benchmarks. SetFit reports mean

\pm

std over 5 seeds. LLM baselines include zero-shot and few-shot. BERT uses full training data.

Figure 2: Benchmark comparison. (a) Accuracy vs. number of classes (log scale). NyayaBench v2 (20 real agentic classes) is substantially harder than established benchmarks, reflecting the difficulty of real-world intent distributions. (b) V-measure decomposition (homogeneity $h$ , completeness $c$ , V-measure $V$ ) across all four benchmarks. SetFit maintains balanced $h$ / $c$ even as class count increases.

Method	Training	Acc. (%)	Latency	Cost/1K	$V$
LLM 0-shot (GPT-oss-20b)	0 examples	68.8	3,447ms	$0.089	0.650
LLM 2-shot $\dagger$	16 examples	${\sim}$ 75	${\sim}$ 4,000ms	$0.11	—
LLM 4-shot $\dagger$	32 examples	${\sim}$ 79	${\sim}$ 4,500ms	$0.13	—
Rule-based (W5H2)	0 examples	60.7	${<}1$ ms	$0.00	—
SetFit-Multi (118M)	64 examples	84.4	${\sim}$ 5ms	$0.00	—
SetFit-EN (22M)	64 examples	91.1 $\pm$ 1.7	${\sim}$ 2–5ms	$0.00	0.813

Table 4: Method comparison on MASSIVE W5H2 (8 classes,

n{=}1{,}102

). LLM cost at GPT-4o-mini pricing. SetFit reports mean

\pm

std over 5 seeds. BERT uses full training data (supervised upper bound).

Benchmark	Method	$h$	$c$	$V$	AMI	Keys
MASSIVE ( $n{=}1{,}102$ , 8 cls)	Oracle	1.000	1.000	1.000	1.000	8
	SetFit-EN 8-shot	0.817	0.810	0.813	0.810	8
	GPTCache ( $k{=}8$ )	0.784	0.788	0.786	0.784	8
	LLM (GPT-oss-20b)	0.638	0.662	0.650	0.645	8
	Random / APC	1.000	0.277	0.433	0.000	1102
BANKING77 ( $n{=}3{,}080$ , 77 cls)	Oracle	1.000	1.000	1.000	1.000	77

Table 5: V-measure decomposition on three benchmarks.

h

= homogeneity (precision),

c

= completeness (consistency), AMI = adjusted mutual information. SetFit outperforms GPTCache at matched key count on all benchmarks. Random/APC correctly scores AMI=0.000.

Figure 3: Cross-lingual transfer on NyayaBench v2. (a) Per-language accuracy for 30 languages, colored by language family. Indo-European (Slavic, Romance) and Kartvelian languages transfer best; Dravidian and Niger-Congo remain challenging. (b) Language family summary with mean accuracy and range (whiskers). All training data is English-only (160 examples).

Family	Languages	Mean Acc. (%)	Range
Indo-European (Indic)	hi, gu, mr, pa, ne, si, ur	33.5	17.0–44.5%
Indo-European (Slavic)	cs, ru, uk	50.7	47.5–54.0%
Indo-European (Germanic)	da, no	46.0	42.0–50.0%
Indo-European (Romance)	ro	51.0	—
Dravidian	ta, te, kn, ml	15.8	12.5–20.5%
Turkic	az, uz	46.0	44.0–48.0%

Table 6: SetFit-Multi cross-lingual transfer on NyayaBench v2 (20-class). Trained on English only (160 examples). Each language has 200 test examples mapped to W5H2 classes. Grouped by language family.

Panel of Figure 4 — Figure 4: Information-theoretic analysis of cache-key quality across three benchmarks.

Panel of Figure 5 — Figure 5: Cache-key quality decomposition. SetFit maximizes mutual information with intents while minimizing conditional entropy in both directions.

Benchmark	Condition	Classes	$h$	$c$	$V$	Rate	Acc
MASSIVE ( $n{=}1{,}102$ )	Raw-17 $\to$ W5H2 map	18	0.842	0.840	0.841	3.0	93.3%
	W5H2-8 (ours)	8	0.791	0.777	0.784	3.0	88.8%
	Target-only (7 cls)	7	0.770	0.843	0.805	2.8	94.1%
	Action-only (5 cls)	5	0.491	0.686	0.573	2.3	81.8%
	Random-8 (control)	8	0.495	0.490	0.492	3.0	68.0%
BANKING77 ( $n{=}3{,}080$ )	Raw-77	77	0.831	0.836	0.833	6.3	78.4%

Table 7: W5H2 decomposition ablation. V-measure computed against W5H2 ground truth for MASSIVE conditions, against respective ground truth for BANKING77.

h

= precision,

c

= consistency.

Decomposition	MASSIVE (8-cls)			NyayaBench v2 (20-cls)
	Acc. (%)	$V$	Classes	Acc. (%)	$V$	Classes
W5H2 (What, Where)	90.1	0.804	8	52.4	0.603	20
Target-only	92.7	0.819	7	52.8	0.452	13
Semantic Role (Action, Patient)	86.8	0.783	11	49.1	0.763	49
Nyaya Pramana Ensemble	82.4	0.787	8	48.0	0.516	20
Verb-only (ablation)	74.1	0.554	5	42.8	0.641	40

Table 8: Decomposition strategy comparison. All use SetFit 8-shot (seed 42). Target-only achieves the highest accuracy on MASSIVE but loses action-verb discrimination critical for cache safety.

Panel of Figure 6 — Figure 6: Risk–coverage curves for flat SetFit with abstention. Higher thresholds $\tau$ reduce coverage but improve accuracy on cached queries. On MASSIVE, $\tau{=}0.25$ achieves the best safe operating point ( $<$ 5% unsafe): 88.0% coverage at 95.4% accuracy. On NyayaBench v2, reaching 100% accuracy requires $\tau{=}0.30$ , leaving only 3.2% coverage—validating the need for the multi-tier cascade.

Benchmark	$\tau$	Coverage	Acc. (cached)	Unsafe %	$n_{\text{cached}}$
MASSIVE	0.00	100%	92.4%	7.6%	1,102
	0.25	88.0%	95.4%	4.6%	970
	0.35	58.5%	98.6%	1.4%	645
	0.60	3.4%	100%	0%	37
NyayaBench v2	0.00	100%	55.7%	44.3%	280
	0.15	42.5%	69.7%	30.3%	119

Table 9: Selected operating points for flat SetFit with abstention. Full curves in Figure 6 .

Benchmark	Method	Accuracy	$V$	Consistency	Train time
MASSIVE (8-class)	Vanilla	91.56 $\pm$ 1.47%	0.819	89.4%	4.5s
MASSIVE (8-class)	Consist-Reg	90.92 $\pm$ 1.59%	0.808	89.0%	15.5s
	$\Delta$	$-0.64$	$-0.011$	$-0.35$	$+11.0$ s
NyayaBench v2 (20-class)	Vanilla	21.6 $\pm$ 1.8%	0.179	66.5%	10.0s
NyayaBench v2 (20-class)	Consist-Reg	21.7 $\pm$ 1.5%	0.188	58.8%	39.0s
	$\Delta$	$+0.07$	$+0.009$	$-7.7$	$+29.0$ s

Table 10: Vanilla vs. consistency-regularized SetFit (5 seeds, 8-shot). Consistency regularization does not improve over baseline SetFit in either regime.

Strategy	Monthly Cost	Savings	Local %
No Cache (all Claude Sonnet)	$31.72	—	0%
APC (6% hit rate)	$29.82	5.9%	6%
GPTCache (37.9% hit rate)	$19.70	37.9%	37.9%
W5H2 (85% local)	$0.80	97.5%	85%

Table 11: Monthly cost comparison at 50 requests/day (five-tier architecture). Tier 3: DeepSeek V3.2 (

0.28/

0.42 per M tokens). Tier 4: Claude Sonnet 4.5 (

3/

15 per M tokens). No-cache baseline uses Claude Sonnet 4.5 for every request. Pricing as of February 2026.

Panel of Figure 7 — Figure 7: Cost model with real February 2026 API pricing.

Figure 8: Reliability diagrams for SetFit confidence calibration. (a) MASSIVE (8-class): moderate calibration; temperature scaling reduces ECE. (b) NyayaBench v2 (20-class): higher ECE reflects the harder task; temperature scaling provides larger improvement. Bars show pre-calibration binned accuracy; dots show post-calibration.

Figure 9: Test coverage as a function of risk tolerance $\alpha$ ( $\delta{=}0.10$ ) across bound variants. (a) MASSIVE: LTT + Emp. Bernstein (green) dominates Hoeffding (orange) across all $\alpha$ ; Wasserstein DRO (blue) is more conservative by design. (b) NyayaBench v2: PAC-Bayes with MASSIVE transfer prior (red) is the only method achieving meaningful coverage at $\alpha{=}0.10$ ; standard bounds are infeasible below $\alpha{=}0.15$ .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Language	Accuracy (%)	Family
az (Azerbaijani)	48.0	Turkic
cs (Czech)	54.0	Slavic
da (Danish)	42.0	Germanic
el (Greek)	49.5	Hellenic
fi (Finnish)	43.5	Uralic
gu (Gujarati)	40.0	Indic

Appendix

Table 12: SetFit-Multi per-language accuracy on NyayaBench v2 (20-class, trained on English only).

n{=}200

per language.

Figure 10: NyayaBench v2 per-class analysis. (a) SetFit-EN accuracy per W5H2 class (8-shot, 5 seeds), sorted by performance. Error bars show $\pm$ 1 std across seeds. Color encodes performance tier (green $>$ 70%, amber 40–70%, red $<$ 40%). (b) Shot-doubling effect: accuracy gain from 8-shot to 16-shot per class. Most classes improve; set_schedule shows the largest absolute gain.

W5H2 Class	EN Test $n$	Accuracy (%)
check_info	155	57.4
control_home	146	86.3
create_content	135	48.9
search_web	109	54.1
manage_tasks	104	26.9
analyze_data	84	46.4

Appendix

Table 13: SetFit-EN per-class accuracy on NyayaBench v2. EN test examples per class vary based on dataset composition.

Benchmark	Method	$h$	$c$	$V$	AMI	FMI	Rate (bits)
MASSIVE	Oracle	1.000	1.000	1.000	1.000	1.000	3.00
	SetFit-EN	0.817	0.810	0.813	0.810	0.833	3.00
	GPTCache ( $k{=}8$ )	0.784	0.788	0.786	0.784	—	3.00
	LLM (GPT-oss-20b)	0.638	0.662	0.650	0.645	—	3.00
	GPTCache ( $k{=}4$ )	0.534	0.835	0.652	0.650	—	2.00
	Majority (1 key)	0.000	1.000	0.000	0.000	0.408	0.00

Appendix

Table 14: Full V-measure results across all methods and benchmarks.

Scale	Req/day	No Cache	GPTCache	W5H2	Savings
Individual	50	$31.72	$19.70	$0.80	97.5%
Power User	200	$126.90	$78.80	$3.19	97.5%
Small Team (5)	1,000	$634.50	$394.02	$15.95	97.5%
Startup (50)	10,000	$6,345	$3,940	$159	97.5%
Enterprise (500)	100,000	$63,450	$39,402	$1,595	97.5%

Appendix

Table 15: Monthly cost scaling across deployment sizes.

LLM Agent Serving

Large language model agent serving runs tool-using, multi-step agents efficiently by coordinating inference, memory, and workflow execution.

See all 35 papers

Silent and Costly: Characterizing and Detecting Prompt-Caching Mis-Integration in LLM Agents
Oct 26, 2026Ying Zhang, Ruoqi Guo, Yi Liu +4LLM Agent Reliability LLM Agent Efficiency
Wake Forest University Winston-Salem, USA · Griffith University Gold Coast, Australia · Nanyang Technological University Singapore +2
Cache Engineering for Agents
Oct 20, 2026Tianyu Fu, Ruoxi Wang, Zihan Min +6KV-Cache Management Memory Management for LLM Agents
Tsinghua University · The Chinese University of Hong Kong
ReCache: Efficient KV Cache Reuse and Compression for Tool-Augmented LLM Agents
Aug 20, 2026Yichu Fang, Sitong Wei, Haozhe Hu +1KV-Cache Reuse LLM Inference Acceleration
Shanghai Jiao Tong University · Eastern Institute of Technology, Ningbo · Xi’an Jiaotong University

Oct 26, 2026J/K move · Enter open · S save

Silent and Costly: Characterizing and Detecting Prompt-Caching Mis-Integration in LLM Agents

Ying Zhang, Ruoqi Guo, Yi Liu, Gelei Deng+3

Wake Forest University Winston-Salem, USA · Griffith University Gold Coast, Australia · Nanyang Technological University Singapore+2

LLM Agent Reliability LLM Agent Efficiency Static Code Analysis

Open paper AskPDF arXiv

Large language model (LLM) services process massive volumes of user requests, resulting in substantial inference costs. To reduce both cost and latency, providers have introduced prompt caching. However, correctly integrating prompt caching is non-trivial due to heterogeneous provider-specific APIs and continuously evolving caching semantics. To understand prompt-caching mis-integration in downstream applications, we conduct the first empirical study of 120 prompt-caching-related pull requests from 10 popular open-source agent systems. Our findings show that prompt-caching mis-integrations arise in both request-side misconfiguration and read-side usage miscount. More importantly, most mis-integrations fail silently without causing observable runtime failures. Motivated by these findings, we design CACHELINT, which models a prompt-caching mis-integration as an invocation site together with the preconditions required for correct cache integration. CACHELINT locates caching invocation sites with CodeQL and discharges each precondition either against a dated provider capability oracle or with an LLM guard check. CACHELINT achieves 0.71 precision and 0.78 recall in identifying the 120 labeled mis-integrations across the 10 studied agent systems. Our evaluation further demonstrates the effectiveness of CACHELINT in real-world settings, identifying 32 mis-integrations across 571 agent projects with 0.86 precision. These findings have received 21 developer confirmations and led to 18 upstream fixes, including one in Anthropic's official SDK.

Abstract

LLM Agent Serving

Explore similar work

Silent and Costly: Characterizing and Detecting Prompt-Caching Mis-Integration in LLM Agents

Cache Engineering for Agents

ReCache: Efficient KV Cache Reuse and Compression for Tool-Augmented LLM Agents