ReCache: Efficient KV Cache Reuse and Compression for Tool-Augmented LLM Agents

Yichu Fang; Sitong Wei; Haozhe Hu; Xiaoyu Shen

cs.CLAug 20, 2026

ReCache: Efficient KV Cache Reuse and Compression for Tool-Augmented LLM Agents

Authors: Yichu Fang, Sitong Wei, Haozhe Hu, Xiaoyu Shen

Organizations: Shanghai Jiao Tong University · Eastern Institute of Technology, Ningbo · Xi’an Jiaotong University

Abstract

Agentic language models repeatedly encode tool and skill schemas that recur across requests in different combinations and orders, preventing standard prefix caching from reusing their key--value (KV) states. We introduce ReCache, a framework for independently caching resource representations while reducing their inference-time computational and memory overhead. Resource-wise attention removes cross-resource interactions and assigns resource-local positions, producing composition-invariant KV blocks. ReCache then restricts resource visibility to contribution-selected layer--KV-head-group routes and retains only invocation-critical fields through structural and semantic pruning. We evaluate ReCache on a benchmark assembled from seven public tool- and skill-use datasets, including resource-disjoint tests. Resource-wise attention matches dense invocation performance (82.3% versus 82.4% Inv-F1) while providing a 3.655 $\times$ time-to-first-token speedup. The complete framework reduces allocated KV-tensor memory by 92.43% and accelerates attention by 1.423 $\times$ . These results show that separating reusable schema encoding from selective resource access substantially reduces agentic inference costs with limited effectiveness loss. The code is available at https://github.com/EIT-NLP/ReCache.

Figures & tables

Panel of Figure 1 — Figure 1: Cross-resource attention on Qwen3-4B.

Figure 2: ReCache constructs reusable KV blocks for each

R_{i}

and progressively reduces their subsequent cache retention and access. (a) Resource-wise attention removes inter-resource attention and resets positions from zero within each

R_{i}

. (b) Structural pruning exposes resource KV states (using

M_{\mathrm{resource}}

) only through selected layer (

L_{i}

)–KV-head-group (

G_{i}

) routes

\Omega^{\star}

(highlighted in blue). The remaining routes use

M_{\mathrm{context}}

. (c) Semantic pruning retains resource names, argument (arg) names and descriptions (desc.), and the final suffix token. The final mask combines these decisions, ensuring that

Y

attends exclusively to retained semantic fields within assigned routes during decoding.

Figure 3: Structural contribution coverage in

Q_{l}

and

Q_{s}

. Cumulative leave-one-in contributions of layers (top) and KV head groups (bottom); vertical lines mark the selected budgets.

Model	$K_{L}/L$	$K_{G}/G$	CC L	CC G	$\rho$	$\Delta$ Inv-F1 (%)
$Q_{l}$	20/36	3/8	97.7	47.3	79.2	$-0.2$
$Q_{s}$	20/28	7/8	99.8	99.3	37.5	$-0.7$

Table 1: Selected structural budgets. CC and

\rho

are percentages;

\Delta

Inv-F1 is measured relative to

\Omega_{\mathrm{full}}

after full-data training.

Method	Inv-F1	ID-F1	Halluc.	TTFT (ms)
Dense	82.4	96.0	0.0	26.319
Block	82.2	95.8	0.1	–
$\Omega_{\mathrm{full}}$	82.3	96.0	0.2	7.200

Table 2: Resource-wise attention ablation.

Structural pruning
Method	Attn. $\boldsymbol{\times}$	Mem. $\downarrow$	Inv-F1	ID-F1	Halluc.
Dense	–	–	82.4	96.0	0.0
$\Omega_{\mathrm{full}}$	1.001 $\times$	0.47%	82.3	96.0	0.2
$\Omega_{20,G}$	1.016 $\times$	44.71%	82.5	95.7	0.3
$\Omega_{\mathrm{full}}$ + SPEED	1.013 $\times$	44.71%	79.3	92.9	2.8
$\Omega_{20,3}$	1.314 $\times$	79.27%	82.1	95.6	0.2

Table 3: Results on

\mathcal{T}_{\mathrm{IND}}

. Efficiency is reported relative to Dense. SMP denotes Semantic Pruning.

Structural pruning
Method	Inv-F1	ID-F1	Halluc.
Dense	66.3	95.6	0.0
$\Omega_{\mathrm{full}}$	64.7	94.4	0.4
$\Omega_{20,G}$	64.3	94.3	0.5
$\Omega_{\mathrm{full}}$ + SPEED	54.3	81.6	13.4
$\Omega_{20,3}$	63.2	93.3	0.5

Table 4:

\mathcal{T}_{\mathrm{OOD}}

Results. SMP denotes Semantic Pruning.

Panel of Figure 4 — Figure 4: Efficiency on varying resource lengths $D_{\mathcal{R}}$ . Requests are grouped by average resource length into Small ( $0\leq D_{\mathcal{R}}<1\mathrm{K}$ ), Medium ( $1\mathrm{K}\leq D_{\mathcal{R}}<5\mathrm{K}$ ), Large ( $5\mathrm{K}\leq D_{\mathcal{R}}<10\mathrm{K}$ ), and XL ( $D_{\mathcal{R}}\geq 10\mathrm{K}$ ). SMP denotes semantic pruning.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Group	Source	Train pool	Distinct names	Subset dup.	Recurring names	IND cand.	OOD cand.
Tool	ToolACE	8,400	14,566	178	7,144	200	195
	APIGEN	10,000	2,806	734	2,805	500	146
	Toucan (all generators)	13,071	4,355	6,901	4,355	600	493
	- Kimi-K2	3,333	1,103	1,772	1,103	200	145
	- GPT-OSS-120B	4,184	1,412	2,162	1,412	200	130
	- Qwen3-32B	5,554	1,840	2,967	1,840	200	218

Appendix

Table 5: Dataset composition and split statistics. Source rows report eligible examples before final selection; the Toucan aggregate is not double-counted in the summary row. Distinct names counts source-level unique declared tools or skills, Subset dup. counts records beyond the first occurrence of an identical candidate configuration, and Recurring names counts resources appearing in multiple records. The summary row corresponds to 18.8% duplicated candidate records and 77.3% recurring resource names. IND cand. and OOD cand. denote eligible in-distribution and out-of-distribution candidates, and dashes mark unavailable statistics.

Config	Inv-F1	Resource ID			Halluc.
		ID-P	ID-R	ID-F1
$\Omega_{\mathrm{full}}$	72.1	92.0	90.8	90.9	0.4
$\Omega_{L,5}$	49.6	74.5	73.5	73.4	4.8
$\Omega_{L,6}$	53.4	79.6	77.9	78.3	3.2
$\Omega_{L,7}$	57.3	83.4	81.8	82.1	2.9
$\Omega_{15,G}$	69.3	89.2	88.3	88.3	1.1

Appendix

Table 6: Structural budget ablation for

Q_{s}

across training data regimes with comparable budgets.

$\boldsymbol{Q_{s}}$
Config	Inv-F1	Resource ID			Halluc.
		ID-P	ID-R	ID-F1
$\Omega_{\mathrm{full}}$	80.7	94.9	94.3	94.4	0.1
$\Omega_{20,5}$	78.8	94.1	93.5	93.6	0.5
$\Omega_{20,7}$	80.0	94.7	94.0	94.1	0.3
$\boldsymbol{Q_{l}}$

Appendix

Table 7: Structural budget ablation trained on

\mathcal{D}

Schema representation ablation
Config	Inv-F1	Resource ID			Halluc.
		ID-P	ID-R	ID-F1
$\Omega_{20,3}$	76.1	93.1	92.8	92.5	0.9
Suffix-only Schema	14.8	22.8	23.0	22.9	76.0
+ Resource Name	45.2	89.9	88.8	88.9	1.8
+ Arg Name	67.7	92.1	91.1	91.2	1.3

Appendix

Table 8: Semantic pruning ablation trained on

\mathcal{D}_{\mathrm{subset}}

. Arg stands for argument, Desc. stands for description.

\Omega_{20,3}

denotes the routing configuration with

K_{L}=20,K_{G}=3

Layers
	$\boldsymbol{Q_{l}}$ (Qwen3-4B)				$\boldsymbol{Q_{s}}$ (Qwen3-1.7B)
rank	unit	$\boldsymbol{w_{i}}$ (%)	CC (%)	rank	unit	$\boldsymbol{w_{i}}$ (%)	CC (%)
1	$L_{34}$	15.56	15.6	1	$L_{22}$	20.09	20.1
2	$L_{35}$	10.53	26.1	2	$L_{25}$	12.90	33.0
3	$L_{23}$	9.37	35.5	3	$L_{20}$	12.14	45.1
4	$L_{25}$	8.31	43.8	4	$L_{26}$	10.90	56.0

Appendix

Table 9: Individual contribution weights and cumulative contribution coverage of structural units. Within each panel, units are ranked independently by decreasing

w_{i}

, and CC at rank

k

is the cumulative positive contribution weight through rank

k

. Bold entries fall within the empirical budgets

K_{L}{=}20,K_{G}{=}3

for

Q_{l}

and

K_{L}{=}20,K_{G}{=}7

for

Q_{s}

KV-Cache Reuse

Key-value (KV) cache reuse lets language models reuse previously computed attention states to cut repeated prefill work and serving latency.

See all 71 papers

AgentKV: Phase-Aware KV Eviction for Agentic LLMs
Sep 14, 2026Taowen Tony Liu, Jeffrey T. H. Wong, Can Xiao +3KV Caching KV-Cache Eviction
Imperial College London, London, United Kingdom · Columbia University, New York, NY, USA
Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devices
Feb 17, 2026Yakov Pyotr ShkolnikovMemory Management for LLM Agents LLM Agent Memory
ActKV: Efficient LLM Agents through Action-Guided KV Cache Management
Sep 25, 2026Zihan Wang, Cheng Tang, Lei Gong +4KV-Cache Management KV-Cache Compression
University of Science and Technology of China Hefei, China · Suzhou Institute for Advanced Research, University of Science and Technology of China Suzhou, China

Sep 14, 2026J/K move · Enter open · S save

AgentKV: Phase-Aware KV Eviction for Agentic LLMs

Taowen Tony Liu, Jeffrey T. H. Wong, Can Xiao, Bowen Yang+2

Imperial College London, London, United Kingdom · Columbia University, New York, NY, USA

KV Caching KV-Cache Eviction LLM Inference Acceleration LLM Agent Serving LLM Agents

Open paper AskPDF arXiv

Agentic serving can consume orders of magnitude more tokens than chatbot workloads, stressing both KV-cache capacity and decode-time bandwidth. Most KV-eviction methods score cached keys against representative queries drawn from the most recent tokens, assuming future attention resembles recent attention. We show that agentic generation violates this assumption: future queries form a mixture over think, act, tool, and others phases, and principal-angle analysis shows these components occupy measurably different query subspaces, so recency representatives systematically undervalue keys that upcoming phases will need. We propose AGENTKV, which maintains a small query buffer per phase and scores cached keys against their union. We further implement AGENTKV in a persistent multi-turn serving path that carries compressed KV state across turns and compacts retained KV pages online. Across two models, six task domains, and three KV budgets each, AGENTKV improves task score by 5.5 points on average over R-KV and 5.3 over Tri-attention. Relative to upstream full-KV SGLang, AGENTKV improves output-token throughput by up to 1.80x. Code: https://github.com/LiuTaowen-Tony/agentkv.

Abstract

KV-Cache Reuse

Explore similar work

AgentKV: Phase-Aware KV Eviction for Agentic LLMs

Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devices

ActKV: Efficient LLM Agents through Action-Guided KV Cache Management