ReCache: Efficient KV Cache Reuse and Compression for Tool-Augmented LLM Agents

Yichu Fang; Sitong Wei; Haozhe Hu; Xiaoyu Shen

cs.CLAug 20, 2026

ReCache: Efficient KV Cache Reuse and Compression for Tool-Augmented LLM Agents

Authors: Yichu Fang, Sitong Wei, Haozhe Hu, Xiaoyu Shen

Organizations: Shanghai Jiao Tong University · Eastern Institute of Technology, Ningbo · Xi’an Jiaotong University

Abstract

Agentic language models repeatedly encode tool and skill schemas that recur across requests in different combinations and orders, preventing standard prefix caching from reusing their key--value (KV) states. We introduce ReCache, a framework for independently caching resource representations while reducing their inference-time computational and memory overhead. Resource-wise attention removes cross-resource interactions and assigns resource-local positions, producing composition-invariant KV blocks. ReCache then restricts resource visibility to contribution-selected layer--KV-head-group routes and retains only invocation-critical fields through structural and semantic pruning. We evaluate ReCache on a benchmark assembled from seven public tool- and skill-use datasets, including resource-disjoint tests. Resource-wise attention matches dense invocation performance (82.3% versus 82.4% Inv-F1) while providing a 3.655 $\times$ time-to-first-token speedup. The complete framework reduces allocated KV-tensor memory by 92.43% and accelerates attention by 1.423 $\times$ . These results show that separating reusable schema encoding from selective resource access substantially reduces agentic inference costs with limited effectiveness loss. The code is available at https://github.com/EIT-NLP/ReCache.

Figures & tables

Panel of Figure 1 — Figure 1: Cross-resource attention on Qwen3-4B.

Figure 2: ReCache constructs reusable KV blocks for each

R_{i}

and progressively reduces their subsequent cache retention and access. (a) Resource-wise attention removes inter-resource attention and resets positions from zero within each

R_{i}

. (b) Structural pruning exposes resource KV states (using

M_{\mathrm{resource}}

) only through selected layer (

L_{i}

)–KV-head-group (

G_{i}

) routes

\Omega^{\star}

(highlighted in blue). The remaining routes use

M_{\mathrm{context}}

. (c) Semantic pruning retains resource names, argument (arg) names and descriptions (desc.), and the final suffix token. The final mask combines these decisions, ensuring that

Y

attends exclusively to retained semantic fields within assigned routes during decoding.

Figure 3: Structural contribution coverage in

Q_{l}

and

Q_{s}

. Cumulative leave-one-in contributions of layers (top) and KV head groups (bottom); vertical lines mark the selected budgets.

Model	$K_{L}/L$	$K_{G}/G$	CC L	CC G	$\rho$	$\Delta$ Inv-F1 (%)
$Q_{l}$	20/36	3/8	97.7	47.3	79.2	$-0.2$
$Q_{s}$	20/28	7/8	99.8	99.3	37.5	$-0.7$

Table 1: Selected structural budgets. CC and

\rho

are percentages;

\Delta

Inv-F1 is measured relative to

\Omega_{\mathrm{full}}

after full-data training.

Method	Inv-F1	ID-F1	Halluc.	TTFT (ms)
Dense	82.4	96.0	0.0	26.319
Block	82.2	95.8	0.1	–
$\Omega_{\mathrm{full}}$	82.3	96.0	0.2	7.200

Table 2: Resource-wise attention ablation.

Structural pruning
Method	Attn. $\boldsymbol{\times}$	Mem. $\downarrow$	Inv-F1	ID-F1	Halluc.
Dense	–	–	82.4	96.0	0.0
$\Omega_{\mathrm{full}}$	1.001 $\times$	0.47%	82.3	96.0	0.2
$\Omega_{20,G}$	1.016 $\times$	44.71%	82.5	95.7	0.3
$\Omega_{\mathrm{full}}$ + SPEED	1.013 $\times$	44.71%	79.3	92.9	2.8
$\Omega_{20,3}$	1.314 $\times$	79.27%	82.1	95.6	0.2

Table 3: Results on

\mathcal{T}_{\mathrm{IND}}

. Efficiency is reported relative to Dense. SMP denotes Semantic Pruning.

Structural pruning
Method	Inv-F1	ID-F1	Halluc.
Dense	66.3	95.6	0.0
$\Omega_{\mathrm{full}}$	64.7	94.4	0.4
$\Omega_{20,G}$	64.3	94.3	0.5
$\Omega_{\mathrm{full}}$ + SPEED	54.3	81.6	13.4
$\Omega_{20,3}$	63.2	93.3	0.5

Table 4:

\mathcal{T}_{\mathrm{OOD}}

Results. SMP denotes Semantic Pruning.

Panel of Figure 4 — Figure 4: Efficiency on varying resource lengths $D_{\mathcal{R}}$ . Requests are grouped by average resource length into Small ( $0\leq D_{\mathcal{R}}<1\mathrm{K}$ ), Medium ( $1\mathrm{K}\leq D_{\mathcal{R}}<5\mathrm{K}$ ), Large ( $5\mathrm{K}\leq D_{\mathcal{R}}<10\mathrm{K}$ ), and XL ( $D_{\mathcal{R}}\geq 10\mathrm{K}$ ). SMP denotes semantic pruning.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Group	Source	Train pool	Distinct names	Subset dup.	Recurring names	IND cand.	OOD cand.
Tool	ToolACE	8,400	14,566	178	7,144	200	195
	APIGEN	10,000	2,806	734	2,805	500	146
	Toucan (all generators)	13,071	4,355	6,901	4,355	600	493
	- Kimi-K2	3,333	1,103	1,772	1,103	200	145
	- GPT-OSS-120B	4,184	1,412	2,162	1,412	200	130
	- Qwen3-32B	5,554	1,840	2,967	1,840	200	218

Appendix

Table 5: Dataset composition and split statistics. Source rows report eligible examples before final selection; the Toucan aggregate is not double-counted in the summary row. Distinct names counts source-level unique declared tools or skills, Subset dup. counts records beyond the first occurrence of an identical candidate configuration, and Recurring names counts resources appearing in multiple records. The summary row corresponds to 18.8% duplicated candidate records and 77.3% recurring resource names. IND cand. and OOD cand. denote eligible in-distribution and out-of-distribution candidates, and dashes mark unavailable statistics.

Config	Inv-F1	Resource ID			Halluc.
		ID-P	ID-R	ID-F1
$\Omega_{\mathrm{full}}$	72.1	92.0	90.8	90.9	0.4
$\Omega_{L,5}$	49.6	74.5	73.5	73.4	4.8
$\Omega_{L,6}$	53.4	79.6	77.9	78.3	3.2
$\Omega_{L,7}$	57.3	83.4	81.8	82.1	2.9
$\Omega_{15,G}$	69.3	89.2	88.3	88.3	1.1

Appendix

Table 6: Structural budget ablation for

Q_{s}

across training data regimes with comparable budgets.

$\boldsymbol{Q_{s}}$
Config	Inv-F1	Resource ID			Halluc.
		ID-P	ID-R	ID-F1
$\Omega_{\mathrm{full}}$	80.7	94.9	94.3	94.4	0.1
$\Omega_{20,5}$	78.8	94.1	93.5	93.6	0.5
$\Omega_{20,7}$	80.0	94.7	94.0	94.1	0.3
$\boldsymbol{Q_{l}}$

Appendix

Table 7: Structural budget ablation trained on

\mathcal{D}

Schema representation ablation
Config	Inv-F1	Resource ID			Halluc.
		ID-P	ID-R	ID-F1
$\Omega_{20,3}$	76.1	93.1	92.8	92.5	0.9
Suffix-only Schema	14.8	22.8	23.0	22.9	76.0
+ Resource Name	45.2	89.9	88.8	88.9	1.8
+ Arg Name	67.7	92.1	91.1	91.2	1.3

Appendix

Table 8: Semantic pruning ablation trained on

\mathcal{D}_{\mathrm{subset}}

. Arg stands for argument, Desc. stands for description.

\Omega_{20,3}

denotes the routing configuration with

K_{L}=20,K_{G}=3

Layers
	$\boldsymbol{Q_{l}}$ (Qwen3-4B)				$\boldsymbol{Q_{s}}$ (Qwen3-1.7B)
rank	unit	$\boldsymbol{w_{i}}$ (%)	CC (%)	rank	unit	$\boldsymbol{w_{i}}$ (%)	CC (%)
1	$L_{34}$	15.56	15.6	1	$L_{22}$	20.09	20.1
2	$L_{35}$	10.53	26.1	2	$L_{25}$	12.90	33.0
3	$L_{23}$	9.37	35.5	3	$L_{20}$	12.14	45.1
4	$L_{25}$	8.31	43.8	4	$L_{26}$	10.90	56.0

Appendix

Table 9: Individual contribution weights and cumulative contribution coverage of structural units. Within each panel, units are ranked independently by decreasing

w_{i}

, and CC at rank

k

is the cumulative positive contribution weight through rank

k

. Bold entries fall within the empirical budgets

K_{L}{=}20,K_{G}{=}3

for

Q_{l}

and

K_{L}{=}20,K_{G}{=}7

for

Q_{s}

KV-Cache Reuse

Key-value (KV) cache reuse lets language models reuse previously computed attention states to cut repeated prefill work and serving latency.

See all 71 papers

Sep 14, 2026

AgentKV: Phase-Aware KV Eviction for Agentic LLMs

Agentic serving can consume orders of magnitude more tokens than chatbot workloads, stressing both KV-cache capacity and decode-time bandwidth. Most KV-eviction methods score cached keys against representative queries drawn from the most recent tokens, assuming future attention resembles recent attention. We show that agentic generation violates this assumption: future queries form a mixture over think, act, tool, and others phases, and principal-angle analysis shows these components occupy measurably different query subspaces, so recency representatives systematically undervalue keys that upcoming phases will need. We propose AGENTKV, which maintains a small query buffer per phase and scores cached keys against their union. We further implement AGENTKV in a persistent multi-turn serving path that carries compressed KV state across turns and compacts retained KV pages online. Across two models, six task domains, and three KV budgets each, AGENTKV improves task score by 5.5 points on average over R-KV and 5.3 over Tri-attention. Relative to upstream full-KV SGLang, AGENTKV improves output-token throughput by up to 1.80x. Code: https://github.com/LiuTaowen-Tony/agentkv.

Taowen Tony Liu, Jeffrey T. H. Wong, Can Xiao +3

Imperial College London, London, United Kingdom · Columbia University, New York, NY, USA

KV Caching KV-Cache Eviction LLM Inference Acceleration LLM Agent Serving LLM Agents

Ask PDF

Feb 17, 2026

Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devices

Multi-agent LLM systems on edge devices face a memory management problem: device RAM is too small to hold every agent's KV cache simultaneously. On Apple M4 Pro with 10.2 GB of cache budget, only 3 agents fit at 8K context in FP16. A 10-agent workflow must constantly evict and reload caches. Without persistence, every eviction forces a full re-prefill through the model -- 15.7 seconds per agent at 4K context. We address this by persisting each agent's KV cache to disk in 4-bit quantized format and reloading it directly into the attention layer, eliminating redundant O(n) prefill computation via direct cache restoration. The system comprises three components: a block pool providing per-agent isolated Q4 KV caches in safetensors format, a BatchQuantizedKVCache for concurrent inference over multiple agents' quantized caches, and cross-phase context injection that accumulates attention state across conversation phases without re-computation. Evaluated on three architectures (Gemma 3 12B, dense GQA, 48 layers; DeepSeek-Coder-V2-Lite 16B, MoE MLA, 27 layers; Llama 3.1 8B, dense GQA, 32 layers), cache restoration reduces time-to-first-token by up to 136x (Gemma: 22--136x at 4K--32K; DeepSeek: 11--76x at 4K--32K; Llama: 24--111x at 4K--16K; 3--10x at 1K). Q4 quantization fits 4x more agent contexts into fixed device memory than FP16. Perplexity measured with actual Q4 KV caches shows -0.7% for Gemma, +2.8% for Llama, and +3.0% for DeepSeek. Open-source at https://github.com/yshk-mxim/agent-memory

Yakov Pyotr Shkolnikov

Memory Management for LLM Agents LLM Agent Memory LLM Inference Efficiency Persistent Memory for Language Models KV-Cache Offloading Multi-Agent LLM Systems+1

Ask PDF

Sep 25, 2026

ActKV: Efficient LLM Agents through Action-Guided KV Cache Management

Agentic LLM inference accumulates long KV caches across iterative observation-reasoning-action loops, imposing substantial memory overhead and limiting serving throughput. Existing compression methods emphasize overall output quality, overlooking the asymmetric importance of actions in driving task progress. Our key idea is to establish a compression criterion that values KV entries by their contribution to action generation and prioritizes action quality. However, iterative execution, dynamic memory demands, and scattered action-critical entries pose challenges to eviction policies, budget allocation, and paged memory integration. To this end, we propose ActKV, the first KV cache compression framework tailored for agentic LLM inference. (i) Action-oriented KV cache eviction exploits stable action access patterns to retain entries critical to future actions, supporting reliable task progress under compression. (ii) Confidence-driven adaptive budget allocation uses LLM's intrinsic confidence to adapt the budget to evolving action-critical memory demands. (iii) Page-aware compression management standardizes compression into three primitives with customized kernels, realizing practical throughput gains. On long-trace tasks, ActKV retains an average of 98.53% of FullKV's accuracy with only 25.98% of its peak KV cache memory. It also achieves 3.97 times and 3.58 times FullKV's token and task throughput, delivering state-of-the-art performance.

Zihan Wang, Cheng Tang, Lei Gong +4

University of Science and Technology of China Hefei, China · Suzhou Institute for Advanced Research, University of Science and Technology of China Suzhou, China

KV-Cache Management KV-Cache Compression LLM Inference Acceleration KV-Cache Pruning

Ask PDF

Abstract

KV-Cache Reuse

Explore similar work

AgentKV: Phase-Aware KV Eviction for Agentic LLMs

Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devices

ActKV: Efficient LLM Agents through Action-Guided KV Cache Management