CacheRouter: A Dual-Path Tool Routing Architecture with Cache-Preserving Main-Model Isolation for Long-Tail Tool Discovery

Donghui Zha; Lingwei Xu; Linxiao Wu; Yixue Dong; Haochen Li

cs.AIAug 24, 2026

CacheRouter: A Dual-Path Tool Routing Architecture with Cache-Preserving Main-Model Isolation for Long-Tail Tool Discovery

Authors: Donghui Zha, Lingwei Xu, Linxiao Wu, Yixue Dong, Haochen Li

Organizations: School of Mathematical Sciences, Beijing University of Posts and Telecommunications, Beijing 100876, China · School of Mathematics and Statistics, Chongqing University, Chongqing 401331, China · School of Science, Beijing Forestry University, Beijing 100083, China

Abstract

Tool use in LLM systems faces a structural trade-off. Progressive disclosure keeps the prompt small by showing only the tools relevant to the current task, while prompt caching rewards a request prefix that stays fixed across calls; every change to the visible tool list invalidates the cached prefix. This paper treats the trade-off as a problem of request architecture and proposes a dual-path routing design that assigns tool selection and tool delivery to separate channels. The main model always sees a small, fixed set of core tools, so the head of its request is unchanged across calls; all other tools are reached through an independent routing channel, in which a router sub-model searches the full tool list, selects one tool, executes it, and returns the result. Tool registration is automated from source code and supports runtime updates, so the tool set can grow without modifying the main model's request prefix. The design generalizes progressive disclosure: capabilities are disclosed through the routing channel, and the main model's prefix stays stable. A prototype implementation was exercised on 55 functional queries and a 30-turn dialogue; token-level cache hit rates reached 90.99% and 95.2%, cutting input cost to about 12.0% and 8.0% of a no-cache baseline under DeepSeek's pricing, where cache-hit input tokens cost roughly 1/30 of cache-miss tokens.

Figures & tables

Figure 1: Five-layer architecture. Each layer encapsulates complexity downward and provides service abstractions upward; inter-layer communication is entirely based on the MCP protocol or inter-process parameter passing, with no shared internal state.

ID	Rounds	Router	Time (s)	Answer summary
1.1	11	5	150.8	Beijing weather query, saved to file and verified via Router (switched to /tmp sandbox path after path restriction)
1.2	3	0	12.3	rag_project1 retrieved campus, 1 hit
1.3	4	0	23.3	Beijing 31.5°C vs. Shanghai 29.4°C, Beijing hotter
1.4	10	6	115.7	faculty retrieved 0 hits, wrote to file via Router (succeeded after whitelist restriction)
1.5	4	1	26.6	Beijing weather vs. campus retrieval, structured output

Table 1: Experiment 1: Results of multi-step sequential tasks

ID	Rounds	Time (s)	Answer summary
2.1	11	25.7	Parallel Beijing/Shanghai weather queries, including humidity and dew point details
2.2	3	19.1	Parallel retrieval across two RAG projects (BUPT + Mathematics), both successful
2.3	4	17.7	Beijing weather and rag_project1 in parallel
2.4	4	25.2	Shanghai weather and rag_project2 in parallel (doctoral retrieval succeeded)
2.5	13	68.3	Parallel weather queries for three cities, summarized

Table 2: Experiment 2: Results of parallel tasks

Group	ID	Rounds	Time (s)	Answer summary
A	Q1	4	15.7	Beijing weather
A	Q2	3	18.2	rag_project1 BUPT retrieval
A	Q3	3	11.1	Shanghai weather
A	Q4	3	10.1	rag_project2 Mathematics retrieval
A	Q5	3	9.3	Guangzhou weather
B	Q1	3	14.3	Beijing weather (cold start)

Table 3: Experiment 3: Results of long-chain dialogue

ID	Rounds	Time (s)	Answer summary
4.1	0	1.7	2 + 2 = 4, direct answer
4.2	0	1.6	The capital of China is Beijing, direct answer
4.3	4	15.2	Beijing weather (tool query)
4.4	3	26.5	rag_project1 BUPT retrieval succeeded
4.5	4	18.4	Shanghai weather

Table 4: Experiment 4: Results of simple queries

ID	Rounds	Router	Time (s)	Answer summary
5.1	15	1	43.6	Four-city weather comparison, Beijing 31.5°C hottest
5.2	5	4	40.4	Beijing/Shanghai four-round weather queries and comparison
5.3	3	0	10.7	Multi-RAG retrieval (campus/mathematics/information/statistics)
5.4	19	7	175.1	Calculator script (add/subtract/multiply/divide) saved to file, run, and verified on three operations; results reported
5.5	0	0	17.9	Student grade management system code output directly

Table 5: Experiment 5: Results of complex queries

ID	Rounds	Router	Time (s)	Answer summary
6.1	10	7	160.0	Mars weather: cited NASA Perseverance MEDA data (Sol 681), honestly stated data source and limitations
6.2	3	0	6.0	xyznonexistent retrieval returned 0 hits, honestly reported
6.3	4	2	10.7	/root/secret_key.txt permission denied, honestly explained
6.4	3	1	11.9	filenotfound.py does not exist, honestly returned the error
6.5	11	1	29.1	ZzzCity does not exist; router’s attempt to generate a script was blocked, honest fallback

Table 6: Experiment 6: Results of unanswerable-query fallback

ID	Group	Rounds	Router	Time (s)	Answer summary
A1	A	4	0	15.8	Beijing weather
A2	A	3	0	11.3	rag_project1 BUPT retrieval
B1	A	3	1	5.8	Ran test1.py (via Router $\rightarrow$ code-runner)
B2	A	0	0	4.4	$2^{100}$ answered directly
B3	A	0	0	5.1	First 20 Fibonacci numbers answered directly
A1	B	4	1	19.8	Beijing weather

Table 7: Experiment 7: Router triggering and filtering effects

ID	Rounds	Time (s)	Answer summary
t1	3	4.9	say_hello consolidated successfully (other package)
t2	3	40.2	greet consolidated successfully (with default parameters)
t3	23	90.4	calc consolidated with int-conversion fix, 7/7 functional tests passed
t4	13	49.7	batch_process consolidated and functionally verified (sum/avg etc.)
t5	19	62.4	generate_report consolidated, 3 output formats and CLI boundary tests

Table 8: Experiment 8: skill-builder consolidation results

Metric	Value
Input tokens (cache hit)	1,424,768
Input tokens (cache miss)	141,098
Total input tokens	1,565,866
Token-level cache hit rate	90.99%

Table 9: Aggregate token-level cache statistics for Experiments 1–8

Metric	Value
Queries/cases	30
Success / timeout / failure	30 / 0 / 0
Router calls	1 (Q20)
Total tokens	249,952 (input 238,701 + output 11,251)
Total in-query time	136.5 s
Input tokens (total)	238,701

Table 10: Long-chain multi-turn dialogue experiment: overall results

Path	Count	Avg. rounds	Avg. time	Avg. tokens
Direct answer ( DIRECT_ANSWER )	16	0	$\approx$ 1.4 s	$\approx$ 1.9K
Tool execution ( NEED_TOOLS )	14	2–3	$\approx$ 8.1 s	$\approx$ 14.9K

Table 11: Statistical comparison of direct-answer and tool-execution paths

ID	Question	Path	Rounds	Tokens	Time (s)
Q1	What is 2 + 2?	Direct answer	0	1,002	1.0
Q6	What is 5 times 6?	Direct answer	0	1,282	0.9
Q20	How many sides does a hexagon have?	Tool loop (with Router)	3	22,458	5.2
Q25	What is 7 times 8?	Tool loop	3	23,875	5.2
Q28	How many minutes are in one hour?	Tool loop	3	24,577	5.3

Table 12: Key comparison cases between hit and miss paths

Tool Discovery for LLM Agents

See all 16 papers

ReCache: Efficient KV Cache Reuse and Compression for Tool-Augmented LLM Agents
Aug 20, 2026Yichu Fang, Sitong Wei, Haozhe Hu +1KV-Cache Reuse LLM Inference Acceleration
Shanghai Jiao Tong University · Eastern Institute of Technology, Ningbo · Xi’an Jiaotong University
CacheRL:Multi-Turn Tool-Calling Agents via Cached Rollouts and Hybrid Reward
Jun 12, 2026Md Amirul Islam, Sumiran Thakur, Huancheng Chen +3LLM Agent Training Tool-Using Language Agents
Center for Advanced AI, Accenture
Irminsul: MLA-Native Position-Independent Caching for Agentic LLM Serving
May 7, 2026Bole Ma, Jan Eitzinger, Harald KöstlerKV Caching LLM Inference Serving
Erlangen National High Performance Computing Center · Erlangen, Germany

Aug 20, 2026J/K move · Enter open · S save

ReCache: Efficient KV Cache Reuse and Compression for Tool-Augmented LLM Agents

Yichu Fang, Sitong Wei, Haozhe Hu, Xiaoyu Shen

Shanghai Jiao Tong University · Eastern Institute of Technology, Ningbo · Xi’an Jiaotong University

KV-Cache Reuse LLM Inference Acceleration Tool-Augmented Language Model Agents Memory-Efficient Inference KV-Cache Compression

Open paper AskPDF arXiv

Agentic language models repeatedly encode tool and skill schemas that recur across requests in different combinations and orders, preventing standard prefix caching from reusing their key--value (KV) states. We introduce ReCache, a framework for independently caching resource representations while reducing their inference-time computational and memory overhead. Resource-wise attention removes cross-resource interactions and assigns resource-local positions, producing composition-invariant KV blocks. ReCache then restricts resource visibility to contribution-selected layer--KV-head-group routes and retains only invocation-critical fields through structural and semantic pruning. We evaluate ReCache on a benchmark assembled from seven public tool- and skill-use datasets, including resource-disjoint tests. Resource-wise attention matches dense invocation performance (82.3% versus 82.4% Inv-F1) while providing a 3.655

\times

time-to-first-token speedup. The complete framework reduces allocated KV-tensor memory by 92.43% and accelerates attention by 1.423

\times

. These results show that separating reusable schema encoding from selective resource access substantially reduces agentic inference costs with limited effectiveness loss. The code is available at https://github.com/EIT-NLP/ReCache.

Abstract

Tool Discovery for LLM Agents

Explore similar work

ReCache: Efficient KV Cache Reuse and Compression for Tool-Augmented LLM Agents

CacheRL:Multi-Turn Tool-Calling Agents via Cached Rollouts and Hybrid Reward

Irminsul: MLA-Native Position-Independent Caching for Agentic LLM Serving