Folding Pooled Usage Examples into the Tool Index Lowers Retrieval of Unused Tools

Abhijit Chakraborty

cs.SEOct 22, 2026

Folding Pooled Usage Examples into the Tool Index Lowers Retrieval of Unused Tools

Authors: Abhijit Chakraborty

Organizations: Arizona State University

Abstract

AI agents often find tools by searching their documentation. Agents that share a tool catalog can also pool usage examples: past requests, each tagged with the tool that handled it. We ask what pooling does to tools no agent has used yet. Adding each usage example to the search index as an extra entry for its tool (folding) helps tools that have examples, but on two of three catalogs it sharply lowers how often a needed unused tool reaches the top five results; on StableToolBench with our main embedding model, the share falls from 73 to 40 percent of queries. When the extra entries hold documentation from other tools instead of requests, the loss is much smaller, though differences in content or writing style could also explain this. We introduce ZoE (Z-scores over Evidence), a scoring rule that scores the documentation and the usage of a tool separately, so a tool without usage keeps its documentation score. On all three catalogs and with both embedding models we tested, ZoE retrieves unused tools more often than a router that reserves result slots for usage matches. With our main embedding model, the router retrieves more of the other tools on two catalogs, and in post hoc comparisons a simpler rule that adds a constant to the scores of tools with usage shows no detected difference from ZoE on unused tools. For five simulated agents on one catalog, folding lowered retrieval of unused tools for every agent; with ZoE no average drop was detected, though some agents still lost. Evaluations of pooled agent experience should report tools no agent has used separately from the average.

Figures & tables

Figure 1: Folding usage into the index lowers held-out Hit@5 on StableToolBench and TOUCAN, while ZoE’s held-out Hit@5 is not detectably below documentation-only. Each row shows one rule’s Hit@5 (the share of queries with a correct tool among the five retrieved) on held-out (circle) and labeled (square) test queries, with Jina embeddings, three seeds and settings chosen on development data; note the different x-axis ranges. Documentation-only searches documentation alone. Untyped grouped is folding without the tool’s catalog fields (its identifier, category, API names and documentation); it was not run on TOUCAN with Jina.

Index	H	L
Documentation only	.733	.658
Folded (real usage)	.400	.801
Other tools’ usage, same length	.422	.474
One example appended to documentation	.429	.671
Documentation + one request per tool	.472	.530
Other tools’ requests (text-type control)	.424	.510

Table 1: StableToolBench controls: Hit@5 on held-out-test (H) and labeled-test (L) queries, Jina, three seeds. Rows below Folded are index variants described in the text; “one request per tool” is a real usage example for used tools and a generated request for the rest.

Figure 2: Text-type control: other tools’ requests used as usage give held-out Hit@5 2.5 points above folding’s on StableToolBench and not detectably different on TOUCAN, and other tools’ documentation matched to each entry’s token length gives a much smaller loss. Both replacements come from the same donor tool and match each entry’s token length; Hit@5, Jina, three seeds.

Figure 3: On StableToolBench and TOUCAN, rules that keep tools without usage at their documentation score sit at the right of each panel (high held-out Hit@5), though the constant offset loses labeled Hit@5 on StableToolBench and TOUCAN (point estimates); folding, additive RRF and, on TOUCAN, best-field RRF lose held-out Hit@5. Held-out-test against labeled-test Hit@5, Jina, three seeds. The ZoE curve runs from documentation only ( $w{=}0$ , diamond) to $w{=}1$ , where $w$ is ZoE’s usage weight; squares are router settings with $d$ documentation slots out of five. Circles mark the settings frozen on development data before test scoring; all other points were selected or plotted after the frozen run. Table 6 gives the values. On StableToolBench and ToolRet, best-field RRF returns the same lists as documentation-only.

	StableToolBench		ToolRet		TOUCAN
Rule	H	L	H	L	H	L
Documentation only	.733	.658	.306	.331	.486	.323
Folded	.400	.801	.290	.374	.203	.612
Additive RRF	.627	.743	.292	.350	.346	.532
Router	.683	.826	.267	.386	.419	.580
Thresholded router	.731	.736	.302	.348	.486	.343

Table 2: Main combination rules on three catalogs: Hit@5 on held-out-test (H) and labeled-test (L) queries, Jina, three seeds. Rules in the lower block leave tools without usage at their documentation score. Only the router and ZoE settings were frozen before test scoring; documentation and folding have no settings.

Figure 4: Pooling lowers every agent’s held-out Hit@5 under folding; under ZoE no average change is detected. Change in each agent’s Hit@5 when it retrieves with all agents’ usage instead of its own (dots: 5 agents

\times

3 seeds; diamonds: means), StableToolBench, Jina. The router’s held-out change is zero because held-out tools have no usage in any pool. Pooled minus local: Hit@5 with all agents’ usage minus Hit@5 with the agent’s own.

\alpha

: Dirichlet concentration, where smaller values give agents more different tool mixes; hash: hash split.

Figure 5: Hash split with similar tool mixes: a single agent’s ZoE Hit@5 drops as the usage pool is split among more agents, most on labeled queries; pooling recovers it. Horizontal lines, marked “pooled (both)”, are pooled Hit@5 (dashed: held-out; dotted: labeled). StableToolBench, averaged over agents and three seeds.

Figure 6: Folding lowers held-out Hit@5 at every held-out share we tried, and ZoE is between 1.4 points below and 1.1 points above documentation-only (point estimates; no paired interval). TOUCAN, the 123 queries held out at all three shares (21 tool clusters); solid lines use all remaining usage, dotted lines subsample it to the same record count at every share; the router is at its frozen setting.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Population	Queries
Held-out, development (tuning only)	135
Labeled, development (tuning only)	119
Held-out, gold tools span halves	13
Labeled, gold tools span halves	72
Gold tools both held out and labeled	80
Total left out	419

Appendix

Table 3: StableToolBench queries left out of the test results, counted once per query.

Catalog	Labeled	All gold covered	Share
StableToolBench	447	394	88%
ToolRet	2,184	449	21%
TOUCAN	7,785	7,533	97%

Appendix

Table 4: Labeled-test rows pooled over three seeds, and rows whose every gold tool has usage.

Figure 7: ZoE’s held-out lead over the router narrows with more candidates on StableToolBench and ToolRet but not on TOUCAN, and its covered-query gain over documentation stays positive at 5, 10 and 30 candidates (the StableToolBench interval at 30 touches zero). Paired differences in Hit@

k

points with tool-clustered 95% intervals (bands); covered labeled means every gold tool has usage. The router keeps its

k{=}5

share of slots for usage (one in five on StableToolBench and ToolRet, two in five on TOUCAN), rounded to whole slots, so at

k{=}1

it equals documentation-only. Settings frozen at

k{=}5

(dotted line); three seeds.

Figure 8: Other tools’ usage text of the same length gives held-out Hit@5 not detectably different from real usage (StableToolBench

+2.3

, TOUCAN

-0.9

points) and much lower labeled Hit@5. Usage entries are replaced by length-matched text from other tools’ usage; Hit@5, Jina, three seeds.

	StableToolBench		ToolRet		TOUCAN
Rule	H	L	H	L	H	L
Documentation	.733	.658	.306	.331	.486	.323
Folded	.400	.801	.290	.374	.203	.612
Untyped grouped	.203	.745	.197	.381	–	–
Router	.683	.826	.267	.386	.419	.580
ZoE	.785	.805	.294	.363	.506	.488

Appendix

Table 5: Values for Figure 1 : Hit@5 on held-out-test (H) and labeled-test (L) queries, three seeds, frozen settings.

	StableToolBench		ToolRet		TOUCAN
Setting	H	L	H	L	H	L
ZoE $w{=}0.0$	.733	.658	.306	.331	.486	.323
ZoE $w{=}0.1$	.745	.707	.305	.332	.501	.341
ZoE $w{=}0.2$	.756	.749	.306	.332	.511	.366
ZoE $w{=}0.3$	.764	.785	.306	.335	.519	.389
ZoE $w{=}0.4$	.778	.805	.306	.331	.526	.415

Appendix

Table 6: Values for Figure 3 : Hit@5 on held-out-test (H) and labeled-test (L) queries for every ZoE usage weight

w

and every router setting

d

(documentation slots out of five), Jina, three seeds.

w{=}0

is documentation-only. ∗ : frozen setting. All other points were computed after the frozen run.

Catalog		Docs	Real	Other	Difference
StableTB	H	.733	.400	.422	$+2.28$ [ $-0.20$ , 5.29]
	L	.658	.801	.474	$-32.66$ [ $-40.04$ , $-24.93$ ]
TOUCAN	H	.486	.203	.194	$-0.87$ [ $-2.06$ , 0.24]
	L	.323	.612	.347	$-26.51$ [ $-30.76$ , $-22.43$ ]

Appendix

Table 7: Values for Figure 8 (StableTB: StableToolBench; H, L: held-out and labeled test queries). Docs: documentation only; Real: real usage (folded); Other: other tools’ usage of the same length. Last column: Other minus Real, in points, with paired tool-clustered 95% interval.

Agents	Local H	Local L	Pooled H	Pooled L
1	.785	.805	.785	.805
2	.780	.781	.785	.805
5	.767	.713	.785	.805
10	.760	.674	.785	.805

Appendix

Table 8: Values for Figure 5 : StableToolBench Hit@5. Local: each agent retrieves with its own usage, averaged over agents; Pooled: with all agents’ usage. H, L: held-out and labeled test queries.

Catalog	$k$	vs router, H	vs router, C	vs docs, H	vs docs, C
StableToolBench	5	$+10.14$ [5.68, 15.05]	$-2.79$ [ $-6.82$ , 0.66]	$+5.18$ [2.31, 8.39]	$+15.99$ [5.75, 24.12]
	10	$+4.55$ [1.83, 7.71]	$+1.27$ [ $-3.37$ , 5.07]	$+3.93$ [1.35, 6.90]	$+13.96$ [3.93, 22.77]
	30	$+6.00$ [2.48, 10.04]	$-0.51$ [ $-4.14$ , 2.40]	$+4.14$ [1.36, 7.26]	$+8.12$ [0.00, 15.40]
ToolRet	5	$+2.75$ [0.67, 5.06]	$-17.15$ [ $-27.45$ , $-8.73$ ]	$-1.18$ [ $-2.85$ , 0.24]	$+16.70$ [4.72, 28.64]
	10	$+2.09$ [0.52, 4.02]	$-21.16$ [ $-29.82$ , $-13.15$ ]	$-0.26$ [ $-0.82$ , 0.00]	$+19.38$ [7.64, 31.27]
	30	$+0.92$ [0.13, 2.05]	$-19.82$ [ $-28.57$ , $-12.41$ ]	$-0.65$ [ $-1.52$ , 0.00]	$+20.71$ [7.18, 34.23]

Appendix

Table 9: Values for Figure 7 at

k\in\{5,10,30\}

: ZoE minus the router and minus documentation-only, in Hit@

k

points with paired tool-clustered 95% intervals. H: held-out-test; C: labeled-test queries whose every gold tool has usage.

Catalog	Rule	H	L	C
StableTB	Documentation	.783	.819	.812
	Folded	.621	.902	.916
	Router	.745	.895	.901
	ZoE	.822	.879	.873
	Untyped grouped	.437	.841	.881
	Same-length	.640	.774	.774

Appendix

Table 10: Nomic Embed v1.5, Hit@5 on held-out-test (H), labeled-test (L) and fully covered labeled (C) queries, three seeds, settings reselected on development tools. StableTB: StableToolBench. Same-length: the same-length control, run on StableToolBench only.

		Documentation		Folded		Folded (1 entry)		ZoE		Offset
Embedder	Catalog	H	L	H	L	H	L	H	L	H	L
Jina	StableToolBench	.733	.658	.400	.801	.553	.682	.785	.805	.770	.626
	ToolRet	.306	.331	.290	.374	.290	.360	.294	.363	.282	.342
	TOUCAN	.486	.323	.203	.612	.287	.503	.506	.488	.509	.286
Nomic	StableToolBench	.783	.819	.621	.902	.687	.839	.822	.879	.826	.776
	ToolRet	.388	.387	.370	.434	.371	.422	.384	.414	.384	.406

Appendix

Table 11: Hit@5 on held-out-test (H) and labeled-test (L) queries for one-entry folding and the constant offset, three seeds. Folded (1 entry): one-entry folding, one usage entry per used tool. Offset: documentation score plus a development-tuned constant for tools with usage.

Emb.	Catalog	H	L
Jina	StableTB	$+5.38$ [2.31, 9.05]	$-54.59$ [ $-61.07$ , $-46.88$ ]
	ToolRet	$+0.92$ [0.14, 1.87]	$-6.18$ [ $-9.33$ , $-3.55$ ]
	TOUCAN	$+1.56$ [0.54, 2.68]	$-46.65$ [ $-54.49$ , $-39.29$ ]
Nomic	StableTB	$+5.80$ [2.12, 10.10]	$-49.89$ [ $-56.13$ , $-41.95$ ]
	ToolRet	$+0.00$ [0.00, 0.00]	$-6.68$ [ $-9.58$ , $-4.23$ ]
	TOUCAN	$+1.39$ [0.47, 2.65]	$-45.87$ [ $-55.16$ , $-37.32$ ]

Appendix

Table 12: Permuted-usage control: ZoE with every tool’s usage replaced by another tool’s, minus ZoE, at the frozen weight (H, L: held-out and labeled test queries; Emb.: embedding model; StableTB: StableToolBench), in Hit@5 points with paired tool-clustered 95% intervals.

Index	H	L
Documentation	.733	.658
Folded	.400	.801
Raw, all (untyped grouped)	.203	.743
Raw, one	.358	.521
Appended, one example	.429	.671
Appended, all usage	.424	.617

Appendix

Table 13: Equal-entry and generated-request controls: StableToolBench Hit@5 on held-out-test (H) and labeled-test (L) queries, Jina, three seeds.

Comparison	H	L
App. one $-$ docs	$-30.43$ [ $-37.50$ , $-23.53$ ]	$+1.34$ [ $-5.37$ , 7.13]
App. all $-$ docs	$-30.85$ [ $-37.53$ , $-24.31$ ]	$-4.03$ [ $-12.38$ , 4.88]
App. all $-$ folded	$+2.48$ [ $-1.39$ , 6.58]	$-18.34$ [ $-27.40$ , $-9.07$ ]
App. one $-$ raw, one	$+7.04$ [3.21, 11.18]	$+14.99$ [8.15, 21.49]
One per tool $-$ docs	$-26.09$ [ $-33.74$ , $-18.43$ ]	$-12.75$ [ $-21.77$ , $-3.07$ ]
One per tool $-$ raw, one	$+11.39$ [6.67, 16.86]	$+0.89$ [0.00, 2.58]

Appendix

Table 14: Paired differences for Table 13 , in Hit@5 points with tool-clustered 95% intervals. App.: appended; Raw+gen.: raw usage plus generated requests; Gen., all: the Re-Invoke-inspired index; docs: documentation-only; other names as in Table 13 . H, L: held-out and labeled test queries.

Split	Rule	H mean	H worst agent	H share losing	L mean	L worst agent
$\alpha{=}0.1$	Folded	$-$ 13.25 [ $-$ 22.13, $-$ 5.29]	$-$ 31.20	1.00	2.89 [0.16, 5.98]	$-$ 2.63
	Router	0.00 [0.00, 0.00]	0.00	0.00	3.02 [0.72, 5.88]	$-$ 3.08
	ZoE	$-$ 1.32 [ $-$ 5.99, 2.98]	$-$ 9.93	0.40	7.25 [4.31, 10.70]	3.64
$\alpha{=}0.3$	Folded	$-$ 12.10 [ $-$ 19.04, $-$ 5.93]	$-$ 20.09	1.00	4.56 [2.40, 7.11]	1.19
	Router	0.00 [0.00, 0.00]	0.00	0.00	4.15 [1.72, 7.01]	$-$ 1.06
	ZoE	$-$ 0.86 [ $-$ 4.58, 2.05]	$-$ 5.70	0.47	6.08 [3.67, 8.93]	2.51

Appendix

Table 15: Pooling with all agents minus retrieving locally, StableToolBench, Jina, five agents, three seeds: mean change in Hit@5 points with tool-clustered 95% interval, worst agent, and share of agent-seed cells that lose. H, L: held-out and labeled test queries.

Catalog	Baseline	H	L
StableTB	Additive RRF	$+15.73$ [10.53, 21.19]	$+6.26$ [0.00, 12.11]
	Renormalized RRF	$+0.62$ [ $-0.22$ , 1.57]	$+3.80$ [0.43, 7.60]
	Best-field RRF	$+5.18$ [2.31, 8.39]	$+14.77$ [5.51, 22.55]
	Thresholded router	$+5.38$ [2.43, 8.71]	$+6.94$ [1.15, 11.75]
ToolRet	Additive RRF	$+0.26$ [ $-0.93$ , 1.50]	$+1.24$ [ $-1.17$ , 4.11]
	Renormalized RRF	$-0.26$ [ $-1.61$ , 1.02]	$+0.14$ [ $-1.80$ , 2.55]

Appendix

Table 16: ZoE minus each fusion baseline, in Hit@5 points with tool-clustered 95% intervals. H, L: held-out and labeled test queries; StableTB: StableToolBench.

Catalog	Comparison	H	L
StableTB	D $-$ R	$+21.53$ [15.04, 28.29]	$+14.32$ [8.18, 21.14]
	R $-$ folded	$+2.48$ [0.21, 5.13]	$-29.08$ [ $-35.66$ , $-20.87$ ]
	D $-$ docs only	$-9.32$ [ $-13.68$ , $-5.32$ ]	$-0.45$ [ $-4.05$ , 2.58]
	No-rep. D $-$ R	$+12.42$ [7.25, 18.13]	$+5.15$ [0.00, 11.99]
TOUCAN	D $-$ R	$+22.97$ [12.54, 32.88]	$+6.80$ [3.24, 11.25]
	R $-$ folded	$+0.61$ [ $-2.00$ , 2.59]	$-30.13$ [ $-35.08$ , $-25.43$ ]

Appendix

Table 17: Text-type control, in Hit@5 points with tool-clustered 95% intervals. D: other tools’ documentation; R: other tools’ requests; docs only: the documentation-only baseline. No-rep.: only entries where neither replacement was repeated, which also changes which tools keep usage. StableTB: StableToolBench.

Share	Usage	Folded $-$ docs	ZoE $-$ folded
10%	all	$-23.31$ [ $-39.93$ , $-13.82$ ]	$+21.95$ [9.79, 42.65]
	matched	$-19.78$ [ $-34.10$ , $-11.78$ ]	$+20.33$ [8.69, 39.58]
30%	all	$-20.87$ [ $-36.82$ , $-12.15$ ]	$+19.51$ [10.84, 35.29]
	matched	$-20.33$ [ $-35.78$ , $-11.85$ ]	$+20.60$ [11.39, 36.81]
50%	all	$-15.72$ [ $-28.02$ , $-8.55$ ]	$+16.80$ [8.91, 30.08]

Appendix

Table 18: Held-out-share sweep on the fixed 123 queries, in Hit@5 points with tool-clustered 95% intervals. Usage “all” keeps every remaining record; “matched” subsamples records to the count at the 50% share. Docs: documentation-only.

Tool Selection for Language Model Agents

See all 33 papers

Oct 16, 2026

Tool Recommendation for Multi-Step Agents: Finding Every Tool a Task Needs

A multi-step agent task fails when a single tool it needs is missing, yet the tools an agent may need are too many to list: the roughly 8,200 tools on the platform we study have definitions that together exceed the context window of any model we evaluate. We present Tool Recommendation, a deployed system that gives an agent all of them, hundreds of times more than a typical MCP server exposes, through a fixed interface of three tools. It runs at the gateway between agent and tools, executing every call under the end user's own credentials and checking it against the user's consent for each service and the tools the agent may use. We evaluate it on both a public retrieval benchmark and billed end-to-end session cost. On ToolRet, Tool Recommendation places every tool a query needs in the top ten 57% more often than BM25, a lexical ranking that commercial tool search offers, and compares favorably with published systems up to nine times its size. In paired multi-step workflows, its first recommendation covered every required tool in 93% of Codex sessions, and against a fixed list of 100 tools it cut input tokens by 17% with Codex and 40% with Claude Code and the total bill in all 14 runs, by up to 40%, with no change in task completion.

Sam Partee, River Stevens, Thierry Damiba +1

Tool Selection for Language Model Agents Tool Discovery for LLM Agents LLM Agent Efficiency

Ask PDF

Apr 12, 2026

Agent-Facing Information Design in LLM Tool Registries: A Preregistered Test of Rhetoric, Position and Structure

AI agents often pick tools from registries, where each tool's provider writes its description. We ask whether sales language in those descriptions changes which tool an agent picks. We built pairs of listings differing in one controlled way (added praise, a verifiable specification, or list order) and asked two OpenAI models to call one tool. In a preregistered study, stacked praise (four kinds combined) raised a tool's pick rate by about 43 percentage points, matching or beating a verifiable specification. Praise also pulled some picks toward tools that could not do the task, but rarely toward tools asking for unneeded data access. With identical listings, the first-listed tool was picked about 72 points more often. On tasks with numeric limits, structured fields helped agents pick the capable tool; adding the provider's sales text beside the fields reduced or erased that gain. Registries could list limits as fields, hide sales text from agents, and randomize order. Stacked praise, but no single kind, replicated on held-out domains. Results are provisional until blind phrase ratings are complete, and cover two small models.

Haochuan Kevin Wang, Zechen Zhang

Massachusetts Institute of Technology, Cambridge, MA, USA.

Tool-Use Evaluation Tool-Augmented Language Model Agents Tool Selection Tool-Using Language Agents LLM Tool Use Information Design+2

Ask PDF

Jul 6, 2026

ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents

Tool calling is central to modern language model agents, but aggregate benchmark scores often hide where tool use fails. A model that never calls a needed tool and a model that calls the tool but ignores the result can look similar under final task accuracy. We introduce ToolFailBench, a diagnostic benchmark for measuring tool-use failures across 1,000 tasks in finance, medicine, law, cybersecurity, and real estate. Tool-required tasks return values the model wouldn't guess, forcing it to trust the tool while control tasks attach the same tools but should be answered directly. We label each trace with Tool-Skip, Result-Ignore, Output-Fabrication, and Unnecessary-Tool-Use, using a rule classifier and two LLM judges aggregated by majority vote. Across 19 headline models, the best reaches 86.33% Clean Tool-Use Rate, showing that faithful tool use is not saturated. More importantly, models with similar aggregate scores fail in different ways: most stay disciplined on no-tool controls, while Llama-3.1 models show an Always-Call pattern, and at the same parameter scale Llama-3.1-70B and Qwen2.5-72B differ by 89 percentage points on control-task accuracy. Tool-use evaluation should measure not only whether agents call tools, but whether they use tool outputs correctly and avoid tools when none is needed.

Harsh Soni

UC Berkeley

AI Agent Reliability Tool-Use Evaluation Tool-Augmented Language Model Agents AI Agent Benchmarks LLM Agent Reliability Failure Attribution for LLM Agents

Ask PDF

Abstract

Tool Selection for Language Model Agents

Explore similar work

Tool Recommendation for Multi-Step Agents: Finding Every Tool a Task Needs

Agent-Facing Information Design in LLM Tool Registries: A Preregistered Test of Rhetoric, Position and Structure

ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents