Large-scale agents increasingly rely on retrieval to access external capabilities. We study this retrieval gate in structured tools and APIs, a measurable class of tool-backed executable skills that must be surfaced before an agent can plan, incorporate, or act. In this setting the retrieval layer can silently fail even when the capability corpus is fixed: on ToolRet, a retriever fine-tuned on one source-specific slice collapses on another source-specific slice of the same benchmark, with FT-1100 despite its higher lexical overlap with the gold tools. We call this failure mode source-style collapse. Query-side TF-IDF fingerprints flag source styles on which the fine-tuned retriever is likely to fail better than semantic or length-based proxies, giving a cheap signal for mismatch over a fixed tool corpus. We propose ToolScout, a source-aware routing method that uses this signal as a routing guard: on the mixed 4,996-query stream, TF-IDF-based routing raises coverage from 22.3% to 86.1%, and across five collapsed sources 20 matched examples raise the coverage-weighted global top-1 proxy from 1.3% to 53.9%. The same failure and routing behaviors persist when tools are rerendered as executable skill cards, which rules out raw API-schema format as the sole cause.
Figures & tables
Figure 1: Tool retrieval is source-style-sensitive on a fixed benchmark corpus. Top: retrieval compresses benchmark-native full-catalog exposure from about 172k tokens to about 407 at retrieved top- 5 . (A) Lexical overlap fails to predict retrievability: APIGen has the highest query-to-gold-tool overlap yet collapses to 0.7% coverage under FT-1100 . (B) TF-IDF distance separates source styles more clearly than BGE-M3 semantic distance ( r=-0.85 vs. -0.58 ). (C) Route, switch, then repair: source-aware routing increases mixed-stream coverage by nearly fourfold, and 20 matched examples substantially improve the collapsed-source coverage-weighted global top- 1 proxy.
Benchmark
Setting
Cov.
P@1
Sem.
Lrn.
ToolRet
bge-small / full 1100
67.7%
25.0%
50.0%
BGE-M3 / full 1100
80.8%
34.9%
55.8%
StableToolBench
bge-small / full 765
97.3%
70.1%
71.0%
BGE-M3 / full 765
98.4%
66.7%
68.5%
Table 1: Off-the-shelf two-stage results under a shared direct retrieval pipeline. “Sem.” denotes retriever-only top- 1 accuracy, and “Lrn.” denotes top- 1 accuracy after the lightweight learned reranker. Coverage is top- 20 candidate coverage. The results show that ToolRet leaves substantially more candidate-generation headroom than StableToolBench under the same protocol. Full covered-query metrics are provided in Appendix A.1 .
Figure 2: Direct-dense retrieval adaptation across ToolRet- 1100 , ToolRet- 4996 , and ToolRet-full. The plotted score is the coverage-weighted top-1 proxy. A substantially larger 2025 off-the-shelf backbone ( Qwen3-Emb-4B ) narrows the gap in every ToolRet setting while leaving sizable room for source-adapted retrieval. Exact values are listed in Appendix A.3 .
FT-1100
20 -shot
FT-aggregate
Source
Cov.
Global
Cov.
Global
Cov.
Global
APIGen
0.7%
0.3%
93.2%
74.4%
96.6%
73.5%
ToolACE
0.0%
0.0%
77.1%
46.0%
81.7%
47.2%
UltraTool
8.3%
5.7%
84.7%
48.1%
98.6%
81.9%
toollens
5.4%
1.3%
58.9%
27.1%
89.3%
47.3%
reversechain
5.1%
2.3%
89.7%
61.5%
79.5%
32.3%
Table 2: Multi-source few-shot repair from a collapsed FT-1100 checkpoint. “Global” is the coverage-weighted multilayer perceptron (MLP) top-1 proxy. The table reports the headline 20 -shot setting only; the full 5/10/20/50 -shot sweep is listed in Appendix A.3 . The reversechain row has only 39 held-out test queries, so we treat its score cautiously. Sample complexity varies by source: the appendix sweep shows that 50 -shot continuation closes the UltraTool gap and narrows toollens further.
Rendering
1100 OTS
1100 FT
FT-1100 → 4996
4996 OTS
4996 FT
Switch
Tool schema
80.8%
91.8%
22.3%
73.7%
85.2%
86.1%
Skill card
72.2%
89.2%
21.7%
64.5%
84.0%
85.2%
Table 3: Skill-card rendering check. Coverage is depth- 20 candidate coverage. The skill-card rendering keeps the same tools, labels, splits, retrievers, and routing policy, while rewriting visible tool text into normalized executable skill cards. Collapse and the switch policy persist under the more skill-like representation. Full per-source values and the deterministic card template are in Appendix A.5 .
Figure 3: ToolRet depth sweep on the full 1100 -query normalized web subset with BGE-M3 . Coverage rises with depth, but the coverage-weighted top-1 proxy peaks around the main 20 -candidate operating point and then declines. Axes use truncated ranges for readability. Exact values are listed in Appendix A.6 .
Table 5: Stronger reranker controls on the main dense runs. Metrics are reported on the same held-out candidate-covered test splits as Table 1 . Cross-encoders substantially raise the covered-query ceiling, and they still operate only after candidate generation has already succeeded.
Reranker
P@1
R@5
nDCG@5
BM25
27.0%
43.0%
35.1%
Semantic
33.2%
57.4%
45.3%
Linear
35.2%
58.1%
47.0%
MLP
41.0%
69.5%
55.6%
Cross-encoder
70.0%
89.6%
79.9%
Appendix
Table 6: ToolRet aggregate stress test on 4996 queries from 19 query configs. Metrics are reported on the held-out candidate-covered test split from the same reranker protocol as Table 1 . The larger aggregate is substantially harder upstream ( 73.7% candidate coverage), and once candidates are present the same reranker ordering reappears.
Source
Retriever
Cov.
P@1
Global
ToolRet-1100
BGE-M3
80.8%
55.8%
45.1%
Qwen3-Emb-4B
87.3%
58.3%
50.9%
BGE-M3 FT on 1100
91.8%
70.3%
64.5%
ToolRet-4996
BGE-M3
73.7%
41.0%
30.2%
Qwen3-Emb-4B
78.9%
46.8%
36.9%
BGE-M3 FT on 4996
85.2%
56.5%
48.1%
Appendix
Table 7: Exact values behind the main-paper retrieval-adaptation summary figure. “Global” is the coverage-weighted top-1 proxy.
Negatives
FT Cov.
Δ Cov.
FT Multi
FT Recall
Semantic hard
88.5%
+16.2
89.3%
71.7%
Semantic + random
88.5%
+16.2
90.0%
68.2%
BM25 hard
87.8%
+15.5
89.3%
69.2%
Random
84.5%
+12.2
85.7%
68.1%
Same-category
78.4%
+6.1
79.3%
58.8%
Appendix
Table 8: Controlled hard-negative ablation for BGE-M3 + MNRL on the 1100 validation split. All rows start from the same off-the-shelf depth- 20 base coverage of 72.3% . Retrieval-confusing semantic negatives are strongest, while same-category negatives are much weaker despite being semantically plausible distractors.
Train Source
Eval Source
Cov.
Multi
Avg Tok.
Off-the-shelf
ToolRet-1100 val
72.3%
72.9%
1104.4
FT on 1100
ToolRet-1100 val
88.5%
90.0%
910.9
FT on 1100
ToolRet-1100 test
91.8%
92.0%
930.6
Off-the-shelf
ToolRet-4996 test
65.5%
63.5%
1194.4
FT on 4996
ToolRet-4996 test
85.2%
89.5%
836.2
FT on 4996
Transfer to 1100 test
87.3%
88.0%
830.3
Appendix
Table 9: Depth- 20 candidate-generation coverage for task-adapted dense retrievers. The 1100 -trained model is strongest on matched data, while the aggregate-trained model generalizes across the broader 4996 -query setting.
Training mix
4996
1100 transfer
Collapsed-5
toolbench
Natural aggregate
85.2%
87.3%
89.3%
87.3%
Balanced
81.8%
74.1%
88.5%
74.1%
Temp.-reweighted
81.8%
76.6%
88.3%
76.6%
Appendix
Table 10: Naive mixed-training controls for aggregate retriever fine-tuning. All rows use the same BGE-M3 + MNRL protocol and the same total number of one-per-query training examples; only the source sampling distribution changes. Coverage is reported at candidate depth 20 . “Collapsed-5” is the query-count-weighted mean over APIGen , ToolACE , UltraTool , toollens , and reversechain .
Retriever
toolbench Cov.
toolbench Margin
Collapsed-5 Cov.
Collapsed-5 Margin
Probe F1
Off-the-shelf
73.4%
-0.033
71.9%
-0.026
82.2%
FT-1100
91.8%
+0.018
2.5%
-0.371
76.6%
FT-aggregate
87.3%
-0.016
89.3%
+0.003
91.3%
Balanced aggregate
74.1%
-0.044
88.5%
-0.009
90.9%
Temp.-reweighted aggregate
76.6%
-0.039
88.3%
-0.002
91.0%
Appendix
Table 11: Representation-geometry probe for the mixed-training control. Margins are the mean best-gold score minus strongest non-gold score at candidate depth 20 ; higher is better. Probe F1 is a cross-validated source-style linear classifier over query embeddings for the six displayed sources.
Source
FT-1100
5 -shot
10 -shot
20 -shot
50 -shot
FT-aggregate
APIGen
0.7%
0.7%
—
93.2%
—
96.6%
ToolACE
0.0%
0.0%
68.0%
77.1%
86.3%
81.7%
UltraTool
8.3%
8.3%
81.9%
84.7%
98.6%
98.6%
toollens
5.4%
5.4%
28.6%
58.9%
87.5%
89.3%
reversechain
5.1%
5.1%
84.6%
89.7%
84.6%
79.5%
Appendix
Table 12: Coverage sweep for lightweight matched-source repair from a collapsed FT-1100 checkpoint. The main paper reports the headline 20 -shot setting; this table gives the full 5/10/20/50 -shot progression. Dashes for APIGen mark shot counts skipped in the initial pilot.
Eval Source
Retriever
Lin. P@1
MLP P@1
MLP Global
ToolRet-1100
Off-the-shelf
55.8%
64.3%
51.9%
FT on 1100
70.3%
69.7%
63.9%
FT on 4996
54.8%
55.5%
48.5%
ToolRet-4996
Off-the-shelf
35.2%
41.0%
30.2%
FT on 1100
36.9%
48.7%
10.9%
FT on 4996
56.3%
56.5%
48.1%
Appendix
Table 13: Direct dense reranker pipelines under task-adapted dense retrievers. “MLP Global” is the coverage-weighted top-1 proxy using the appropriate candidate-generation coverage for each source. The 1100 -trained checkpoint is strongest on the main ToolRet slice, while the aggregate-trained checkpoint remains stable on the broader 4996 -query aggregate.
Retriever
Cov.
Multi
Avg Tok.
Lin. Global
MLP P@1
MLP Global
Off-the-shelf
56.4%
62.8%
1334.7
20.8%
42.0%
23.7%
FT on full
79.9%
86.6%
1454.5
42.5%
54.3%
43.4%
Appendix
Table 14: Full ToolRet scale test on the merged 44,453 -tool / 7,726 -query build spanning web , code , and customized corpora. The same task-adapted dense-retrieval conclusion survives at full scale: candidate coverage rises by +23.5 points and the direct dense MLP global proxy rises from 23.7% to 43.4% , while average candidate tokens increase by only about 9% .
Retriever
Cov.
Avg Tok.
Lin. P@1
MLP P@1
MLP Global
Off-the-shelf
96.3%
1508.0
68.5%
68.5%
66.0%
FT on StableToolBench
99.1%
1304.5
75.0%
75.0%
74.3%
Appendix
Table 15: StableToolBench direct dense reranker pipeline under the same BGE-M3 + MNRL retriever fine-tuning protocol. Because the benchmark already starts at very high depth- 20 coverage, the gain is smaller than on ToolRet in coverage terms ( 96.3% to 99.1% ), but the learned reranker still improves from 68.5% to 75.0% P@1 and from 66.0% to 74.3% on the coverage-weighted global proxy, while average candidate tokens decrease.
Eval Source
Retriever
CE P@1
CE Global
ToolRet-1100
Off-the-shelf
76.7%
61.9%
ToolRet-1100
FT on 1100
73.1%
67.1%
ToolRet-4996
Off-the-shelf
70.0%
51.6%
ToolRet-4996
FT on 4996
65.0%
55.4%
Appendix
Table 16: Compact cross-encoder view of the same retriever fine-tuning intervention. Candidate-covered cross-encoder P@1 is already high before retriever adaptation, so the fine-tuned checkpoints leave that covered-query margin similar; the global proxy still improves because matched-source fine-tuning raises candidate coverage substantially before reranking.
Comparison
Baseline
Candidate
Δ
95% CI
1100 direct FT (Linear)
45.6%
64.6%
+19.0
[10.8, 27.2]
4996 direct FT (MLP)
30.6%
49.9%
+19.3
[15.4, 23.1]
Appendix
Table 17: Paired bootstrap significance for the main direct dense fine-tuning gains. Means are computed over the held-out test queries underlying the global top-1 proxy; both improvements have p<0.001 .
Backbone
Setting
Cov.
Lin. P@1
MLP P@1
MLP Global
bge-small
Off-the-shelf
67.7%
50.0%
60.5%
38.1%
FT on 1100
86.1%
68.3%
69.7%
57.6%
BGE-M3
Off-the-shelf
80.8%
55.8%
64.3%
51.9%
FT on 1100
91.8%
70.3%
69.7%
63.9%
Appendix
Table 18: Cross-backbone validation of the direct retriever fine-tuning effect on the 1100 ToolRet slice. Both bge-small and BGE-M3 improve substantially under the same MNRL fine-tuning protocol, even though the larger backbone remains stronger in absolute terms.
Source
Backbone
Cov.
Avg Tok.
MLP P@1
MLP Global
ToolRet-1100
BGE-M3
80.8%
—
64.3%
51.9%
Qwen3-Emb-4B
87.3%
924.6
58.3%
50.9%
e5-large-v2
73.4%
1141.6
56.5%
41.5%
ToolRet-4996
BGE-M3
73.7%
—
41.0%
30.2%
Qwen3-Emb-4B
78.9%
905.7
46.8%
36.9%
ToolRet-full
BGE-M3
56.4%
1334.7
42.0%
23.7%
Appendix
Table 19: Additional off-the-shelf backbone comparisons on ToolRet under the same direct dense MLP reranking pipeline. The BGE-M3 rows reuse the paper’s off-the-shelf reference coverage together with the appendix MLP reranker results from Table 13 ; the Qwen3-Emb-4B and e5-large-v2 rows come from corresponding held-out direct-dense runs on the same sources. The comparison tests whether changing the untuned encoder family closes the gap; a full retriever bake-off is outside the scope of this paper.
Rendering
Adapter training
1100 test
4996 test
Collapsed-5
TF-IDF switch
Tool schema
1100
86.7%
57.8%
55.1%
83.6%
Tool schema
4996
85.4%
83.3%
88.5%
—
Skill card
1100
88.0%
58.7%
57.4%
84.0%
Skill card
4996
83.5%
83.3%
88.1%
—
Appendix
Table 20: LCO-Omni projection-adapter validation. Values are depth- 20 candidate coverage using official last-token text embeddings from LCO-Embedding-Omni-3B-2605 . The backbone is frozen and a shared linear projection is trained over query/tool embeddings. The pattern matches the main story: matched-source adaptation is strong, transfer to the mixed stream drops sharply, and the TF-IDF switch restores aggregate-level coverage.
Config
Qs.
Base
FT-1100
FT-4996
toolbench
158
73.4%
91.8%
87.3%
APIGen
146
82.9%
0.7%
96.6%
ToolACE
175
79.4%
0.0%
81.7%
UltraTool
72
69.4%
8.3%
98.6%
toollens
56
21.4%
5.4%
89.3%
reversechain
39
74.4%
5.1%
79.5%
Appendix
Table 21: Per-config candidate coverage on the 4996 aggregate test split. The 1100 -trained checkpoint remains strong on the embedded toolbench source, while the aggregate-trained checkpoint improves coverage across heterogeneous source-query generators.
Config
Qs.
Avg Tok.
Multi-step
Lex. Overlap
FT-1100 Cov.
toolbench
158
43.3
94.9%
0.195
91.8%
APIGen
146
18.0
24.0%
0.427
0.7%
ToolACE
175
58.3
18.3%
0.196
0.0%
UltraTool
72
37.3
75.0%
0.157
8.3%
Appendix
Table 22: Simple query-side feature probe for the held-out aggregate test rows of the main source-query generators discussed in the paper. “Lex. Overlap” is the average fraction of query content tokens that also appear in the corresponding gold-tool names, descriptions, or tags. A separate full-source query-only TF-IDF classifier over the 12 source styles with at least 50 queries reaches 87.5% accuracy and 81.0 macro-F1, showing that the source footprint is recoverable from query-side features. The routing experiments still use the fixed centroid rule.
Source
Qs.
BM25
Dense
BM25+Dense
toolbench
1099
51.0%
60.1%
64.9%
APIGen
1000
75.0%
77.9%
82.1%
ToolACE
1000
67.4%
72.7%
77.8%
UltraTool
500
41.8%
67.4%
62.4%
toollens
314
20.4%
21.3%
21.7%
reversechain
200
46.5%
61.0%
63.5%
Appendix
Table 23: Lexical BM25, off-the-shelf dense BGE-M3 , and BM25+dense reciprocal-rank fusion on key ToolRet source-query configurations under the same top- 20 candidate budget. BM25 is competitive on high-overlap sources such as APIGen , and fusion usually improves further, but the gains remain source-dependent and leave hard cases. We use this table as a lexical control for the source-style transfer analysis.
Source
Qs.
BM25
FT-1100
BM25+FT
toolbench
1099
51.0%
94.7%
91.3%
APIGen
1000
75.0%
0.7%
68.3%
ToolACE
1000
67.4%
0.0%
63.8%
UltraTool
500
41.8%
6.2%
36.2%
toollens
314
20.4%
2.5%
15.0%
reversechain
200
46.5%
1.5%
42.0%
Appendix
Table 24: Lexical BM25 versus the actually collapsed FT-1100 dense checkpoint, plus reciprocal-rank fusion between the two, under the same top- 20 candidate budget. In the real mismatched setting, naive lexical+dense fusion underperforms BM25 alone across all five collapsed sources, even though the same FT-1100 checkpoint remains excellent on the matched toolbench slice. This is why the paper routes away from an unsafe dense specialist instead of blending it with a lexical branch.
Retriever
Original
Oracle category
FT-1100
22.3%
32.1%
FT-4996
85.2%
88.4%
Appendix
Table 25: Oracle category-filtered candidate coverage on the mixed 4996 -query stream. The filter restricts retrieval to tools sharing a gold category, so it should be read as an upper-bound category restriction. It reduces part of the candidate-space difficulty while leaving checkpoint compatibility differences visible.
Figure 4: Full 19 -config source-distance probe. Each point is one aggregate source-query configuration; point size reflects held-out query count. Across all configs the unweighted correlation is r=-0.61 ; weighting by query count gives r=-0.85 ; and restricting to configs with at least 10 held-out queries gives r=-0.84 .
Proxy
LOO Acc.
95% Acc. CI
LOO F1
TF-IDF centroid distance
78.9%
[54.4, 93.9]
0.88
BGE-M3 centroid distance
73.7%
[48.8, 90.9]
0.85
Query-length gap
68.4%
[43.4, 87.4]
0.80
Multi-step-rate gap
47.4%
[24.4, 71.1]
0.58
Appendix
Table 26: Leave-one-config-out compatibility check for unsafe transfer sources, defined as FT-1100 coverage below 30% . F1 is reported for the positive unsafe class.
Window
Acc.
F1
1--10
78.9%
0.882
20
83.1%
0.903
50--100
84.2%
0.909
Appendix
Table 27: Small-window application of the fixed TF-IDF compatibility rule. Each row uses sampled incoming windows and the same unsafe threshold as Table 26 .
Feature
Weighted r
LOO Acc.
LOO F1
Query-side TF-IDF
-0.86
78.9%
0.88
Tool-side TF-IDF
-0.77
73.7%
0.85
Average(query, tool)
-0.83
78.9%
0.88
Max(query, tool)
-0.86
78.9%
0.88
Appendix
Table 28: Query-side versus tool-side source fingerprints on ToolRet- 4996 . Tool-side distance is computed over the normalized tool texts actually encoded by the retriever (name, description, arguments, examples) for each config’s gold tools. The tool-side signal remains informative and tightly correlated with the query-side one, while the query-side detector stays strongest.
Target
TF-IDF gap
Sem. gap
Off-the-shelf cov.
FT-1100 cov.
APIGen
71.1%
30.7%
75.0 → 77.0
92.0 → 90.0
ToolACE
142.0%
87.2%
75.0 → 66.0
92.0 → 83.0
UltraTool
40.1%
10.3%
75.0 → 76.0
92.0 → 93.0
rotbench
52.1%
32.4%
75.0 → 72.0
92.0 → 89.0
reversechain
37.8%
23.0%
75.0 → 76.0
92.0 → 89.0
Appendix
Table 29: Synthetic query-only rewrite control on 100 held-out toolbench -style queries. Gap columns measure how much of the natural target-source centroid-distance gap from toolbench_1100 is closed by rewriting, under the same TF-IDF and BGE-M3 semantic views used in the main source-style probe. Coverage columns report original-query → rewritten-query top- 20 coverage under off-the-shelf BGE-M3 and the source-specific FT-1100 retriever. Values above 100% indicate that the rewrite moves beyond the natural target source in that representation space.
Qwen-only
Qwen/MirrorAPI
Train / Eval
Natural
API
Natural
API
Off-the-shelf BGE-M3
72.1%
56.3%
72.1%
39.1%
FT natural
89.5%
60.6%
88.4%
42.0%
FT API-template
55.8%
84.5%
43.0%
82.6%
Appendix
Table 30: Paired synthetic-source transfer over a shared ToolRet tool corpus. Values report top- 20 coverage. The Qwen-only variant uses the same Qwen2.5-14B-Instruct generator for both styles; the Qwen/MirrorAPI variant keeps the natural style fixed and uses MirrorAPI for the API-template style. Both settings show transfer loss under a shared tool corpus, with stronger loss in the cross-generator setting.
Train ↓ / Eval →
Task1
Similar
Scenario
task1
96.3%
91.0%
92.0%
similar
100.0%
91.7%
96.7%
scenario
93.8%
88.5%
100.0%
Off-the-shelf BGE-M3
82.7%
78.8%
74.3%
Appendix
Table 31: MetaTool task-family comparison. Rows fine-tune BGE-M3 on one single-tool task family and evaluate top- 20 coverage on another. MetaTool is useful as a second tool corpus, but its task-family labels are shared-process task labels. The absence of collapse here supports a bounded interpretation: ToolRet’s source-style collapse is tied to independent generation sources more than arbitrary task-family differences inside a mostly shared generation process.
Field
Template content
Skill
{tool_name}
Capability
This skill enables the agent to {description}.
When to use
Use this skill when the user request matches {tags/categories/description}.
Inputs
{arguments or schema fields}
Output/effect
{available output or API-effect description}
Metadata
{tags and categories}
Appendix
Table 32: Deterministic skill-card rendering template. The rendering is template-based and changes the visible capability representation while keeping labels, queries, and supervision fixed.
Rendering
1100 OTS
1100 FT
FT-1100 → 4996
4996 OTS
4996 FT
Switch
Tool schema
80.8%
91.8%
22.3%
73.7%
85.2%
86.1%
Skill card
72.2%
89.2%
21.7%
64.5%
84.0%
85.2%
Appendix
Table 33: Core skill-card rendering check. Values are depth- 20 candidate coverage. Rerendering tools as executable skill cards lowers the off-the-shelf baseline but preserves the main transfer-collapse and switch-policy pattern.
Source
Queries
OTS
FT-1100
FT-aggregate
APIGen
146
82.9%
2.1%
96.6%
ToolACE
175
75.4%
0.6%
85.7%
UltraTool
72
66.7%
1.4%
98.6%
toollens
56
23.2%
0.0%
87.5%
reversechain
39
66.7%
15.4%
82.0%
Appendix
Table 34: Collapsed-source coverage under skill-card rendering. The natural collapsed sources remain collapsed under FT-1100 , supporting the interpretation that raw schema formatting alone fails to explain the failure.
Setting
Original
MCP-lite
Δ
1100 off-the-shelf
80.8%
68.4%
-12.4
1100 FT (matched)
91.8%
90.5%
-1.3
FT-1100 → 4996
22.3%
25.1%
+2.8
4996 off-the-shelf
73.7%
66.8%
-6.9
4996 FT (matched)
85.2%
84.3%
-0.9
FT-4996 → 1100
87.3%
87.3%
+0.0
Appendix
Table 35: Paired retrain/eval under a stricter MCP-lite tool rendering. Both ToolRet- 1100 and ToolRet- 4996 are rebuilt with unified name + description + arguments fields and the retrievers are retrained on the corresponding normalized corpora before evaluation. Schema normalization leaves the narrow-checkpoint collapse in place and lowers off-the-shelf coverage substantially. All values are retriever coverage, before reranked global top-1.
Source
FT-1100
FT-aggregate
Δ
APIGen
0.337
0.924
+0.587
ToolACE
0.341
0.621
+0.279
UltraTool
0.454
0.915
+0.461
toollens
0.470
0.747
+0.277
reversechain
0.451
0.705
+0.254
Mean (all 5)
0.411
0.782
+0.372
Appendix
Table 36: Representation-level invariance under MCP-lite normalization on collapsed-source gold tools. Each row reports the mean cosine similarity between the embedding of a tool’s original text and the embedding of its MCP-lite text under the same retriever checkpoint, using 200 sampled gold tools per source ( 1,000 tools total). The aggregate-trained checkpoint produces more stable tool representations than FT-1100 in every source. The especially high APIGen value reflects that its source format is already close to the MCP-lite template.
Depth
Cov.
P@1
R@5
nDCG@5
Covered
Global
Covered
Global
Covered
10
69.4%
64.3%
44.6%
90.7%
62.9%
77.1%
20
76.2%
58.7%
44.7%
78.7%
60.0%
67.6%
50
83.9%
47.4%
39.8%
64.8%
54.4%
55.2%
Appendix
Table 37: Exact ToolRet depth-sweep values behind Figure 3 . Increasing depth raises coverage, but covered-query quality decays enough that the global top-1 proxy peaks at the main 20 -candidate operating point.
Phenomenon
Query Form
Gold Tools
Why It Matters
Same-corpus source-style shift
APIGen : “Please provide the latest songs from the artist Beyoncé and also list the countries in ascending order.”
all_songs_from_artist ; getallcountry
The query has obvious lexical access to the tool schemas, yet it is phrased as a compact API-style command instead of a longer end-user task description. This is the kind of high-overlap cross-source case where the 1100 -trained retriever still collapses.
Compositional tail / staged repair
Yosemite query: weather forecast + trail recommendations + scenic-view filtering in one request
Single-shot retrieval must compress heterogeneous intents into one embedding. The conservative splitter rewrites this into “Provide the weather forecast for Yosemite …” and “Suggest popular hiking trails in Yosemite …”, which produces two cleaner retrieval problems before the candidates are merged.
Appendix
Table 38: Two concrete cases behind the main findings. The first illustrates why same-corpus source-style shift goes beyond missing keywords. The second illustrates why staged retrieval helps the compositional tail even when direct dense retrieval is already strong on average.
Setting
Cov.
P@1
R@5
Global top-1
Single + original reranker
76.2%
45.6%
77.5%
34.7%
Split + original reranker
82.9%
39.2%
77.3%
32.5%
Split + adapted linear
82.9%
46.8%
78.8%
38.8%
Split + adapted MLP
82.9%
55.7%
83.6%
46.2%
Split + adapted cross-encoder
82.9%
67.1%
93.1%
55.6%
Appendix
Table 39: Staged-retrieval follow-up on ToolRet under the same candidate-generation setup as the main split-and-merge probe. The merged pool hurts top-1 quality under the unchanged reranker, but adapted rerankers recover and then surpass the single-query global proxy. A compact cross-encoder pushes the same staged pool further upward, showing that the staged candidate generator remains competitive under stronger rerankers as well.
top- k skills
Manual skill → tool
Automatic skill → tool
P@1
R@5
Cov.
Cand. Red.
P@1
R@5
Cov.
Cand. Red.
1
34.9%
46.4%
49.8%
96.5%
42.6%
57.1%
65.0%
97.2%
3
47.1%
63.0%
67.8%
89.8%
53.7%
71.8%
80.5%
92.3%
5
51.8%
69.5%
77.3%
83.5%
58.2%
76.9%
89.2%
87.4%
7
55.3%
73.9%
84.4%
77.6%
60.4%
79.2%
92.3%
82.5%
10
59.6%
79.0%
90.3%
69.6%
62.1%
82.6%
95.4%
75.2%
Appendix
Table 40: Main StableToolBench hierarchical-routing Pareto table. Tool-only retrieval is the accuracy anchor; the hierarchy rows trade quality for candidate-context reduction.
Routing Mode
Cand. Tok.
Final Tok.
Cov.
P@1
R@5
Cand. Red.
Full Catalog Injection
172,426
172,426
100.0%
–
–
0.0%
Tool-Only Retrieval
172,426
407
100.0%
66.1%
87.2%
0.0%
Manual Skill → Tool ( k=5 )
28390
410
77.3%
51.8%
69.5%
83.5%
Automatic Skill → Tool ( k=5 )
21785
411
89.2%
58.2%
76.9%
87.4%
Appendix
Table 41: Routing-stage token efficiency on StableToolBench. Full catalog versus tool-only explains final-prompt compression, while tool-only versus hierarchy explains candidate-pool compression.
Group
Count
Official
Auto
Δ Auto
G1_instruction
74
35.1%
31.1%
-4.1
G1_category
65
33.8%
46.2%
+12.3
G1_tool
57
31.6%
29.8%
-1.8
G2_instruction
26
26.9%
26.9%
+0.0
G2_category
16
25.0%
6.2%
-18.8
G3_instruction
21
23.8%
33.3%
+9.5
Appendix
Table 42: Benchmark-native local execution on the 259 -query paired common subset shared by official_api_list and auto_skill_tool across all six StableToolBench groups. The two methods use the same local Qwen-14B planner and the same local function-call-accuracy-style judge. Average end-to-end token counts stay nearly identical ( 5,807 vs. 5,822 ) because planner/execution overhead dominates the full trace length, but token cost per solved task still favors the automatic hierarchy ( 17.7k vs. 18.3k ).
Anthropic's Agent Skills package reusable procedural know-how for an LLM agent into SKILL.md directories, and open-source aggregations have grown past 230,000 skills, making selection rather than authoring the bottleneck. The standing answer in the literature outsources selection to the agent itself: an LLM-mediated retrieval loop that rewrites queries and refines candidates inside the agent's decision loop, paying LLM tokens on every task. We present SkillSeek, an open-source two-stage skill retriever built from the standard IR recipe (a BGE-base bi-encoder feeding a small cross-encoder, exposed over MCP). Across a 4×11 grid of pool, backbone, and method on the 89-task SkillsBench benchmark, SkillSeek reaches observed parity with the LLM-mediated loop of Liu et al. at essentially no extra cost: plain bm25 alone records a pass rate at or above their refined loop on three of four settings, and a small cross-encoder covers the remaining difference on the fourth. A first-stage recall ceiling explains the pattern, and total per-trial spend drops from USD 51.30 to USD 27.54 (within fifty cents of the no-skill baseline). Under the SkillsBench tasks and OpenHands harness we tested, this positions the standard IR recipe as a strong default for agent-skill retrieval, with LLM-mediated alternatives a natural fit for cases where deterministic methods fall short.
Guanqun Yang, Wenlong Zhang, Tian Shi +1
Stevens Institute of Technology, Hoboken, NJ, USA · Independent Researcher
As large language model agents gain access to increasingly large skill libraries, retrieving the right skill becomes critical to reliable capability selection and execution. Existing retrievers often treat skill contents as ordinary documents, overlooking their highly regular structure: shared descriptive patterns recur across many skills while providing little evidence for distinguishing the required capability. We show that this shared descriptive background is reflected in dense relevance scores, induces a pronounced energy gap between queries and skill documents, and obscures discriminative signals, especially for structurally similar hard negatives. Based on this observation, we propose SkillSight, a training-free retrieval framework that calibrates shared background in both semantic and lexical spaces. Semantic Background Calibration estimates a background subspace from generic tokens identified by IDF, reducing similarity induced by shared descriptive patterns, while Lexical Evidence Calibration downweights shared background tokens to recover discriminative token-level evidence. Experiments on SRA-Bench and SkillBench-Supp demonstrate consistent improvements across retrieval metrics, with SkillSight improving Recall@10 by up to 20.21 percentage points over the original dense retriever. It is up to 1,248 times faster than the Dense + Reranker baseline. In end-to-end evaluation, SkillSight achieves the best overall performance across three agent models and outperforms LLM Selection by up to 4.97 percentage points. These results identify shared descriptive background as a source of ranking interference in skill retrieval and demonstrate that calibrating it enables accurate and efficient skill selection without additional training. Our code can be found at https://github.com/xiaojinying/SkillSight
Jinying Xiao, Bin Li, Xiaopeng Li +7
1National University of Defense Technology · 2Qinghai Normal University · 3Xizang University
Large Language Model (LLM) agents increasingly rely on external skills, yet standard evaluations obscure whether retrieving these skills actually helps. Aggregate metrics often compare retrieved versus non-retrieved tasks, introducing severe selection bias and failing to isolate the true effect of skill use. To measure this actual-use capability-which we formalize as Skill Following (SF)-we introduce the Retrieval-Invoked Actual-Use Effect (RAE). RAE computes the same-task outcome difference between matched skill-enabled and skill-disabled executions, conditioned exclusively on tasks where the agent actively retrieved a skill. Evaluating 17 LLMs across coding and mathematical domains, we uncover a stark evaluation paradox: models frequently show positive aggregate retrieval lift but negative RAE. On MBPP+, multiple models that appear to benefit system-wide actually harm their own performance on the exact tasks where retrieval occurred. These findings demonstrate that aggregate averages can create a misleading illusion of tool-use proficiency, whereas RAE directly measures whether the retrieval-to-answer pipeline genuinely rescues more outcomes than it harms.