When Tool-Backed Skill Retrieval Fails: Source-Style Collapse in Executable Capability Retrieval

Yiqi Liu; Joseph James; Yang Wang; Chenghao Xiao; Chenghua Lin

cs.LGAug 17, 2026

When Tool-Backed Skill Retrieval Fails: Source-Style Collapse in Executable Capability Retrieval

Authors: Yiqi Liu, Joseph James, Yang Wang, Chenghao Xiao, Chenghua Lin

Organizations: University of Manchester · University of Sheffield · Shanghai University of Finance and Economics

Abstract

Large-scale agents increasingly rely on retrieval to access external capabilities. We study this retrieval gate in structured tools and APIs, a measurable class of tool-backed executable skills that must be surfaced before an agent can plan, incorporate, or act. In this setting the retrieval layer can silently fail even when the capability corpus is fixed: on ToolRet, a retriever fine-tuned on one source-specific slice collapses on another source-specific slice of the same benchmark, with FT-1100 despite its higher lexical overlap with the gold tools. We call this failure mode source-style collapse. Query-side TF-IDF fingerprints flag source styles on which the fine-tuned retriever is likely to fail better than semantic or length-based proxies, giving a cheap signal for mismatch over a fixed tool corpus. We propose ToolScout, a source-aware routing method that uses this signal as a routing guard: on the mixed 4,996-query stream, TF-IDF-based routing raises coverage from 22.3% to 86.1%, and across five collapsed sources 20 matched examples raise the coverage-weighted global top-1 proxy from 1.3% to 53.9%. The same failure and routing behaviors persist when tools are rerendered as executable skill cards, which rules out raw API-schema format as the sole cause.

Figures & tables

Figure 1: Tool retrieval is source-style-sensitive on a fixed benchmark corpus. Top: retrieval compresses benchmark-native full-catalog exposure from about 172k tokens to about 407 at retrieved top- 5 . (A) Lexical overlap fails to predict retrievability: APIGen has the highest query-to-gold-tool overlap yet collapses to 0.7% coverage under FT-1100 . (B) TF-IDF distance separates source styles more clearly than BGE-M3 semantic distance ( r=-0.85 vs. -0.58 ). (C) Route, switch, then repair: source-aware routing increases mixed-stream coverage by nearly fourfold, and 20 matched examples substantially improve the collapsed-source coverage-weighted global top- 1 proxy.

Benchmark	Setting	Cov.	P@1
			Sem.	Lrn.
ToolRet	bge-small / full 1100	67.7%	25.0%	50.0%
ToolRet	BGE-M3 / full 1100	80.8%	34.9%	55.8%
StableToolBench	bge-small / full 765	97.3%	70.1%	71.0%
StableToolBench	BGE-M3 / full 765	98.4%	66.7%	68.5%

Table 1: Off-the-shelf two-stage results under a shared direct retrieval pipeline. “Sem.” denotes retriever-only top- 1 accuracy, and “Lrn.” denotes top- 1 accuracy after the lightweight learned reranker. Coverage is top- 20 candidate coverage. The results show that ToolRet leaves substantially more candidate-generation headroom than StableToolBench under the same protocol. Full covered-query metrics are provided in Appendix A.1 .

Figure 2: Direct-dense retrieval adaptation across ToolRet- 1100 , ToolRet- 4996 , and ToolRet-full. The plotted score is the coverage-weighted top-1 proxy. A substantially larger 2025 off-the-shelf backbone ( Qwen3-Emb-4B ) narrows the gap in every ToolRet setting while leaving sizable room for source-adapted retrieval. Exact values are listed in Appendix A.3 .

	FT-1100		20 -shot		FT-aggregate
Source	Cov.	Global	Cov.	Global	Cov.	Global
APIGen	0.7%	0.3%	93.2%	74.4%	96.6%	73.5%
ToolACE	0.0%	0.0%	77.1%	46.0%	81.7%	47.2%
UltraTool	8.3%	5.7%	84.7%	48.1%	98.6%	81.9%
toollens	5.4%	1.3%	58.9%	27.1%	89.3%	47.3%
reversechain	5.1%	2.3%	89.7%	61.5%	79.5%	32.3%

Table 2: Multi-source few-shot repair from a collapsed FT-1100 checkpoint. “Global” is the coverage-weighted multilayer perceptron (MLP) top-1 proxy. The table reports the headline 20 -shot setting only; the full 5/10/20/50 -shot sweep is listed in Appendix A.3 . The reversechain row has only 39 held-out test queries, so we treat its score cautiously. Sample complexity varies by source: the appendix sweep shows that 50 -shot continuation closes the UltraTool gap and narrows toollens further.

Rendering	1100 OTS	1100 FT	FT-1100 $\rightarrow$ 4996	4996 OTS	4996 FT	Switch
Tool schema	80.8%	91.8%	22.3%	73.7%	85.2%	86.1%
Skill card	72.2%	89.2%	21.7%	64.5%	84.0%	85.2%

Table 3: Skill-card rendering check. Coverage is depth- 20 candidate coverage. The skill-card rendering keeps the same tools, labels, splits, retrievers, and routing policy, while rewriting visible tool text into normalized executable skill cards. Collapse and the switch policy persist under the more skill-like representation. Full per-source values and the deterministic card template are in Appendix A.5 .

Figure 3: ToolRet depth sweep on the full 1100 -query normalized web subset with BGE-M3 . Coverage rises with depth, but the coverage-weighted top-1 proxy peaks around the main 20 -candidate operating point and then declines. Axes use truncated ranges for readability. Exact values are listed in Appendix A.6 .

Appendix figures & tables40 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Ret.	Meta.	Lex.	Query	Exec.
ToolRet	48.2%	44.1%	7.7%	0.0%	0.0%
StableToolBench	71.2%	14.3%	12.7%	1.8%	0.0%

Appendix

Table 4: Grouped reranker coefficient mass. Ret. = retrieval scores, reciprocal-rank features, and candidate-presence indicators; Meta. = metadata-structure features; Lex. = lexical overlap; Query = query-length features; Exec. = execution-feedback features.

Benchmark	Reranker	P@1	R@5	nDCG@5
ToolRet	Linear	55.8%	73.0%	63.2%
ToolRet	MLP	64.3%	78.6%	70.4%
ToolRet	Cross-enc.	76.7%	93.1%	84.7%
StableTB	Linear	68.5%	78.8%	71.6%
StableTB	MLP	68.5%	81.2%	73.5%
StableTB	Cross-enc.	85.2%	89.5%	85.8%

Appendix

Table 5: Stronger reranker controls on the main dense runs. Metrics are reported on the same held-out candidate-covered test splits as Table 1 . Cross-encoders substantially raise the covered-query ceiling, and they still operate only after candidate generation has already succeeded.

Reranker	P@1	R@5	nDCG@5
BM25	27.0%	43.0%	35.1%
Semantic	33.2%	57.4%	45.3%
Linear	35.2%	58.1%	47.0%
MLP	41.0%	69.5%	55.6%
Cross-encoder	70.0%	89.6%	79.9%

Appendix

Table 6: ToolRet aggregate stress test on 4996 queries from 19 query configs. Metrics are reported on the held-out candidate-covered test split from the same reranker protocol as Table 1 . The larger aggregate is substantially harder upstream ( 73.7% candidate coverage), and once candidates are present the same reranker ordering reappears.

Source	Retriever	Cov.	P@1	Global
ToolRet-1100	BGE-M3	80.8%	55.8%	45.1%
	Qwen3-Emb-4B	87.3%	58.3%	50.9%
	BGE-M3 FT on 1100	91.8%	70.3%	64.5%
ToolRet-4996	BGE-M3	73.7%	41.0%	30.2%
	Qwen3-Emb-4B	78.9%	46.8%	36.9%
	BGE-M3 FT on 4996	85.2%	56.5%	48.1%

Appendix

Table 7: Exact values behind the main-paper retrieval-adaptation summary figure. “Global” is the coverage-weighted top-1 proxy.

Negatives	FT Cov.	$\Delta$ Cov.	FT Multi	FT Recall
Semantic hard	88.5%	+16.2	89.3%	71.7%
Semantic + random	88.5%	+16.2	90.0%	68.2%
BM25 hard	87.8%	+15.5	89.3%	69.2%
Random	84.5%	+12.2	85.7%	68.1%
Same-category	78.4%	+6.1	79.3%	58.8%

Appendix

Table 8: Controlled hard-negative ablation for BGE-M3 + MNRL on the 1100 validation split. All rows start from the same off-the-shelf depth- 20 base coverage of 72.3% . Retrieval-confusing semantic negatives are strongest, while same-category negatives are much weaker despite being semantically plausible distractors.

Train Source	Eval Source	Cov.	Multi	Avg Tok.
Off-the-shelf	ToolRet-1100 val	72.3%	72.9%	1104.4
FT on 1100	ToolRet-1100 val	88.5%	90.0%	910.9
FT on 1100	ToolRet-1100 test	91.8%	92.0%	930.6
Off-the-shelf	ToolRet-4996 test	65.5%	63.5%	1194.4
FT on 4996	ToolRet-4996 test	85.2%	89.5%	836.2
FT on 4996	Transfer to 1100 test	87.3%	88.0%	830.3

Appendix

Table 9: Depth- 20 candidate-generation coverage for task-adapted dense retrievers. The 1100 -trained model is strongest on matched data, while the aggregate-trained model generalizes across the broader 4996 -query setting.

Training mix	4996	1100 transfer	Collapsed-5	toolbench
Natural aggregate	85.2%	87.3%	89.3%	87.3%
Balanced	81.8%	74.1%	88.5%	74.1%
Temp.-reweighted	81.8%	76.6%	88.3%	76.6%

Appendix

Table 10: Naive mixed-training controls for aggregate retriever fine-tuning. All rows use the same BGE-M3 + MNRL protocol and the same total number of one-per-query training examples; only the source sampling distribution changes. Coverage is reported at candidate depth 20 . “Collapsed-5” is the query-count-weighted mean over APIGen , ToolACE , UltraTool , toollens , and reversechain .

Retriever	toolbench Cov.	toolbench Margin	Collapsed-5 Cov.	Collapsed-5 Margin	Probe F1
Off-the-shelf	73.4%	-0.033	71.9%	-0.026	82.2%
FT-1100	91.8%	+0.018	2.5%	-0.371	76.6%
FT-aggregate	87.3%	-0.016	89.3%	+0.003	91.3%
Balanced aggregate	74.1%	-0.044	88.5%	-0.009	90.9%
Temp.-reweighted aggregate	76.6%	-0.039	88.3%	-0.002	91.0%

Appendix

Table 11: Representation-geometry probe for the mixed-training control. Margins are the mean best-gold score minus strongest non-gold score at candidate depth 20 ; higher is better. Probe F1 is a cross-validated source-style linear classifier over query embeddings for the six displayed sources.

Source	FT-1100	5 -shot	10 -shot	20 -shot	50 -shot	FT-aggregate
APIGen	0.7%	0.7%	—	93.2%	—	96.6%
ToolACE	0.0%	0.0%	68.0%	77.1%	86.3%	81.7%
UltraTool	8.3%	8.3%	81.9%	84.7%	98.6%	98.6%
toollens	5.4%	5.4%	28.6%	58.9%	87.5%	89.3%
reversechain	5.1%	5.1%	84.6%	89.7%	84.6%	79.5%

Appendix

Table 12: Coverage sweep for lightweight matched-source repair from a collapsed FT-1100 checkpoint. The main paper reports the headline 20 -shot setting; this table gives the full 5/10/20/50 -shot progression. Dashes for APIGen mark shot counts skipped in the initial pilot.

Eval Source	Retriever	Lin. P@1	MLP P@1	MLP Global
ToolRet-1100	Off-the-shelf	55.8%	64.3%	51.9%
	FT on 1100	70.3%	69.7%	63.9%
	FT on 4996	54.8%	55.5%	48.5%
ToolRet-4996	Off-the-shelf	35.2%	41.0%	30.2%
	FT on 1100	36.9%	48.7%	10.9%
	FT on 4996	56.3%	56.5%	48.1%

Appendix

Table 13: Direct dense reranker pipelines under task-adapted dense retrievers. “MLP Global” is the coverage-weighted top-1 proxy using the appropriate candidate-generation coverage for each source. The 1100 -trained checkpoint is strongest on the main ToolRet slice, while the aggregate-trained checkpoint remains stable on the broader 4996 -query aggregate.

Retriever	Cov.	Multi	Avg Tok.	Lin. Global	MLP P@1	MLP Global
Off-the-shelf	56.4%	62.8%	1334.7	20.8%	42.0%	23.7%
FT on full	79.9%	86.6%	1454.5	42.5%	54.3%	43.4%

Appendix

Table 14: Full ToolRet scale test on the merged 44,453 -tool / 7,726 -query build spanning web , code , and customized corpora. The same task-adapted dense-retrieval conclusion survives at full scale: candidate coverage rises by +23.5 points and the direct dense MLP global proxy rises from 23.7% to 43.4% , while average candidate tokens increase by only about 9% .

Retriever	Cov.	Avg Tok.	Lin. P@1	MLP P@1	MLP Global
Off-the-shelf	96.3%	1508.0	68.5%	68.5%	66.0%
FT on StableToolBench	99.1%	1304.5	75.0%	75.0%	74.3%

Appendix

Table 15: StableToolBench direct dense reranker pipeline under the same BGE-M3 + MNRL retriever fine-tuning protocol. Because the benchmark already starts at very high depth- 20 coverage, the gain is smaller than on ToolRet in coverage terms ( 96.3% to 99.1% ), but the learned reranker still improves from 68.5% to 75.0% P@1 and from 66.0% to 74.3% on the coverage-weighted global proxy, while average candidate tokens decrease.

Eval Source	Retriever	CE P@1	CE Global
ToolRet-1100	Off-the-shelf	76.7%	61.9%
ToolRet-1100	FT on 1100	73.1%	67.1%
ToolRet-4996	Off-the-shelf	70.0%	51.6%
ToolRet-4996	FT on 4996	65.0%	55.4%

Appendix

Table 16: Compact cross-encoder view of the same retriever fine-tuning intervention. Candidate-covered cross-encoder P@1 is already high before retriever adaptation, so the fine-tuned checkpoints leave that covered-query margin similar; the global proxy still improves because matched-source fine-tuning raises candidate coverage substantially before reranking.

Comparison	Baseline	Candidate	$\Delta$	95% CI
1100 direct FT (Linear)	45.6%	64.6%	+19.0	[10.8, 27.2]
4996 direct FT (MLP)	30.6%	49.9%	+19.3	[15.4, 23.1]

Appendix

Table 17: Paired bootstrap significance for the main direct dense fine-tuning gains. Means are computed over the held-out test queries underlying the global top-1 proxy; both improvements have

p<0.001

Backbone	Setting	Cov.	Lin. P@1	MLP P@1	MLP Global
bge-small	Off-the-shelf	67.7%	50.0%	60.5%	38.1%
	FT on 1100	86.1%	68.3%	69.7%	57.6%
BGE-M3	Off-the-shelf	80.8%	55.8%	64.3%	51.9%
	FT on 1100	91.8%	70.3%	69.7%	63.9%

Appendix

Table 18: Cross-backbone validation of the direct retriever fine-tuning effect on the 1100 ToolRet slice. Both bge-small and BGE-M3 improve substantially under the same MNRL fine-tuning protocol, even though the larger backbone remains stronger in absolute terms.

Source	Backbone	Cov.	Avg Tok.	MLP P@1	MLP Global
ToolRet-1100	BGE-M3	80.8%	—	64.3%	51.9%
	Qwen3-Emb-4B	87.3%	924.6	58.3%	50.9%
	e5-large-v2	73.4%	1141.6	56.5%	41.5%
ToolRet-4996	BGE-M3	73.7%	—	41.0%	30.2%
ToolRet-4996	Qwen3-Emb-4B	78.9%	905.7	46.8%	36.9%
ToolRet-full	BGE-M3	56.4%	1334.7	42.0%	23.7%

Appendix

Table 19: Additional off-the-shelf backbone comparisons on ToolRet under the same direct dense MLP reranking pipeline. The BGE-M3 rows reuse the paper’s off-the-shelf reference coverage together with the appendix MLP reranker results from Table 13 ; the Qwen3-Emb-4B and e5-large-v2 rows come from corresponding held-out direct-dense runs on the same sources. The comparison tests whether changing the untuned encoder family closes the gap; a full retriever bake-off is outside the scope of this paper.

Rendering	Adapter training	1100 test	4996 test	Collapsed-5	TF-IDF switch
Tool schema	1100	86.7%	57.8%	55.1%	83.6%
Tool schema	4996	85.4%	83.3%	88.5%	—
Skill card	1100	88.0%	58.7%	57.4%	84.0%
Skill card	4996	83.5%	83.3%	88.1%	—

Appendix

Table 20: LCO-Omni projection-adapter validation. Values are depth- 20 candidate coverage using official last-token text embeddings from LCO-Embedding-Omni-3B-2605 . The backbone is frozen and a shared linear projection is trained over query/tool embeddings. The pattern matches the main story: matched-source adaptation is strong, transfer to the mixed stream drops sharply, and the TF-IDF switch restores aggregate-level coverage.

Config	Qs.	Base	FT-1100	FT-4996
toolbench	158	73.4%	91.8%	87.3%
APIGen	146	82.9%	0.7%	96.6%
ToolACE	175	79.4%	0.0%	81.7%
UltraTool	72	69.4%	8.3%	98.6%
toollens	56	21.4%	5.4%	89.3%
reversechain	39	74.4%	5.1%	79.5%

Appendix

Table 21: Per-config candidate coverage on the 4996 aggregate test split. The 1100 -trained checkpoint remains strong on the embedded toolbench source, while the aggregate-trained checkpoint improves coverage across heterogeneous source-query generators.

Config	Qs.	Avg Tok.	Multi-step	Lex. Overlap	FT-1100 Cov.
toolbench	158	43.3	94.9%	0.195	91.8%
APIGen	146	18.0	24.0%	0.427	0.7%
ToolACE	175	58.3	18.3%	0.196	0.0%
UltraTool	72	37.3	75.0%	0.157	8.3%

Appendix

Table 22: Simple query-side feature probe for the held-out aggregate test rows of the main source-query generators discussed in the paper. “Lex. Overlap” is the average fraction of query content tokens that also appear in the corresponding gold-tool names, descriptions, or tags. A separate full-source query-only TF-IDF classifier over the 12 source styles with at least 50 queries reaches 87.5% accuracy and 81.0 macro-F1, showing that the source footprint is recoverable from query-side features. The routing experiments still use the fixed centroid rule.

Source	Qs.	BM25	Dense	BM25+Dense
toolbench	1099	51.0%	60.1%	64.9%
APIGen	1000	75.0%	77.9%	82.1%
ToolACE	1000	67.4%	72.7%	77.8%
UltraTool	500	41.8%	67.4%	62.4%
toollens	314	20.4%	21.3%	21.7%
reversechain	200	46.5%	61.0%	63.5%

Appendix

Table 23: Lexical BM25, off-the-shelf dense BGE-M3 , and BM25+dense reciprocal-rank fusion on key ToolRet source-query configurations under the same top- 20 candidate budget. BM25 is competitive on high-overlap sources such as APIGen , and fusion usually improves further, but the gains remain source-dependent and leave hard cases. We use this table as a lexical control for the source-style transfer analysis.

Source	Qs.	BM25	FT-1100	BM25+FT
toolbench	1099	51.0%	94.7%	91.3%
APIGen	1000	75.0%	0.7%	68.3%
ToolACE	1000	67.4%	0.0%	63.8%
UltraTool	500	41.8%	6.2%	36.2%
toollens	314	20.4%	2.5%	15.0%
reversechain	200	46.5%	1.5%	42.0%

Appendix

Table 24: Lexical BM25 versus the actually collapsed FT-1100 dense checkpoint, plus reciprocal-rank fusion between the two, under the same top- 20 candidate budget. In the real mismatched setting, naive lexical+dense fusion underperforms BM25 alone across all five collapsed sources, even though the same FT-1100 checkpoint remains excellent on the matched toolbench slice. This is why the paper routes away from an unsafe dense specialist instead of blending it with a lexical branch.

Retriever	Original	Oracle category
FT-1100	22.3%	32.1%
FT-4996	85.2%	88.4%

Appendix

Table 25: Oracle category-filtered candidate coverage on the mixed 4996 -query stream. The filter restricts retrieval to tools sharing a gold category, so it should be read as an upper-bound category restriction. It reduces part of the candidate-space difficulty while leaving checkpoint compatibility differences visible.

Figure 4: Full 19 -config source-distance probe. Each point is one aggregate source-query configuration; point size reflects held-out query count. Across all configs the unweighted correlation is r=-0.61 ; weighting by query count gives r=-0.85 ; and restricting to configs with at least 10 held-out queries gives r=-0.84 .

Proxy	LOO Acc.	95% Acc. CI	LOO F1
TF-IDF centroid distance	78.9%	[54.4, 93.9]	0.88
BGE-M3 centroid distance	73.7%	[48.8, 90.9]	0.85
Query-length gap	68.4%	[43.4, 87.4]	0.80
Multi-step-rate gap	47.4%	[24.4, 71.1]	0.58

Appendix

Table 26: Leave-one-config-out compatibility check for unsafe transfer sources, defined as FT-1100 coverage below 30% . F1 is reported for the positive unsafe class.

Window	Acc.	F1
1--10	78.9%	0.882
20	83.1%	0.903
50--100	84.2%	0.909

Appendix

Table 27: Small-window application of the fixed TF-IDF compatibility rule. Each row uses sampled incoming windows and the same unsafe threshold as Table 26 .

Feature	Weighted r	LOO Acc.	LOO F1
Query-side TF-IDF	-0.86	78.9%	0.88
Tool-side TF-IDF	-0.77	73.7%	0.85
Average(query, tool)	-0.83	78.9%	0.88
Max(query, tool)	-0.86	78.9%	0.88

Appendix

Table 28: Query-side versus tool-side source fingerprints on ToolRet- 4996 . Tool-side distance is computed over the normalized tool texts actually encoded by the retriever (name, description, arguments, examples) for each config’s gold tools. The tool-side signal remains informative and tightly correlated with the query-side one, while the query-side detector stays strongest.

Target	TF-IDF gap	Sem. gap	Off-the-shelf cov.	FT-1100 cov.
APIGen	71.1%	30.7%	75.0 $\rightarrow$ 77.0	92.0 $\rightarrow$ 90.0
ToolACE	142.0%	87.2%	75.0 $\rightarrow$ 66.0	92.0 $\rightarrow$ 83.0
UltraTool	40.1%	10.3%	75.0 $\rightarrow$ 76.0	92.0 $\rightarrow$ 93.0
rotbench	52.1%	32.4%	75.0 $\rightarrow$ 72.0	92.0 $\rightarrow$ 89.0
reversechain	37.8%	23.0%	75.0 $\rightarrow$ 76.0	92.0 $\rightarrow$ 89.0

Appendix

Table 29: Synthetic query-only rewrite control on 100 held-out toolbench -style queries. Gap columns measure how much of the natural target-source centroid-distance gap from toolbench_1100 is closed by rewriting, under the same TF-IDF and BGE-M3 semantic views used in the main source-style probe. Coverage columns report original-query

\rightarrow

rewritten-query top- 20 coverage under off-the-shelf BGE-M3 and the source-specific FT-1100 retriever. Values above 100% indicate that the rewrite moves beyond the natural target source in that representation space.

	Qwen-only		Qwen/MirrorAPI
Train / Eval	Natural	API	Natural	API
Off-the-shelf BGE-M3	72.1%	56.3%	72.1%	39.1%
FT natural	89.5%	60.6%	88.4%	42.0%
FT API-template	55.8%	84.5%	43.0%	82.6%

Appendix

Table 30: Paired synthetic-source transfer over a shared ToolRet tool corpus. Values report top- 20 coverage. The Qwen-only variant uses the same Qwen2.5-14B-Instruct generator for both styles; the Qwen/MirrorAPI variant keeps the natural style fixed and uses MirrorAPI for the API-template style. Both settings show transfer loss under a shared tool corpus, with stronger loss in the cross-generator setting.

Train $\downarrow$ / Eval $\rightarrow$	Task1	Similar	Scenario
task1	96.3%	91.0%	92.0%
similar	100.0%	91.7%	96.7%
scenario	93.8%	88.5%	100.0%
Off-the-shelf BGE-M3	82.7%	78.8%	74.3%

Appendix

Table 31: MetaTool task-family comparison. Rows fine-tune BGE-M3 on one single-tool task family and evaluate top- 20 coverage on another. MetaTool is useful as a second tool corpus, but its task-family labels are shared-process task labels. The absence of collapse here supports a bounded interpretation: ToolRet’s source-style collapse is tied to independent generation sources more than arbitrary task-family differences inside a mostly shared generation process.

Field	Template content
Skill	{tool_name}
Capability	This skill enables the agent to {description}.
When to use	Use this skill when the user request matches {tags/categories/description}.
Inputs	{arguments or schema fields}
Output/effect	{available output or API-effect description}
Metadata	{tags and categories}

Appendix

Table 32: Deterministic skill-card rendering template. The rendering is template-based and changes the visible capability representation while keeping labels, queries, and supervision fixed.

Rendering	1100 OTS	1100 FT	FT-1100 $\rightarrow$ 4996	4996 OTS	4996 FT	Switch
Tool schema	80.8%	91.8%	22.3%	73.7%	85.2%	86.1%
Skill card	72.2%	89.2%	21.7%	64.5%	84.0%	85.2%

Appendix

Table 33: Core skill-card rendering check. Values are depth- 20 candidate coverage. Rerendering tools as executable skill cards lowers the off-the-shelf baseline but preserves the main transfer-collapse and switch-policy pattern.

Source	Queries	OTS	FT-1100	FT-aggregate
APIGen	146	82.9%	2.1%	96.6%
ToolACE	175	75.4%	0.6%	85.7%
UltraTool	72	66.7%	1.4%	98.6%
toollens	56	23.2%	0.0%	87.5%
reversechain	39	66.7%	15.4%	82.0%

Appendix

Table 34: Collapsed-source coverage under skill-card rendering. The natural collapsed sources remain collapsed under FT-1100 , supporting the interpretation that raw schema formatting alone fails to explain the failure.

Setting	Original	MCP-lite	$\Delta$
1100 off-the-shelf	80.8%	68.4%	-12.4
1100 FT (matched)	91.8%	90.5%	-1.3
FT-1100 $\rightarrow$ 4996	22.3%	25.1%	+2.8
4996 off-the-shelf	73.7%	66.8%	-6.9
4996 FT (matched)	85.2%	84.3%	-0.9
FT-4996 $\rightarrow$ 1100	87.3%	87.3%	+0.0

Appendix

Table 35: Paired retrain/eval under a stricter MCP-lite tool rendering. Both ToolRet- 1100 and ToolRet- 4996 are rebuilt with unified name + description + arguments fields and the retrievers are retrained on the corresponding normalized corpora before evaluation. Schema normalization leaves the narrow-checkpoint collapse in place and lowers off-the-shelf coverage substantially. All values are retriever coverage, before reranked global top-1.

Source	FT-1100	FT-aggregate	$\Delta$
APIGen	0.337	0.924	+0.587
ToolACE	0.341	0.621	+0.279
UltraTool	0.454	0.915	+0.461
toollens	0.470	0.747	+0.277
reversechain	0.451	0.705	+0.254
Mean (all 5)	0.411	0.782	+0.372

Appendix

Table 36: Representation-level invariance under MCP-lite normalization on collapsed-source gold tools. Each row reports the mean cosine similarity between the embedding of a tool’s original text and the embedding of its MCP-lite text under the same retriever checkpoint, using 200 sampled gold tools per source ( 1,000 tools total). The aggregate-trained checkpoint produces more stable tool representations than FT-1100 in every source. The especially high APIGen value reflects that its source format is already close to the MCP-lite template.

Depth	Cov.	P@1		R@5		nDCG@5
		Covered	Global	Covered	Global	Covered
10	69.4%	64.3%	44.6%	90.7%	62.9%	77.1%
20	76.2%	58.7%	44.7%	78.7%	60.0%	67.6%
50	83.9%	47.4%	39.8%	64.8%	54.4%	55.2%

Appendix

Table 37: Exact ToolRet depth-sweep values behind Figure 3 . Increasing depth raises coverage, but covered-query quality decays enough that the global top-1 proxy peaks at the main 20 -candidate operating point.

Phenomenon	Query Form	Gold Tools	Why It Matters
Same-corpus source-style shift	APIGen : “Please provide the latest songs from the artist Beyoncé and also list the countries in ascending order.”	all_songs_from_artist ; getallcountry	The query has obvious lexical access to the tool schemas, yet it is phrased as a compact API-style command instead of a longer end-user task description. This is the kind of high-overlap cross-source case where the 1100 -trained retriever still collapses.
Compositional tail / staged repair	Yosemite query: weather forecast + trail recommendations + scenic-view filtering in one request	VALUE_SERP_Google_Search ; VALUE_SERP_Google_Place _Details ; VALUE_SERP_Google_Video	Single-shot retrieval must compress heterogeneous intents into one embedding. The conservative splitter rewrites this into “Provide the weather forecast for Yosemite …” and “Suggest popular hiking trails in Yosemite …”, which produces two cleaner retrieval problems before the candidates are merged.

Appendix

Table 38: Two concrete cases behind the main findings. The first illustrates why same-corpus source-style shift goes beyond missing keywords. The second illustrates why staged retrieval helps the compositional tail even when direct dense retrieval is already strong on average.

Setting	Cov.	P@1	R@5	Global top-1
Single + original reranker	76.2%	45.6%	77.5%	34.7%
Split + original reranker	82.9%	39.2%	77.3%	32.5%
Split + adapted linear	82.9%	46.8%	78.8%	38.8%
Split + adapted MLP	82.9%	55.7%	83.6%	46.2%
Split + adapted cross-encoder	82.9%	67.1%	93.1%	55.6%

Appendix

Table 39: Staged-retrieval follow-up on ToolRet under the same candidate-generation setup as the main split-and-merge probe. The merged pool hurts top-1 quality under the unchanged reranker, but adapted rerankers recover and then surpass the single-query global proxy. A compact cross-encoder pushes the same staged pool further upward, showing that the staged candidate generator remains competitive under stronger rerankers as well.

top- $k$ skills	Manual skill $\rightarrow$ tool				Automatic skill $\rightarrow$ tool
top- $k$ skills	P@1	R@5	Cov.	Cand. Red.	P@1	R@5	Cov.	Cand. Red.
1	34.9%	46.4%	49.8%	96.5%	42.6%	57.1%	65.0%	97.2%
3	47.1%	63.0%	67.8%	89.8%	53.7%	71.8%	80.5%	92.3%
5	51.8%	69.5%	77.3%	83.5%	58.2%	76.9%	89.2%	87.4%
7	55.3%	73.9%	84.4%	77.6%	60.4%	79.2%	92.3%	82.5%
10	59.6%	79.0%	90.3%	69.6%	62.1%	82.6%	95.4%	75.2%

Appendix

Table 40: Main StableToolBench hierarchical-routing Pareto table. Tool-only retrieval is the accuracy anchor; the hierarchy rows trade quality for candidate-context reduction.

Routing Mode	Cand. Tok.	Final Tok.	Cov.	P@1	R@5	Cand. Red.
Full Catalog Injection	172,426	172,426	100.0%	–	–	0.0%
Tool-Only Retrieval	172,426	407	100.0%	66.1%	87.2%	0.0%
Manual Skill $\rightarrow$ Tool ( k=5 )	28390	410	77.3%	51.8%	69.5%	83.5%
Automatic Skill $\rightarrow$ Tool ( k=5 )	21785	411	89.2%	58.2%	76.9%	87.4%

Appendix

Table 41: Routing-stage token efficiency on StableToolBench. Full catalog versus tool-only explains final-prompt compression, while tool-only versus hierarchy explains candidate-pool compression.

Group	Count	Official	Auto	$\Delta$ Auto
G1_instruction	74	35.1%	31.1%	-4.1
G1_category	65	33.8%	46.2%	+12.3
G1_tool	57	31.6%	29.8%	-1.8
G2_instruction	26	26.9%	26.9%	+0.0
G2_category	16	25.0%	6.2%	-18.8
G3_instruction	21	23.8%	33.3%	+9.5

Appendix

Table 42: Benchmark-native local execution on the 259 -query paired common subset shared by official_api_list and auto_skill_tool across all six StableToolBench groups. The two methods use the same local Qwen-14B planner and the same local function-call-accuracy-style judge. Average end-to-end token counts stay nearly identical ( 5,807 vs. 5,822 ) because planner/execution overhead dominates the full trace length, but token cost per solved task still favors the automatic hierarchy ( 17.7k vs. 18.3k ).

LLM Agent Skill Retrieval

Large language model agent skill retrieval finds and selects reusable procedures to help agents complete tasks.

See all 101 papers

SkillSeek: Revisiting Agent Skill Retrieval at Marketplace Scale
Sep 30, 2026Guanqun Yang, Wenlong Zhang, Tian Shi +1LLM Agent Skill Retrieval Information Retrieval
Stevens Institute of Technology, Hoboken, NJ, USA · Independent Researcher
SkillSight: Calibrating Generic Content Bias for Skill Retrieval
Jul 21, 2026Jinying Xiao, Bin Li, Xiaopeng Li +7Lexical Retrieval LLM Agent Skill Retrieval
1National University of Defense Technology · 2Qinghai Normal University · 3Xizang University
Skill Following: Evaluating Actual Skill Use in Retrieval-Enabled LLM Agents
Sep 1, 2026Seonghyeon Cho, Chanjun ParkTool-Use Evaluation Tool-Augmented Language Model Agents
Soongsil University

Sep 30, 2026J/K move · Enter open · S save

SkillSeek: Revisiting Agent Skill Retrieval at Marketplace Scale

Guanqun Yang, Wenlong Zhang, Tian Shi, Ping Wang

Stevens Institute of Technology, Hoboken, NJ, USA · Independent Researcher

LLM Agent Skill Retrieval Information Retrieval

Open paper AskPDF arXiv

Anthropic's Agent Skills package reusable procedural know-how for an LLM agent into SKILL.md directories, and open-source aggregations have grown past 230,000 skills, making selection rather than authoring the bottleneck. The standing answer in the literature outsources selection to the agent itself: an LLM-mediated retrieval loop that rewrites queries and refines candidates inside the agent's decision loop, paying LLM tokens on every task. We present SkillSeek, an open-source two-stage skill retriever built from the standard IR recipe (a BGE-base bi-encoder feeding a small cross-encoder, exposed over MCP). Across a

4 \times 11

grid of pool, backbone, and method on the 89-task SkillsBench benchmark, SkillSeek reaches observed parity with the LLM-mediated loop of Liu et al. at essentially no extra cost: plain bm25 alone records a pass rate at or above their refined loop on three of four settings, and a small cross-encoder covers the remaining difference on the fourth. A first-stage recall ceiling explains the pattern, and total per-trial spend drops from USD 51.30 to USD 27.54 (within fifty cents of the no-skill baseline). Under the SkillsBench tasks and OpenHands harness we tested, this positions the standard IR recipe as a strong default for agent-skill retrieval, with LLM-mediated alternatives a natural fit for cases where deterministic methods fall short.

Abstract

LLM Agent Skill Retrieval

Explore similar work

SkillSeek: Revisiting Agent Skill Retrieval at Marketplace Scale

SkillSight: Calibrating Generic Content Bias for Skill Retrieval

Skill Following: Evaluating Actual Skill Use in Retrieval-Enabled LLM Agents