From Relevance to Validity: Zero-Shot Tool Selection via Functional and Contextual Validation

Xiaoxuan Dong; Jin Zhang; Zichen Yuan; Yue Zhou; Zihao Rao; Dexu Yu; Tingyong WU; Wenhao Deng

cs.AIOct 14, 2026

From Relevance to Validity: Zero-Shot Tool Selection via Functional and Contextual Validation

Authors: Xiaoxuan Dong, Jin Zhang, Zichen Yuan, Yue Zhou, Zihao Rao, Dexu Yu, Tingyong WU, Wenhao Deng,

Organizations: UESTC · LZU · CityUHK · Northwestern ·

Abstract

Large language model (LLM) agents increasingly face tasks from unseen domains and execution contexts, making zero-shot tool selection essential for identifying tools that can satisfy task requirements. Existing dominant methods mainly rely on relevance ranking, but semantic matching alone cannot ensure that a tool is functionally compatible or remains applicable in a new context. We preliminarily identify a transfer-validity gap, where semantically relevant tools may still fail to satisfy functional or contextual requirements after transfer. To address this gap, we propose \methodname{}, a progressive validation framework that complements semantic relevance with endpoint-grounded functional compatibility and task-conditioned contextual evidence. Across cross-domain and cross-dataset transfer, \methodname{} consistently improves capability ranking, with larger gains under functional or contextual ambiguity. Ablation results further confirm the complementary contributions of functional and contextual validation.

Figures & tables

Figure 1: Zero-shot capability transfer involves three questions: whether a candidate is relevant, whether its endpoints can satisfy the task, and whether source-side support remains applicable to the current task.

Figure 2: Controlled diagnostics of the transfer-validity gap. Left: functionally different candidates frequently remain close under frozen semantic relevance. Right: changing one task-context condition while preserving the capability and callable function flips applicability in

15.20\%

of supported-base, determinate pairs (

62/408

Figure 3: Overview of TACT . Semantic Proposal supplies the anchor $A$ ; Functional Validation estimates Endpoint-Grounded compatibility $I$ ; Contextual Validation produces the signed applicability correction $D$ ; and Validated Ranking combines them through bounded gates. The semantic retrieval panel shows cosine similarities before the log-sigmoid calibration in Equation ( 3 ); final-panel scores are schematic illustrations of the resulting order.

Protocol	Model	Natural			Semantic			Context			Mixed
		MRR	N@5	H@3	MRR	N@5	H@3	MRR	N@5	H@3	MRR	N@5	H@3
	Random	0.227	0.157	0.220	0.222	0.157	0.209	0.222	0.154	0.212	0.219	0.153	0.205
		$\pm$ 0.001	$\pm$ 0.001	$\pm$ 0.003	$\pm$ 0.004	$\pm$ 0.006	$\pm$ 0.008	$\pm$ 0.001	$\pm$ 0.001	$\pm$ 0.004	$\pm$ 0.001	$\pm$ 0.003	$\pm$ 0.004
	Semantic Cosine	0.881	0.840	0.911	0.263	0.246	0.305	0.259	0.225	0.297	0.213	0.165	0.190
		$\pm$ 0.000	$\pm$ 0.000	$\pm$ 0.000	$\pm$ 0.000	$\pm$ 0.000	$\pm$ 0.000	$\pm$ 0.000	$\pm$ 0.000	$\pm$ 0.000	$\pm$ 0.000	$\pm$ 0.000	$\pm$ 0.000
	Pointwise MLP	0.740	0.699	0.823	0.440	0.410	0.500	0.435	0.392	0.487	0.399	0.355	0.442

Table 1: Overall performance on zero-shot capability ranking under cross-domain and cross-dataset transfer across four candidate views, reported as mean

\pm

standard deviation over three runs. Best and second-best means are bolded and underlined, respectively. The final row of each protocol reports TACT’s absolute gain over the strongest non-TACT baseline in percentage points (pp).

Protocol	Variant	Natural			Semantic			Context			Mixed
		MRR	N@5	H@3	MRR	N@5	H@3	MRR	N@5	H@3	MRR	N@5	H@3
Cross domain	TACT (Full)	0.915	0.873	0.952	0.629	0.604	0.726	0.672	0.624	0.755	0.564	0.529	0.644
	w/o Context	0.899	0.853	0.939	0.620	0.592	0.720	0.594	0.552	0.677	0.521	0.483	0.594
	w/o Functional	0.903	0.859	0.941	0.568	0.545	0.660	0.605	0.557	0.683	0.504	0.467	0.579
	Anchor only	0.881	0.840	0.911	0.263	0.246	0.305	0.259	0.225	0.297	0.213	0.165	0.190
Cross dataset	TACT (Full)	0.763	0.746	0.836	0.613	0.580	0.668	0.574	0.543	0.613	0.538	0.500	0.572

Table 2: Ablation Study. Cross-domain and cross-dataset transfer without Functional or Contextual Validation; Anchor only uses the frozen semantic path. Trainable variants are independently fitted and selected on source data, unlike RQ4’s fixed-checkpoint diagnostic. Best results per setting are bold.

Figure 4: Downstream Execution. Completion rates on 120 solvable StableToolBench queries under cached execution with four frozen agent backbones. None denotes no tools; Sem., Pool., ReInv., and Full use Top-5 capabilities from Semantic Cosine, MultiPool, Re-Invoke, and TACT, respectively.

Scale	Natural		Semantic		Context		Mixed
Scale	Sem.	TACT	Sem.	TACT	Sem.	TACT	Sem.	TACT
Base	0.8397	0.8725	0.2458	0.6035	0.2246	0.6244	0.1654	0.5288
100	0.4319	0.4454	0.0406	0.2930	0.0780	0.3242	0.0600	0.2862
500	0.4015	0.4100	0.0346	0.2478	0.0626	0.2782	0.0508	0.2462
Full	0.3822	0.4011	0.0333	0.2315	0.0563	0.2602	0.0483	0.2311

Table 3: Library Scaling. Cross-domain NDCG@5 by candidate-pool size. Base: evaluation manifest; Full: 3,442 capabilities; Sem.: Semantic Cosine. Pairwise best in bold.

Figure 5: Cross-domain NDCG@5 gains from contextual validation across source-association retention levels. Error bars indicate paired 95% cluster-bootstrap intervals.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Diagnostic statistic	Value
Eligible functional-diagnostic tasks	29,178
$d_{t}<0.005$	31.41%
$d_{t}<0.01$	44.25%
$d_{t}<0.02$	57.14%
$d_{t}<0.05$	73.62%
Constructed contextual pairs	500

Appendix

Table 4: Functional ambiguity and contextual instability corresponding to Figure 2 and Section 3.1 of the main paper.

Changed condition	Included	Flips	Rate (%)
Access policy	64	6	9.38
Execution environment	68	5	7.35
Geographic scope	62	8	12.90
Locale	71	9	12.68
Platform	73	23	31.51
Temporal scope	70	11	15.71

Appendix

Table 5: Contextual instability by changed condition. Each denominator contains only supported-base, determinate pairs. These descriptive rates are not ranking metrics or cross-domain test results.

Dataset	View	Tasks	Caps.	Pairs	Pos. (%)	Sem. (%)	Ctx. (%)	Other (%)	Cand./Task
ToolBench	Natural	201,774	3,442	4,370,648	7.7	0.0	0.0	92.3	21.66
	Semantic	50,254	3,442	1,084,884	7.4	30.4	0.0	62.3	21.59
	Context	75,709	3,442	1,637,800	7.5	0.0	51.4	41.1	21.63
	Mixed	62,266	3,442	1,345,846	7.5	22.0	24.7	45.9	21.61
TaskBench	Natural	17,136	103	205,535	23.7	0.0	0.0	76.3	11.99
	Semantic	17,136	103	220,048	22.1	77.9	0.0	0.0	12.84

Appendix

Table 7: Per-dataset scale and candidate composition of the four views under cross-dataset leave-one-dataset-out evaluation. Caps. denotes the number of distinct capabilities appearing in each view manifest. Percentages are independently rounded and may not sum to exactly 100.0. Pos., Sem., and Ctx. denote observed positives, Semantic candidates, and Context candidates, respectively. View-specific eligibility conditions may produce different task populations, but all methods share the same frozen manifest within each dataset–view combination.

View	Domains	Tasks/Pools	Caps.	Pairs	Pos. (%)	Sem. (%)	Ctx. (%)	Other (%)	Cand./Task
Natural	49	201,774	3,442	4,370,648	7.7	0.0	0.0	92.3	21.66
Semantic	42	50,254	3,442	1,084,884	7.4	30.4	0.0	62.3	21.59
Context	49	75,709	3,442	1,637,800	7.5	0.0	51.4	41.1	21.63
Mixed	46	62,266	3,442	1,345,846	7.5	22.0	24.7	45.9	21.61

Appendix

Table 8: Aggregate scale and candidate composition of the four views under the ToolBench candidate manifests. Domains denotes the number of domains containing candidate records; it is not the scored-domain count used in the ranking macro average. Caps. is the number of distinct candidate capabilities. Pos., Sem., and Ctx. denote observed positives, Semantic candidates, and Context candidates, respectively.

Dataset	View	Unit	$N$	Valid	Valid (%)	Invalid	Unresolved
ToolBench	Semantic	Pair	160	152	95.0	8	0
	Context	Pair	160	148	92.5	12	0
	Mixed	Pool	100	92	92.0	8	0
TaskBench	Semantic	Pair	160	154	96.3	6	0
	Context	Pair	160	150	93.8	10	0
	Mixed	Pool	100	93	93.0	7	0

Appendix

Table 9: Per-dataset AI construction audit. Two independent AI judges apply the same view-specific rubric under double-blind review. Valid counts require approval from both judges. Semantic and Context are evaluated at the pair level, whereas Mixed is evaluated at the candidate-pool level.

View	Audited records	Unit	$N$	Valid	Valid (%)	Agreement (%)	Sem. $\kappa$	Func. $\kappa$
Natural	98	Pair	85	82	96.5	92.7	0.920	0.767
Semantic	126	Pair	84	78	92.9	89.1	0.750	0.844
Context	147	Pair	98	92	93.9	90.3	0.797	0.740
Mixed	496	Pool	46	43	93.5	93.8	0.687	0.747

Appendix

Table 10: Double-blind human audit under ToolBench cross-domain transfer. Acceptance requires approval from both human reviewers and passage of the AI audit under the same view-specific criteria. For Semantic and Context,

N

denotes audited hard candidates; for Mixed, it denotes audited candidate pools. Agreement and Cohen’s

\kappa

are computed from the two independent human reviews. Natural is included as a reference for reviewer agreement rather than as a candidate-view construction test.

Configuration	Value
Frozen encoder	all-MiniLM-L6-v2
Embedding dimension	384
Maximum requirements $N_{\max}$	4
Endpoint retrieval $K$	4
Maximum exact assignments	209
Initial null score $\nu$	0.15

Appendix

Table 11: Main implementation configuration of TACT. Discrete sets denote source-only search spaces. All correction grids contain a zero-valued setting to permit exact fallback to the preceding component.

Method	Configuration
Logistic Ranker	Eight pair features; StandardScaler ; balanced pointwise logistic regression with $C=1$ , lbfgs , and 500 maximum iterations.
Pointwise MLP	Input $[e_{t},e_{c},e_{t}\odot e_{c},\|e_{t}-e_{c}\|]$ ; $4D\!\rightarrow\!256\!\rightarrow\!64\!\rightarrow\!1$ with LayerNorm, ReLU, and dropout $0.10$ ; weighted BCE with positive weight $\min(n_{-}/n_{+},20)$ ; AdamW ( $10^{-3}$ , weight decay $10^{-4}$ ); batch size 2048; cosine schedule; clip 1.0; optimized for up to 20 cross-domain or 12 cross-dataset epochs.
MultiPool	Separate $D\!\rightarrow\!256\!\rightarrow\!128$ task/capability encoders, dropout $0.05$ , raw-cosine plus zero-initialized bounded residual; pairwise hinge margin $0.03$ ; AdamW ( $10^{-3}$ , $10^{-4}$ ); at most 50,000 pairs, two positives and five negatives per task; batch size 1024/2048 and up to 20/12 training epochs for cross-domain/cross-dataset. The cross-dataset rows use label-free transductive model_signature alignment based on unlabeled target embedding statistics.
ZESRec	Separate $D\!\rightarrow\!256\!\rightarrow\!128$ towers; normalized dot product; 128-d zero-initialized capability-ID residual (weight 0.20) and ID bias (weight 0.05); no domain ID. Weighted BCE and the Pointwise-MLP optimizer, batch size, schedule, clipping, and training budget.
UniSRec	Separate $D\!\rightarrow\!128$ MoE adaptors with eight parametric-whitening experts, dropout $0.05$ , noisy soft gates, and learned temperature initialized at 10 and clipped to $[1,30]$ ; no domain ID. The same weighted BCE objective and optimization schedule as Pointwise MLP.
AREAD	Invariant and domain towers $D\!\rightarrow\!192\!\rightarrow\!96$ ; 64-d source-domain embedding, FiLM conditioning, four experts, domain-correction gate, $\alpha=1$ , and centroid inference temperature $0.10$ . It uses the same weighted BCE objective and optimization schedule as Pointwise MLP.

Appendix

Table 12: Configurations of baselines and controls. ZESRec, UniSRec, and AREAD are task-to-capability adaptations. Re-Invoke, ToolDreamer, and PORTS retain the authors’ model configurations and use the same task–capability ranking interface.

Method	Generation and retrieval configuration
Re-Invoke	text-bison@001 generates ten synthetic queries per capability at temperature 0.7 and extracts task intents at temperature 0.0. textembedding-gecko@003 embeds augmented documents and intents; the expanded document embeddings are averaged.
ToolDreamer	GPT-4.1 generates hypothetical tools. Each tool contributes its thought, name, and description alongside the original query (QTND). Retrieval uses the Qwen3-8B dense and BM25 configurations with RRF over the resulting ranked lists.

Appendix

Table 13: Generation and retrieval settings used by Re-Invoke and ToolDreamer.

Natural : held-out domain	$N$	MRR	NDCG@5	HR@3
Advertising	2,189	0.7984 (0.0213)	0.7479 (0.0220)	0.8502 (0.0242)
Artificial Intelligence / ML	2,076	0.9270 (0.0046)	0.8832 (0.0083)	0.9478 (0.0041)
Business	7,229	0.8522 (0.0051)	0.7828 (0.0076)	0.8974 (0.0051)
Business Software	2,828	0.8509 (0.0089)	0.7948 (0.0098)	0.9004 (0.0071)
Commerce	2,686	0.9050 (0.0095)	0.8532 (0.0127)	0.9443 (0.0092)
Communication	4,015	0.8737 (0.0004)	0.8211 (0.0015)	0.9310 (0.0031)

Appendix

Table 16: Per-domain TACT (Full) results: Natural . Values are seed means (sample standard deviations), using seeds 42, 43, and 44.

N

is the recorded task count. A dash denotes no reported score, including when

N>0

, and is not zero. This view has 49/49 scored domains; ML denotes Machine Learning.

Semantic : held-out domain	$N$	MRR	NDCG@5	HR@3
Advertising	15	0.5617 (0.0380)	0.5291 (0.0410)	0.6000 (0.0667)
Artificial Intelligence / ML	229	0.6276 (0.0150)	0.5987 (0.0121)	0.7074 (0.0131)
Business	1,628	0.5841 (0.0061)	0.5101 (0.0055)	0.6396 (0.0065)
Business Software	615	0.6250 (0.0124)	0.5737 (0.0115)	0.6678 (0.0199)
Commerce	224	0.8091 (0.0249)	0.8094 (0.0293)	0.9137 (0.0269)
Communication	695	0.7016 (0.0073)	0.6748 (0.0040)	0.8072 (0.0094)

Appendix

Table 17: Per-domain TACT (Full) results: Semantic . Values are seed means (sample standard deviations), using seeds 42, 43, and 44.

N

is the recorded task count. A dash denotes no reported score, including when

N>0

, and is not zero. This view has 39/49 scored domains; ML denotes Machine Learning.

Context : held-out domain	$N$	MRR	NDCG@5	HR@3
Advertising	1,674	0.6202 (0.0193)	0.5858 (0.0202)	0.6808 (0.0247)
Artificial Intelligence / ML	685	0.6761 (0.0109)	0.6279 (0.0038)	0.7611 (0.0034)
Business	3,475	0.6159 (0.0017)	0.5450 (0.0028)	0.6718 (0.0012)
Business Software	1,643	0.6307 (0.0136)	0.5735 (0.0129)	0.6963 (0.0171)
Commerce	1,131	0.6615 (0.0145)	0.6143 (0.0167)	0.7586 (0.0248)
Communication	2,081	0.6157 (0.0042)	0.5519 (0.0039)	0.6761 (0.0046)

Appendix

Table 18: Per-domain TACT (Full) results: Context . Values are seed means (sample standard deviations), using seeds 42, 43, and 44.

N

is the recorded task count. A dash denotes no reported score, including when

N>0

, and is not zero. This view has 49/49 scored domains; ML denotes Machine Learning.

Mixed : held-out domain	$N$	MRR	NDCG@5	HR@3
Advertising	344	0.5005 (0.0279)	0.4675 (0.0307)	0.5514 (0.0244)
Artificial Intelligence / ML	477	0.5930 (0.0192)	0.5656 (0.0118)	0.6953 (0.0087)
Business	2,295	0.5649 (0.0044)	0.4932 (0.0028)	0.6115 (0.0038)
Business Software	913	0.6063 (0.0118)	0.5555 (0.0125)	0.6484 (0.0162)
Commerce	667	0.6508 (0.0181)	0.6186 (0.0216)	0.7261 (0.0350)
Communication	1,316	0.5869 (0.0057)	0.5499 (0.0035)	0.6654 (0.0076)

Appendix

Table 19: Per-domain TACT (Full) results: Mixed . Values are seed means (sample standard deviations), using seeds 42, 43, and 44.

N

is the recorded task count. A dash denotes no reported score, including when

N>0

, and is not zero. This view has 44/49 scored domains; ML denotes Machine Learning.

View	Target dataset	MRR	NDCG@5	HR@3
Natural	ToolBench	0.8975 (0.0055)	0.8648 (0.0045)	0.9326 (0.0020)
	TaskBench	0.8387 (0.0097)	0.7366 (0.0129)	0.9160 (0.0095)
	MetaTool	0.6839 (0.0032)	0.6981 (0.0024)	0.7429 (0.0054)
	APIBench	0.6334 (0.0027)	0.6847 (0.0027)	0.7514 (0.0046)
Semantic	ToolBench	0.4167 (0.0033)	0.3817 (0.0025)	0.4859 (0.0033)
	TaskBench	0.9170 (0.0075)	0.8107 (0.0042)	0.9700 (0.0035)

Appendix

Table 20: Per-dataset TACT (Full) results. Each entry is the mean (sample standard deviation) across seeds 42, 43, and 44. Higher is better.

Targets	View	Scored	$N_{\mathrm{scored}}$	MRR	NDCG@5	HR@3
Domains	Natural	49/49	201,774	0.915159	0.872471	0.951539
	Semantic	39/49	50,249	0.629185	0.603492	0.726141
	Context	49/49	75,709	0.671839	0.624384	0.755076
	Mixed	44/49	62,260	0.564361	0.528805	0.644375
Datasets	Natural	4/4	–	0.763375	0.746050	0.835725
	Semantic	4/4	–	0.613150	0.580325	0.667875

Appendix

Table 21: Equal-target means calculated from the rounded per-target means. Six decimals retain the arithmetic of displayed inputs rather than imply six-decimal measurement precision. Missing scores are excluded under the evaluation eligibility rule.

N_{\mathrm{scored}}

sums counts only over scored domain cells; dataset-level counts are omitted.

Group	Semantic scope	# Domains	Included ToolBench domains	Natural tasks
G1	Data, Security, and Communication	10	Cybersecurity, Cryptography, Data, Database, Communication, Email, Monitoring, SMS, Text_Analysis, and Translation	41,503
G2	Business, Finance, and Commerce	10	Advertising, Business, Business_Software, Commerce, eCommerce, Finance, Financial, Jobs, Payments, and Reward	39,275
G3	Media, Leisure, and Social	10	Entertainment, Events, Gaming, Media, Movies, Music, News_Media, Social, Sports, and Video_Images	63,943
G4	Mobility and Infrastructure	10	Devices, Energy, Logistics, Location, Mapping, Other, Storage, Transportation, Travel, and Weather	26,603
G5	AI, Life, and Knowledge	9	Artificial_Intelligence_Machine_Learning, Education, Food, Health_and_Fitness, Medical, Science, Search, Tools, and Visual_Recognition	30,450

Appendix

Table 22: Fixed ToolBench domain groups used for the supplementary leave-one-group-out evaluation. Each group is held out once, and the remaining four groups form the outer-source set. The five groups jointly cover all 49 ToolBench domains without overlap. Natural tasks denote Natural-view evaluation tasks in each group and sum to 201,774.

Protocol	Model	Natural			Semantic			Context			Mixed
		MRR	N@5	H@3	MRR	N@5	H@3	MRR	N@5	H@3	MRR	N@5	H@3
	Random	0.2841	0.1926	0.2735	0.1682	0.2134	0.2817	0.1745	0.2284	0.1836	0.2712	0.1947	0.1423
		$\pm$ 0.0021	$\pm$ 0.0004	$\pm$ 0.0036	$\pm$ 0.0049	$\pm$ 0.0068	$\pm$ 0.0071	$\pm$ 0.0030	$\pm$ 0.0027	$\pm$ 0.0025	$\pm$ 0.0022	$\pm$ 0.0014	$\pm$ 0.0044
	Semantic Cosine	0.8264	0.7513	0.8472	0.3416	0.1892	0.2438	0.1974	0.3125	0.2251	0.2985	0.2417	0.2766
		$\pm$ 0.0000	$\pm$ 0.0000	$\pm$ 0.0000	$\pm$ 0.0000	$\pm$ 0.0000	$\pm$ 0.0000	$\pm$ 0.0000	$\pm$ 0.0000	$\pm$ 0.0000	$\pm$ 0.0000	$\pm$ 0.0000	$\pm$ 0.0000
	Pointwise MLP	0.6428	0.7861	0.7315	0.3674	0.4912	0.4136	0.3487	0.4732	0.5794	0.4618	0.3105	0.3642

Appendix

Table 23: Supplementary ToolBench leave-one-group-out capability ranking. Each of the five groups is held out once and the remaining four are used as sources. Means appear above smaller sample standard deviations over seeds 42, 43, and 44; best and second-best displayed means are bold and underlined. The final row gives the absolute TACT gain over the strongest non-TACT baseline in percentage points (pp). N@5 and H@3 denote NDCG@5 and HR@3; Endpoint Schema denotes Endpoint Schema Match. Deterministic methods have zero seed variance.

Pre-evaluation filter stage	Remaining queries
Initial candidate query pool	5000
Annotated capability is present in the frozen 3,442-capability library	3078
At least one public GET / HEAD endpoint is available	1528
Required arguments can be instantiated from the public schema	745
At least one permitted endpoint has a valid MirrorAPI-Cache response	354
No authentication, write operation, or live external side effect is required	147

Appendix

Table 24: Pre-evaluation solvability filtering for cached execution. Counts are obtained from the frozen sampling manifest before any ranking policy is evaluated.

	Full vs. Semantic Cosine			Full vs. Re-Invoke
Agent Backbone	$\Delta$ Completion	Bootstrap 95% CI	Exact $p$	$\Delta$ Completion	Bootstrap 95% CI	Exact $p$
GPT-5.6 Terra	+13.33 pp	[+5.83, +20.83]	0.0015	+8.33 pp	[+1.67, +15.83]	0.0414
Grok-4.5	+9.17 pp	[+2.50, +15.83]	0.0127	+8.33 pp	[+4.17, +13.33]	0.0020
Claude Sonnet 5	+9.17 pp	[+3.33, +15.83]	0.0074	+8.33 pp	[+2.50, +15.00]	0.0213
Gemini 3.1 Pro	+13.33 pp	[+5.83, +20.83]	0.0009	+9.17 pp	[+2.50, +15.83]	0.0192

Appendix

Table 25: Paired completion-rate comparisons of TACT (Full) against Semantic Cosine Top-5 and Re-Invoke Top-5. Intervals are bootstrap 95% confidence intervals, and

p

values are obtained from exact McNemar tests.

Tool Selection

Tool selection studies how AI agents decide when to use external tools and which APIs, models, or physical tools best fit a task.

See all 25 papers

Toolcompass: Guiding Tool Trialing, Not Suppressing It
Sep 22, 2026Junlin Fang, Chong Zhang, Do Nguyen-Thanh +3Representation Learning Tool-Augmented Language Model Agents
College of Computing and Data Science, Nanyang Technological University · Zhejiang University · Australian Artificial Intelligence Institute, University of Technology Sydney +1
Evaluating Tool-Using Language Agents: Judge Reliability, Propagation Cascades, and Runtime Mitigation in AgentProp-Bench
Apr 17, 2026Bhaskar GurramAI Agent Reliability Tool-Augmented Language Model Agents
Zasti Inc., Ashburn, VA, USA
Training Small Language Models for Transferable Tool Selection
Oct 22, 2026Zhaoyi Zhou, Akshayaa Magesh, Jeongyeol Kwon +4Tool Selection for Language Model Agents Small Language Models
Meta · Carnegie Mellon University

Sep 22, 2026J/K move · Enter open · S save

Toolcompass: Guiding Tool Trialing, Not Suppressing It

Junlin Fang, Chong Zhang, Do Nguyen-Thanh, Xiaogang Xu+2

College of Computing and Data Science, Nanyang Technological University · Zhejiang University · Australian Artificial Intelligence Institute, University of Technology Sydney+1

Representation Learning Tool-Augmented Language Model Agents LLM Tool Use LLM Post-Training Tool Use Tool Selection for Language Model Agents

Open paper AskPDF arXiv

Large language model (LLM) agents must generalize from tools seen during training to unseen tools at deployment. A key challenge is tool trialing, i.e., excessive trials waste the interaction budget, whereas selective trials enable exploration of unfamiliar tools. Existing outcome-based post-training leaves wasteful trials unguided, while turn-level supervision may suppress necessary exploration. We introduce ToolCompass, a post-training framework that guides tool trialing by organizing tool-call representations according to shared functions. Specifically, ToolCompass models each function class as a von Mises--Fisher distribution and jointly reduces intra-function variation across domains and increases inter-function separation. This structure transfers experience from seen tools to functionally similar unseen tools, directing exploration away from unrelated alternatives. ToolCompass requires no ground-truth call traces or unseen-tool access and incurs no inference overhead. Experiments on AppWorld and FTRL show consistent gains across GRPO, RFT, and DMPO. improves AppWorld OOD task success by up to 10.71 percentage points over vanilla post-training and performs best among competitive baselines on both benchmarks.

Abstract

Tool Selection

Explore similar work

Toolcompass: Guiding Tool Trialing, Not Suppressing It

Evaluating Tool-Using Language Agents: Judge Reliability, Propagation Cascades, and Runtime Mitigation in AgentProp-Bench

Training Small Language Models for Transferable Tool Selection