Large language model (LLM) agents increasingly face tasks from unseen domains and execution contexts, making zero-shot tool selection essential for identifying tools that can satisfy task requirements. Existing dominant methods mainly rely on relevance ranking, but semantic matching alone cannot ensure that a tool is functionally compatible or remains applicable in a new context. We preliminarily identify a transfer-validity gap, where semantically relevant tools may still fail to satisfy functional or contextual requirements after transfer. To address this gap, we propose \methodname{}, a progressive validation framework that complements semantic relevance with endpoint-grounded functional compatibility and task-conditioned contextual evidence. Across cross-domain and cross-dataset transfer, \methodname{} consistently improves capability ranking, with larger gains under functional or contextual ambiguity. Ablation results further confirm the complementary contributions of functional and contextual validation.
Figures & tables
Figure 1: Zero-shot capability transfer involves three questions: whether a candidate is relevant, whether its endpoints can satisfy the task, and whether source-side support remains applicable to the current task.
Figure 2: Controlled diagnostics of the transfer-validity gap. Left: functionally different candidates frequently remain close under frozen semantic relevance. Right: changing one task-context condition while preserving the capability and callable function flips applicability in 15.20% of supported-base, determinate pairs ( 62/408 ).
Figure 3: Overview of TACT . Semantic Proposal supplies the anchor A ; Functional Validation estimates Endpoint-Grounded compatibility I ; Contextual Validation produces the signed applicability correction D ; and Validated Ranking combines them through bounded gates. The semantic retrieval panel shows cosine similarities before the log-sigmoid calibration in Equation ( 3 ); final-panel scores are schematic illustrations of the resulting order.
Protocol
Model
Natural
Semantic
Context
Mixed
MRR
N@5
H@3
MRR
N@5
H@3
MRR
N@5
H@3
MRR
N@5
H@3
Random
0.227
0.157
0.220
0.222
0.157
0.209
0.222
0.154
0.212
0.219
0.153
0.205
± 0.001
± 0.001
± 0.003
± 0.004
± 0.006
± 0.008
± 0.001
± 0.001
± 0.004
± 0.001
± 0.003
± 0.004
Semantic Cosine
0.881
0.840
0.911
0.263
0.246
0.305
0.259
0.225
0.297
0.213
0.165
0.190
± 0.000
± 0.000
± 0.000
± 0.000
± 0.000
± 0.000
± 0.000
± 0.000
± 0.000
± 0.000
± 0.000
± 0.000
Pointwise MLP
0.740
0.699
0.823
0.440
0.410
0.500
0.435
0.392
0.487
0.399
0.355
0.442
Table 1: Overall performance on zero-shot capability ranking under cross-domain and cross-dataset transfer across four candidate views, reported as mean ± standard deviation over three runs. Best and second-best means are bolded and underlined, respectively. The final row of each protocol reports TACT’s absolute gain over the strongest non-TACT baseline in percentage points (pp).
Protocol
Variant
Natural
Semantic
Context
Mixed
MRR
N@5
H@3
MRR
N@5
H@3
MRR
N@5
H@3
MRR
N@5
H@3
Cross domain
TACT (Full)
0.915
0.873
0.952
0.629
0.604
0.726
0.672
0.624
0.755
0.564
0.529
0.644
w/o Context
0.899
0.853
0.939
0.620
0.592
0.720
0.594
0.552
0.677
0.521
0.483
0.594
w/o Functional
0.903
0.859
0.941
0.568
0.545
0.660
0.605
0.557
0.683
0.504
0.467
0.579
Anchor only
0.881
0.840
0.911
0.263
0.246
0.305
0.259
0.225
0.297
0.213
0.165
0.190
Cross dataset
TACT (Full)
0.763
0.746
0.836
0.613
0.580
0.668
0.574
0.543
0.613
0.538
0.500
0.572
Table 2: Ablation Study. Cross-domain and cross-dataset transfer without Functional or Contextual Validation; Anchor only uses the frozen semantic path. Trainable variants are independently fitted and selected on source data, unlike RQ4’s fixed-checkpoint diagnostic. Best results per setting are bold.
Figure 4: Downstream Execution. Completion rates on 120 solvable StableToolBench queries under cached execution with four frozen agent backbones. None denotes no tools; Sem., Pool., ReInv., and Full use Top-5 capabilities from Semantic Cosine, MultiPool, Re-Invoke, and TACT, respectively.
Scale
Natural
Semantic
Context
Mixed
Sem.
TACT
Sem.
TACT
Sem.
TACT
Sem.
TACT
Base
0.8397
0.8725
0.2458
0.6035
0.2246
0.6244
0.1654
0.5288
100
0.4319
0.4454
0.0406
0.2930
0.0780
0.3242
0.0600
0.2862
500
0.4015
0.4100
0.0346
0.2478
0.0626
0.2782
0.0508
0.2462
Full
0.3822
0.4011
0.0333
0.2315
0.0563
0.2602
0.0483
0.2311
Table 3: Library Scaling. Cross-domain NDCG@5 by candidate-pool size. Base: evaluation manifest; Full: 3,442 capabilities; Sem.: Semantic Cosine. Pairwise best in bold.
Figure 5: Cross-domain NDCG@5 gains from contextual validation across source-association retention levels. Error bars indicate paired 95% cluster-bootstrap intervals.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Diagnostic statistic
Value
Eligible functional-diagnostic tasks
29,178
dt<0.005
31.41%
dt<0.01
44.25%
dt<0.02
57.14%
dt<0.05
73.62%
Constructed contextual pairs
500
Appendix
Table 4: Functional ambiguity and contextual instability corresponding to Figure 2 and Section 3.1 of the main paper.
Changed condition
Included
Flips
Rate (%)
Access policy
64
6
9.38
Execution environment
68
5
7.35
Geographic scope
62
8
12.90
Locale
71
9
12.68
Platform
73
23
31.51
Temporal scope
70
11
15.71
Appendix
Table 5: Contextual instability by changed condition. Each denominator contains only supported-base, determinate pairs. These descriptive rates are not ranking metrics or cross-domain test results.
Dataset
View
Tasks
Caps.
Pairs
Pos. (%)
Sem. (%)
Ctx. (%)
Other (%)
Cand./Task
ToolBench
Natural
201,774
3,442
4,370,648
7.7
0.0
0.0
92.3
21.66
Semantic
50,254
3,442
1,084,884
7.4
30.4
0.0
62.3
21.59
Context
75,709
3,442
1,637,800
7.5
0.0
51.4
41.1
21.63
Mixed
62,266
3,442
1,345,846
7.5
22.0
24.7
45.9
21.61
TaskBench
Natural
17,136
103
205,535
23.7
0.0
0.0
76.3
11.99
Semantic
17,136
103
220,048
22.1
77.9
0.0
0.0
12.84
Appendix
Table 7: Per-dataset scale and candidate composition of the four views under cross-dataset leave-one-dataset-out evaluation. Caps. denotes the number of distinct capabilities appearing in each view manifest. Percentages are independently rounded and may not sum to exactly 100.0. Pos., Sem., and Ctx. denote observed positives, Semantic candidates, and Context candidates, respectively. View-specific eligibility conditions may produce different task populations, but all methods share the same frozen manifest within each dataset–view combination.
View
Domains
Tasks/Pools
Caps.
Pairs
Pos. (%)
Sem. (%)
Ctx. (%)
Other (%)
Cand./Task
Natural
49
201,774
3,442
4,370,648
7.7
0.0
0.0
92.3
21.66
Semantic
42
50,254
3,442
1,084,884
7.4
30.4
0.0
62.3
21.59
Context
49
75,709
3,442
1,637,800
7.5
0.0
51.4
41.1
21.63
Mixed
46
62,266
3,442
1,345,846
7.5
22.0
24.7
45.9
21.61
Appendix
Table 8: Aggregate scale and candidate composition of the four views under the ToolBench candidate manifests. Domains denotes the number of domains containing candidate records; it is not the scored-domain count used in the ranking macro average. Caps. is the number of distinct candidate capabilities. Pos., Sem., and Ctx. denote observed positives, Semantic candidates, and Context candidates, respectively.
Dataset
View
Unit
N
Valid
Valid (%)
Invalid
Unresolved
ToolBench
Semantic
Pair
160
152
95.0
8
0
Context
Pair
160
148
92.5
12
0
Mixed
Pool
100
92
92.0
8
0
TaskBench
Semantic
Pair
160
154
96.3
6
0
Context
Pair
160
150
93.8
10
0
Mixed
Pool
100
93
93.0
7
0
Appendix
Table 9: Per-dataset AI construction audit. Two independent AI judges apply the same view-specific rubric under double-blind review. Valid counts require approval from both judges. Semantic and Context are evaluated at the pair level, whereas Mixed is evaluated at the candidate-pool level.
View
Audited records
Unit
N
Valid
Valid (%)
Agreement (%)
Sem. κ
Func. κ
Natural
98
Pair
85
82
96.5
92.7
0.920
0.767
Semantic
126
Pair
84
78
92.9
89.1
0.750
0.844
Context
147
Pair
98
92
93.9
90.3
0.797
0.740
Mixed
496
Pool
46
43
93.5
93.8
0.687
0.747
Appendix
Table 10: Double-blind human audit under ToolBench cross-domain transfer. Acceptance requires approval from both human reviewers and passage of the AI audit under the same view-specific criteria. For Semantic and Context, N denotes audited hard candidates; for Mixed, it denotes audited candidate pools. Agreement and Cohen’s κ are computed from the two independent human reviews. Natural is included as a reference for reviewer agreement rather than as a candidate-view construction test.
Configuration
Value
Frozen encoder
all-MiniLM-L6-v2
Embedding dimension
384
Maximum requirements Nmax
4
Endpoint retrieval K
4
Maximum exact assignments
209
Initial null score ν
0.15
Appendix
Table 11: Main implementation configuration of TACT. Discrete sets denote source-only search spaces. All correction grids contain a zero-valued setting to permit exact fallback to the preceding component.
Method
Configuration
Logistic Ranker
Eight pair features; StandardScaler ; balanced pointwise logistic regression with C=1 , lbfgs , and 500 maximum iterations.
Pointwise MLP
Input [et,ec,et⊙ec,∣et−ec∣] ; 4D→256→64→1 with LayerNorm, ReLU, and dropout 0.10 ; weighted BCE with positive weight min(n−/n+,20) ; AdamW ( 10−3 , weight decay 10−4 ); batch size 2048; cosine schedule; clip 1.0; optimized for up to 20 cross-domain or 12 cross-dataset epochs.
MultiPool
Separate D→256→128 task/capability encoders, dropout 0.05 , raw-cosine plus zero-initialized bounded residual; pairwise hinge margin 0.03 ; AdamW ( 10−3 , 10−4 ); at most 50,000 pairs, two positives and five negatives per task; batch size 1024/2048 and up to 20/12 training epochs for cross-domain/cross-dataset. The cross-dataset rows use label-free transductive model_signature alignment based on unlabeled target embedding statistics.
ZESRec
Separate D→256→128 towers; normalized dot product; 128-d zero-initialized capability-ID residual (weight 0.20) and ID bias (weight 0.05); no domain ID. Weighted BCE and the Pointwise-MLP optimizer, batch size, schedule, clipping, and training budget.
UniSRec
Separate D→128 MoE adaptors with eight parametric-whitening experts, dropout 0.05 , noisy soft gates, and learned temperature initialized at 10 and clipped to [1,30] ; no domain ID. The same weighted BCE objective and optimization schedule as Pointwise MLP.
AREAD
Invariant and domain towers D→192→96 ; 64-d source-domain embedding, FiLM conditioning, four experts, domain-correction gate, α=1 , and centroid inference temperature 0.10 . It uses the same weighted BCE objective and optimization schedule as Pointwise MLP.
Appendix
Table 12: Configurations of baselines and controls. ZESRec, UniSRec, and AREAD are task-to-capability adaptations. Re-Invoke, ToolDreamer, and PORTS retain the authors’ model configurations and use the same task–capability ranking interface.
Method
Generation and retrieval configuration
Re-Invoke
text-bison@001 generates ten synthetic queries per capability at temperature 0.7 and extracts task intents at temperature 0.0. textembedding-gecko@003 embeds augmented documents and intents; the expanded document embeddings are averaged.
ToolDreamer
GPT-4.1 generates hypothetical tools. Each tool contributes its thought, name, and description alongside the original query (QTND). Retrieval uses the Qwen3-8B dense and BM25 configurations with RRF over the resulting ranked lists.
Appendix
Table 13: Generation and retrieval settings used by Re-Invoke and ToolDreamer.
Natural : held-out domain
N
MRR
NDCG@5
HR@3
Advertising
2,189
0.7984 (0.0213)
0.7479 (0.0220)
0.8502 (0.0242)
Artificial Intelligence / ML
2,076
0.9270 (0.0046)
0.8832 (0.0083)
0.9478 (0.0041)
Business
7,229
0.8522 (0.0051)
0.7828 (0.0076)
0.8974 (0.0051)
Business Software
2,828
0.8509 (0.0089)
0.7948 (0.0098)
0.9004 (0.0071)
Commerce
2,686
0.9050 (0.0095)
0.8532 (0.0127)
0.9443 (0.0092)
Communication
4,015
0.8737 (0.0004)
0.8211 (0.0015)
0.9310 (0.0031)
Appendix
Table 16: Per-domain TACT (Full) results: Natural . Values are seed means (sample standard deviations), using seeds 42, 43, and 44. N is the recorded task count. A dash denotes no reported score, including when N>0 , and is not zero. This view has 49/49 scored domains; ML denotes Machine Learning.
Semantic : held-out domain
N
MRR
NDCG@5
HR@3
Advertising
15
0.5617 (0.0380)
0.5291 (0.0410)
0.6000 (0.0667)
Artificial Intelligence / ML
229
0.6276 (0.0150)
0.5987 (0.0121)
0.7074 (0.0131)
Business
1,628
0.5841 (0.0061)
0.5101 (0.0055)
0.6396 (0.0065)
Business Software
615
0.6250 (0.0124)
0.5737 (0.0115)
0.6678 (0.0199)
Commerce
224
0.8091 (0.0249)
0.8094 (0.0293)
0.9137 (0.0269)
Communication
695
0.7016 (0.0073)
0.6748 (0.0040)
0.8072 (0.0094)
Appendix
Table 17: Per-domain TACT (Full) results: Semantic . Values are seed means (sample standard deviations), using seeds 42, 43, and 44. N is the recorded task count. A dash denotes no reported score, including when N>0 , and is not zero. This view has 39/49 scored domains; ML denotes Machine Learning.
Context : held-out domain
N
MRR
NDCG@5
HR@3
Advertising
1,674
0.6202 (0.0193)
0.5858 (0.0202)
0.6808 (0.0247)
Artificial Intelligence / ML
685
0.6761 (0.0109)
0.6279 (0.0038)
0.7611 (0.0034)
Business
3,475
0.6159 (0.0017)
0.5450 (0.0028)
0.6718 (0.0012)
Business Software
1,643
0.6307 (0.0136)
0.5735 (0.0129)
0.6963 (0.0171)
Commerce
1,131
0.6615 (0.0145)
0.6143 (0.0167)
0.7586 (0.0248)
Communication
2,081
0.6157 (0.0042)
0.5519 (0.0039)
0.6761 (0.0046)
Appendix
Table 18: Per-domain TACT (Full) results: Context . Values are seed means (sample standard deviations), using seeds 42, 43, and 44. N is the recorded task count. A dash denotes no reported score, including when N>0 , and is not zero. This view has 49/49 scored domains; ML denotes Machine Learning.
Mixed : held-out domain
N
MRR
NDCG@5
HR@3
Advertising
344
0.5005 (0.0279)
0.4675 (0.0307)
0.5514 (0.0244)
Artificial Intelligence / ML
477
0.5930 (0.0192)
0.5656 (0.0118)
0.6953 (0.0087)
Business
2,295
0.5649 (0.0044)
0.4932 (0.0028)
0.6115 (0.0038)
Business Software
913
0.6063 (0.0118)
0.5555 (0.0125)
0.6484 (0.0162)
Commerce
667
0.6508 (0.0181)
0.6186 (0.0216)
0.7261 (0.0350)
Communication
1,316
0.5869 (0.0057)
0.5499 (0.0035)
0.6654 (0.0076)
Appendix
Table 19: Per-domain TACT (Full) results: Mixed . Values are seed means (sample standard deviations), using seeds 42, 43, and 44. N is the recorded task count. A dash denotes no reported score, including when N>0 , and is not zero. This view has 44/49 scored domains; ML denotes Machine Learning.
View
Target dataset
MRR
NDCG@5
HR@3
Natural
ToolBench
0.8975 (0.0055)
0.8648 (0.0045)
0.9326 (0.0020)
TaskBench
0.8387 (0.0097)
0.7366 (0.0129)
0.9160 (0.0095)
MetaTool
0.6839 (0.0032)
0.6981 (0.0024)
0.7429 (0.0054)
APIBench
0.6334 (0.0027)
0.6847 (0.0027)
0.7514 (0.0046)
Semantic
ToolBench
0.4167 (0.0033)
0.3817 (0.0025)
0.4859 (0.0033)
TaskBench
0.9170 (0.0075)
0.8107 (0.0042)
0.9700 (0.0035)
Appendix
Table 20: Per-dataset TACT (Full) results. Each entry is the mean (sample standard deviation) across seeds 42, 43, and 44. Higher is better.
Targets
View
Scored
Nscored
MRR
NDCG@5
HR@3
Domains
Natural
49/49
201,774
0.915159
0.872471
0.951539
Semantic
39/49
50,249
0.629185
0.603492
0.726141
Context
49/49
75,709
0.671839
0.624384
0.755076
Mixed
44/49
62,260
0.564361
0.528805
0.644375
Datasets
Natural
4/4
–
0.763375
0.746050
0.835725
Semantic
4/4
–
0.613150
0.580325
0.667875
Appendix
Table 21: Equal-target means calculated from the rounded per-target means. Six decimals retain the arithmetic of displayed inputs rather than imply six-decimal measurement precision. Missing scores are excluded under the evaluation eligibility rule. Nscored sums counts only over scored domain cells; dataset-level counts are omitted.
Artificial_Intelligence_Machine_Learning, Education, Food, Health_and_Fitness, Medical, Science, Search, Tools, and Visual_Recognition
30,450
Appendix
Table 22: Fixed ToolBench domain groups used for the supplementary leave-one-group-out evaluation. Each group is held out once, and the remaining four groups form the outer-source set. The five groups jointly cover all 49 ToolBench domains without overlap. Natural tasks denote Natural-view evaluation tasks in each group and sum to 201,774.
Protocol
Model
Natural
Semantic
Context
Mixed
MRR
N@5
H@3
MRR
N@5
H@3
MRR
N@5
H@3
MRR
N@5
H@3
Random
0.2841
0.1926
0.2735
0.1682
0.2134
0.2817
0.1745
0.2284
0.1836
0.2712
0.1947
0.1423
± 0.0021
± 0.0004
± 0.0036
± 0.0049
± 0.0068
± 0.0071
± 0.0030
± 0.0027
± 0.0025
± 0.0022
± 0.0014
± 0.0044
Semantic Cosine
0.8264
0.7513
0.8472
0.3416
0.1892
0.2438
0.1974
0.3125
0.2251
0.2985
0.2417
0.2766
± 0.0000
± 0.0000
± 0.0000
± 0.0000
± 0.0000
± 0.0000
± 0.0000
± 0.0000
± 0.0000
± 0.0000
± 0.0000
± 0.0000
Pointwise MLP
0.6428
0.7861
0.7315
0.3674
0.4912
0.4136
0.3487
0.4732
0.5794
0.4618
0.3105
0.3642
Appendix
Table 23: Supplementary ToolBench leave-one-group-out capability ranking. Each of the five groups is held out once and the remaining four are used as sources. Means appear above smaller sample standard deviations over seeds 42, 43, and 44; best and second-best displayed means are bold and underlined. The final row gives the absolute TACT gain over the strongest non-TACT baseline in percentage points (pp). N@5 and H@3 denote NDCG@5 and HR@3; Endpoint Schema denotes Endpoint Schema Match. Deterministic methods have zero seed variance.
Pre-evaluation filter stage
Remaining queries
Initial candidate query pool
5000
Annotated capability is present in the frozen 3,442-capability library
3078
At least one public GET / HEAD endpoint is available
1528
Required arguments can be instantiated from the public schema
745
At least one permitted endpoint has a valid MirrorAPI-Cache response
354
No authentication, write operation, or live external side effect is required
147
Appendix
Table 24: Pre-evaluation solvability filtering for cached execution. Counts are obtained from the frozen sampling manifest before any ranking policy is evaluated.
Full vs. Semantic Cosine
Full vs. Re-Invoke
Agent Backbone
Δ Completion
Bootstrap 95% CI
Exact p
Δ Completion
Bootstrap 95% CI
Exact p
GPT-5.6 Terra
+13.33 pp
[+5.83, +20.83]
0.0015
+8.33 pp
[+1.67, +15.83]
0.0414
Grok-4.5
+9.17 pp
[+2.50, +15.83]
0.0127
+8.33 pp
[+4.17, +13.33]
0.0020
Claude Sonnet 5
+9.17 pp
[+3.33, +15.83]
0.0074
+8.33 pp
[+2.50, +15.00]
0.0213
Gemini 3.1 Pro
+13.33 pp
[+5.83, +20.83]
0.0009
+9.17 pp
[+2.50, +15.83]
0.0192
Appendix
Table 25: Paired completion-rate comparisons of TACT (Full) against Semantic Cosine Top-5 and Re-Invoke Top-5. Intervals are bootstrap 95% confidence intervals, and p values are obtained from exact McNemar tests.
Large language model (LLM) agents must generalize from tools seen during training to unseen tools at deployment. A key challenge is tool trialing, i.e., excessive trials waste the interaction budget, whereas selective trials enable exploration of unfamiliar tools. Existing outcome-based post-training leaves wasteful trials unguided, while turn-level supervision may suppress necessary exploration. We introduce ToolCompass, a post-training framework that guides tool trialing by organizing tool-call representations according to shared functions. Specifically, ToolCompass models each function class as a von Mises--Fisher distribution and jointly reduces intra-function variation across domains and increases inter-function separation. This structure transfers experience from seen tools to functionally similar unseen tools, directing exploration away from unrelated alternatives. ToolCompass requires no ground-truth call traces or unseen-tool access and incurs no inference overhead. Experiments on AppWorld and FTRL show consistent gains across GRPO, RFT, and DMPO. improves AppWorld OOD task success by up to 10.71 percentage points over vanilla post-training and performs best among competitive baselines on both benchmarks.
Junlin Fang, Chong Zhang, Do Nguyen-Thanh +3
College of Computing and Data Science, Nanyang Technological University · Zhejiang University · Australian Artificial Intelligence Institute, University of Technology Sydney +1
Automated evaluation of tool-using large language model (LLM) agents is widely assumed to be reliable, but this assumption has rarely been validated against human annotation. We introduce AgentProp-Bench, a 2,000-task benchmark with 2,300 traces across four domains, nine production LLMs, and a 100-label human-validated subset. We quantify judge reliability, characterize error propagation, and evaluate a runtime mitigation. Substring-based judging agrees with human annotation at kappa=0.049 (chance-level); a three-LLM ensemble reaches kappa=0.432 (moderate) with a conservative bias. Under validated evaluation, a parameter-level injection propagates to a wrong final answer with human-calibrated probability approximately 0.62 (range 0.46-0.73 across models). Rejection (catching bad parameters) and recovery (correcting after acceptance) are independent model capabilities (Spearman rho=0.126, p=0.747). A tuned runtime interceptor reduces hallucination on GPT-4o-mini by 23.0 percentage points under a concurrent n=600 control, but shows no significant effect on Gemini-2.0-Flash, whose aggressive parameter rejection eliminates the target failure mode. All code, data, traces, and human labels are released at https://github.com/bhaskargurram-ai/agenthallu-bench.
As LLM agents operate over corpora of more than ten thousand tools, selecting the right tool for a complex multi-step user instruction becomes a central bottleneck. Tool descriptions cannot all fit in context, and relevance-based retrievers only rank tools without knowing which are actually needed. Recent work instead leverages proprietary commercial LLMs like Claude or ChatGPT to drive a retriever, decomposing the instruction and selecting over retrieved candidates. We show that a small, open model can be trained via reinforcement learning to perform the same functionality. On ToolBench, a Qwen3-8B model post-trained with our recipe outperforms Llama-4-Maverick (402B). In addition, the trained model transfers directly to different retrievers and generalizes to new tool corpora in different domains.