As LLM agents operate over corpora of more than ten thousand tools, selecting the right tool for a complex multi-step user instruction becomes a central bottleneck. Tool descriptions cannot all fit in context, and relevance-based retrievers only rank tools without knowing which are actually needed. Recent work instead leverages proprietary commercial LLMs like Claude or ChatGPT to drive a retriever, decomposing the instruction and selecting over retrieved candidates. We show that a small, open model can be trained via reinforcement learning to perform the same functionality. On ToolBench, a Qwen3-8B model post-trained with our recipe outperforms Llama-4-Maverick (402B). In addition, the trained model transfers directly to different retrievers and generalizes to new tool corpora in different domains.
Figures & tables
Figure 1: Illustration of the Plan – Retrieve – Select tool selection workflow. The same trained model decomposes the request and selects a tool for each substep. Each substep issues one query to an external retriever, which filters candidates by score and retains at most K for the Select stage.
Figure 2: Once trained, our model directly transfers to other tool corpora in new domains and is compatible with new retrievers, without additional model training.
Figure 3: Effect of training on ToolBench G3 (test split, 2,124 instructions). Success rate for Qwen3 at three scales, run through the same Plan – Retrieve – Select workflow untrained (Base), after SFT, and after SFT followed by RL (SFT+RL).
Method
Success Rate (%)
Llama-4-Maverick (prompted)
+ ReAct
28.4
+ hierarchical search
8.6
+ Search-R1 style
30.6
+ our workflow
10.1
Qwen3 (SFT+RL) + our workflow
Table 1: Tool-selection success rate on the ToolBench G3 test split (2,124 instructions) under exact match: an instruction succeeds if and only if the selected tool set equals the gold set. Llama-4-Maverick is prompted under four interaction protocols. Qwen3 models are trained with SFT followed by RL (§ 3 ). The best run is underlined .
Success Rate (%)
Method
ToolBench G2
Gorilla
CRAFT
TaskBench
Llama-4-Maverick (prompted)
+ ReAct
15.6
16.0
18.0
60.2
+ Search-R1 style
20.8
15.9
41.0
80.6
+ our workflow
4.4
9.2
47.9
36.9
Qwen3 (untrained) + our workflow
Table 2: Transferability results to new tool domains. The Qwen3 models are trained on the training split of ToolBench G3, the same checkpoints as in Table 1 . All numbers are success rates in percent (%). Column max underlined .
Figure 5: Planner
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Instr.
tools
Gold
τ
K
ToolBench G3 (test)
2,124
1,419
multi
3
15
ToolBench G2 (test)
7,238
11,474
multi
3
15
Gorilla (3 splits)
598
43–907 †
1
3
20
CRAFT (3 splits)
654
180–525 †
1
2
20
TaskBench (3 splits)
103
23–40 †
multi
0
15
Appendix
Table 3: Evaluation benchmarks. “Instr.” is the number of evaluated instructions, “tools” the size of the corpus retrieved over, “Gold” the number of gold tools per instruction, and τ and K the BM25 score threshold and candidate cap. About 9% of the G2 tools also appear in G3. † Range over the source splits, each retrieved over separately.
SFT
RL (GRPO)
Framework
LLaMA-Factory
VeRL + vLLM
Trainable parameters
full
full
Learning rate (AdamW)
10−5
10−6
LR schedule
cosine
constant
Epochs or steps
3 epochs ‡
300 steps
Effective batch size
128 sequences
64 prompts
Appendix
Table 4: Training hyperparameters. The same values are used at all three model scales. The SFT schedule has a 3% linear warm-up.
Table 5: Public locations and licenses of the resources used in this paper. Top: models and retrievers. Middle: benchmarks. Bottom: software. “Unspecified” means that no license is declared at the public location; these resources are used for research evaluation only.
Large language models (LLMs) excel at natural language processing but struggle to interact with external environments. Tool learning provides a promising way to extend LLMs into actionable agents, where tool selection is a critical prerequisite for successful tool use. Existing work often assumes a small or predefined set of tools, leaving large-scale tool selection underexplored. Real-world repositories contain a vast and diverse array of tools, making it difficult for LLMs to effectively search, distinguish, and compose tools under context-length constraints. We identify large-scale tool selection as a new challenge for agentic reinforcement learning, highlighting that existing RL methods for knowledge-based question answering are inadequate for selecting tools while considering compatibility. To address this challenge, we propose ToolSearcher, a novel RL framework for effective multi-turn search and fine-grained optimization in large-scale tool selection. Specifically, we introduce category-constrained tool discrimination to improve the model's ability to distinguish functionally similar tools, event-level search modeling to explicitly optimize the discovery of target tools during multi-turn search, and trajectory-aligned credit allocation to provide fine-grained reward signals for different stages of the search-selection process. Extensive experiments on large-scale tool selection benchmarks demonstrate that ToolSearcher consistently outperforms a set of strong baselines in challenging settings involving iterative search and complex tool composition.
Integrating external tools with Large Language Models (LLMs) has emerged as a promising paradigm for accomplishing complex tasks. Since LLMs still struggle to effectively manage large tool collections, researchers have begun exploring retrieval-based methods to pre-select the most relevant options, addressing input length and latency constraints. However, existing retrievers are often misaligned with tool-calling LLMs due to their separate training processes. This paper presents PORTS, a novel odds ratio preference optimization method for training retrievers aimed at tool selection. Using a perplexity-inspired preference signal from a frozen LLM, our approach fine-tunes a retriever to find helpful tools by optimizing the correlation between the selection probabilities and the downstream performances while jointly enforcing a contrastive semantic loss between documentation strings. The versatility of PORTS and its ability to significantly improve tool selection accuracy are demonstrated through extensive experiments on six datasets, two encoder models, and three LLMs with diverse prior knowledge. With low computational demands, our alignment process facilitates generalization to new queries and tools, proving valuable for practical applications with evolving toolsets.
Lorenzo Molfetta, Giacomo Frisoni, Nicolò Monaldini +1
Department of Computer Science and Engineering, University of Bologna
Tool-augmented language models are bounded by the APIs humans bothered to write; existing tool-creation systems patch this by prompting a frozen LLM at inference time, leaving the model that writes a tool decoupled from the one that uses it, with no signal that the schemas it produces are schemas it can invoke. We propose SMITH (Schema-grounded Multi-task Iterative Tool Honing), a reinforcement learning framework that jointly trains tool creation and tool use inside a single policy. Each rollout is either a build task (write a tool from a few examples) or a use task (invoke a pooled tool on a held-out question). Three separate reward axes catch schema, code, and outcome failures independently, so each failure mode contributes its own gradient. A 4B Qwen3 trained with SMITH on 13 procedural reasoning tasks with exact verifiers reaches 79.8 macro-average accuracy on held-out tasks, the best across all evaluated methods and ahead of an untrained 30B-A3B tool-writer. It also reaches 40.4 on TabMWP-Hard and 42.6 on out-of-domain GQA (+7.6 over the best same-backbone inference-time baseline), without any visual or tabular training data. Tools written by our 4B models also lifted the performance of LFM-2.5-350M and Qwen3-30B-A3B under same reasoning tasks.