Tool Retrieval Bridge: Aligning Vague Instructions with Retriever Preferences via Bridge Model
Authors: Kunfeng Chen, Luyao Zhuang, Fei Liao, Juhua Liu, Jian Wang, Bo Du
Organizations: Department of Gastroenterology, Renmin Hospital, Wuhan University, Wuhan, China. · School of Computer Science, Wuhan University, Wuhan, China. · State Grid Corporation, China.
Tool learning has emerged as a promising paradigm for large language models (LLMs) to address real-world challenges. Due to the extensive and irregularly updated number of tools, tool retrieval for selecting the desired tool subset is essential. However, current tool retrieval methods are usually based on academic benchmarks containing overly detailed instructions (e.g., specific API names and parameters), while real-world instructions are more vague. Such a discrepancy would hinder the tool retrieval in real-world applications. In this paper, we first construct a new benchmark, VGToolBench, to simulate human vague instructions. Based on this, we conduct a series of preliminary analyses and find that vague instructions indeed damage the performance of tool retrieval. To this end, we propose a simple-yet-effective Tool Retrieval Bridge (TRB) approach to boost the performance of tool retrieval for vague instructions. The principle of TRB is to introduce a bridge model to rewrite the vague instructions into more specific ones and alleviate the gap between vague instructions and retriever preferences.We conduct extensive experiments under multiple commonly used retrieval settings, and the results show that TRB effectively mitigates the ambiguity of vague instructions while delivering consistent and substantial improvements across all baseline retrievers. For example, with the help of TRB, BM25 achieves a relative improvement of up to 111.51%, i.e., increasing the average NDCG score from 9.73 to 19.59. The source code and models are publicly available at https://github.com/kfchenhn/TRB.
Figures & tables
Figure 1: Instruction Comparison between ToolBench ( Qin et al., 2023 ) and our VGToolBench . As seen, ToolBench contains more detailed and specific instructions (highlighted in red ), while the instructions of our VGToolBench are vague and more aligned with real-world scenarios.
Given a user instruction and a list of related APIs, your task is to generate a fuzzier version of the user instruction by simplifying or replacing technical terms with synonyms, without changing the user’s core requirements. You can follow these steps:
- Analyze the user instruction: identify how many tasks the user has and what the specific needs are.
- Compare with the API list: match the user’s tasks with the relevant APIs. Remove any references to specific APIs or redundant technical details.
- Simplify technical terms: replace highly specialized or technical terms with more common, everyday language. The goal is to make the instruction sound like something a regular user would say in casual conversation.
- Rephrase the instruction: use simpler language or synonyms where appropriate, but ensure the core intent remains unchanged.
- Output the result: provide the final fuzzier version of the instruction.
Table 1: The prompt template for vague instruction generation
Dataset
I1
I2
I3
#Training
86,826
84,425
25,151
#Test
600
400
100
Table 2: Statistics of VGToolBench .
Figure 2: Retrieval Performance comparison of VGToolBench v.s. ToolBench. The x-axis denotes the different types of sub-sets in VGToolBench and ToolBench, and the y-axis denotes the tool retrieval performance, evaluated by the average of NDCG@5 and NDCG@10, where the evaluation details can be found in Section 5.1 . The numerical results represent the relative decrease compared to the results on Toolbench. We can observe that the performance declines significantly, e.g. , the performance drops on BM25 is up to 50.39 % .
Figure 3: Overview of our proposed TRB . (a) The pipeline of TRB , where the core is to introduces a bridge model to enhance the instruction into a more specific version. (b) The training scheme of the bridge model, which consists of a two-stage process: ❶ Supervised Fine-tuning , ❷ Reinforcement Learning . We first construct paired input–output instances by aligning data correspondences between VGToolBench and ToolBench, and train an initial model through supervised fine-tuning. Building on this initial model, we further apply reinforcement learning to align its preference by using the retrieval performance as reward signal.
Hyperparameter
Stage 1(SFT)
Stage 2 (RL)
epoch
3
3
batch_size
16
12
learning_rate
5e-5
2e-5
warmup_ratio
0.0
0.02
optimizer
Adam
Adam
max_sequence_length
1024
1024
Table 3: Training hyperparameters used in TRB .
Methods
Datasets
I1
I2
I3
NDCG@5
NDCG@10
Avg.
NDCG@5
NDCG@10
Avg.
NDCG@5
NDCG@10
Avg.
BM25
ToolBench
22.73
24.55
23.64
17.55
19.85
18.70
24.73
27.75
26.24
VGToolBench
12.49
13.50
13.00
8.81
9.73
9.27
13.94
15.70
14.82
% Δ↓
-45.05
-45.01
-45.03
-49.80
-50.98
-50.39
-43.63
-43.42
-43.53
+TRB
VGToolBench
23.32
25.29
24.31
19.06
20.11
19.59
18.54
21.97
20.26
% Δ↑
86.71
87.33
87.02
116.35
106.68
111.51
33.00
39.94
36.47
Table 4: Experimental results on ToolBench and VGToolBench . “Avg.” means the average performance of NDCG@5 and NDCG@10. " %Δ↓ " represents the relative performance degradation (%) of VGToolBench compared to the original ToolBench, while " %Δ↑ " indicates the relative performance improvement (%) of the four tool retrievers equipped with our proposed TRB on VGToolBench . Red results indicate that the retrievers’ performance on VGToolBench degrades compared to ToolBench, while green results denote that the retrievers equipped with TRB improves its performance on VGToolBench . We observe that TRB consistently outperforms the baselines in both single-tool and multi-tool situations scenarios.
Method
I1
I2
I3
N@5
N@10
N@5
N@10
N@5
N@10
BM25
12.49
13.50
8.81
9.73
13.94
15.70
-w/ SFT-only
18.18
19.98
14.13
15.43
16.14
18.86
-w/ Full
23.32
25.29
19.06
20.11
18.54
21.97
TF-IDF
31.50
34.19
23.60
25.54
17.64
22.35
-w/ SFT-only
38.80
42.62
29.82
32.43
25.01
31.38
Table 5: Impact of SFT and RL training stages of TRB . Notably, taking the BM25 as an example, “BM25” means the vanilla tool retriever that does not incorporate with the bridge model. “-w/ SFT-only” denotes that we only use the SFT training stage, and "-w/ Full" refers to the full training scheme.
Method
BM25
ToolRetriever
N@5
N@10
N@5
N@10
-w/o RL
16.14
18.86
56.23
65.07
-w/ PPO
16.24
19.48
56.87
65.95
-w/ GRPO
15.87
19.00
57.09
65.98
-w/ DPO
18.54
21.97
57.70
66.58
Table 6: Performance of various RL algorithms on TRB . We report tool retrieval scores with TRB under different RL training strategies. “-w/o RL” denotes training without reinforcement learning (only using the SFT), while “-w/ *” refers to using different RL algorithms to continue training the SFT model.
Method
Model
I3
Efficiency
N@5
N@10
w/o TRB
-
13.94
15.70
0.09s
TRB w/ BM25
Llama-3.2-1B
17.66
21.55
0.43s
Llama-3.2-3B
18.54
21.97
0.67s
Llama-3.1-8B
18.33
22.37
1.54s
w/o TRB
-
48.50
56.34
0.11s
Table 7: Performance comparison between different model scales of bridge model. We report the results of TRB with BM25 and ToolRetriever under VGToolBench (I3). The efficiency is measured by the time consumption to complete one instruction.
Figure 4: Effect of the iteration number T in iterative DPO. We show the retrieval performance (NDCG@5 and NDCG@10) of TRB with BM25 on VGToolBench ( I3 ) across different iterations. Both metrics display a noticeable upward trend followed by a subsequent downturn as T increases, indicating that moderate iterative refinement improves retrieval quality whereas excessive iterations may hinder performance.
Figure 5: Retrieval performance with state-of-the-art retrieval methods on VGToolBench (I3). Integrating TRB with the base ToolRetriever surpasses all stronger retrieval methods. Moreover, applying TRB to the hybrid, re-ranking, and ColBERT pipelines yields consistent further improvements.
Figure 6: Case study between the vague instruction in VGToolBench and the instruction enhanced via our TRB . In this case, we use the BM25 as the base tool retriever.
Task
Method
Settings
Overall
Simple
Multiple
Parallel
Parallel Multiple
Tool Retrieval
w/o TRB
46.68
63.54
29.52
53.57
48.33
w/ TRB
46.33
67.56
35.70
61.51
52.78
Δ
-0.35
+4.02
+6.18
+7.94
+4.45
Tool Calling
w/o TRB
51.16
57.64
37.50
50.00
49.08
w/ TRB
55.43
59.54
50.00
54.17
54.79
Table 8: Performance on the real-world BFCL-live benchmark ( Patil et al., 2025 ) . We evaluate TRB on both tool retrieval and calling tasks. It can be seen that TRB also works well in this real-world benchmark.