Outcome-Aware Tool Selection for Semantic Routers: Latency-Constrained Learning Without LLM Inference

Huamin Chen; Xunzhuo Liu; Junchen Jiang; Bowei He; Xue Liu

cs.LGMar 13, 2026

Outcome-Aware Tool Selection for Semantic Routers: Latency-Constrained Learning Without LLM Inference

Authors: Huamin Chen, Xunzhuo Liu, Junchen Jiang, Bowei He, Xue Liu

Organizations: vLLM Semantic Router Project · Tensormesh Inc / UChicago · MBZUAI / McGill University

Abstract

Semantic routers in LLM inference gateways select tools in the critical request path, where every millisecond of added latency compounds across millions of requests. We propose Outcome-Aware Tool Selection (OATS), which interpolates tool embeddings toward the centroid of queries where they historically succeed -- an offline process that adds no parameters, latency, or GPU cost at serving time. On MetaTool (199~~tools, 4,287~~queries), this improves NDCG@5 from 0.869 to 0.940; on ToolBench (2,413~APIs), from 0.834 to 0.848. We also evaluate two learned extensions: a 2,625-parameter MLP re-ranker and a 197K-parameter contrastive adapter. The MLP re-ranker hurts or matches baseline when outcome data is sparse relative to the tool set; the contrastive adapter provides comparable gains on MetaTool (NDCG@5: 0.931). All methods are evaluated on the same held-out 30% test split. The practical takeaway is to start with the zero-cost refinement and add learned components only when data density warrants it. All mechanisms run within single-digit millisecond CPU budgets.

Figures & tables

Method	Latency	Parameters	GPU	Viable at
	(ms)		Required	10K rps?
BM25 (lexical)	$\sim$ 7	0	No	✓
Static embedding	$\sim$ 5	22M †	No ∗	✓
OATS-S1 (offline)	$\sim$ 4	22M †	No ∗	✓
OATS-S2 (+ MLP)	$\sim$ 5	22M + 2.6K	No ∗	✓
OATS-S3 (+ adapter)	$\sim$ 5	22M + 197K	No ∗	✓

Table 1: Cost of tool selection mechanisms. Latency is per-request p50 on 2,413 tools (ToolBench). All methods run on CPU.

Panel of Figure 1 — Figure 1: (a) LLM-based tool selection requires a GPU-bound orchestrator in the request path (500–2,000 ms). (b) OATS performs selection on CPU in the router (3–7 ms), with all learning offline. LLM serving latency is the same in both cases.

Panel of Figure 2 — Figure 2: The OATS pipeline. Top: the online serving path runs on CPU in 3–7 ms. Bottom: offline learning from outcome logs. The core mechanism (Stage 1, dashed arrow) refines tool embeddings in the Tool DB at zero serving cost. Stages 2 and 3 are ablation mechanisms that optionally update the Re-Rank and Embed components respectively.

Panel of Figure 3 — Figure 3: Geometry of OATS-S1 for the buildbetter example. The original embedding (blue circle) sits in a generic “SaaS” region, far from the test query (star). Positive training queries $Q^{+}$ (teal dots) cluster around “meeting transcripts”; negative queries $Q^{-}$ (red dots) cluster around “call management.” The refinement pulls the tool embedding toward $\bar{e}^{+}$ and away from $\bar{e}^{-}$ , placing the refined embedding (teal circle) closer to the test query. The description text never changes.

Method	$\Delta$ NDCG@5	AG/ms	$\Delta$ NDCG@5	AG/ms
	MetaTool		ToolBench
SE (baseline)	—	—	—	—
Emb. refinement (core)	+0.071	$\infty$	+0.014	$\infty$
+ Contrastive (ablation B)	+0.062	$\infty$	+0.008	$\infty$
SE + Lexical	-0.053	—	+0.020	$\infty$

Table 2: Cost efficiency: NDCG@5 gain per additional millisecond of p50 latency vs. SE baseline.

\infty

means accuracy improved with no latency increase.

Method	Accuracy	Latency	Hardware
LLM-based (from published benchmarks)
ChatGPT (GPT-3.5) †	69.1%	$\sim$ 1–3 s	API / GPU
Vicuna-7b †	73.5%	$\sim$ 2–5 s	GPU
Vicuna-13b †	58.2%	$\sim$ 3–8 s	GPU
LLaMA2-13b †	44.1%	$\sim$ 2–5 s	GPU
Average (9 LLMs) †	$\sim$ 57%	—	GPU

Table 3: Comparison with LLM-based tool selection on the MetaTool “similar choices” subtask (295 test queries). LLM results report Correct Selection Rate (CSR) from Huang et al. (2024) ; retrieval-based results report Recall@1 on the same subtask and split. Both metrics measure whether the top-selected tool is correct.

	ToolBench				MetaTool
Method	R@1	R@3	R@5	NDCG@5	R@1	R@3	R@5	NDCG@5
Random	0.238	0.656	0.829	0.692	0.096	0.308	0.492	0.298
BM25	0.392	0.772	0.921	0.853	0.397	0.617	0.753	0.595
SE	0.382	0.765	0.897	0.834	0.716	0.904	0.958	0.869
SE + Lexical	0.388	0.783	0.919	0.854	0.640	0.860	0.927	0.816
OATS-S1	0.381	0.793	0.915	0.848	0.830	0.964	0.986	0.940

Table 4: Selection performance on ToolBench (2,413 tools, 180 test queries) and MetaTool (199 tools, 1,287 test queries). Same 30% test split for all methods. Best in bold , second underlined .

	Serving	Added	ToolBench		MetaTool
	params	latency	NDCG@5	$\Delta$	NDCG@5	$\Delta$
SE (baseline)	0	—	0.834	–	0.869	–
+ Emb. refinement (core)	0	0 ms	0.848	+0.014	0.940	+0.071
+ MLP re-ranker (ablation A)	2.6K	$<$ 0.5 ms	0.823	-0.011	0.869	+0.000
+ Contrastive (ablation B)	197K	$<$ 0.1 ms	0.841	+0.007	0.931	+0.062

Table 5: Ablation: incremental cost and contribution of each OATS component. Same 30% test split. Embedding refinement adds zero cost; the MLP re-ranker degrades ToolBench due to data sparsity (

<

0.15 examples/tool).

Method	p50 (ms)	p99 (ms)	p50 (ms)	p99 (ms)	GPU
	MetaTool		ToolBench
SE (baseline)	3.66	6.47	4.59	9.83	No
OATS-S1	3.53	5.99	4.42	6.19	No
OATS-S2	3.92	6.91	5.04	16.38	No
OATS-S3	3.87	7.25	4.95	11.11	No

Table 6: Inference latency per request (CPU-only). All methods stay within single-digit millisecond p50.

Panel of Figure 4 — Figure 4: Stage 1 convergence over iterations. Left: MetaTool. Right: ToolBench.

Panel of Figure 5 — Figure 5: Selection performance across all methods. Left: MetaTool. Right: ToolBench.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Tool	Description
buildbetter ⋆	Chat with the knowledge of all your calls in BuildBetter (Zoom, GMeet, Webex). Start for free @ BuildBetter.ai
QuiverQuantitative	Access data on congressional stock trading, lobbying, insider trading, and proposed legislation.
MixerBox_WebSearch	Search and summarize the web with our customized search engine powered by Google Search API!
brandfetch	Retrieve company and brand data including logos, colors, fonts, and other brand information.
C3_Glide	Get live aviation data for pilots. Ask questions about METARs, TAFs, NOTAMs…
locator	Displaying the current coordinates of the ISS and the names of the current crew.

Appendix

Table 7: Candidate tools for the example query, with original descriptions from MetaTool.

Query (truncated)	Correct	SE top-1	SE	S1
“Show me diverse and amusing memes…”	lsongai	PolishTool	0.287	0.450
“Discount codes for booking hotels or flights?”	Discount	TripTool	0.585	0.691
“Meaning of arigatou in Japanese?”	speak	ResearchFinder	0.077	0.149

Appendix

Table 8: Additional OATS-S1 corrections on MetaTool test queries. “SE sim” and “S1 sim” are cosine similarities to the correct tool before and after refinement. “SE top-1” is the wrong tool selected by static embedding.

LLM Routing

Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.

See all 349 papers

Latency-Quality Routing for Functionally Equivalent Tools in LLM Agents
May 14, 2026Kexin Chu, Dawei Xiang, Wei ZhangTool-Augmented Language Model Agents LLM Tool Use
University of Connecticut
Lookahead-R: Budget-Aware Tool Retrieval via Execution-Centric Planning
Sep 20, 2026Zongze Wu, Yani Guo, Runnan LiLLM Tool Use Tool-Augmented Language Model Agents
Beijing University of Posts and Telecommunications Beijing, China
OrcaRouter: A Production-Oriented LLM Router with Hybrid Offline-Online Learning
May 29, 2026Zhenghua Bao, Fengya Tian, Chris Zhang +3Model Selection LLM Routing
Continuum AI

May 14, 2026J/K move · Enter open · S save

Latency-Quality Routing for Functionally Equivalent Tools in LLM Agents

Kexin Chu, Dawei Xiang, Wei Zhang

University of Connecticut

Tool-Augmented Language Model Agents LLM Tool Use LLM Agents Contextual Bandits LLM Agent Routing

Open paper AskPDF arXiv

Tool-augmented LLM agents increasingly access the same tool type through multiple functionally equivalent providers, such as web-search APIs, retrievers, or LLM backends exposed behind a shared interface. This creates a provider-routing problem under runtime load: the router must choose among providers that differ in latency, reliability, and answer quality, often without gold labels at deployment time. We introduce LQM-ContextRoute, a contextual bandit router for same-function tool providers. Its key design is latency-quality matching: instead of letting low latency offset poor answers in an additive reward, the router ranks providers by expected answer quality per service cycle. It combines this capacity-aware score with query-specific quality estimation and LLM-as-judge feedback, allowing it to adapt online to both load changes and provider-quality differences. On the main web-search load benchmark, LQM-ContextRoute improves F1 by +2.18 pp over SW-UCB while staying on the latency-quality frontier. In a high-heterogeneity StrategyQA setting, LQM-ContextRoute avoids additive-reward collapse and improves accuracy by up to +18 pp over SW-UCB; on heterogeneous retriever pools, it improves NDCG by +2.91--+3.22 pp over SW-UCB. These results show that same-function tool routing benefits from treating latency as service capacity, especially when runtime pressure and provider-quality heterogeneity coexist.

Abstract

LLM Routing

Explore similar work

Latency-Quality Routing for Functionally Equivalent Tools in LLM Agents

Lookahead-R: Budget-Aware Tool Retrieval via Execution-Centric Planning

OrcaRouter: A Production-Oriented LLM Router with Hybrid Offline-Online Learning