cs.AIOct 19, 2026
SaveRouterWorld: Scaling Conversational AI with Latency-Aware LLM Routing
Organizations: The University of Hong Kong · Amazon · Centre for Artificial Intelligence, University College London
Abstract
Conversational AI systems built on Large Language Models (LLMs) require routing mechanisms that balance answer quality and response latency at scale. However, existing LLM routing benchmarks assume full observability of all candidate model outcomes, under-emphasize latency as a first-class metric, and fragment evaluation across heterogeneous protocols misaligned with real-world settings. We introduce RouterWorld, a comprehensive benchmark for LLM routing in conversational GenAI. RouterWorld unifies 371k single-turn responses and 76k multi-turn agentic trajectories, adopts natural user interactions with a standardized LLM-as-a-judge protocol, and decouples efficiency from transient pricing via hardware-grounded latency annotations. Crucially, RouterWorld formalizes routing under partial observability through two setups: Cold-Start, which restricts learning to sparse, static logs, and Voyager, which models continuous data streams where routing decisions shape future training distributions. Evaluating 13 routing baselines, we find that most routers over-emphasize accuracy while ignoring latency under partial observability. Code and benchmark data are publicly available.
Figures & tables
Figure 1: Routing observability taxonomy. ( Left ) Fully-Observable routing, where feedback from all candidate models is available. ( Center ) Cold-Start Partially-Observable routing, where the router observes outcomes only for selected models. ( Right ) Voyager Partially-Observable routing, where routing decisions affect future data collection and training. refers to candidate model. denotes models not selected by the router, denotes incorrect outputs, while vivid colors indicate correct predictions. Color intensity reflects inference latency (darker indicates higher latency), and dark borders mark optimal models that are both correct and fastest.
| Benchmark | #Instances | #Models | #Datasets | #Base. | Part. Obs. | Stream. | Latency Ann. | Agentic Tasks | CF-Query |
| RouterBench Hu et al. (2024) | 405K | 11 | 8 | 4 | ✗ | ✗ | ✗ | ✗ | ✗ |
| RouteLLM Ong et al. (2025) | — | 2 | 3 | 4 | ✗ | ✗ | ✗ | ✗ | ✗ |
| LLMRouterBench Li et al. (2026) | 400K | 33 | 21 | 10 | ✗ | ✗ | ✗ | ✗ | ✗ |
| RouterEval Huang et al. (2025) | 200M | 8,500+ | 12 | 5 | ✗ | ✗ | ✗ | ✗ | ✗ |
| RouterArena Lu et al. (2026) | 8.4K | 42 | 23 | 14 | ✗ | ✗ | ✗ | ✗ | ✗ |
| RouterWorld-Core | 53K | 7 | 12 | 17 | ✓ | ✗ | ✓ | ✗ | ✓ |
Table 1: Comparison of LLM routing benchmarks. “Part. Obs.” = partially observable setting; “Stream.” = streaming rerouting; “Latency Ann.” = per-instance latency annotations; “#Base.” = number of routing baselines evaluated; “CF-Query” = user query normalization as router input.
Figure 2: Task category distribution in Router-Core and Router-Agentic .
| Statistic | Core | Agentic |
|---|---|---|
| Total Instances | 48,232 | 12,740 |
| # Training | 40,265 | 11,292 |
| # Testing | 7,967 | 1,448 |
| # Task Categories | 7 | 3 |
| # Datasets | 12 | 3 |
| # Models | 7 | 6 |
Table 2: Data characteristics of RouterWorld .
Figure 3: Performance compression in agentic routing. Each dot is a non-collapsed routing baseline. Under both 3-model and 6-model pools, methods cluster tightly with low variance across all metrics. Boxes show interquartile range; values shown below.
| Fully-Observable | Partially-Observable | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Method | Base Model | Acc@1 (%) | Acc@1-L (%) | DCG | L-DCG | Acc@1 (%) | Acc@1-L (%) | DCG | L-DCG |
| Naive Baselines | |||||||||
| Oracle (Perfect) | – | 100.00 | 100.00 | 1.0000 | 1.0000 | 100.00 | 100.00 | 1.0000 | 1.0000 |
| Oracle (Best-of-8) | – | 80.04 | 67.97 | 0.7521 | 0.7140 | 96.81 | 54.51 | 0.7901 | 0.6462 |
| Random | – | 47.63 | 12.80 | 0.2995 | 0.1784 | 47.63 | 12.80 | 0.2995 | 0.1784 |
| Route-To-Largest | – | 89.11 | 9.10 | 0.4731 | 0.1494 | 89.11 | 9.10 | 0.4731 | 0.1494 |
Table 3: Performance on RouterWorld -Core under fully- and partially-observable settings. Oracle (Perfect) and Oracle (Best-of-8) denote theoretical and empirical upper bounds, respectively. Orange cells indicate collapsed routing that always selects a single model and are excluded from best/second-best comparisons. All pairwise improvements of the best-performing method over the second-best baseline are statistically significant ( ).
Appendix figures & tables39 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Routing method performance across distillation policies in the partially-observable setting. All methods exhibit a high correlation with the distillation policy trend
Figure 5: Voyager round-by-round L-DCG.
| Model | Pool | Params | TTFT (ms) | ITL (ms) | Total (s) |
|---|---|---|---|---|---|
| Gemma-2B | Core | 2B | 12.64 | 3.96 | 3.97 |
| CodeGemma-7B | Core | 7B | 14.22 | 5.20 | 5.21 |
| Ministral-14B | Agentic | 14B | 18.26 | 3.50 | 3.51 |
| Mistral-7B | Core | 7B | 100.21 | 4.13 | 4.23 |
| DS-Math-7B | Core | 7B | 117.16 | 4.36 | 4.48 |
| Devstral-123B | Agentic | 123B | 493.30 | 14.56 | 15.04 |
Table 4: Unified latency profile of all candidate models, measured with GenAI-Perf (2,000 input tokens, 1,000 output tokens, concurrency 10, streaming). Models sorted by total latency.
Figure 6: Per-benchmark model correctness for RouterWorld -Core. Models exhibit distinct specialization patterns across task types, with smaller models often outperforming generalists in specific domains.
Figure 9: Fastest correct model distribution for RouterWorld -Core (train and test). Small models are the optimal choice for a large share of queries.
Figure 10: Per-benchmark model correctness for RouterWorld -Agentic. Performance varies substantially across the three agentic task families.
Figure 11: Records per benchmark in RouterWorld -Agentic across three agentic task families.
Figure 12: Median episode latency per model in RouterWorld -Agentic. Latency ranges from 0.8s to 15.7s across models.
Figure 13: Fastest correct model distribution for RouterWorld -Agentic (train and test).
Figure 14: Screenshot of the web-based annotation tool developed for the LLM judge validation study. The interface displays the source dataset and model (colored tags), the user query, the full prompt, the ground-truth answer, and the model response. Annotators label each triple as correct or incorrect by comparing the model response against the ground truth. The tool and annotation guidelines will be released as open source.
| Metric | Value |
|---|---|
| Fleiss’ | 0.835 |
| Krippendorff’s | 0.867 |
| Avg. pairwise | 0.892 |
| Avg. pairwise agreement | 0.950 |
Table 5: Inter-annotator agreement among 6 experts on 150 blind annotations.
| Comparison | Agreement | Precision / Recall | F1 | |
|---|---|---|---|---|
| Consensus vs. LLM Judge | 0.920 | 0.835 | 0.98 / 0.89 | 0.932 |
Table 6: Majority-vote consensus (6 annotators) vs. LLM judge ( ).
| Annotator | Agreement | ||
|---|---|---|---|
| Annotator-S | 87 | 94.3% | 0.880 |
| Annotator-A | 86 | 94.2% | 0.871 |
| Annotator-D | 88 | 96.6% | 0.929 |
| Annotator-H | 100 | 89.0% | 0.772 |
| Annotator-P | 100 | 87.0% | 0.730 |
| Annotator-W | 97 | 93.8% | 0.874 |
Table 7: Individual annotator agreement with the LLM judge on the 150-sample set.
| Model | TTFT (ms) | ITL (ms) | Throughput (tok/s) | Med. Episode Lat. (s) |
|---|---|---|---|---|
| Ministral-14B | 18 | 3.5 | 2,764 | 0.6 |
| Qwen3-Coder-30B | 532 | 7.2 | 1,126 | 5.5 |
| Devstral-123B | 493 | 14.6 | 574 | 4.9 |
| GPT-OSS-20B | 1,137 | 4.1 | 987 | 12.7 |
| GPT-OSS-120B | 1,213 | 6.0 | 1,169 | 11.1 |
| Qwen3-235B | 2,258 | 18.8 | 370 | 16.4 |
Table 8: Agentic candidate model pool. TTFT = time to first token; ITL = inter-token latency. Episode latency is the median total time across all tasks.
Figure 15: Median interaction steps per model by benchmark. Models with stronger planning capabilities (e.g., Qwen3-235B, Devstral-123B) resolve tasks in fewer steps, particularly on ALFWorld and WebShop.
Figure 16: Median episode latency per model. The conventional assumption that larger models are slower does not hold in agentic settings. Planning efficiency can offset higher per-step inference cost.
| Fully-Observable | Partially-Observable | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Method | Base Model | Acc@1 (%) | Acc@1-L (%) | DCG | L-DCG | Acc@1 (%) | Acc@1-L (%) | DCG | L-DCG |
| Naive Baselines | |||||||||
| Oracle (Perfect) | – | 100.00 | 100.00 | 1.0000 | 1.0000 | 100.00 | 100.00 | 1.0000 | 1.0000 |
| Oracle (Best-of-N) | – | 82.67 | 70.65 | 0.7791 | 0.7521 | 94.89 | 77.35 | 0.8781 | 0.8313 |
| Random | – | 41.64 | 16.99 | 0.3010 | 0.2211 | 41.64 | 16.99 | 0.3010 | 0.2211 |
| Route-To-Devstral | – | 44.41 | 21.20 | 0.3519 | 0.2674 | 44.41 | 21.20 | 0.3519 | 0.2674 |
Table 9: Routing performance on RW-Agentic (6-model pool) under fully- and partially-observable settings. Bold = best; underline = second best.
| Fully-Observable | Partially-Observable | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Method | Base Model | Acc@1 (%) | Acc@1-L (%) | DCG | L-DCG | Acc@1 (%) | Acc@1-L (%) | DCG | L-DCG |
| Naive Baselines | |||||||||
| Oracle (Perfect) | – | 100.00 | 100.00 | 1.0000 | 1.0000 | 100.00 | 100.00 | 1.0000 | 1.0000 |
| Oracle (Best-of-8) | – | 86.81 | 73.46 | 0.8176 | 0.7931 | 88.16 | 75.19 | 0.8325 | 0.8051 |
| Random | – | 41.89 | 26.38 | 0.3589 | 0.3161 | 41.89 | 26.38 | 0.3589 | 0.3161 |
| Route-To-Qwen | – | 67.95 | 38.92 | 0.5706 | 0.4971 | 67.95 | 38.92 | 0.5706 | 0.4971 |
Table 10: Routing performance on RW-Agentic under fully- and partially-observable settings. Oracle (Perfect) and Oracle (Best-of- ) denote theoretical and empirical upper bounds, respectively. Orange cells indicate collapsed routing (always selects a single model) and are excluded from best/second-best comparisons. Bold = best; underline = second best among non-collapsed methods.
| Fully-Obs | Partially-Obs | |||||
|---|---|---|---|---|---|---|
| 0.6B | 8B | 0.6B | 8B | |||
| KNN | .397 | .405 | +0.8 | .222 | .223 | +0.2 |
| MLP | .430 | .439 | +1.0 | .216 | .259 | +4.3 |
| BaRP | .298 | .162 | -13.6 | .157 | .200 | +4.4 |
| Avenger | .371 | .379 | +0.8 | .149 | .149 | +0.0 |
Table 11: L-DCG: 0.6B vs. 8B encoder ( in pp).
Figure 17: Rollout prompt for mathematical reasoning benchmarks.
Figure 18: Rollout prompt for multiple-choice benchmarks.
Figure 19: Rollout prompt for code generation benchmarks.
Figure 20: Rollout prompt for BBH logical reasoning (3-shot with chain-of-thought).
Figure 21: Rollout prompt for Knights & Knaves deductive reasoning.
Figure 22: Rollout prompt for Winogrande commonsense reasoning.
Figure 23: ALFWorld agent rollout prompt with ReAct few-shot demonstrations.
Figure 24: Rollout prompt for emotion recognition (shared across EmoryNLP, MELD, DailyDialog).
Figure 25: Rollout prompt for open-ended factual QA.
Figure 26: WorkArena web agent system prompt for ServiceNow knowledge-work tasks.
Figure 27: LLM-as-Judge prompt for code generation tasks.
Figure 28: LLM-as-Judge prompt for open-ended QA tasks.
Figure 29: LLM-as-Judge prompt for multiple-choice tasks.
Figure 30: LLM-as-Judge prompt for SQL generation tasks (relaxed variant).
Figure 31: LLM-as-Judge prompt for tool use / function calling tasks.
Figure 32: LLM-as-Judge prompt for personal chat tasks.
Figure 33: User query annotation prompt. Converts formal benchmark descriptions into natural conversational-form queries for the routing dataset.
Figure 34: Generative router prompt used for SFT training and inference. The model zoo section is populated dynamically with measured latency statistics.
Figure 35: RAG-augmented router prompt. Retrieved similar queries from the training set are included as few-shot examples to improve routing accuracy.
More in this topic
LLM Routing
Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.
See all 355 papers