Organizations: Institute of Information Science and Technologies, CNR, Italy · University of Pisa, Department of Computer Science, Italy · Institute of Informatics and Telematics, CNR, Italy
Large language models (LLMs) increasingly support Web-based services that must balance response quality and inference cost across heterogeneous requests. Conventional routing selects one model per prompt, while fixed ensembles cannot adapt their composition and cost to individual requests. We introduce PISA, a prompt-adaptive, cost-aware single-model router, and PISA-E, its extension to ensemble routing. At the core of both methods, PISA combines soft cluster memberships with model-specific predictors to estimate individual model capabilities. Building on these estimates, PISA-E jointly selects ensemble composition and cardinality under an explicit cost--quality preference that can be adjusted at inference time without retraining. A tractable ensemble-success surrogate and efficient candidate selection enable this extension without ensemble-specific training. Across four LLM routing benchmarks under in-distribution (ID) and out-of-distribution (OOD) conditions, PISA achieves the highest macro-average cost--performance hypervolume among the evaluated single-model routers in both settings. Meanwhile, PISA-E achieves the highest macro-average task performance among the evaluated ensemble approaches when prioritizing response quality, surpassing the strongest competitors by 2.09 and 2.87 percentage points under ID and OOD conditions, respectively. At selected lower-cost operating points, it also exceeds the strongest ensemble competitors' macro-average task performance while reducing average inference cost by approximately 36% and 25%, respectively. These results demonstrate the effectiveness of prompt-adaptive capability estimation for both single-model routing and selective ensembling.
Figures & tables
Figure 1. Inference-time overview of PISA and PISA-E. PISA maps a prompt embedding to soft memberships over X-means centroids, estimates each model’s success probability, and selects the model with highest cost-aware utility. PISA-E reuses the capability estimates to rank models by utility using its ensemble cost normalization, and evaluates odd-cardinality ranking prefixes using a strict-majority success surrogate and normalized ensemble cost. In the illustrated five-model example, the candidates are E1 , E3 , and E5 . The selected ensemble is queried in parallel, and self-weighted BERTScore selects the final response using the estimated model capabilities.
Table 1. Normalized hypervolume (HV) of single-model routers across ID and OOD benchmarks. HV is multiplied by 100 and uses the p40 full-pool cost reference. Parentheses report sample standard deviations across five seeds.
Approach
Track-2
RouterBench
FusionBench
SPROUT
Macro average
Perf. (%)
Cost
Perf. (%)
Cost
Perf. (%)
Cost
Perf. (%)
Cost
Perf. (%)
Cost
Oracle
85.93
213
92.15
13.7
91.01
4.48
98.04
62.2
91.78
73.3
Prior ensemble approaches
PairRanker-PT
60.87
946
67.71
87
51.03
8.05
68.54
132
62.04
293
PairRanker-FT
62.18
1200
77.62
87
52.04
8.05
75.75
162
66.90
365
MBR
60.75
1040
70.22
87
62.40
45.6
71.31
139
66.17
329
Table 2. ID test task performance (%) and inference cost (USD/query ×104 ) across four benchmarks and their macro average. Parentheses report sample standard deviations across five seeds where available.
Approach
Track-2
RouterBench
FusionBench
SPROUT
Macro average
Perf. (%)
Cost
Perf. (%)
Cost
Perf. (%)
Cost
Perf. (%)
Cost
Perf. (%)
Cost
Oracle
77.09
303
87.17
12.3
88.61
3.97
96.27
50.1
87.29
92.3
Prior ensemble approaches
PairRanker-PT
56.94
1920
47.94
82.6
27.42
8.73
40.83
122
43.28
534
PairRanker-FT
50.97
1940
66.57
82.6
26.81
11.7
65.27
122
52.40
539
MBR
52.96
1840
50.58
82.6
29.28
41.5
47.56
135
45.10
524
Table 3. OOD test task performance (%) and inference cost (USD/query ×104 ) on held-out task groups across four benchmarks and their macro average. Parentheses report sample standard deviations across five seeds where available.
Split
α
Models/query
Singleton (%)
Perf. (%)
Cost (USD/query ×104 )
ID
0.3
1.32 (0.05)
84.33 (2.46)
70.75 (0.18)
13 (0.23)
ID
0.6
2.00 (0.04)
55.70 (2.02)
72.07 (0.42)
50.8 (3.1)
ID
1.0
6.42 (0.15)
21.56 (1.39)
75.27 (0.27)
280 (8.4)
OOD
0.3
1.35 (0.01)
82.41 (0.62)
50.54 (0.09)
8.01 (0.25)
OOD
0.6
2.41 (0.02)
37.44 (1.23)
55.33 (0.15)
30.1 (0.42)
OOD
1.0
7.39 (0.15)
9.31 (0.09)
63.24 (0.15)
854 (6.8)
Table 4. Ensemble size, performance, and cost of PISA-E versus α . Singleton (%) denotes queries using one model. “Both” averages all eight scenarios; parentheses give sample standard deviations over five seeds.
Scenario
PISA-E
Fixed k=1
Fixed k=3
Fixed k=5
Fixed k=7
Fixed k=9
Fixed k=11
ID
Track-2
28.57 (0.59)
27.89 (0.49)
27.80 (0.53)
26.69 (0.26)
24.06 (0.38)
19.90 (0.41)
14.55 (0.40)
RouterBench
46.37 (0.31)
46.99 (0.25)
44.71 (0.30)
41.66 (0.38)
38.75 (0.37)
28.95 (0.21)
2.67 (0.02)
FusionBench
44.89 (0.87)
42.74 (0.98)
44.72 (0.75)
44.42 (0.66)
43.74 (0.64)
42.90 (0.89)
41.60 (0.60)
SPROUT
37.57 (0.67)
38.65 (0.49)
36.13 (1.13)
35.86 (0.89)
34.27 (0.92)
31.83 (0.45)
27.61 (0.60)
ID macro average
39.35 (0.44)
39.07 (0.37)
38.34 (0.52)
37.16 (0.40)
35.21 (0.42)
30.89 (0.36)
21.61 (0.29)
Table 5. Normalized HV ( ×100 ) of adaptive and fixed-cardinality ensemble routing under ID and OOD conditions, using the p70 cost reference. Parentheses report sample standard deviations across five seeds.
Selector
Resolver
ID
OOD
Both
PB
Self-weighted BERTScore
75.27 (0.27)
63.24 (0.15)
69.25 (0.17)
PB
Other-weighted BERTScore
73.04 (0.26)
59.88 (0.26)
66.46 (0.21)
PB
BERTScore
74.35 (0.19)
62.03 (0.15)
68.19 (0.06)
PB
PairRanker (pretrained)
68.00 (0.48)
48.68 (0.17)
58.34 (0.23)
PB
PairRanker (fine-tuned)
71.65 (0.31)
52.07 (0.24)
61.86 (0.19)
GC
Self-weighted BERTScore
75.23 (0.13)
62.36 (0.22)
68.80 (0.13)
Table 6. Task performance (%) at α=1 for different ensemble-success estimators and response resolvers. The ensemble-success estimators are Poisson-binomial (PB) and Gaussian copula (GC). ID and OOD results are macro-averaged across the four benchmarks, while “Both” averages all eight scenarios. Parentheses report sample standard deviations across five seeds.
Scenario
PISA-E
Fixed global + PB
ID
Track-2
28.57 (0.59)
22.86
RouterBench
46.37 (0.31)
39.12
FusionBench
44.89 (0.87)
41.26
SPROUT
37.57 (0.67)
34.31
ID macro average
39.35 (0.44)
34.39
Table 7. Effect of prompt-dependent model ranking on normalized hypervolume (HV, ×100 ) under ID and OOD conditions, using the p70 cost reference. PISA-E is compared with a variant that ranks models globally by their task performance on the fit set while retaining Poisson-binomial (PB) ensemble-success estimation.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Prompts
Candidate models
LLMRouterBench Track-2
12,446
13
RouterBench
34,684
11
FusionBench
6,398
40
SPROUT
44,241
13
Appendix
Table 8. Number of prompts and candidate LLMs in each evaluation benchmark.
Table 9. Task-group partitions used for OOD evaluation. The final column reports the number and percentage of held-out prompts.
Figure 2. Sensitivity of PISA’s macro-average HV advantage over competing routers to the performance reference (left) and full-pool cost reference (right). Positive values favor PISA.
Figure 3. Sensitivity of PISA-E’s macro-average HV advantage over fixed-cardinality variants to the performance reference (left) and full-pool cost reference (right). Positive values favor PISA-E.
Scenario
kmax=1
kmax=3
kmax=5
kmax=7
kmax=9
kmax=11
ID
Track-2
27.89 (0.49)
28.26 (0.64)
28.60 (0.60)
28.58 (0.54)
28.51 (0.58)
28.57 (0.59)
RouterBench
46.99 (0.25)
46.38 (0.26)
46.25 (0.33)
46.31 (0.29)
46.38 (0.28)
46.37 (0.31)
FusionBench
42.74 (0.98)
44.40 (0.69)
44.68 (0.69)
44.83 (0.72)
44.90 (1.04)
44.89 (0.87)
SPROUT
38.65 (0.49)
37.67 (0.82)
38.03 (1.21)
37.91 (1.05)
37.60 (0.76)
37.57 (0.67)
ID average
39.07 (0.37)
39.18 (0.48)
39.39 (0.50)
39.41 (0.48)
39.35 (0.46)
39.35 (0.44)
Appendix
Table 10. Normalized HV ( ×100 ) of PISA-E for different maximum ensemble cardinalities under ID and OOD conditions, using the p70 cost reference. Parentheses report sample standard deviations across five seeds.
Figure 4. Cost–performance trade-offs of greedy candidate selection and exhaustive surrogate optimization on Track-2 and RouterBench, with kmax=7 .
Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.
No single large language model (LLM) is optimal across all queries and budget constraints, making model routing essential for cost-effective deployment. Existing routers adopt diverse formulations and implementations, making fair comparison and extension difficult. We present a unified formulation of LLM routing as a sequential decision process characterized by five components: context encoders, model encoders, scoring functions, decision rules, and learning signals, covering single-turn, multi-turn, and personalized routing. Based on this formulation, we develop an automated pipeline for constructing routing supervision and evaluating routers jointly on response quality and inference cost. The resulting benchmark, xRouteBench, spans generic LLM, memory-augmented, vision, time-series, and personalized routing tasks. We further introduce LLMRouter, an open-source modular infrastructure with more than 16 representative routers. Our empirical study shows that learned routers outperform the strongest fixed-model baseline by 14.6% relatively, lightweight routers become more competitive under tight cost constraints, and user-conditioned routing consistently improves personalization.
Tao Feng, Fangxu Yu, Haozhen Zhang +9
University of Illinois Urbana-Champaign · University of Maryland, College Park · 3Nanyang Technological University +2
Dynamic LLM routers promise to cut inference costs by sending each query to the cheapest model that can answer it correctly. We analyze six commercial routers across 14 settings on a diverse benchmark spanning eight task categories, finding that none of them outperforms a router that randomly selects between two well-chosen models at matched cost. Some underperform by more than 10 percentage points. We trace this gap to four patterns prevalent across routers: difficulty blindness, length reversal, semantic matching, and roster suboptimality. We show that the first three are what the standard objective rewards: cost-accuracy Pareto efficiency on realized costs favors escalating moderately hard queries over the hardest ones, shorter queries over longer ones, and routing by a query's source over its difficulty. We also argue that the two assumptions that would justify large rosters, model granularity and model specialization, do not hold empirically. We propose an alternative evaluation methodology that does not reward these patterns, and as a proof of concept, we design a simple two-model router that avoids all four. Nevertheless, its gain over random routing is limited, because a well-chosen roster leaves little to route.
LLM routing aims to reduce inference costs by assigning easy queries to less expensive models and reserving stronger models for cases in which they are needed. However, we find that existing routers often violate this principle under loose cost budgets: as the budget increases, they increasingly select the strongest and most expensive model even when less expensive feasible candidates provide comparable or identical utility. We refer to this behavior as strong-model over-selection under loose budgets. Our analysis shows that ambiguity alone does not explain this phenomenon. Substantial overpayment can arise when differently priced models offer equivalent or near-equivalent utility and the router retains a persistent model-level preference for costly models; budget relaxation exposes this preference by making premium candidates feasible. Motivated by this diagnosis, we propose EquiRouter, a decision-aligned router that learns query-dependent model rankings, preserves meaningful utility differences, and explicitly prefers lower-cost models among equivalent candidates through a Cost-Aware Ranking Objective and a lightweight Query--Model Interaction Representation. Experiments across diverse text, multimodal, continuous-utility, and large-model-pool settings show that EquiRouter consistently improves cost efficiency while maintaining competitive peak and whole-curve performance. Code is available at https://github.com/LAMDA-Model-Reuse/EquiRouter.
Guannan Lai, Long Chen, Han-Jia Ye
School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China.