Large language models (LLMs) are increasingly accessed as remotely hosted services by edge and enterprise clients that cannot run frontier models locally. Since models vary widely in capability and price, routing queries to models that balance quality and inference cost is essential. Existing router approaches assume access to centralized query-model evaluation data. However, these data are often fragmented across clients, such as end users and organizations, and are privacy-sensitive, which makes centralizing data infeasible. Additionally, per-client router training is ineffective since local evaluation data is limited and covers only a restricted query distribution and a biased subset of model evaluations. We introduce the first federated framework for LLM routing, enabling clients to learn a shared routing policy from local offline query-model evaluation data. Our framework supports both parametric multilayer perceptron router and nonparametric K-means router under heterogeneous client query distributions and non-uniform model coverage. Across three benchmarks, federated collaboration improves the accuracy-cost frontier over client-local routers, both via increased effective model coverage and better query generalization. Our theoretical results also validate that federated training reduces routing suboptimality.
Figures & tables
Figure 1: Federated LLM Router System Description. (i) Clients possess queries from different tasks, and each query is evaluated only on some LMs (in our experiments, we consider the extreme case of one LM per query, where the choice of LM is client-specific and can be highly non-uniform). (ii) Federated Learning setup where clients train their local router on local data, and central server aggregates these local models and returns global model to clients. (iii) Functioning of router, where modelwise accuracy and cost is calculated for the given query, which is routed to the most suitable model that maximizes the objective value.
Figure 2: Federated vs. client-local routers on the global test distribution. Accuracy–cost curves for MLP-Router and K-Means-Router by sweeping λ . AUC scores are shown in parentheses.
Figure 3: Federated vs. client-local routers evaluated on local test sets. Representative clients for MLP-Router and K-Means-Router are shown. Federated training improves the in-distribution accuracy–cost frontier via better effective model coverage.
Figure 5
Figure 6: Adaptive personalization under extreme heterogeneity. Evaluated on local test sets of clients with federated training, isolated local training, and the proposed adaptive mixture.
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: t-SNE visualization of query distributions. We run t-SNE once on the full dataset and reuse the same 2D embedding for all panels. The grid highlights each client’s local subset, illustrating (subsampled) the resulting heterogeneity across clients.
Figure 8: t-SNE visualization of query distributions for ProxRouter-Data .
Figure 9: Client-specific model heterogeneity for RouterBench-Data . Bubble plot of the per-model proportions in each client local dataset Di under the Dirichlet assignment scheme ( α=0.45 ).
MLP-Router
K-Means-Router
N
Federated
Client-local
Federated
Client-local
5
0.747±0.004
0.679±0.010
0.733±0.007
0.656±0.023
10
0.750±0.005
0.679±0.008
0.742±0.007
0.646±0.018
20
0.750±0.003
0.661±0.007
0.745±0.003
0.601±0.011
40
0.749±0.009
0.652±0.003
0.749±0.005
0.584±0.008
Appendix
Table 1: Sensitivity to the number of clients on RouterBench-Data . Global-test normalized AUCs are reported as mean ± standard error across seeds; client-local results are averaged over clients.
MLP-Router
K-Means-Router
α
Federated
Client-local
Federated
Client-local
0.03
0.742±0.007
0.640±0.015
0.731±0.007
0.601±0.033
0.10
0.743±0.012
0.666±0.010
0.742±0.005
0.614±0.014
0.30
0.744±0.012
0.669±0.010
0.734±0.015
0.619±0.014
0.60
0.746±0.015
0.672±0.011
0.741±0.008
0.650±0.018
5.00
0.748±0.012
0.669±0.013
0.749±0.004
0.625±0.015
Appendix
Table 2: Sensitivity to query heterogeneity on RouterBench-Data . Global-test normalized AUCs are reported as mean ± standard error across seeds; client-local results are averaged over clients.
Figure 10: All-clients local test results ( MLP-Router ). Each panel corresponds to one client and reports accuracy–cost trade-offs for the federated router versus the client-local (no-FL) router, evaluated on that client’s local test set.
Figure 11: All-clients local test results ( K-Means-Router ). Each panel corresponds to one client and reports accuracy–cost trade-offs for the federated router versus the client-local (no-FL) router, evaluated on that client’s local test set.
Figure 12: Adapting to new clients. Routers are trained initially by 7 clients, then adapted by 3 new clients, without additional participation from the original clients. MLP-Router uses continued training with a distillation regularizer to retain the base policy; K-Means-Router updates cluster-level statistics via weighted averaging. We report accuracy–cost frontiers on the global test set.
Figure 13: Per-client accuracy–cost frontiers under extreme heterogeneity ( α=0.03 ) for MLP-Router .
Figure 14: Per-client accuracy–cost frontiers under extreme heterogeneity ( α=0.03 ) for K-Means-Router .
Sentence Encoder (Emb. Dim.)
RouterBench-Data
ProxRouter-Data
K-Means-Router
MLP-Router
K-Means-Router
MLP-Router
all_mpnet_base_v2 (768)
0.747
0.759
0.759
0.760
all_minilm_l6_v2 (384)
0.737
0.761
0.728
0.751
paraphrase_albert_
0.746
0.758
0.751
0.755
small_v2 (768)
Appendix
Table 3: Performance (through normalized AUC) of K-Means-Router and MLP-Router in centralized training regime across different choices of sentence encoders used for representing queries, for both RouterBench-Data and ProxRouter-Data .
Figure 15: ( ProxRouter-Data ) Federated vs. client-local (no-FL) routers on the global test distribution. On ProxRouter-Data , we evaluate federated model and locally trained models on the global test set as in Figure 2 .
Figure 16: ( ProxRouter-Data - Subsampled Clients) Local test set results of subsampled clients. We evaluate the federated and local models on ProxRouter-Data , as in Figure 3 .
Figure 17: ( ProxRouter-Data ) Local test set results of all clients with MLP-Router . We evaluate the federated and local models on ProxRouter-Data , as Figure 10 .
Figure 18: ( ProxRouter-Data ) Local test set results of all clients with K-Means-Router . We evaluate the federated and local models on ProxRouter-Data , as Figure 11 .
Figure 19: ( ProxRouter-Data ) Federated vs. centralized training.
Figure 20: ( ProxRouter-Data ) Adapting to newly joined models. We observe that our framework allows successful integration of the new models introduced in the system.
Figure 21: ( ProxRouter-Data ) Adapting to newly joined clients. We observe that our framework allows successful integration of the new clients joining in the system.
Figure 22: ( ProxRouter-Data ) Adaptive personalization under high heterogeneity for representative clients with both MLP-Router and K-Means-Router . Local test set performance of client-local, federated, and adaptive routers.
Figure 23: ( ProxRouter-Data ) Adaptive personalization under high heterogeneity for all clients with MLP-Router . Local test set performance of client-local (no-FL), federated, and personalized routers.
Figure 24: ( ProxRouter-Data ) Adaptive personalization under high heterogeneity for all clients with K-Means-Router . Local test set performance of client-local (no-FL), federated, and personalized routers.
Figure 25: SPROUT global-test performance. Federated and client-local routers for MLP-Router and K-Means-Router . The accuracy axis denotes mean benchmark quality score, and parentheses denote normalized AUC.
Figure 26: SPROUT local-test performance. Federated and client-local routers evaluated on the local test distributions of representative clients. The accuracy axis denotes mean benchmark quality score, and parentheses denote normalized AUC.
Figure 27: Federated routers on the global test distribution with EmbedLLM. We compare the proposed federated MLP-Router and federated K-Means-Router routers against a federated adaptation of EmbedLLM [ 82 ] . Results are shown on RouterBench-Data and ProxRouter-Data . Numbers in parentheses denote normalized AUC. The proposed methods outperform or are competitive with the federated EmbedLLM baseline across datasets.
Figure 28: Federated EmbedLLM vs. client-local EmbedLLM on the global test distribution. We compare federated EmbedLLM against client-local, no-FL EmbedLLM routers trained independently on each client’s local data. Results are shown on RouterBench-Data and ProxRouter-Data , with normalized AUC shown in parentheses. Federated training consistently improves global-distribution generalization over client-local training, supporting the value of federated collaboration beyond the proposed MLP-Router and K-Means-Router router architectures.
Figure 29: RouterBench-Data global-generalization results under different seeds. Federated routers consistently outperform client-local routers across both random seeds.
Figure 30: ProxRouter-Data global-generalization results under different seeds. Federated routers consistently outperform client-local routers across both random seeds.
No single large language model (LLM) is optimal across all queries and budget constraints, making model routing essential for cost-effective deployment. Existing routers adopt diverse formulations and implementations, making fair comparison and extension difficult. We present a unified formulation of LLM routing as a sequential decision process characterized by five components: context encoders, model encoders, scoring functions, decision rules, and learning signals, covering single-turn, multi-turn, and personalized routing. Based on this formulation, we develop an automated pipeline for constructing routing supervision and evaluating routers jointly on response quality and inference cost. The resulting benchmark, xRouteBench, spans generic LLM, memory-augmented, vision, time-series, and personalized routing tasks. We further introduce LLMRouter, an open-source modular infrastructure with more than 16 representative routers. Our empirical study shows that learned routers outperform the strongest fixed-model baseline by 14.6% relatively, lightweight routers become more competitive under tight cost constraints, and user-conditioned routing consistently improves personalization.
Tao Feng, Fangxu Yu, Haozhen Zhang +9
University of Illinois Urbana-Champaign · University of Maryland, College Park · 3Nanyang Technological University +2
Dynamic LLM routers promise to cut inference costs by sending each query to the cheapest model that can answer it correctly. We analyze six commercial routers across 14 settings on a diverse benchmark spanning eight task categories, finding that none of them outperforms a router that randomly selects between two well-chosen models at matched cost. Some underperform by more than 10 percentage points. We trace this gap to four patterns prevalent across routers: difficulty blindness, length reversal, semantic matching, and roster suboptimality. We show that the first three are what the standard objective rewards: cost-accuracy Pareto efficiency on realized costs favors escalating moderately hard queries over the hardest ones, shorter queries over longer ones, and routing by a query's source over its difficulty. We also argue that the two assumptions that would justify large rosters, model granularity and model specialization, do not hold empirically. We propose an alternative evaluation methodology that does not reward these patterns, and as a proof of concept, we design a simple two-model router that avoids all four. Nevertheless, its gain over random routing is limited, because a well-chosen roster leaves little to route.
Large language models (LLMs) achieve impressive performance across multiple domains, but using the most capable model for every query is prohibitive at scale. LLM routing exploits diversity in model capability and cost by assigning each query to a suitable model to balance utility and budget. Current methods have two limitations: (i) they either use heuristics that do not always enforce the budget constraint or impose a fixed per-query budget that cannot adapt across the workload and leads to suboptimal performance; (ii) they require supervised learning on a dense dataset with statistics for every query-model pair, which is expensive to collect. To address these challenges, we formulate LLM routing as a constrained contextual multi-armed bandit problem and introduce WISERouter (WR for short), a framework that supports offline learning from historical interactions as well as online learning with exploration. We further prove that WR-Online achieves a sublinear regret bound of O(T) over a time horizon T. Empirical results on RouterBench and SWE-Bench demonstrate that (i) WR-Offline surpasses existing baselines in performance under a fixed budget and adheres more closely to budget constraints, and (ii) WR-Online achieves comparable performance to the baselines, while using substantially less exploration data.