cs.LGJan 29, 2026
SaveFederate the Router: Learning Language Model Routers with Sparse and Decentralized Evaluations
Organizations: Carnegie Mellon University.
Abstract
Large language models (LLMs) are increasingly accessed as remotely hosted services by edge and enterprise clients that cannot run frontier models locally. Since models vary widely in capability and price, routing queries to models that balance quality and inference cost is essential. Existing router approaches assume access to centralized query-model evaluation data. However, these data are often fragmented across clients, such as end users and organizations, and are privacy-sensitive, which makes centralizing data infeasible. Additionally, per-client router training is ineffective since local evaluation data is limited and covers only a restricted query distribution and a biased subset of model evaluations. We introduce the first federated framework for LLM routing, enabling clients to learn a shared routing policy from local offline query-model evaluation data. Our framework supports both parametric multilayer perceptron router and nonparametric K-means router under heterogeneous client query distributions and non-uniform model coverage. Across three benchmarks, federated collaboration improves the accuracy-cost frontier over client-local routers, both via increased effective model coverage and better query generalization. Our theoretical results also validate that federated training reduces routing suboptimality.
Figures & tables
Figure 1: Federated LLM Router System Description. (i) Clients possess queries from different tasks, and each query is evaluated only on some LMs (in our experiments, we consider the extreme case of one LM per query, where the choice of LM is client-specific and can be highly non-uniform). (ii) Federated Learning setup where clients train their local router on local data, and central server aggregates these local models and returns global model to clients. (iii) Functioning of router, where modelwise accuracy and cost is calculated for the given query, which is routed to the most suitable model that maximizes the objective value.
| \displaystyle\bar{a}^{(m)}_{i,k}=\mathord{\raise 0.49991pt\hbox{\displaystyle\genfrac{}{}{0.4pt}{}{1}{n^{(m)}{i,k}}}}\!\!\sum_{\begin{subarray}{c}\mathbf{x}\in\mathcal{D}_{i}\\ k_{\mu}(\mathbf{x})=k\end{subarray}}\!\!\widehat{\text{acc}}(\mathbf{x},m),\bar{c}^{(m)}_{i,k}=\mathord{\raise 0.49991pt\hbox{\displaystyle\genfrac{}{}{0.4pt}{}{1}{n^{(m)}{i,k}}}}\!\!\sum_{\begin{subarray}{c}\mathbf{x}\in\mathcal{D}_{i}\\ k_{\mu}(\mathbf{x})=k\end{subarray}}\!\!\widehat{\text{cost}}(\mathbf{x},m) |
Algorithm 2 Federated K-Means-Router training
Figure 2: Federated vs. client-local routers on the global test distribution. Accuracy–cost curves for MLP-Router and K-Means-Router by sweeping . AUC scores are shown in parentheses.
Figure 3: Federated vs. client-local routers evaluated on local test sets. Representative clients for MLP-Router and K-Means-Router are shown. Federated training improves the in-distribution accuracy–cost frontier via better effective model coverage.
Figure 6: Adaptive personalization under extreme heterogeneity. Evaluated on local test sets of clients with federated training, isolated local training, and the proposed adaptive mixture.
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: t-SNE visualization of query distributions. We run t-SNE once on the full dataset and reuse the same 2D embedding for all panels. The grid highlights each client’s local subset, illustrating (subsampled) the resulting heterogeneity across clients.
Figure 8: t-SNE visualization of query distributions for ProxRouter-Data .
Figure 9: Client-specific model heterogeneity for RouterBench-Data . Bubble plot of the per-model proportions in each client local dataset under the Dirichlet assignment scheme ( ).
| MLP-Router | K-Means-Router | |||
|---|---|---|---|---|
| Federated | Client-local | Federated | Client-local | |
| 5 | ||||
| 10 | ||||
| 20 | ||||
| 40 | ||||
Table 1: Sensitivity to the number of clients on RouterBench-Data . Global-test normalized AUCs are reported as mean standard error across seeds; client-local results are averaged over clients.
| MLP-Router | K-Means-Router | |||
|---|---|---|---|---|
| Federated | Client-local | Federated | Client-local | |
| 0.03 | ||||
| 0.10 | ||||
| 0.30 | ||||
| 0.60 | ||||
| 5.00 | ||||
Table 2: Sensitivity to query heterogeneity on RouterBench-Data . Global-test normalized AUCs are reported as mean standard error across seeds; client-local results are averaged over clients.
Figure 10: All-clients local test results ( MLP-Router ). Each panel corresponds to one client and reports accuracy–cost trade-offs for the federated router versus the client-local (no-FL) router, evaluated on that client’s local test set.
Figure 11: All-clients local test results ( K-Means-Router ). Each panel corresponds to one client and reports accuracy–cost trade-offs for the federated router versus the client-local (no-FL) router, evaluated on that client’s local test set.
Figure 12: Adapting to new clients. Routers are trained initially by 7 clients, then adapted by 3 new clients, without additional participation from the original clients. MLP-Router uses continued training with a distillation regularizer to retain the base policy; K-Means-Router updates cluster-level statistics via weighted averaging. We report accuracy–cost frontiers on the global test set.
Figure 13: Per-client accuracy–cost frontiers under extreme heterogeneity ( ) for MLP-Router .
Figure 14: Per-client accuracy–cost frontiers under extreme heterogeneity ( ) for K-Means-Router .
| Sentence Encoder (Emb. Dim.) | RouterBench-Data | ProxRouter-Data | ||
|---|---|---|---|---|
| K-Means-Router | MLP-Router | K-Means-Router | MLP-Router | |
| all_mpnet_base_v2 (768) | 0.747 | 0.759 | 0.759 | 0.760 |
| all_minilm_l6_v2 (384) | 0.737 | 0.761 | 0.728 | 0.751 |
| paraphrase_albert_ | 0.746 | 0.758 | 0.751 | 0.755 |
| small_v2 (768) | ||||
Table 3: Performance (through normalized AUC) of K-Means-Router and MLP-Router in centralized training regime across different choices of sentence encoders used for representing queries, for both RouterBench-Data and ProxRouter-Data .
Figure 15: ( ProxRouter-Data ) Federated vs. client-local (no-FL) routers on the global test distribution. On ProxRouter-Data , we evaluate federated model and locally trained models on the global test set as in Figure 2 .
Figure 16: ( ProxRouter-Data - Subsampled Clients) Local test set results of subsampled clients. We evaluate the federated and local models on ProxRouter-Data , as in Figure 3 .
Figure 17: ( ProxRouter-Data ) Local test set results of all clients with MLP-Router . We evaluate the federated and local models on ProxRouter-Data , as Figure 10 .
Figure 18: ( ProxRouter-Data ) Local test set results of all clients with K-Means-Router . We evaluate the federated and local models on ProxRouter-Data , as Figure 11 .
Figure 19: ( ProxRouter-Data ) Federated vs. centralized training.
Figure 20: ( ProxRouter-Data ) Adapting to newly joined models. We observe that our framework allows successful integration of the new models introduced in the system.
Figure 21: ( ProxRouter-Data ) Adapting to newly joined clients. We observe that our framework allows successful integration of the new clients joining in the system.
Figure 22: ( ProxRouter-Data ) Adaptive personalization under high heterogeneity for representative clients with both MLP-Router and K-Means-Router . Local test set performance of client-local, federated, and adaptive routers.
Figure 23: ( ProxRouter-Data ) Adaptive personalization under high heterogeneity for all clients with MLP-Router . Local test set performance of client-local (no-FL), federated, and personalized routers.
Figure 24: ( ProxRouter-Data ) Adaptive personalization under high heterogeneity for all clients with K-Means-Router . Local test set performance of client-local (no-FL), federated, and personalized routers.
Figure 25: SPROUT global-test performance. Federated and client-local routers for MLP-Router and K-Means-Router . The accuracy axis denotes mean benchmark quality score, and parentheses denote normalized AUC.
Figure 26: SPROUT local-test performance. Federated and client-local routers evaluated on the local test distributions of representative clients. The accuracy axis denotes mean benchmark quality score, and parentheses denote normalized AUC.
Figure 27: Federated routers on the global test distribution with EmbedLLM. We compare the proposed federated MLP-Router and federated K-Means-Router routers against a federated adaptation of EmbedLLM [ 82 ] . Results are shown on RouterBench-Data and ProxRouter-Data . Numbers in parentheses denote normalized AUC. The proposed methods outperform or are competitive with the federated EmbedLLM baseline across datasets.
Figure 28: Federated EmbedLLM vs. client-local EmbedLLM on the global test distribution. We compare federated EmbedLLM against client-local, no-FL EmbedLLM routers trained independently on each client’s local data. Results are shown on RouterBench-Data and ProxRouter-Data , with normalized AUC shown in parentheses. Federated training consistently improves global-distribution generalization over client-local training, supporting the value of federated collaboration beyond the proposed MLP-Router and K-Means-Router router architectures.
Figure 29: RouterBench-Data global-generalization results under different seeds. Federated routers consistently outperform client-local routers across both random seeds.
Figure 30: ProxRouter-Data global-generalization results under different seeds. Federated routers consistently outperform client-local routers across both random seeds.