When LLM Routers Overpay: The Cost of Strong-Model Preference
Organizations: School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China.
Abstract
LLM routing aims to reduce inference costs by assigning easy queries to less expensive models and reserving stronger models for cases in which they are needed. However, we find that existing routers often violate this principle under loose cost budgets: as the budget increases, they increasingly select the strongest and most expensive model even when less expensive feasible candidates provide comparable or identical utility. We refer to this behavior as strong-model over-selection under loose budgets. Our analysis shows that ambiguity alone does not explain this phenomenon. Substantial overpayment can arise when differently priced models offer equivalent or near-equivalent utility and the router retains a persistent model-level preference for costly models; budget relaxation exposes this preference by making premium candidates feasible. Motivated by this diagnosis, we propose EquiRouter, a decision-aligned router that learns query-dependent model rankings, preserves meaningful utility differences, and explicitly prefers lower-cost models among equivalent candidates through a Cost-Aware Ranking Objective and a lightweight Query--Model Interaction Representation. Experiments across diverse text, multimodal, continuous-utility, and large-model-pool settings show that EquiRouter consistently improves cost efficiency while maintaining competitive peak and whole-curve performance. Code is available at https://github.com/LAMDA-Model-Reuse/EquiRouter.
Figures & tables
| Router | RouterBench | MMR-Bench | MixInstruct | RouterEval | ||||
|---|---|---|---|---|---|---|---|---|
| QNC | QNC | QNC | QNC | |||||
| Existing routers | ||||||||
| kNNRouter | / | 0.8049 | 0.9895 | 0.7416 | 0.9819 | 0.7484 | 0.9349 | 0.7860 |
| MLPRouter | / | 0.8054 | 1.0000 | 0.7384 | 1.0000 | 0.7477 | 0.9785 | 0.7821 |
| EmbedLLM | 0.9752 | 0.8056 | / | 0.7369 | 0.9927 | 0.7487 | 0.9192 | 0.7790 |
| GraphRouter | / | 0.8026 | 1.0000 | 0.7384 | 1.0000 | 0.7477 | / | 0.7513 |
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Benchmark | |||
|---|---|---|---|
| RouterBench | 10 | 1 | 0.1 |
| MMR-Bench | 33 | 0.15 | 1 |
| MixInstruct | 1 | 50 | 1 |
| RouterEval | 10 | 10 | 0.001 |
| Opportunity rate (%) | |
|---|---|
| 0 | 0.5 |
| 0.0025 | 8.1 |
| 0.005 | 14.2 |
| 0.010 | 25.7 |
| 0.020 | 44.4 |
| Benchmark | Symmetric noise | Aligned direction | Identity permutation |
|---|---|---|---|
| RouterBench | 50.0 | 79.3 | 48.7 |
| MMR-Bench | 50.0 | 78.1 | 49.2 |
| MixInstruct | 50.6 | 73.9 | 50.2 |
| RouterEval | 50.0 | 68.4 | 50.1 |
| Router paradigm | Original | Mean only | Residual only | Variance equalized |
|---|---|---|---|---|
| Regression | 97.6 | 98.6 | 46.0 | 98.0 |
| kNN | 78.8 | 94.8 | 47.7 | 80.5 |
| Ranking | 81.6 | 92.6 | 49.4 | 84.5 |
| Classification | 53.0 | 56.6 | 48.4 | 53.3 |
| Benchmark | Opportunity | Overpayment | All-query OP | Quality Regret |
|---|---|---|---|---|
| RouterBench | 94.7% | 97.66% | 79.05% | 0.1199 |
| MMR-Bench | 93.2% | 95.65% | 69.96% | 0.1855 |
| MixInstruct | 14.2% | 5.73% | 2.57% | 0.0319 |
| RouterEval | 97.0% | 98.58% | 72.09% | 0.2145 |
| Benchmark | QNC | nAUC | |
|---|---|---|---|
| RouterBench | |||
| MMR-Bench | |||
| MixInstruct | |||
| RouterEval |
| 60/40 split | 80/20 split | |||||
|---|---|---|---|---|---|---|
| Method | QNC | nAUC | QNC | nAUC | ||
| EquiRouter | 0.8039 | 0.9194 | 0.7680 | 0.8048 | 0.8789 | 0.7671 |
| kNNRouter | 0.7904 | / | 0.7619 | 0.8040 | / | 0.7679 |
| MLPRouter | 0.8036 | 0.9836 | 0.7588 | 0.8050 | 0.9688 | 0.7622 |
| EmbedLLM | 0.8038 | 0.9073 | 0.7704 | 0.8053 | 0.8888 | 0.7670 |
| GraphRouter | 0.8034 | / | 0.6674 | 0.8062 | 0.8909 | 0.6919 |
| 60/40 split | 80/20 split | |||||
|---|---|---|---|---|---|---|
| Method | QNC | nAUC | QNC | nAUC | ||
| EquiRouter | 0.7423 | 0.8733 | 0.7062 | 0.7469 | 0.8017 | 0.7119 |
| kNNRouter | 0.7440 | 0.8884 | 0.7087 | 0.7473 | 0.8682 | 0.7097 |
| MLPRouter | 0.7413 | 0.9994 | 0.6946 | 0.7416 | 1.0000 | 0.6954 |
| EmbedLLM | 0.7413 | 1.1345 | 0.6897 | 0.7396 | / | 0.6911 |
| GraphRouter | 0.7030 | / | 0.6540 | 0.7401 | / | 0.6758 |
| 60/40 split | 80/20 split | |||||
|---|---|---|---|---|---|---|
| Method | QNC | nAUC | QNC | nAUC | ||
| EquiRouter | 0.7497 | 0.9581 | 0.7371 | 0.7493 | 0.9361 | 0.7370 |
| kNNRouter | 0.7469 | / | 0.7350 | 0.7466 | / | 0.7347 |
| MLPRouter | 0.7475 | 1.0000 | 0.7345 | 0.7473 | 1.0000 | 0.7341 |
| EmbedLLM | 0.7492 | 0.9746 | 0.7360 | 0.7491 | 0.9741 | 0.7356 |
| GraphRouter | 0.7483 | 0.9997 | 0.7348 | 0.7484 | 0.9966 | 0.7346 |
| 60/40 split | 80/20 split | |||||
|---|---|---|---|---|---|---|
| Method | QNC | nAUC | QNC | nAUC | ||
| EquiRouter | 0.8024 | 0.7987 | 0.6803 | 0.8119 | 0.7903 | 0.6943 |
| kNNRouter | 0.7844 | 0.8742 | 0.6779 | 0.7929 | 0.8696 | 0.6852 |
| MLPRouter | 0.7844 | 0.9666 | 0.6693 | 0.7935 | 0.9722 | 0.6756 |
| EmbedLLM | 0.7992 | 0.8090 | 0.6830 | 0.8083 | 0.8034 | 0.6903 |
| GraphRouter | 0.7661 | / | 0.6625 | 0.7737 | / | 0.6672 |
| QNC | Saving (%) | nAUC | ||
|---|---|---|---|---|
| 0 | ||||
| 0.001 | ||||
| 0.0025 | ||||
| 0.0075 | ||||
| 0.01 |
LLM Routing
Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.
See all 380 papers