When LLM Routers Overpay: The Cost of Strong-Model Preference
Authors: Guannan Lai, Long Chen, Han-Jia Ye
Organizations: School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China.
LLM routing aims to reduce inference costs by assigning easy queries to less expensive models and reserving stronger models for cases in which they are needed. However, we find that existing routers often violate this principle under loose cost budgets: as the budget increases, they increasingly select the strongest and most expensive model even when less expensive feasible candidates provide comparable or identical utility. We refer to this behavior as strong-model over-selection under loose budgets. Our analysis shows that ambiguity alone does not explain this phenomenon. Substantial overpayment can arise when differently priced models offer equivalent or near-equivalent utility and the router retains a persistent model-level preference for costly models; budget relaxation exposes this preference by making premium candidates feasible. Motivated by this diagnosis, we propose EquiRouter, a decision-aligned router that learns query-dependent model rankings, preserves meaningful utility differences, and explicitly prefers lower-cost models among equivalent candidates through a Cost-Aware Ranking Objective and a lightweight Query--Model Interaction Representation. Experiments across diverse text, multimodal, continuous-utility, and large-model-pool settings show that EquiRouter consistently improves cost efficiency while maintaining competitive peak and whole-curve performance. Code is available at https://github.com/LAMDA-Model-Reuse/EquiRouter.
Figures & tables
Figure 1: Illustration of strong-model over-selection under loose budgets on RouterBench [ 11 ] . Left: As the cost budget increases, existing routers rapidly increase their call rate to the strongest model, GPT-4, and several methods nearly saturate at 100%. Right: In contrast, the Oracle, which has access to per-query outcomes and applies cost-based tie breaking, selects GPT-4 for fewer than 20% of queries and continues to use cheaper models for a substantial fraction of cases.
Figure 2: Mechanism analysis. Left: prevalence of quality-equivalent choices with different costs. Middle: symmetric noise produces no directional preference, whereas a premium-aligned model component does; permuting model identities removes the effect. Right: in learned routers, the model-level score mean preserves most premium preference, while query-specific residuals remain close to the 50% baseline.
Figure 3: Overview of EquiRouter training. EquiRouter constructs query–model interaction representations and learns to prioritize higher-utility models, breaking near-ties in favor of lower-cost alternatives.
Router
RouterBench
MMR-Bench
MixInstruct
RouterEval
QNC ↓
Ps↑
QNC ↓
Ps↑
QNC ↓
Ps↑
QNC ↓
Ps↑
Existing routers
kNNRouter
/
0.8049
0.9895
0.7416
0.9819
0.7484
0.9349
0.7860
MLPRouter
/
0.8054
1.0000
0.7384
1.0000
0.7477
0.9785
0.7821
EmbedLLM
0.9752
0.8056
/
0.7369
0.9927
0.7487
0.9192
0.7790
GraphRouter
/
0.8026
1.0000
0.7384
1.0000
0.7477
/
0.7513
Table 1: Main results across four routing benchmarks. Lower QNC and higher Ps are better. Best and second-best results among complete methods are shown in bold and underline , respectively. “/” indicates that the router does not reach the strongest-model target.
Figure 4: Whole-curve performance and routed-model reliability. Left: top-6 methods by normalized AUC on each benchmark. Right: success rate of each model when selected by EquiRouter.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
λMSE
λCARO
ρ
RouterBench
10
1
0.1
MMR-Bench
33
0.15
1
MixInstruct
1
50
1
RouterEval
10
10
0.001
Appendix
Table A1 : Loss coefficients and cost-aware tie-breaking weights used by EquiRouter across the four benchmarks.
δanalysis
Opportunity rate (%)
0
0.5
0.0025
8.1
0.005
14.2
0.010
25.7
0.020
44.4
Appendix
Table A2 : Sensitivity of overpayment opportunities to the utility equivalence tolerance on MixInstruct.
Figure A1 : Feasible-set utility margin distributions on MMR-Bench and MixInstruct. Both benchmarks contain many close-utility comparisons. These margins characterize decision sensitivity; premium-model preference describes its direction.
Benchmark
Symmetric noise
Aligned direction
Identity permutation
RouterBench
50.0
79.3
48.7
MMR-Bench
50.0
78.1
49.2
MixInstruct
50.6
73.9
50.2
RouterEval
50.0
68.4
50.1
Appendix
Table A3 : Premium-model preference (%) under controlled score interventions. The identity-permutation condition averages over 200 permutations.
Router paradigm
Original
Mean only
Residual only
Variance equalized
Regression
97.6
98.6
46.0
98.0
kNN
78.8
94.8
47.7
80.5
Ranking
81.6
92.6
49.4
84.5
Classification
53.0
56.6
48.4
53.3
Appendix
Table A4 : Premium-model preference (%) under score decomposition interventions in representative learned routers.
Figure A2 : Training-set evaluation on RouterBench. Learned routers are evaluated on their training queries to remove the train–test generalization gap. Performance and model-selection frequencies are reported across budget levels.
Benchmark
Opportunity
Overpayment
All-query OP
Quality Regret
RouterBench
94.7%
97.66%
79.05%
0.1199
MMR-Bench
93.2%
95.65%
69.96%
0.1855
MixInstruct
14.2%
5.73%
2.57%
0.0319
RouterEval
97.0%
98.58%
72.09%
0.2145
Appendix
Table A5 : Overpayment analysis of Jev 1.14 under unconstrained budgets. Opportunity follows the main diagnostic setting in Figure 2 and measures the fraction of queries with differently priced acceptable models. Overpayment is conditioned on acceptable selections, while All-query OP is measured over all test queries.
Figure 13
Figure A5 : Per-model call rates under budget sweep. EquiRouter reduces strongest-model reliance relative to competing routers while retaining greater utilization of cheaper candidates.
Benchmark
Ps↑
QNC ↓
nAUC ↑
RouterBench
0.8044±0.0021
0.8955±0.0058
0.7697±0.0011
MMR-Bench
0.7521±0.0065
0.8661±0.0264
0.7160±0.0052
MixInstruct
0.7497±0.0001
0.9574±0.0036
0.7370±0.0001
RouterEval
0.8013±0.0028
0.7841±0.0086
0.6867±0.0025
Appendix
Table A6: Five-seed robustness. Results are reported as mean ± sample standard deviation. QNC statistics are computed over runs with defined values.
60/40 split
80/20 split
Method
Ps↑
QNC ↓
nAUC ↑
Ps↑
QNC ↓
nAUC ↑
EquiRouter
0.8039
0.9194
0.7680
0.8048
0.8789
0.7671
kNNRouter
0.7904
/
0.7619
0.8040
/
0.7679
MLPRouter
0.8036
0.9836
0.7588
0.8050
0.9688
0.7622
EmbedLLM
0.8038
0.9073
0.7704
0.8053
0.8888
0.7670
GraphRouter
0.8034
/
0.6674
0.8062
0.8909
0.6919
Appendix
Table A7: Split sensitivity on RouterBench. Comparison under 60/40 and 80/20 train/test splits. Lower QNC and higher Ps and nAUC are better.
60/40 split
80/20 split
Method
Ps↑
QNC ↓
nAUC ↑
Ps↑
QNC ↓
nAUC ↑
EquiRouter
0.7423
0.8733
0.7062
0.7469
0.8017
0.7119
kNNRouter
0.7440
0.8884
0.7087
0.7473
0.8682
0.7097
MLPRouter
0.7413
0.9994
0.6946
0.7416
1.0000
0.6954
EmbedLLM
0.7413
1.1345
0.6897
0.7396
/
0.6911
GraphRouter
0.7030
/
0.6540
0.7401
/
0.6758
Appendix
Table A8: Split sensitivity on MMR-Bench. Comparison under 60/40 and 80/20 train/test splits. Lower QNC and higher Ps and nAUC are better.
60/40 split
80/20 split
Method
Ps↑
QNC ↓
nAUC ↑
Ps↑
QNC ↓
nAUC ↑
EquiRouter
0.7497
0.9581
0.7371
0.7493
0.9361
0.7370
kNNRouter
0.7469
/
0.7350
0.7466
/
0.7347
MLPRouter
0.7475
1.0000
0.7345
0.7473
1.0000
0.7341
EmbedLLM
0.7492
0.9746
0.7360
0.7491
0.9741
0.7356
GraphRouter
0.7483
0.9997
0.7348
0.7484
0.9966
0.7346
Appendix
Table A9: Split sensitivity on MixInstruct. Comparison under 60/40 and 80/20 train/test splits. Lower QNC and higher Ps and nAUC are better.
60/40 split
80/20 split
Method
Ps↑
QNC ↓
nAUC ↑
Ps↑
QNC ↓
nAUC ↑
EquiRouter
0.8024
0.7987
0.6803
0.8119
0.7903
0.6943
kNNRouter
0.7844
0.8742
0.6779
0.7929
0.8696
0.6852
MLPRouter
0.7844
0.9666
0.6693
0.7935
0.9722
0.6756
EmbedLLM
0.7992
0.8090
0.6830
0.8083
0.8034
0.6903
GraphRouter
0.7661
/
0.6625
0.7737
/
0.6672
Appendix
Table A10: Split sensitivity on RouterEval. Comparison under 60/40 and 80/20 train/test splits. Lower QNC and higher Ps and nAUC are better.
δ
Ps↑
QNC ↓
Saving (%) ↑
nAUC ↑
0
0.749869±0.000053
0.973393±0.005741
2.66±0.57
0.736584±0.000239
0.001
0.749807±0.000031
0.970515±0.005845
2.95±0.58
0.736639±0.000222
0.0025
0.749782±0.000072
0.967879±0.005150
3.21±0.51
0.736710±0.000197
0.005†
0.749570±0.000147
0.961802±0.005831
3.82±0.58
0.736776±0.000155
0.0075
0.749369±0.000244
0.953031±0.006654
4.70±0.67
0.736880±0.000106
0.01
0.749030±0.000378
0.947220±0.005035
5.28±0.50
0.736921±0.000099
Appendix
Table A11: Sensitivity to the utility-indifference threshold on MixInstruct. Results are means ± standard deviations over five optimization seeds. The dagger indicates the default threshold. Lower QNC and higher Ps , saving, and nAUC are better. “–” indicates undefined values.
Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.