Organizations: University of Alberta Edmonton, Alberta, Canada · University of California San Diego La Jolla, California, USA · Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China · New York University New York, New York, USA
Large language models (LLMs) and agentic AI systems are creating rapidly growing inference energy demands as model sizes grow and reasoning trajectories extend. While in practice, many queries do not require the capabilities of the largest available model, and routinely directing such queries to a high-capability model can introduce unnecessary, considerable computation and energy consumption. In this paper, we investigate whether adaptive routing across a heterogeneous pool of LLMs can reduce this energy burden without substantially compromising task performance. We design a language-model-based router that reads in each query and selects an answer model from a fixed candidate pool. The candidate models are first profiled through an offline tournament that records their correctness, latency, power, and GPU energy for each query. Using these measurements, the router is trained through supervised fine-tuning followed by group relative policy optimization (GRPO) with the tailored paradigms. Results demonstrate that learned routing can selectively allocate expensive model capacity based on query context and improve the accuracy-energy tradeoff in multi-LLM serving. Across seven benchmark tasks, we also observe a sharp accuracy-energy phase transition among routers, providing practical insights into improving energy efficiency while maintaining LLM performance.
Figures & tables
Figure 1. Overview of LLM-based routing learned from measured inference outcomes and RL feedbacks. We collect and construct offline profiling dataset from LLM queries to record correctness, GPU energy, and latency to train a small LLM routing controller through SFT and GRPO; At serving time, the intended controller selects one candidate from the fixed executer pool, which achieves improved tradeoff between LLM serving energy and performance.
Model
Nominal Size
ARC Acc.
Energy (J)
Qwen 2.5
0.5B
0.314
66.1
Qwen 2.5
1.5B
0.665
84.3
Qwen 2.5
3B
0.727
131.7
Phi-3 Mini
3.8B
0.803
170.5
Gemma 2
2B
0.722
56.2
Qwen 2.5
7B
0.871
140.8
Table 1. Candidate language models used in the evaluation, with ARC Challenge ( Clark et al., 2018 ) accuracy and mean effective answer energy per query on all 472 collected questions.
Policy
Accuracy
Reward
Duration (ms)
Model Energy (J)
Power (W)
always_smallest
0.189
-
3780
91.8
23.2
always_medium
0.791
-
28672
4127.9
125.9
always_large
0.735
-
25706
5112.4
174.5
K-NN
0.790
-
23366
3697.6
121.3
UniRoute
0.618
-
15371
2192.2
84.8
Smoothie
0.599
-
9269
1070.6
59.4
Table 2. Held-out avg. results by routing policies.
Figure 3. Model selection mix on the held-out set.
Figure 4. Energy–Accuracy Pareto comparison.
always smallest
GreenServ ( λ=0.5 )
Smoothie
SFT ( λe=0.5 )
GRPO (step 750)
KL-GRPO (step 1500)
GRPO ( λe=0.3 )
UniRoute
K-NN
RouteLLM
always largest
MMLU
14.8
51.7
49.0
40.9
38.9
45.6
50.3
58.4
63.1
65.1
65.1
GSM8K
27.1
82.6
86.8
38.2
75.7
61.8
81.2
70.8
89.6
89.6
89.6
BBH
2.1
15.2
42.8
33.8
40.7
52.4
37.9
19.3
69.7
27.6
27.6
ARC Challenge
35.4
77.6
76.2
74.8
76.2
78.2
78.9
87.1
88.4
91.2
92.5
HellaSwag
20.4
57.1
53.7
40.8
57.8
58.5
57.8
62.6
72.8
66.7
72.8
Math500
17.2
57.4
62.3
73.0
67.8
73.8
36.9
78.7
86.1
85.2
86.9
Table 3. Held-out accuracy (%) by benchmark and method.
Figure 5. SFT versus KL-GRPO effective GPU power on four held-out questions. Each panel shows the controller decode followed by the answer-model draft, with energy for the router, the answer, and their sum. Title deltas are relative to that total.
always smallest
GreenServ ( λ=0.5 )
Smoothie
SFT ( λe=0.5 )
GRPO (step 750)
KL-GRPO (step 1500)
GRPO ( λe=0.3 )
UniRoute
K-NN
RouteLLM
always largest
MMLU
116.5
213.6
228.8
146.3
123.9
154.2
263.4
2.8k
1.1k
3.9k
3.9k
GSM8K
91.1
306.0
1.5k
159.5
389.2
489.1
331.9
572.6
2.3k
2.4k
2.4k
BBH
62.7
122.2
1.4k
1.6k
2.3k
2.5k
305.8
234.0
3.6k
4.3k
4.4k
ARC Challenge
72.0
157.3
161.8
77.6
129.7
180.1
204.4
1.9k
2.0k
3.2k
3.3k
HellaSwag
45.5
127.5
154.3
100.5
382.0
162.3
176.6
1.5k
2.9k
3.4k
3.8k
Math500
167.6
854.0
3.8k
3.5k
2.0k
6.6k
571.4
6.3k
9.6k
8.6k
9.1k
Table 4. Mean effective answer energy (J) by benchmark and method.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Collected
Train
Held-out
MMLU
475
325
149
GSM8K
473
325
144
BBH
472
325
145
ARC Challenge
472
325
147
HellaSwag
472
325
147
MATH-500
472
325
122
Appendix
Table 5. Number of questions per benchmark used for training and held-out evaluation. Collected counts are before removing 37 incomplete tournament items.
Figure 10
Figure 9. Held-out model-selection mix on GSM8K, MATH-500, and ARC Challenge for the non-always policies. KL-GRPO step 1500 is mostly Qwen2.5-1.5B on GSM8K, DeepSeek-R1-8B on MATH-500, and Gemma2-2B on ARC; GRPO step 750 is 7B-heavy on GSM8K/MATH and Gemma-heavy on ARC; SFT still places mass on the 0.5 B model on GSM8K. RouteLLM stays on R1-32B and GRPO λe=0.3 locks onto Llama 3.1 8B on every task.
Checkpoint
Acc.
Dur. (ms)
Energy (J)
Power (W)
GRPO step 1000 (SFT λe=0.3 , RL λe=0.3 )
0.573
6002
297.0
46.8
GRPO step 750 (SFT λe=0.7 , RL λe=0.7 , run2)
0.421
5634
354.7
30.4
GRPO step 1000 (SFT λe=0.7 , RL λe=0.7 , run3)
0.432
6821
561.8
31.8
GRPO step 1500 (SFT λe=0.7 , RL λe=0.7 )
0.384
6070
569.3
27.9
KL-GRPO step 500 (SFT λe=0.7 , RL λe=0.7 )
0.527
7286
648.7
45.5
KL-GRPO step 1536 (SFT λe=0.3 , RL λe=0.3 )
0.503
7410
719.2
44.4
Appendix
Table 6. TRL RL checkpoints from Table 2 and the blue-circle sweep in Fig. 4 , ordered by increasing mean energy. Choice only means the controller is trained to emit a route letter ( Choice: A ) without a free-form reasoning line.
Checkpoint
Acc.
Dur. (ms)
Energy (J)
Power (W)
SFT ( λe=0.7 )
0.468
7517
749.9
45.5
SFT ( λe=0.3 )
0.456
7588
776.7
45.0
SFT ( λe=0.5 , choice only)
0.544
7912
779.9
47.0
SFT ( λe=0.5 )
0.502
7821
822.8
45.9
SFT ( λe=1.0 )
0.738
24414
4706.6
160.3
Appendix
Table 7. SFT warm-start checkpoints for the TRL sweep and Table 2 . The labelled SFT λe=0.5 marker in Fig. 4 matches the λe=0.5 row. SFT ( λe=1.0 ) is the accuracy-oracle warm start sft_acc , not a hybrid λe=1.0 label set.
Model
MMLU
GSM8K
BBH
ARC
HellaSwag
MATH-500
HumanEval
Overall
Qwen 2.5 0.5B
Acc.
14.8
27.1
2.1
35.4
20.4
17.2
10.2
18.9
Energy
116.5
91.1
62.7
72.0
45.5
167.6
110.3
91.8
Qwen 2.5 1.5B
Acc.
37.6
59.0
8.3
68.7
44.9
32.8
30.5
41.4
Energy
126.5
129.7
84.0
92.9
66.1
226.7
105.0
117.1
Qwen 2.5 3B
Acc.
53.7
64.6
11.0
74.8
59.9
49.2
33.9
51.2
Energy
198.4
233.5
195.3
136.7
124.4
497.6
167.7
219.6
Appendix
Table 8. Held-out per-model accuracy (%) and mean effective answer energy (J) on 913 questions. Overall is item-weighted.
Model
MMLU
GSM8K
BBH
ARC
HellaSwag
MATH-500
HumanEval
Overall
Qwen 2.5 0.5B
Acc.
20.9
34.5
6.8
29.5
29.5
11.7
30.0
22.5
Energy
125.8
300.7
859.4
63.4
57.6
2063.2
117.1
555.8
Qwen 2.5 1.5B
Acc.
42.5
61.5
18.8
65.5
40.0
32.0
64.0
44.4
Energy
151.3
888.4
803.3
80.4
72.9
2000.3
100.8
638.5
Qwen 2.5 3B
Acc.
45.8
52.0
23.7
71.7
56.6
46.2
52.0
49.5
Energy
203.6
250.4
698.4
129.4
135.1
1198.4
172.6
423.1
Appendix
Table 9. Train per-model accuracy (%) and mean effective answer energy (J) on 2,050 questions. Overall is item-weighted.
Figure 10. Same four held-out questions as Fig. 5 on a later NVIDIA A100-SXM4- 80 ,GB session. KL-GRPO step 1500 of a different KL beta variation versus SFT. Router energy is about 30J for KL-GRPO and a few hundred joules for the longer SFT decode. Answer curves reuse tournament logs.
No single large language model (LLM) is optimal across all queries and budget constraints, making model routing essential for cost-effective deployment. Existing routers adopt diverse formulations and implementations, making fair comparison and extension difficult. We present a unified formulation of LLM routing as a sequential decision process characterized by five components: context encoders, model encoders, scoring functions, decision rules, and learning signals, covering single-turn, multi-turn, and personalized routing. Based on this formulation, we develop an automated pipeline for constructing routing supervision and evaluating routers jointly on response quality and inference cost. The resulting benchmark, xRouteBench, spans generic LLM, memory-augmented, vision, time-series, and personalized routing tasks. We further introduce LLMRouter, an open-source modular infrastructure with more than 16 representative routers. Our empirical study shows that learned routers outperform the strongest fixed-model baseline by 14.6% relatively, lightweight routers become more competitive under tight cost constraints, and user-conditioned routing consistently improves personalization.
Tao Feng, Fangxu Yu, Haozhen Zhang +9
University of Illinois Urbana-Champaign · University of Maryland, College Park · 3Nanyang Technological University +2
Large language models (LLMs) achieve impressive performance across multiple domains, but using the most capable model for every query is prohibitive at scale. LLM routing exploits diversity in model capability and cost by assigning each query to a suitable model to balance utility and budget. Current methods have two limitations: (i) they either use heuristics that do not always enforce the budget constraint or impose a fixed per-query budget that cannot adapt across the workload and leads to suboptimal performance; (ii) they require supervised learning on a dense dataset with statistics for every query-model pair, which is expensive to collect. To address these challenges, we formulate LLM routing as a constrained contextual multi-armed bandit problem and introduce WISERouter (WR for short), a framework that supports offline learning from historical interactions as well as online learning with exploration. We further prove that WR-Online achieves a sublinear regret bound of O(T) over a time horizon T. Empirical results on RouterBench and SWE-Bench demonstrate that (i) WR-Offline surpasses existing baselines in performance under a fixed budget and adheres more closely to budget constraints, and (ii) WR-Online achieves comparable performance to the baselines, while using substantially less exploration data.
Dynamic LLM routers promise to cut inference costs by sending each query to the cheapest model that can answer it correctly. We analyze six commercial routers across 14 settings on a diverse benchmark spanning eight task categories, finding that none of them outperforms a router that randomly selects between two well-chosen models at matched cost. Some underperform by more than 10 percentage points. We trace this gap to four patterns prevalent across routers: difficulty blindness, length reversal, semantic matching, and roster suboptimality. We show that the first three are what the standard objective rewards: cost-accuracy Pareto efficiency on realized costs favors escalating moderately hard queries over the hardest ones, shorter queries over longer ones, and routing by a query's source over its difficulty. We also argue that the two assumptions that would justify large rosters, model granularity and model specialization, do not hold empirically. We propose an alternative evaluation methodology that does not reward these patterns, and as a proof of concept, we design a simple two-model router that avoids all four. Nevertheless, its gain over random routing is limited, because a well-chosen roster leaves little to route.