Measured Joules, Learned Routes: Learning to Route for Energy-Efficient LLM Serving

Muhammad Abdur Rab Siddiqui; Daniela Rojas; Chen Yang; Wenqi Cui; Yuanyuan Shi; Yize Chen

cs.PFSep 19, 2026

Measured Joules, Learned Routes: Learning to Route for Energy-Efficient LLM Serving

Authors: Muhammad Abdur Rab Siddiqui, Daniela Rojas, Chen Yang, Wenqi Cui, Yuanyuan Shi, Yize Chen

Organizations: University of Alberta Edmonton, Alberta, Canada · University of California San Diego La Jolla, California, USA · Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China · New York University New York, New York, USA

Abstract

Large language models (LLMs) and agentic AI systems are creating rapidly growing inference energy demands as model sizes grow and reasoning trajectories extend. While in practice, many queries do not require the capabilities of the largest available model, and routinely directing such queries to a high-capability model can introduce unnecessary, considerable computation and energy consumption. In this paper, we investigate whether adaptive routing across a heterogeneous pool of LLMs can reduce this energy burden without substantially compromising task performance. We design a language-model-based router that reads in each query and selects an answer model from a fixed candidate pool. The candidate models are first profiled through an offline tournament that records their correctness, latency, power, and GPU energy for each query. Using these measurements, the router is trained through supervised fine-tuning followed by group relative policy optimization (GRPO) with the tailored paradigms. Results demonstrate that learned routing can selectively allocate expensive model capacity based on query context and improve the accuracy-energy tradeoff in multi-LLM serving. Across seven benchmark tasks, we also observe a sharp accuracy-energy phase transition among routers, providing practical insights into improving energy efficiency while maintaining LLM performance.

Figures & tables

Figure 1. Overview of LLM-based routing learned from measured inference outcomes and RL feedbacks. We collect and construct offline profiling dataset from LLM queries to record correctness, GPU energy, and latency to train a small LLM routing controller through SFT and GRPO; At serving time, the intended controller selects one candidate from the fixed executer pool, which achieves improved tradeoff between LLM serving energy and performance.

Model	Nominal Size	ARC Acc.	Energy (J)
Qwen 2.5	0.5B	0.314	66.1
Qwen 2.5	1.5B	0.665	84.3
Qwen 2.5	3B	0.727	131.7
Phi-3 Mini	3.8B	0.803	170.5
Gemma 2	2B	0.722	56.2
Qwen 2.5	7B	0.871	140.8

Table 1. Candidate language models used in the evaluation, with ARC Challenge ( Clark et al., 2018 ) accuracy and mean effective answer energy per query on all 472 collected questions.

Policy	Accuracy	Reward	Duration (ms)	Model Energy (J)	Power (W)
always_smallest	0.189	-	3780	91.8	23.2
always_medium	0.791	-	28672	4127.9	125.9
always_large	0.735	-	25706	5112.4	174.5
K-NN	0.790	-	23366	3697.6	121.3
UniRoute	0.618	-	15371	2192.2	84.8
Smoothie	0.599	-	9269	1070.6	59.4

Table 2. Held-out avg. results by routing policies.

Figure 3. Model selection mix on the held-out set.

Figure 4. Energy–Accuracy Pareto comparison.

	always smallest	GreenServ ( $\lambda{=}0.5$ )	Smoothie	SFT ( $\lambda_{e}{=}0.5$ )	GRPO (step 750)	KL-GRPO (step 1500)	GRPO ( $\lambda_{e}{=}0.3$ )	UniRoute	K-NN	RouteLLM	always largest
MMLU	14.8	51.7	49.0	40.9	38.9	45.6	50.3	58.4	63.1	65.1	65.1
GSM8K	27.1	82.6	86.8	38.2	75.7	61.8	81.2	70.8	89.6	89.6	89.6
BBH	2.1	15.2	42.8	33.8	40.7	52.4	37.9	19.3	69.7	27.6	27.6
ARC Challenge	35.4	77.6	76.2	74.8	76.2	78.2	78.9	87.1	88.4	91.2	92.5
HellaSwag	20.4	57.1	53.7	40.8	57.8	58.5	57.8	62.6	72.8	66.7	72.8
Math500	17.2	57.4	62.3	73.0	67.8	73.8	36.9	78.7	86.1	85.2	86.9

Table 3. Held-out accuracy (%) by benchmark and method.

Figure 5. SFT versus KL-GRPO effective GPU power on four held-out questions. Each panel shows the controller decode followed by the answer-model draft, with energy for the router, the answer, and their sum. Title deltas are relative to that total.

	always smallest	GreenServ ( $\lambda{=}0.5$ )	Smoothie	SFT ( $\lambda_{e}{=}0.5$ )	GRPO (step 750)	KL-GRPO (step 1500)	GRPO ( $\lambda_{e}{=}0.3$ )	UniRoute	K-NN	RouteLLM	always largest
MMLU	116.5	213.6	228.8	146.3	123.9	154.2	263.4	2.8k	1.1k	3.9k	3.9k
GSM8K	91.1	306.0	1.5k	159.5	389.2	489.1	331.9	572.6	2.3k	2.4k	2.4k
BBH	62.7	122.2	1.4k	1.6k	2.3k	2.5k	305.8	234.0	3.6k	4.3k	4.4k
ARC Challenge	72.0	157.3	161.8	77.6	129.7	180.1	204.4	1.9k	2.0k	3.2k	3.3k
HellaSwag	45.5	127.5	154.3	100.5	382.0	162.3	176.6	1.5k	2.9k	3.4k	3.8k
Math500	167.6	854.0	3.8k	3.5k	2.0k	6.6k	571.4	6.3k	9.6k	8.6k	9.1k

Table 4. Mean effective answer energy (J) by benchmark and method.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Collected	Train	Held-out
MMLU	475	325	149
GSM8K	473	325	144
BBH	472	325	145
ARC Challenge	472	325	147
HellaSwag	472	325	147
MATH-500	472	325	122

Appendix

Table 5. Number of questions per benchmark used for training and held-out evaluation. Collected counts are before removing

37

incomplete tournament items.

Figure 9. Held-out model-selection mix on GSM8K, MATH-500, and ARC Challenge for the non-always policies. KL-GRPO step 1500 is mostly Qwen2.5-1.5B on GSM8K, DeepSeek-R1-8B on MATH-500, and Gemma2-2B on ARC; GRPO step 750 is 7B-heavy on GSM8K/MATH and Gemma-heavy on ARC; SFT still places mass on the $0.5$ B model on GSM8K. RouteLLM stays on R1-32B and GRPO $\lambda_{e}{=}0.3$ locks onto Llama 3.1 8B on every task.

Checkpoint	Acc.	Dur. (ms)	Energy (J)	Power (W)
GRPO step 1000 (SFT $\lambda_{e}{=}0.3$ , RL $\lambda_{e}{=}0.3$ )	0.573	6002	297.0	46.8
GRPO step 750 (SFT $\lambda_{e}{=}0.7$ , RL $\lambda_{e}{=}0.7$ , run2)	0.421	5634	354.7	30.4
GRPO step 1000 (SFT $\lambda_{e}{=}0.7$ , RL $\lambda_{e}{=}0.7$ , run3)	0.432	6821	561.8	31.8
GRPO step 1500 (SFT $\lambda_{e}{=}0.7$ , RL $\lambda_{e}{=}0.7$ )	0.384	6070	569.3	27.9
KL-GRPO step 500 (SFT $\lambda_{e}{=}0.7$ , RL $\lambda_{e}{=}0.7$ )	0.527	7286	648.7	45.5
KL-GRPO step 1536 (SFT $\lambda_{e}{=}0.3$ , RL $\lambda_{e}{=}0.3$ )	0.503	7410	719.2	44.4

Appendix

Table 6. TRL RL checkpoints from Table 2 and the blue-circle sweep in Fig. 4 , ordered by increasing mean energy. Choice only means the controller is trained to emit a route letter ( Choice: A ) without a free-form reasoning line.

Checkpoint	Acc.	Dur. (ms)	Energy (J)	Power (W)
SFT ( $\lambda_{e}{=}0.7$ )	0.468	7517	749.9	45.5
SFT ( $\lambda_{e}{=}0.3$ )	0.456	7588	776.7	45.0
SFT ( $\lambda_{e}{=}0.5$ , choice only)	0.544	7912	779.9	47.0
SFT ( $\lambda_{e}{=}0.5$ )	0.502	7821	822.8	45.9
SFT ( $\lambda_{e}{=}1.0$ )	0.738	24414	4706.6	160.3

Appendix

Table 7. SFT warm-start checkpoints for the TRL sweep and Table 2 . The labelled SFT

\lambda_{e}{=}0.5

marker in Fig. 4 matches the

\lambda_{e}{=}0.5

row. SFT (

\lambda_{e}{=}1.0

) is the accuracy-oracle warm start sft_acc , not a hybrid

\lambda_{e}{=}1.0

label set.

Model		MMLU	GSM8K	BBH	ARC	HellaSwag	MATH-500	HumanEval	Overall
Qwen 2.5 0.5B	Acc.	14.8	27.1	2.1	35.4	20.4	17.2	10.2	18.9
	Energy	116.5	91.1	62.7	72.0	45.5	167.6	110.3	91.8
Qwen 2.5 1.5B	Acc.	37.6	59.0	8.3	68.7	44.9	32.8	30.5	41.4
	Energy	126.5	129.7	84.0	92.9	66.1	226.7	105.0	117.1
Qwen 2.5 3B	Acc.	53.7	64.6	11.0	74.8	59.9	49.2	33.9	51.2
	Energy	198.4	233.5	195.3	136.7	124.4	497.6	167.7	219.6

Appendix

Table 8. Held-out per-model accuracy (%) and mean effective answer energy (J) on

913

questions. Overall is item-weighted.

Model		MMLU	GSM8K	BBH	ARC	HellaSwag	MATH-500	HumanEval	Overall
Qwen 2.5 0.5B	Acc.	20.9	34.5	6.8	29.5	29.5	11.7	30.0	22.5
	Energy	125.8	300.7	859.4	63.4	57.6	2063.2	117.1	555.8
Qwen 2.5 1.5B	Acc.	42.5	61.5	18.8	65.5	40.0	32.0	64.0	44.4
	Energy	151.3	888.4	803.3	80.4	72.9	2000.3	100.8	638.5
Qwen 2.5 3B	Acc.	45.8	52.0	23.7	71.7	56.6	46.2	52.0	49.5
	Energy	203.6	250.4	698.4	129.4	135.1	1198.4	172.6	423.1

Appendix

Table 9. Train per-model accuracy (%) and mean effective answer energy (J) on

2{,}050

questions. Overall is item-weighted.

Figure 10. Same four held-out questions as Fig. 5 on a later NVIDIA A100-SXM4- $80$ ,GB session. KL-GRPO step 1500 of a different KL beta variation versus SFT. Router energy is about $30\,\mathrm{J}$ for KL-GRPO and a few hundred joules for the longer SFT decode. Answer curves reuse tournament logs.

Energy-Aware LLM Inference

See all 25 papers

LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers
Aug 7, 2026Tao Feng, Fangxu Yu, Haozhen Zhang +9LLM Evaluation LLM Routing
University of Illinois Urbana-Champaign · University of Maryland, College Park · 3Nanyang Technological University +2
WISERouter: LLM Routing with Workload Budget Constraint
Jul 26, 2026Yifei Li, Zihui Gao, Laks V. S. LakshmananLLM Routing Contextual Bandits
The University of British Columbia
Dynamic LLM Routers are Often Misguided
Oct 2, 2026Sam Wang, Julia White, Sahibzada Allahyar +3LLM Routing Cost-Aware Inference
Fastino Labs

Aug 7, 2026J/K move · Enter open · S save

LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers

Tao Feng, Fangxu Yu, Haozhen Zhang, Zhongjie Dai+8

University of Illinois Urbana-Champaign · University of Maryland, College Park · 3Nanyang Technological University+2

LLM Evaluation LLM Routing

Open paper AskPDF arXiv

No single large language model (LLM) is optimal across all queries and budget constraints, making model routing essential for cost-effective deployment. Existing routers adopt diverse formulations and implementations, making fair comparison and extension difficult. We present a unified formulation of LLM routing as a sequential decision process characterized by five components: context encoders, model encoders, scoring functions, decision rules, and learning signals, covering single-turn, multi-turn, and personalized routing. Based on this formulation, we develop an automated pipeline for constructing routing supervision and evaluating routers jointly on response quality and inference cost. The resulting benchmark, xRouteBench, spans generic LLM, memory-augmented, vision, time-series, and personalized routing tasks. We further introduce LLMRouter, an open-source modular infrastructure with more than 16 representative routers. Our empirical study shows that learned routers outperform the strongest fixed-model baseline by 14.6% relatively, lightweight routers become more competitive under tight cost constraints, and user-conditioned routing consistently improves personalization.

Abstract

Energy-Aware LLM Inference

Explore similar work

LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers

WISERouter: LLM Routing with Workload Budget Constraint

Dynamic LLM Routers are Often Misguided