Tug of Watts: A study on the Energy - Accuracy Trade-off of Large Language Models on Medical Question Answering Tasks

Viju Sudhi; Giulio Billi; Andreea Scrob; Regina Stodden; Philipp Cimiano

cs.CLNov 2, 2026

Tug of Watts: A study on the Energy - Accuracy Trade-off of Large Language Models on Medical Question Answering Tasks

Authors: Viju Sudhi, Giulio Billi, Andreea Scrob, Regina Stodden, Philipp Cimiano

Organizations: Cluster of Excellence Cognitive Interaction Technology (CITEC), Bielefeld, Germany · Bielefeld University, Bielefeld, Germany · University of Bologna, Bologna, Italy

Abstract

Large Language Models (LLMs) are increasingly used in medical question answering (QA) systems. While recent works aim at incrementally improving the downstream task accuracy, an empirical estimation of the models' energy consumption and the associated carbon footprints is often overlooked. In this study, we demonstrate how practitioners can estimate energy consumption and arrive at reliable emission estimates for medical QA tasks available in the OpenMedical-LLM leaderboard by investigating 13 LLMs from 3 model families ranging from 270M to 14B parameters. We discuss how one can critically interpret energy versus accuracy trade-offs for a set of models for a given task. With this work, we underline that reporting disjoint energy and accuracy metrics alone does not suffice for careful trade-off assessments, but it also demands systematic construction and interpretation of joint trade-off metrics. We envisage such studies aiding practitioners in adopting better and more efficient models for their task and helping them become active participants in building sustainable LLM pipelines.

Figures & tables

Figure 1: Example Pareto-optimal Frontier.

Model	Accuracy ( $\uparrow$ )		Energy (kWh) ( $\downarrow$ )		CO2 Emission (kg) ( $\downarrow$ )
Model	Pre-trained	Fine-tuned	Pre-trained	Fine-tuned	Pre-trained	Fine-tuned
qwen3-0.6b	48.73 $\pm$ 3.33	49.52 $\pm$ 3.33	0.022	0.031	0.008	0.012
qwen3-1.7b	54.83 $\pm$ 3.29	62.23 $\pm$ 3.18	0.023	0.036	0.009	0.014
qwen3-4b	68.56 $\pm$ 3.08	74.21 $\pm$ 2.75	0.04	0.054	0.015	0.02
qwen3-8b	67.13 $\pm$ 3.00	76.89 $\pm$ 2.65	0.066	0.064	0.025	0.024
qwen3-14b	74.73 $\pm$ 2.74	81.03 $\pm$ 2.45	0.077	0.089	0.029	0.034

Table 1: Accuracy, Energy Consumption and CO2 Emission Results

Figure 2: Accuracy and CO2 Emissions versus Model Size

Model	Relative Gain	Relative Costs
Family	(Accuracy)	(Energy)
Qwen3	+64%	+187%
Gemma2	+34%	+95%
Gemma3	+134%	+146%
Llama3	+61%	+186%

Table 2: Comparison of relative accuracy gains versus relative energy costs

Figure 3: Interpreting Energy-Accuracy Trade-offs with Pareto-optimal frontier (left) and CIRC rating (right).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Type	# Train	# Validation	# Test	Evaluation
		Samples	Samples	Samples	Metric
MedQA	Open MCQA	10178	1272	1273	log-likelihood over
					A / B / C / D
MedMCQA	Open MCQA	18000	4183	2000	log-likelihood over
					A / B / C / D
PubMedQA	Closed MCQA	900	50	500	log-likelihood over

Appendix

Table 3: Datasets included in Open Medical-LLM Leaderboard

Model	Runtime	Energy	CO2 Emission
Model	(s)	(kWh)	(kg)
qwen3-0.6b	2767.698	0.255	0.097
qwen3-1.7b	3704.089	0.353	0.134
qwen3-4b	10920.611	1.071	0.408
qwen3-8b	20619.091	1.986	0.756
qwen3-14b	27956.034	2.752	1.048

Appendix

Table 4: Estimates of energy consumption and CO2 emission for fine-tuning runs

Model	PubMedQA	MedMCQA	MedQA	MMLU Subsets	Avg. Accuracy
qwen3-0.6b	70.20 $\pm$ 2.05	38.82 $\pm$ 0.75	38.57 $\pm$ 1.36	48.50 $\pm$ 3.88	48.73 $\pm$ 3.33
qwen3-1.7b	72.60 $\pm$ 2.00	48.10 $\pm$ 0.77	48.08 $\pm$ 1.40	54.11 $\pm$ 3.83	54.83 $\pm$ 3.29
qwen3-4b	75.80 $\pm$ 1.92	57.09 $\pm$ 0.77	62.06 $\pm$ 1.36	70.34 $\pm$ 3.56	68.56 $\pm$ 3.08
qwen3-8b	76.20 $\pm$ 1.91	62.47 $\pm$ 0.75	65.83 $\pm$ 1.33	66.62 $\pm$ 3.48	67.13 $\pm$ 3.00
qwen3-14b	76.40 $\pm$ 1.90	66.34 $\pm$ 0.73	71.25 $\pm$ 1.27	76.43 $\pm$ 3.18	74.73 $\pm$ 2.74
gemma-2-2b	74.00 $\pm$ 1.96	40.86 $\pm$ 0.76	35.04 $\pm$ 1.34	51.78 $\pm$ 3.85	51.18 $\pm$ 3.30

Appendix

Table 5: Evaluation results of pre-trained models

Model	PubMedQA	MedMCQA	MedQA	MMLU Subsets	Avg. Accuracy
qwen3-0.6b	59.40 $\pm$ 2.20	43.08 $\pm$ 0.77	38.96 $\pm$ 1.37	50.71 $\pm$ 3.86	49.52 $\pm$ 3.33
qwen3-1.7b	72.40 $\pm$ 2.00	50.01 $\pm$ 0.77	48.78 $\pm$ 1.40	64.82 $\pm$ 3.70	62.23 $\pm$ 3.18
qwen3-4b	77.80 $\pm$ 1.86	60.29 $\pm$ 0.76	63.08 $\pm$ 1.35	77.79 $\pm$ 3.16	74.21 $\pm$ 2.75
qwen3-8b	79.40 $\pm$ 1.81	64.36 $\pm$ 0.74	68.50 $\pm$ 1.30	79.96 $\pm$ 3.06	76.89 $\pm$ 2.65
qwen3-14b	79.80 $\pm$ 1.80	68.04 $\pm$ 0.72	72.27 $\pm$ 1.26	84.86 $\pm$ 2.78	81.03 $\pm$ 2.45
gemma-2-2b	75.60 $\pm$ 1.92	44.23 $\pm$ 0.77	41.08 $\pm$ 1.38	57.50 $\pm$ 3.85	56.21 $\pm$ 3.29

Appendix

Table 6: Evaluation results of fine-tuned models

Model	Mode	$Acc$	$Acc_{norm}$	$Energy$	$Eff_{norm}$	Distance	Rating
qwen3-0.6b	Pre-trained	48.7	0.424	0.022	0.871	0.590	3
qwen3-0.6b	Fine-tuned	49.5	0.439	0.031	0.757	0.612	3
qwen3-1.7b	Pre-trained	54.8	0.533	0.023	0.858	0.488	4
qwen3-1.7b	Fine-tuned	62.2	0.665	0.036	0.686	0.459	4
qwen3-4b	Pre-trained	68.6	0.779	0.040	0.637	0.425	4
qwen3-4b	Fine-tuned	74.2	0.879	0.054	0.460	0.554	4

Appendix

Table 7: CIRC Ratings

Energy-Aware LLM Inference

See all 39 papers

Measuring the Inference Energy Efficiency of Open-Weight LLMs across Benchmarks, Reasoning Modes, and Hardware
Oct 9, 2026Zeyu Yang, Wes ArmourEnergy-Aware LLM Inference LLM Inference Efficiency
Department of Engineering Science, University of Oxford
From Small to Large: Assessing Medical Reasoning, Benchmark Contamination, and Model Size in Language Models
Oct 30, 2026Navapat Nananukul, Mayank KejriwalMedical LLMs Medical QA
Information Sciences Institute, University of Southern California, Marina del Rey, CA, USA · Lead contact
Evaluating Small Open LLMs for Medical Question Answering: A Practical Framework
Apr 12, 2026Avi-ad Avraam BuskilaLLM Evaluation LLM Reliability
Department of Information Science and Applied Artificial Intelligence, Bar-Ilan University, Ramat-Gan, Israel

Oct 9, 2026J/K move · Enter open · S save

Measuring the Inference Energy Efficiency of Open-Weight LLMs across Benchmarks, Reasoning Modes, and Hardware

Zeyu Yang, Wes Armour

Department of Engineering Science, University of Oxford

Energy-Aware LLM Inference LLM Inference Efficiency Energy-Efficient ML Open-Weight Language Models LLM Evaluation Efficient Language Model Reasoning

Open paper AskPDF arXiv

Benchmark scores tell us what an LLM can do, but not how much energy is required to deliver that capability. We directly measure this trade-off for 86 open-weight model checkpoints, evaluated on seven benchmarks and six hardware platforms ranging from a consumer GPU to an eight-GPU server. We find large differences in energy efficiency even among models with similar capability. Most configurations lie away from the score--energy Pareto frontier: they use a median of 2.6--4.8

\times

more energy than another configuration that scores at least as well, with differences reaching 27

\times

. Reasoning can improve capability, but enabling thinking increases energy by about 3.7

\times

on average, and the benefit depends strongly on the model family and output budget. Generated-token count is also an imperfect proxy for energy, misordering 18--24% of model pairs. In contrast, FP8 reduces energy by roughly a quarter with little change in score, while changing hardware largely rescales energy without changing the relative ranking of models. These results show that capability alone gives an incomplete picture of efficiency. We argue that benchmark scores should be reported together with measured energy consumption under a specified serving regime and output budget.

Abstract

Energy-Aware LLM Inference

Explore similar work

Measuring the Inference Energy Efficiency of Open-Weight LLMs across Benchmarks, Reasoning Modes, and Hardware

From Small to Large: Assessing Medical Reasoning, Benchmark Contamination, and Model Size in Language Models

Evaluating Small Open LLMs for Medical Question Answering: A Practical Framework