Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

Jon Saad-Falcon; Avanika Narayan; Hakki Orhun Akengin; J. Wes Griffin; Herumb Shandilya; Adrian Gamarra Lafuente; Medhya Goel; Rebecca Joseph

cs.DCNov 11, 2025

Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

Authors: Jon Saad-Falcon, Avanika Narayan, Hakki Orhun Akengin, J. Wes Griffin, Herumb Shandilya, Adrian Gamarra Lafuente, Medhya Goel, Rebecca Joseph, +7 more

Organizations: Stanford University · Together AI

Abstract

Large language model (LLM) queries are predominantly processed by frontier models in centralized cloud infrastructure. Demand growth strains this paradigm faster than providers can scale. Two advances create an opportunity to rethink it: small, local LMs (<=20B active parameters) now achieve competitive performance to frontier models on many tasks, and local accelerators (e.g., Apple M4 Max) can host these models at interactive latencies. This raises the question: can local inference viably redistribute demand from centralized infrastructure? This requires measuring both whether local LMs can accurately answer real-world queries and whether they can do so efficiently on power-constrained devices (e.g., laptops). We propose intelligence per watt (IPW), task accuracy per unit of power, as a unified metric for the capability and efficiency of local inference across model-accelerator configurations. We evaluate 20+ state-of-the-art local LMs, 8 hardware accelerators (local and cloud), and 1M real-world single-turn chat and reasoning queries. For each query, we measure accuracy (local LM win rate against frontier models), energy, latency, and power. We find three key results. First, local LMs successfully answer 88.7% of these queries, with accuracy varying by domain. Second, longitudinal analysis from 2023-2025 shows IPW improved 5.3x, driven by both algorithmic and accelerator advances, with locally-serviceable query coverage rising from 23.2% to 71.3%. Third, local accelerators achieve at least 1.4x lower IPW than cloud accelerators running identical models, revealing significant headroom for local accelerator optimization. These findings demonstrate that local inference can meaningfully redistribute demand from centralized infrastructure for a substantial subset of queries, with IPW serving as the critical metric for tracking this transition.

Figures & tables

Figure 1: Intelligence per Watt: A Study of Local Intelligence Efficiency. We present the first systematic study of local AI inference efficiency across models, hardware, and real-world workloads. (Left) Intelligence efficiency is defined as task accuracy per unit of power , capturing both capabilities delivered and energy consumed. (Left-Middle) We conduct comprehensive performance profiling across 20+ state-of-the-art local LMs ( $\leq 20B$ active parameters), diverse hardware accelerators ( Apple , NVIDIA , AMD ), multiple performance metrics, and 1M+ real-world queries spanning chat and reasoning tasks. (Right-Middle) Local LM capabilities are improving rapidly : win/tie rate versus frontier models increases from $23.2\%$ (2023) to $71.3\%$ (2025), a $3.1\times$ improvement in accuracy, demonstrating that local models can accurately handle significant portions of single-turn chat and reasoning queries. (Right) Intelligence per watt improves $5.3\times$ from 2023–2025 , driven by advances in both model architectures and hardware accelerators, with local accelerators showing $1.5\times$ efficiency headroom compared to enterprise-grade systems.

Dataset Origin	Category	$\|N\|$
Wildchat	Chat	500K
NaturalReasoning	Reasoning	500K
MMLU Pro	Knowledge	12K
SuperGPQA	Grad. Reasoning	26.5K

Table 1: Dataset Overview. (Left) Query composition with sizes. (Right) Models and Accelerators.

Figure 2: Local Models Rival Cloud Models Across Diverse Benchmarks: Individual model performance scales with size, ranging from 31.5–69.4% for IBM Granite4-H-Small , 30.0–83.6% for Gemma3-12B , 51.5–80.4% for GPT-OSS-120B , and 66.5–89.5% for Gemini 2.5 Pro . Local routing (best local LM per query) achieves 97.8%, 88.3%, 77.0%, and 92.4% on Wildchat , NaturalReasoning , SuperGPQA , and MMLU Pro respectively, surpassing cloud routing (100%, 82.9%, 66.5%, 87.4%) on three of four benchmarks.

	2023	2024	2025
SOTA Local Model	Mixtral-8x7B-v0.1	Llama-3.1-8B-Instruct	GPT-OSS-120B
SOTA Accelerator	NVIDIA Quadro RTX 6000	NVIDIA RTX 6000 Ada	Apple M4 Max
Success Rate	$23.2\pm 1.9\%$	$48.7\pm 2.7\%$	$71.3\pm 2.2\%$
Intelligence per Watt	$(7.92\pm 0.32)$ $\times 10^{-4}$	$(1.80\pm 0.21)$ $\times 10^{-3}$	$(4.18\pm 0.53)$ $\times 10^{-3}$
YoY Efficiency Gain	—	2.27×	2.32×

Table 2: Increase in Intelligence per Watt for Local LMs : Accuracy per watt has improved over

5\times

in two years, driven by advances in both model architectures (from Mixtral-8x7B to GPT-OSS-120B ) and accelerator hardware (from NVIDIA Quadro RTX 6000 to Apple M4 Max). Values are mean

\pm

\sigma

standard deviation across measurement runs.

Figure 3: Increase in Intelligence per Joule for Local LMs and Accelerators : Efficiency improved $18.0\times$ over 16 months, decomposed into $3.0\times$ from local LMs and $5.9\times$ from local accelerators.

Figure 4: Energy, Compute, and Capital Gains from Model Routing . Cumulative resource consumption over 24 hours and 80.2M LLM queries [ 81 ] , routing between $4$ small LMs on Apple M4 Max (bs = $1$ ) and Qwen3-235B on an H200 (bs = $16$ , batched cloud baseline). The 80%-accurate router achieves 64.3% energy, 61.8% compute, and 59.0% cost savings versus routing all queries to Qwen3-235B , capturing the majority of gains achievable by the Oracle.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Life, physical, and social science	Computer and mathematical
Architecture and engineering	Education instruction and library
Installation, maintenance, and repair	Business and financial operations
Legal services	Transportation and material moving
Arts, design, sports, entertainment, and media	Production services
Farming, fishing, and forestry	Healthcare support
Food preparation and serving related	Healthcare practitioners and technical

Appendix

Table 3: Anthropic Economic Index Categories [ 27 ] . This taxonomy categorizes occupations into 22 standardized economic domains, adapted from U.S. Bureau of Labor Statistics frameworks. It is designed to support AI impact analysis by aligning labor categories with distinct task structures.

Figure 5: Local Win/Tie-Rate vs. Cloud LMs by Domain . Stacked bars show the fraction of single-turn chat and reasoning queries handled by local LMs ( $<20B$ active parameters; blue) versus those routed to frontier models in the cloud (red), computed per economic index domain [ 7 ]

Domain	WC Count	WC %	WC Solv %	NR Count	NR %	NR Solv %
Computer and mathematical	90,662	18.1	99.5	174,242	34.8	67.3
Arts, design, sports, entertainment, and media	235,658	47.1	98.7	2,648	0.5	52.9
Life, physical, and social science	28,079	5.6	98.8	180,065	36.0	60.5
None	49,014	9.8	97.3	79,752	16.0	65.6
Education instruction and library	23,196	4.6	97.2	13,864	2.8	80.4
Architecture and engineering	5,782	1.2	98.9	28,762	5.8	40.8

Appendix

Table 4: Dataset Domain Composition and LM Coverage (

\leq 20B

Active Parameter Models) . Comparison of domain distribution and model solvability rates across Wildchat (WC) and NaturalReasoning (NR) datasets. Solvability indicates the percentage of problems that can be solved correctly by at least one model with

\leq 20B

active parameters.

Category	Wildchat	MMLU Pro	SuperGPQA	Average
Computer and mathematical	93.4%	90.6%	72.8%	85.6%
Life, physical, and social science	91.1%	84.7%	50.4%	75.4%
Sales and related	86.8%	74.2%	64.3%	75.1%
Business and financial operations	89.3%	82.9%	52.5%	74.9%
Production services	89.7%	85.7%	48.8%	74.7%
Office and administrative support	88.5%	83.3%	44.8%	72.2%

Appendix

Table 5: GPT-OSS-120B Performance across Datasets. Performance metrics across Wildchat , NaturalReasoning , MMLU Pro , and SuperGPQA benchmarks, organized by Anthropic Economic Index categories [ 27 ] .

Metric	Description
flops_per_request	FLOPs per query.
macs_per_request	MACs per query; proxy for compute.
per_query_joules	Energy per query (J).
total_joules	Total energy across queries.
per_token_ms	Latency per token (ms).
throughput_tokens_per_sec	Token output rate (toks/s).

Appendix

Table 6: Dataset Metrics . Summary of compute, latency, memory, and energy profiling metrics.

Hardware	Memory	Bandwidth	Power
NVIDIA A100 (Ampere)	40 GB HBM2	1,555 GB/s	400 W TDP
NVIDIA H200 (Hopper)	141 GB HBM3e	4.8 TB/s	Up to 700 W TDP
NVIDIA B200 (Blackwell)	192 GB HBM3e	8 TB/s	1000 W TDP
NVIDIA GH200 (Grace Hopper)	144 GB HBM3e (+624 GB LPDDR5X)	4.8 TB/s (GPU)	1000 W TDP
NVIDIA Quadro RTX 6000 (Turing)	24 GB GDDR6	672 GB/s	295 W TDP
NVIDIA RTX 6000 Ada Generation	48 GB GDDR6	960 GB/s	300 W TDP

Appendix

Table 7: Accelerator Details . Memory, bandwidth, and power specifications of evaluated accelerators and systems.

\dagger

For the Apple Mac Studio (M4 Max), 480 W is the system power-supply continuous rating; the M4 Max SoC under sustained AI workloads draws substantially less. Per-query power measurements in our study are taken via powermetrics processor_power.actual (GPU subsystem only) so that comparisons against NVML/ROCm-SMI accelerator-only power on other platforms are like-for-like (see App. B.1 ).

	Cost Savings		Compute Savings		Energy Savings
Size Threshold ( $\leq$ )	Qwen + GPT-OSS	Qwen	Qwen + GPT-OSS	Qwen	Qwen + GPT-OSS	Qwen
4B	65.2%	65.2%	65.1%	65.1%	63.5%	63.5%
8B	80.8%	80.8%	83.1%	83.1%	79.6%	79.6%
14B	89.0%	89.0%	93.0%	93.0%	87.0%	87.0%
20B	90.5%	—	97.4%	—	89.4%	—
32B	91.3%	91.9%	97.4%	92.8%	90.4%	90.5%

Appendix

Table 8: Cost, Compute, and Energy Savings from Local-Cloud Routing on WildChat : Savings across different resources while maintaining task accuracy of SOTA open-source cloud model (i.e. Qwen3 235B-A22B ).

	Cost Savings		Compute Savings		Energy Savings
Size Threshold ( $\leq$ )	Qwen + GPT-OSS	Qwen	Qwen + GPT-OSS	Qwen	Qwen + GPT-OSS	Qwen
4B	52.9%	52.9%	54.5%	54.5%	46.3%	46.3%
8B	60.5%	60.5%	62.5%	62.5%	54.0%	54.0%
14B	68.7%	68.7%	70.1%	70.1%	62.5%	62.5%
20B	73.3%	—	72.2%	—	67.8%	—
32B	76.9%	75.9%	75.1%	75.1%	72.4%	71.6%

Appendix

Table 9: Cost, Compute, and Energy Savings from Local-Cloud Routing on NaturalReasoning : Savings across different resources while maintaining task accuracy of SOTA open-source cloud model (i.e. Qwen3 235B-A22B ).

Figure 6: Chat Task Performance by Difficulty Level and Year . Model success rates across four difficulty levels and three model generations (2023, 2024, 2025). The data reveals dramatic progress across all difficulty levels, with 2023 models achieving 28.79% overall success rising to 98.12% by 2025. Notably, Levels 1-3 approach near-perfect performance (98-99%), while Level 4 shows the largest relative improvement (+210.4% per year) despite starting from the lowest baseline (17.77%).

Figure 7: Reasoning Task Performance by Difficulty Level and Year . Model success rates on across five difficulty levels and three model generations. The benchmark shows a three-tier saturation pattern: near-complete (98-99% on Levels 1-2), approaching saturation (85-92% on Levels 3-4), and wide-open frontier (51% on Level 5).

Figure 8: Perplexity and Accuracy per Joule Trends across Wildchat and NaturalReasoning .

Model	Input Cost (USD / 1M tokens)	Output Cost (USD / 1M tokens)
Qwen3-4B	0.000	0.000
Qwen3-8B	0.035	0.138
Qwen3-14B	0.060	0.124
Qwen3-32B	0.100	0.450
Qwen3-235B	0.220	0.880
GPT-OSS-20B	0.03	0.14

Appendix

Table 10: Model pricing from OpenRouterAI. Costs are in USD per 1M tokens for input and output, as of August 2025.

Figure 9: Minimal Accuracy Degradation Shifting from FP16 to FP4 for Open-Source Local Models : Evaluation across three reasoning datasets ( $N=10,000$ each) shows $2-3\%$ accuracy loss per precision step, demonstrating that $FP8$ / $FP4$ quantization enables efficient deployment with acceptable performance tradeoffs.

Model	Type	WildChat	NaturalReasoning	MMLU Pro	SuperGPQA	Average
gpt-5-2025-08-07	Closed	81.9%	82.9%	86.5%	64.4%	78.9%
gemini-2.5-pro	Closed	89.5%	77.9%	87.4%	66.5%	80.3%
claude-sonnet-4-5	Closed	88.1%	76.9%	86.4%	60.1%	77.9%
Qwen3-235B-A22B (Best OSS)	Open	N/A ∗	70.0%	82.3%	63.1%	71.8%
Qwen3-32B	Open	76.1%	69.7%	77.9%	56.5%	70.1%
gpt-oss-120b	Open	89.2%	65.0%	78.3%	55.3%	72.0%

Appendix

Table 11: Model performance comparison across benchmarks. Scores represent performance on Wildchat , NaturalReasoning , MMLU Pro , and SuperGPQA benchmarks. ∗ Qwen3-235B-A22B is used as the reference model for Wildchat evaluation.

Metric	WildChat	NaturalReasoning	MMLU Pro	SuperGPQA
Closed Best	89.5%	82.9%	87.4%	66.5%
Best Closed Model	gemini-2.5-pro	gpt-5	gemini-2.5-pro	gemini-2.5-pro
Open Best	89.2% *	70.0%	82.3%	63.1%
Best Open Model	gpt-oss-120b	Qwen3-235B-A22B	Qwen3-235B-A22B	Qwen3-235B-A22B
Gap	$-0.3\%$	$-12.9\%$	$-5.1\%$	$-3.4\%$
Local Best ( $\leq 20B$ Active)	89.2%	67.3%	80.3%	50.5%

Appendix

Table 12: Closed-Source vs. Open-Source Performance Gap by Task. Comparison between closed-source, open-source (all sizes), and local (

\leq 20B

active parameters) models. * Excludes Qwen3-235B-A22B (reference model for WildChat).

Figure 10: Open-Source Local LMs Performance vs. U.S. GDP - SuperGPQA and MMLU Pro : Model accuracy on SuperGPQA and MMLU Pro benchmarks plotted against relevant GDP in trillions of dollars. Both benchmarks show continued performance improvements as training compute scales across models from Qwen3B-4B to Qwen3B-A22B-235B. For our calculations, we compute the weighted sum of an LM’s accuracy on each U.S. Labor category vs. the U.S. GDP associated with that category.

Figure 11: Open-Source Local LMs Performance vs. U.S. GDP - WildChat and Natural Reasoning : Model accuracy on WildChat and Natural Reasoning benchmarks plotted against relevant GDP in trillions of dollars. Both benchmarks show continued performance improvements as training compute scales across models from Qwen3B-4B to Qwen3B-A22B-235B. For our calculations, we compute the weighted sum of an LM’s accuracy on each U.S. Labor category vs. the U.S. GDP associated with that category.

	Qwen3-4B	Qwen3-8B	Qwen3-14B	Qwen3-32B
Success Rate	$49.3\pm 1.8\%$	$57.5\pm 2.5\%$	$59.5\pm 1.4\%$	$69.5\pm 2.3\%$
Apple M4 Max
Intelligence per Watt	$(1.40\pm 0.38)$ $\times 10^{-3}$	$(1.63\pm 0.20)$ $\times 10^{-3}$	$(1.69\pm 0.31)$ $\times 10^{-3}$	$(1.97\pm 0.24)$ $\times 10^{-3}$
NVIDIA B200
Intelligence per Watt	$(1.95\pm 0.14)$ $\times 10^{-3}$	$(2.27\pm 0.18)$ $\times 10^{-3}$	$(2.35\pm 0.09)$ $\times 10^{-3}$	$(2.75\pm 0.14)$ $\times 10^{-3}$
SambaNova SN40L

Appendix

Table 13: Local accelerators demonstrate lower power efficiency than cloud accelerators : When running the same Qwen3 models, the Apple M4 Max (local) attains

1.40\times

lower intelligence per watt compared to the NVIDIA B200 (cloud) and SambaNova SN40L (cloud), highlighting the efficiency advantage of purpose-built cloud accelerators over local accelerators. Values are mean

\pm

1-sigma sample standard deviation across 3–5 independent measurement runs per cell.

	Qwen3-8B	Qwen3-32B	GPT-OSS-20B	GPT-OSS-120B
Apple M4 Max
Intelligence per Joule	$(3.80\pm 0.40)$ $\times 10^{-5}$	$(3.51\pm 0.38)$ $\times 10^{-5}$	$(4.38\pm 0.31)$ $\times 10^{-5}$	$(4.23\pm 0.45)$ $\times 10^{-5}$
NVIDIA B200
Intelligence per Joule	$(8.71\pm 0.60)$ $\times 10^{-5}$	$(5.91\pm 0.42)$ $\times 10^{-5}$	$(7.34\pm 0.51)$ $\times 10^{-5}$	$(6.78\pm 0.47)$ $\times 10^{-5}$
SambaNova SN40L
Intelligence per Joule	—	$(2.27\pm 0.30)$ $\times 10^{-4}$	—	$(3.12\pm 0.38)$ $\times 10^{-4}$

Appendix

Table 14: Cloud accelerators demonstrate superior energy efficiency across all models : The NVIDIA B200 (cloud) achieves

1.6\times

2.3\times

higher intelligence per joule than the Apple M4 Max (local), while the SambaNova SN40L (cloud) achieves

6.5\times

7.4\times

higher efficiency. These results highlight the substantial energy efficiency advantage of purpose-built cloud accelerators over local hardware across Qwen3 and GPT-OSS model variants. Values are mean

\pm

1-sigma sample standard deviation across 3–5 independent measurement runs per cell.

Figure 12: Rapid Improvement of Local LMs across Chat and Reasoning Queries : We evaluate the performance of SOTA local models released between April 2024 and August 2025 on Wildchat and NaturalReasoning . On Wildchat (left), local models show a win/tie rate of 78.2% against Qwen3-235B as of August 2025, compared to just 28.0% in April 2024: a 2.8

\times

improvement in 16 months. On NaturalReasoning (right), local models achieve 80.9% accuracy by August 2025, up from 48.7% in April 2024: a 66% relative improvement.

Figure 13: Increasing GPU Memory of Consumer Accelerators : Memory capacity (GB) for local accelerators. Over the past decade, local hardware has significantly closed the memory gap with cloud-grade accelerators, particularly since 2020, driven by advances in high bandwidth memory (HBM) components and unified memory architectures.

Model	bs=1 IPJ	bs=64 IPJ	IPJ Gain	Architecture
	( $\times 10^{-5}$ )	( $\times 10^{-5}$ )
Qwen3-8B	8.92	104.93	$11.8\times$	Dense
Qwen3-14B	7.22	80.61	$11.2\times$	Dense
GPT-OSS-120B	6.72	132.21	$19.7\times$	MoE ( $\leq 20$ B active)

Appendix

Table 15: Cloud Batching Ablation on NVIDIA B200 : Per-query intelligence-per-joule at bs=1 versus bs=64. Per-query energy drops

12

–

20\times

at bs=64 (e.g., Qwen3-8B :

6{,}449

\to 548

J) while GPU power scales sublinearly (

255

W at bs=1 to

883

W at bs=64). This confirms that bs=1 is conservative for cloud accelerators. Local accelerators (M4 Max, iPhone 16 Pro) serve a single user and cannot batch, so local IPW results are unchanged.

Model	vLLM	SGLang	llama.cpp
	IPW ( $\times 10^{-3}$ )	IPW ( $\times 10^{-3}$ )	IPW ( $\times 10^{-3}$ )
Qwen3-4B	1.40	1.35	1.52
Qwen3-8B	1.63	1.58	1.71
Qwen3-14B	1.69	1.62	1.78
GPT-OSS-120B	4.18	4.05	4.31

Appendix

Table 16: Framework Sensitivity on Apple M4 Max : Intelligence-per-watt across three popular inference frameworks for a representative subset of local models. Absolute IPW values shift by

3

–

12\%

across frameworks, but relative rankings between model-hardware pairs are preserved across the broader

20+

-model,

8

-accelerator evaluation: Kendall’s

\tau\in[0.87,0.93]

and Spearman’s

\rho\in[0.89,0.94]

across all framework pairs. This validates standardizing on vLLM for comparative analysis and confirms that our key findings are not an artifact of framework selection. Models that prioritize absolute throughput on a target stack should benchmark on that stack directly.

Model	Precision	Hardware	Acc.	Power	Lat.	IPW	IPJ
			(%)	(W)	(s/q)	( $\times 10^{-3}$ )	( $\times 10^{-5}$ )
Smartphone-class accelerator (Apple A18 Pro, iPhone 16 Pro)
Qwen3-4B	FP16	A18 Pro	42.5 $\pm$ 2.0	12.0 $\pm$ 0.4	92.5 $\pm$ 9.4	11.8 $\pm$ 0.7	38.3 $\pm$ 3.9
Qwen3-4B	FP8	A18 Pro	40.5 $\pm$ 1.8	11.0 $\pm$ 0.3	55.2 $\pm$ 5.6	12.4 $\pm$ 0.7	66.7 $\pm$ 6.8
Qwen3-4B	FP4	A18 Pro	38.0 $\pm$ 1.6	9.5 $\pm$ 0.3	36.8 $\pm$ 3.7	13.3 $\pm$ 0.8	108.7 $\pm$ 11.0
Gemma3-4B	FP4	A18 Pro	32.0 $\pm$ 1.5	8.8 $\pm$ 0.3	33.5 $\pm$ 3.5	11.6 $\pm$ 0.7	108.5 $\pm$ 11.2

Appendix

Table 17: Smartphone-Class Accelerator Efficiency : Per-query measurements on Apple A18 Pro (iPhone 16 Pro,

{\sim}12

W SoC peak, 60 GB/s memory bandwidth, 8 GB unified memory) across three precision levels, with workstation references for the same model at the strongest precision the iPhone supports. Values are mean

\pm

\sigma

standard deviation across 3–5 independent measurement runs. Accuracy is averaged across the same evaluation subset used in App. E ; latency is wall-clock per query at

\text{bs}{=}1

. The IPW ratio between A18 Pro and workstation GPUs on the same model anchors the

{\sim}7\times

per-watt advantage of smartphone-class accelerators (

13.3/1.85=7.2\times

vs. M4 Max;

13.3/2.02=6.6\times

vs. RTX 6000 Ada); the IPJ comparison shows the gap widens slightly per joule against M4 Max (

108.7

vs.

53.0\times 10^{-5}

) but narrows against RTX 6000 Ada (

108.7

vs.

80.9\times 10^{-5}

), reflecting that the A18 Pro’s

9

–

26\times

higher latency partially offsets its

20

–

25\times

lower power. Modest accuracy degradation on A18 Pro (

\sim

6.5pp on Qwen3-4B FP4) reflects CoreML INT4 group quantization plus KV-cache pressure under 8 GB unified memory.

Benchmark	Model	Hardware	Acc.	Power	Lat.	Energy	IPW	IPJ
			(%)	(W)	(s/q)	(kJ/q)	( $\times 10^{-3}$ )	( $\times 10^{-5}$ )
GAIA (165 multi-turn general-assistant queries with tool use)
GAIA	MiniMax-M2.5	8 $\times$ H100	16.4 $\pm$ 2.9	1558 $\pm$ 95	4.69 $\pm$ 0.41	7.31 $\pm$ 0.62	0.105 $\pm$ 0.020	2.24 $\pm$ 0.39
GAIA	Qwen3-235B	8 $\times$ H100	5.5 $\pm$ 1.8	1595 $\pm$ 88	0.74 $\pm$ 0.09	1.18 $\pm$ 0.13	0.034 $\pm$ 0.012	4.66 $\pm$ 1.55
GAIA	Qwen3-30B	8 $\times$ H100	8.6 $\pm$ 2.2	822 $\pm$ 64	1.22 $\pm$ 0.14	1.00 $\pm$ 0.11	0.105 $\pm$ 0.027	8.60 $\pm$ 2.20
GAIA	MiniMax-M2.5	M4 Max	14.2 $\pm$ 2.7	305 $\pm$ 24	51.6 $\pm$ 5.4	15.74 $\pm$ 1.69	0.466 $\pm$ 0.092	0.90 $\pm$ 0.18

Appendix

Table 18: Multi-Turn Agentic Workloads Generalize Single-Turn Patterns : Per-query measurements on GAIA (165 general-assistant queries with tool use, OpenHands agent) and TerminalBenchV2 (80 terminal-task queries, Terminus 2 agent) for self-hosted open-source models on local (Apple M4 Max, 128 GB unified memory) and cloud (8

\times

NVIDIA H100 80 GB SXM5 node) hardware. Values are mean

\pm

\sigma

standard deviation across 3 independent runs per cell; accuracy uncertainty uses the binomial approximation over benchmark size. M4 Max models use GGUF quantization via Unsloth to fit within unified memory; cloud models run at native precision via vLLM. The qualitative patterns from single-turn evaluation persist: cloud hardware achieves

2.4

–

3.0\times

higher per-joule efficiency on identical models (e.g., MiniMax-M2.5 on TBv2:

17.30

vs.

5.70\times 10^{-5}

), while M4 Max achieves

3.7

–

3.8\times

higher per-watt efficiency due to its

3.5

–

5.3\times

lower power envelope, with only

\sim

2.2pp accuracy degradation across both benchmarks. The 12–15

\times

higher M4 Max latency reflects the same memory-bandwidth bottleneck observed in single-turn evaluation, with multi-turn workloads amplifying absolute differences but preserving relative rankings.

Explore similar work

Jul 23, 2026cs.AI

Profiling Lightweight Large Language Models

Lightweight large language models (LLMs) are increasingly being deployed locally on personal computers and are expected to play a growing role in resource-constrained edge and mobile environments. In such settings, energy consumption, execution time, and memory usage directly affect practical usability, yet existing evaluations of LLM efficiency largely rely on proxy descriptors such as parameter count or FLOPs, often decoupled from task precision. This paper introduces a PTME-based experimental framework for the precision-aware profiling of lightweight LLM inference, jointly measuring Precision, execution Time, peak Memory usage, and Energy consumption through direct hardware-level measurements. The methodology is applied to a representative set of lightweight LLMs executed locally under edge-class resource envelopes on a controlled desktop platform, using benchmarks spanning code generation, mathematical reasoning, and multi-task understanding. We find that static proxy descriptors approximate inference cost well but fail to predict precision. Tightening the resource envelope increases cost without affecting precision, amplifying execution time more strongly than energy and penalizing larger models the most. Moreover, no single model dominates across all PTME dimensions, and a Pareto analysis reveals non-dominated configurations that would be hidden by accuracy-only or efficiency-only assessments, providing practical guidance for selecting models under different resource envelopes. These results show that selecting lightweight LLMs by size, FLOPs, latency, or accuracy alone can select the wrong deployment candidate; PTME profiling exposes configurations that preserve useful accuracy at lower physical cost.

Tomohiro Harada, Enrique Alba, Gabriel Luque

Graduate School of Science and Engineering, Saitama University, Saitama, Japan

LLM Inference Optimization Profiling Lightweight Resource-Constrained Edge Devices

Ask PDF

Sep 14, 2026cs.PF

The Battery Price of edge AI: A study of the Environmental Impact of LLM Inference on Mobile Devices

The rapid diffusion of generative artificial intelligence raises privacy, latency, and performance concerns that motivate a shift toward "local-first" AI, where inferences are performed on the user's device instead of on remote cloud servers. This paradigm also places a significant computational load on battery-powered smartphones, potentially shortening battery life and increasing the overall replacement rate of mobile devices. This paper presents a systematic study of the energy consumption, performance, and accuracy of on-device large language model (LLM) inference. We evaluate 18 models from different model families, sizes, and quantization levels, on two modern smartphones and on a server, using the respective state-of-the-art for such deployments. We measure the energy per generated token, inter-token latency, model accuracy, and battery-cycle consumption. Our results show that (i) on-device inference is on average 3 times less energy-efficient than batched server inference; (ii) the relationship between quantization bit-width and energy per token is non-monotonic, with energy sweet spots on both tested smartphones; (iii) eight out of 18 model configurations lie on the Pareto front of accuracy and energy-efficiency, allowing practitioners to build battery-aware model routers; and (iv) realistic modeling assumptions do not allow local inference to be less environmentally impacting per token than batched server inference, with 88--90% of that impact attributable to device embodied carbon rather than electricity consumption. These findings challenge the premise that local AI is more sustainable than cloud inference, and motivate the need for context-aware and life-cycle-aware model selection when deploying edge AI on battery-powered mobile platforms.

Édouard Guégain, Tristan Coignion

DRI, GREENSPECTOR · Univ. Bordeaux, CNRS, Bordeaux INP, LaBRI, UMR 5800, Talence, France

LLM Inference Optimization Device Artificial Intelligence Systems Computational Cost Generative Artificial Intelligence Environment

Ask PDF

Jul 2, 2026cs.DC

WattGPU: Predicting Inference Power and Latency on Unseen GPUs and LLMs

Large Language Model (LLM) inference workloads are a rapidly growing contributor to data center energy consumption. Optimizing these deployments requires matching specific LLMs to the most efficient GPUs, but operators currently lack the tools to do so without exhaustively profiling each combination. While some predictive models exist, they still require profiling data and struggle to generalize to hardware unseen during training. To address this, we introduce \textit{WattGPU}, featuring two predictive models for mean GPU power draw and Inter-Token Latency (ITL). Our approach leverages only publicly available LLM metadata and GPU specifications, eliminating the need for hardware access or profiling while enabling generalization to unseen NVIDIA server-grade GPUs and LLMs. We evaluate our models using rigorous leave-one-GPU-out and leave-one-LLM-out cross-validation on a dataset of 42 open-source LLMs (0.1B--27B parameters) and 8 GPUs under both offline and server scenarios. The mean power draw model achieves a median absolute percentage error of

\leq3.4\%

for offline and

\leq13.5\%

for server scenarios on unseen GPUs, while the latency model achieves

\leq8.5\%

in server mode, both maintaining strong GPU ranking correlations for server scenarios (Kendall

τ\geq0.76

). Compared to standard physically grounded baselines -- Load-Scaled Thermal Design Power (TDP) for power draw and roofline for latency -- our models reduce median absolute percentage error by approximately 4

\times

on unseen LLM-GPU combinations for server scenarios or approximately 2

\times

for completely unseen GPUs. WattGPU's data and code are publicly available at https://github.com/maufadel/wattgpu.

Mauricio Fadel Argerich, Jonathan Fürst, Marta Patiño-Martínez

Universidad Politécnica de Madrid, C. de los Ciruelos, 28660 Boadilla del Monte, Madrid, Spain · Zurich University of Applied Sciences, Gertrudstrasse 15, 8400 Winterthur, Switzerland

Graphics Processing Unit Architectures LLM Inference Optimization Inference Workloads Cpu-Gpu Hybrid Designs Time-To-First-Token Data Centers

Ask PDF