LLM Inference at the Edge: Mobile, NPU, and GPU Performance Efficiency Trade-offs Under Sustained Load

Pranay Tummalapalli; Sahil Arayakandy; Ritam Pal; Kautuk Kundan

cs.DCMar 24, 2026

LLM Inference at the Edge: Mobile, NPU, and GPU Performance Efficiency Trade-offs Under Sustained Load

Authors: Pranay Tummalapalli, Sahil Arayakandy, Ritam Pal, Kautuk Kundan

Organizations: Conscious Engines

Abstract

Deploying large language models on-device for always-on personal agents demands sustained inference from hardware tightly constrained in power, thermal envelope, and memory. We benchmark Qwen 2.5 1.5B (4-bit quantised) across four platforms: a Raspberry Pi 5 with Hailo-10H NPU, a Samsung Galaxy S24 Ultra, an iPhone 16 Pro, and a laptop NVIDIA RTX 4050 GPU. Using a fixed 258-token prompt over 20 warm-condition iterations per device, we measure throughput, latency, power, and thermal behaviour. For mobile platforms, thermal management supersedes peak compute as the primary constraint: the iPhone 16 Pro loses nearly half its throughput within two iterations, and the S24 Ultra suffers a hard OS-enforced GPU frequency floor that terminates inference entirely. On dedicated hardware, distinct constraints dominate: the RTX 4050 is bounded by its battery power ceiling, while the Hailo-10H is limited by on-module memory bandwidth. The RTX 4050 sustains 131.7 tok/s at 34.1 W; the Hailo-10H sustains 6.9 tok/s at under 2 W with near-zero variance, matching the RTX 4050 in energy proportionality at 19x lower throughput. Results should be interpreted as platform-level deployment characterisations for a single model and prompt type, reflecting hardware and software combined, rather than general claims about hardware capability alone.

Figures & tables

	RPi 5 + Hailo-10H	S24 Ultra	iPhone 16 Pro	RTX 4050 Laptop
SoC / CPU	BCM2712 4 $\times$ A76 @ 2.4 GHz	Snapdragon 8 Gen 3	Apple A18 Pro	Intel Core i5-13420H
AI Accel.	Hailo-10H NPU (40 TOPS, $<$ 5 W)	Hexagon NPU (45 TOPS)	Neural Engine 17-core ( $\sim$ 35 TOPS)	RTX 4050 (80 Tensor Cores)
RAM	8 GB LPDDR4X	12 GB LPDDR5X	8 GB Unified	16 GB DDR5
GPU	VideoCore VII	Adreno 750	6-core Apple GPU	RTX 4050 (2560 CUDA cores)
VRAM	Shared	Shared	Shared	6 GB GDDR6
OS	Raspberry Pi OS	Android 16	iOS 26	Ubuntu 22.04.3

Table 1: Hardware specifications across evaluated platforms.

	RPi 5 + Hailo-10H	S24 Ultra	iPhone 16 Pro	RTX 4050 Laptop
Framework	hailo-ollama	MLC-LLM 0.1.0	MLX Swift v0.29.1	vLLM (CUDA 12.1)
Backend	HailoRT 4.17.0	OpenCL 3.0 + TVM 0.14	Metal (MLX)	PyTorch 2.1.0
Language	Python 3.10	Java / NDK r26b	Swift 5.9 (SwiftUI)	Python 3.10
Model format	GGUF Q4_0	MLC binary q4f16_2	MLX safetensors Q4_0	GPTQ Int4 safetensors
Context window ‡	2,048	2,048	2,048 (KV)	2,048
Power monitor	INA219 (1 kHz)	BatteryManager (per-iter) ∗	UIDevice batteryLevel	nvidia-smi (100 ms)

Table 2: Software stack versions across platforms.

Attribute	Value
Parameters	1.5B
Architecture	Transformer decoder, GQA (2 groups)
Layers	28
Hidden size	1536
Attention heads	12
Vocabulary	151,936 tokens

Table 3: Qwen 2.5 1.5B model specifications.

Metric	Unit	Definition
Decode tokens	count	Tokens generated during the decode phase
Decode time	ms	Wall-clock duration of the decode phase
Prefill Time	ms	Time from prompt submission to first output token; equivalent to the duration of the prompt encoding phase
Throughput (TPS)	tok/s	$N_{\text{decode}}/t_{\text{decode}}$
Avg power	W	Mean system power draw over inference duration
Peak power	W	Maximum instantaneous power observed

Table 4: Metrics collected across platforms.

Panel of Figure 1 — Figure 1: Experimental protocol flowchart. Each platform undergoes this procedure independently. Cold-start prefill-time from the warm-up inference is recorded separately and excluded from throughput analysis.

Metric	Mean	Std Dev	Min	Max
Decode tokens	1789	0	1789	1789
Decode time (ms)	13,697	274	12,881	13,906
Throughput (tok/s)	131.70	2.87	128.65	138.88
Prefill time (ms)	1,998	42	1,894	2,028
Avg system power (W)	34.12	0.17	33.75	34.40
Peak system power (W)	35.28	0.21	35.05	35.73

Table 5: RTX 4050 inference performance (runs 2–20, warm condition). Power values derived from system-level measurement at 100 ms polling.

Figure 2: RTX 4050 per-run throughput (left axis) and GPU/CPU temperature (right axis) across runs 2–20. Mean throughput of 131.70 tok/s (CV = 2.2%) confirms stable battery-throttled performance; GPU rises from 55°C to 70°C with no throttling observed.

Metric	Mean	Std Dev	Min	Max
Decode tokens	564	0	564	564
Decode time (ms)	81,569	34	81,516	81,640
Throughput (tok/s)	6.914	0.003	6.908	6.919
Prefill time (ms)	1,287	1	1,286	1,290
Avg system power (W)	1.870	0.016	1.851	1.921
Peak system power (W)	3.065	1.099	2.383	6.662

Table 6: RPi 5 + Hailo-10H inference performance (runs 2–20, warm condition).

Figure 3: RPi 5 + Hailo-10H throughput (line, left axis) and NPU/CPU temperature (lines, right axis) across runs 2–20. Throughput CV of 0.04% and stable temperatures confirm no throttling at any point.

Metric	Mean	Std Dev	Min	Max
Decode tokens	819	0	819	819
Decode time (s)	32.19	3.69	20.23	35.06
Throughput (tok/s)	25.89	4.11	23.36	40.49
TTFT (s)	0.466	0.064	0.268	0.544
Prompt throughput (tok/s)	606.5	122.9	505.7	1027.5
Battery drain (total)	5% over 20 iterations (100% $\to$ 95%)

Table 7: iPhone 16 Pro inference performance (20 iterations, warm condition, greedy decoding, maxKVSize = 2,048).

Figure 4: iPhone 16 Pro per-iteration throughput across 20 iterations. Background shading indicates thermal state: green (Normal, iter 1–2), orange (Warm, iter 3–16), red (Hot, iter 17–20). Dashed line marks the sustained Hot-state mean of 23.67 tok/s.

Metric	Mean	Std Dev	Min	Max
Decode tokens	646	0.3	645	646
Decode time (ms)	59,908	4,122	52,906	67,619
Throughput (tok/s)	10.83	0.75	9.55	12.21
Prefill time (ms)	91,376	3,077	89,899	104,108
Max GPU temperature (°C)	64.6	1.9	61.1	68.5
Max CPU temperature (°C)	61.3	1.2	59.7	64.0

Table 8: Samsung S24 Ultra inference performance (20 iterations, warm condition, greedy decoding, 2,048-token context, 128-token prefill chunk). Power and energy figures are collected via the Android BatteryManager API with the display off; absolute accuracy remains lower than hardware-instrumented measurement.

Figure 5: Samsung S24 Ultra per-iteration throughput and temperature across all 20 iterations. Mean throughput of 10.83 tok/s (CV = 6.9%). No OS-enforced frequency floor observed in this run; GPU frequency varied between 720–1,000 MHz.

∗ Power not available: iOS exposes no per-component API.
Platform	TPS	CV	Avg Power (W)	W/tok/s	mJ/token
RTX 4050 †	$131.70\pm 2.87$	2.2%	34.12 †	0.259 †	297.3 †
iPhone 16 Pro (Hot)	$23.67\pm 0.28$	1.2%	∗	∗	∗
S24 Ultra (plateau)	$10.38\pm 0.44$	4.2%	1.486 §	0.143 §	143.0 §
RPi 5 + Hailo-10H	$6.914\pm 0.003$	0.04%	1.870 ‡	0.271 ‡	270.5 ‡
Energy figures are not directly comparable across platforms due to differing

Table 9: Cross-platform sustained inference performance and power efficiency summary. iPhone figures reflect sustained Hot state (iterations 17–20); S24 Ultra figures reflect the throttled plateau (iterations 8–20); RTX and Hailo figures reflect runs 2–20 after cold-start exclusion. † = battery-throttled; power reflects GPU-level draw via nvidia-smi . ‡ = whole-system draw via INA219. § = BatteryManager with display off; absolute accuracy caveated (Section 5 ). ∗ = not available due to platform measurement limitations.

Scenario	RTX 4050	iPhone 16 Pro	S24 Ultra	RPi + Hailo
Interactive assistant (AC-powered)	✓	$\sim$	$\sim$	$\sim$
Intermittent queries (5–10/hr)	$\sim$	✓	✓	✓
Sustained agent ( $>$ 20/hr)	$\sim$	×	$\sim$	✓
Battery-powered always-on	$\sim$	×	$\sim$	✓
Low-latency streaming response	✓	$\sim$	×	×

Table 10: Deployment scenario suitability based on empirical results. Ratings reflect observed platform behaviour under tested framework configurations only. ✓ = suitable based on observed thermal stability and throughput;

\sim

= marginal, either due to thermal degradation, low throughput, or framework limitations; × = unsuitable based on observed thermal failure or throughput floor. S24 Ultra ratings assume the display-off, 2,048-token context, 128-token prefill chunk configuration documented here.

On-Device Language Model Inference

On-device language model inference runs language models locally on phones, computers, and edge devices under tight memory, power, and latency limits.

See all 164 papers

Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference
Jul 6, 2026Guanyu Cai, Ruiming Tian, Lang Yang +4LLM Inference Efficiency AI Accelerator Inference
Tsinghua University · Beijing Jiaotong University
EnerInfer: Energy-Aware On-Device LLM Inference
Jun 22, 2026Bohua Zou, Nian Liu, Binqi Sun +6On-Device Language Model Inference LLM Inference Acceleration
Technical University of Munich · Shanghai Jiao Tong University · Huawei Hilbert Research Center (Dresden) +1
BaseRT: Advancing Best-in-Class LLM Inference with Apple M5 Neural Accelerators
Jul 21, 2026Fabian Waschkowski, Prabod Rathnayaka, Lukas WesemannAI Accelerator Inference On-Device Language Model Inference
Base Compute, Melbourne, Australia

Jul 6, 2026J/K move · Enter open · S save

Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference

Guanyu Cai, Ruiming Tian, Lang Yang, Zhouhong Ren+3

Tsinghua University · Beijing Jiaotong University

LLM Inference Efficiency AI Accelerator Inference On-Device Language Model Inference LLM Inference

Open paper AskPDF arXiv

Deploying Large Language Models (LLMs) on mobile devices enhances privacy and reduces latency, but is severely bottlenecked by hardware inefficiency. We present the first comprehensive, cross-layer measurement study of mobile LLM inference, uniquely spanning five mainstream frameworks (e.g., llama.cpp, GENIE) and three hardware backends (CPU, GPU, NPU). To enable this analysis, we develop PowerBench, a fine-grained profiling tool that provides the first backend-specific energy attribution, moving beyond traditional device-level measurements. Our study yields three critical insights: (1) Framework-induced performance gaps are substantially amplified on NPUs, reaching up to 10x using custom operators due to divergent offloading and quantization strategies. (2) We identify a distinct phase split where NPUs excel at compute-bound prefilling, while CPUs outperform all other backends in memory-bound decoding. This is driven by the NPU's preference for large, fixed-shape workloads, which conflicts with the small-kernel, dynamic nature of decoding. (3) Backend-specific profiling uncovers substantial scheduling headroom missed by prior work. Suboptimal thread configurations, uncoordinated NPU sleep latencies, and CPU polling intervals result in up to 40% energy waste. Leveraging these findings, we present an energy-oriented best-practice configuration for mobile LLM inference. We estimate that this configuration could reduce energy consumption by up to 54.8% on the NPU backend across three datasets.

Abstract

On-Device Language Model Inference

Explore similar work

Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference

EnerInfer: Energy-Aware On-Device LLM Inference

BaseRT: Advancing Best-in-Class LLM Inference with Apple M5 Neural Accelerators