Seeing at the Edge: Energy Characterization of Vision-Language Model Inference on Edge Platforms

Junlin Liao; Harshit Kondle; Di Liu; Zonghua Gu

cs.LGOct 14, 2026

Seeing at the Edge: Energy Characterization of Vision-Language Model Inference on Edge Platforms

Authors: Junlin Liao, Harshit Kondle, Di Liu, Zonghua Gu

Organizations: Hofstra University, Hempstead, NY, USA · Bridgewater-Raritan High School, Bridgewater, NJ, USA · Norwegian University of Science and Technology, Trondheim, Norway

Abstract

Vision-language models (VLMs) are moving onto edge devices, but little is known about how much energy they draw on single-board computers with NPU or GPU accelerators, or where that energy goes. We measure total-system energy, latency, and thermal behavior of Qwen2-VL-2B-Instruct and its text-only sibling on a Raspberry Pi 5 with a Hailo-10H NPU and on a Jetson Orin Nano Super, on both CPU and accelerator, and separate the fixed energy of each request from the energy that grows with output length. Because the accelerators use different quantization and serving software, we compare deployment stacks rather than silicon. Three findings stand out. First, NPU offloading makes the Pi a much more efficient VLM platform: it cuts energy per generated token by about two-thirds, leaves the host CPU largely free, and avoids the thermal throttling that the CPU backend reaches in sustained use. Second, an image adds both fixed and per-token energy, but at typical response lengths generating the answer costs far more than processing the image. Third, the Jetson GPU favors short responses through its lower fixed cost per request, whereas the Pi NPU favors longer ones through its lower cost per output token; their energy curves cross near 40 output tokens.

Figures & tables

Component	Configuration
SBC1	Raspberry Pi 5 Model B Rev 1.1, BCM2712, 4 $\times$ A76 @ 2.4 GHz, 8 GB
NPU	AI HAT+ 2 (Hailo-10H, 40 TOPS, 8 GB, PCIe Gen 3)
NPU stack	HailoRT 4.21.0 (hailort 4.21.0-1, Hailo-10H firmware 4.21.0), Hailo GenAI serving path (Ollama-compatible /api/chat )
CPU stack	llama.cpp release b9412 (commit 7c4e9e0) llama-server ; launch: 4 threads, ctx-size 4096
OS	Debian GNU/Linux 12 (bookworm), Raspberry Pi OS 64-bit
SBC2	NVIDIA Jetson Orin Nano Developer Kit (8 GB), Ubuntu 22.04.5 LTS (JetPack 6.2, L4T R36.4.3), MAXN SUPER mode (nvpmodel mode 2, verified), schedutil CPU governor, jetson_clocks not applied, unpinned GPU/EMC frequency ranges; CUDA 12.6.11, cuDNN 9.3.0, TensorRT 10.3.0; kernel 5.15.148-tegra

TABLE I : Testbed configuration

Metric	CPU LLM-T	CPU VLM-V	NPU LLM-T	NPU VLM-V
Mean power (W)	8.66 ( $\pm$ 0.10)	9.22 ( $\pm$ 0.05)	5.97 ( $\pm$ 0.04)	6.56 ( $\pm$ 0.03)
Energy/token (mJ)	1156 ( $\pm$ 7)	1959 ( $\pm$ 115)	425 ( $\pm$ 50)	676 ( $\pm$ 96)
Energy/inference (J)	135.3 ( $\pm$ 34.7)	211.6 ( $\pm$ 49.0)	39.6 ( $\pm$ 8.5)	64.2 ( $\pm$ 12.3)
Wall time (s)	15.35 ( $\pm$ 3.88)	22.75 ( $\pm$ 5.22)	6.55 ( $\pm$ 1.38)	9.72 ( $\pm$ 1.83)
Prefill time (s)	0.803 ( $\pm$ 0.182)	2.659 ( $\pm$ 0.164)	1.424 ( $\pm$ 0.019)	2.325 ( $\pm$ 0.031)
Decode rate (tok/s)	8.07 ( $\pm$ 0.02)	5.92 ( $\pm$ 0.02)	22.33 ( $\pm$ 0.08)	16.31 ( $\pm$ 0.05)

TABLE II : Raspberry Pi main benchmark: mean per-condition results over 30 prompts

\times

3 trials (prompt-level means; 95% CIs). Energy/token is a mean of per-prompt ratios, not total energy over total tokens.

Fig. 1 : Energy per generated token by backend and condition (prompt-level means, 95% CI error bars, log scale). Three-condition modality ablation on the Pi (left) and Jetson (right). The LLM-T

\rightarrow

VLM-T step is positive on the Pi but negative on the Jetson, whereas image conditioning (VLM-T

\rightarrow

VLM-V) increases energy on every backend.

Stack	Cond.	Text in	Visual in	Generated	EOS/cap/other (%)
Pi-CPU	LLM-T	125	0	117	97/3/0
	VLM-T	101	0	135	93/7/0
	VLM-V	101	1204	118	97/3/0
Pi-NPU	LLM-T	125	0	113	97/3/0
	VLM-T	101	0	128	93/7/0
	VLM-V	101	1204	119	97/3/0

TABLE III : Per-condition request diagnostics (natural suite; prompt-level means over 30 prompts

\times

3 trials). Stop reasons: end of sequence (EOS), 256-token cap, or other.

Fig. 2 : One-token request energy vs. reference vision-token count (computed by the Qwen reference processor after preprocessing; level means, 3 trials/level, linear fits). The Orin GPU and Pi NPU slopes (5.90 and 4.82 mJ/token, respectively) are within 1.22

\times

of each other.

Fig. 3 : Pi CPU vs. NPU temperature over one representative 30-minute session per backend (of 3 replicated sessions each; full ranges in Section V-G ), with a marker at the CPU’s throttle onset (

\approx

17 min in this session) and the aligned inference completion rate (requests/min, 60-s rolling window) below.

	Pi CPU	Pi NPU	Orin CPU	Orin GPU
Mean power (W)	9.22	6.56	12.97	13.50
Energy/token (mJ)	1959	676 †	1206	717 †
Energy/inference (J)	211.6	64.2	123.8	77.6
Decode (tok/s)	5.92	16.31	16.05	24.88
Wall time (s)	22.75	9.72	9.41	5.58
Prefill (s)	2.659	2.325	1.882	0.741

TABLE IV : Cross-platform comparison (VLM-V; prompt-level means, campaigns matched by prompt; energy/token is a mean of per-prompt ratios as in Table II ). Bold marks the better value where the direction is unambiguous;

\dagger

: difference not significant (Section V-H ).

Fig. 4 : Cross-platform throughput–energy trade-off for image-conditioned VLM inference (VLM-V). Each point is one prompt’s mean over three trials (30 points per stack). The Pi-NPU and Orin-GPU stacks have similar mean per-prompt energy per token, and the Orin-GPU stack decodes faster. Which stack uses less energy depends on output length (Section V-I ). Logarithmic energy axis.

Fig. 5 : Left: fitted total request energy versus forced output length at 672 px, with the cross-platform NPU=GPU energy crossover marked. Right: two distinct output-length equalities versus image resolution. Squares and circles show the within-backend length at which image-attributable fixed energy equals autoregressive decode energy. Diamonds show the cross-platform length at which total NPU and GPU request energy are equal. Error bars denote the bootstrap 95% CI for the cross-platform crossover.

Fig. 6 : Observed minus fitted request energy for the fixed-length VLM-V fits (ordinary least squares on achieved length; 21 trials per fit) on the Pi NPU (left) and Orin GPU (right) at three image resolutions; markers are offset horizontally within each length for legibility.

Res. (px)	$A$ (J), $\alpha$ (J)	$\beta$ [95% CI] (mJ/tok)	$A$ (J), $\alpha$ (J)	$\beta$ [95% CI] (mJ/tok)	$N^{*}_{\text{img/dec}}$ (NPU/GPU, tok)	$N^{*}_{\text{NPU=GPU}}$ [95% CI] (tok)
	Pi NPU		Orin GPU
448	9.23, 0.74	384 [380.2, 387.7]	1.71, 1.01	556 [554.5, 557.8]	1.94 / 1.82	43.7 [43.1, 44.3]
672	10.31, 1.82	386 [382.4, 389.8]	2.78, 2.09	563 [559.6, 568.2]	4.72 / 3.70	42.3 [41.8, 42.9]
1120	13.41, 4.92	404 [402.3, 405.8]	6.59, 5.90	583 [578.6, 589.9]	12.18 / 10.12	38.2 [37.6, 38.7]

TABLE V : Fixed-length image-ingestion/decode and platform-crossover fits.

A

is the complete VLM-image request intercept,

\beta

is marginal decode energy (95% CI, bootstrap, in brackets),

\alpha

is the fixed image-attributable energy (

\alpha=A_{\text{VLM-V}}-A_{\text{VLM-T}}

N^{*}_{\text{img/dec}}=\alpha/\beta

is the output length where image-attributable fixed energy equals marginal decode energy, and

N^{*}_{\text{NPU=GPU}}

is the cross-platform energy crossover (95% CI, trial-resampling bootstrap). All fits

R^{2}\geq 0.9997

n=21

trials/path.

Fig. 7 : Exploratory quality–energy sweeps: accuracy and energy per correct answer vs. resolution (top) and output cap (bottom), Pi NPU and Orin GPU. Single run per setting on 25 gradable prompts; error bars are prompt-level bootstrap 95% intervals (markers offset horizontally for legibility).

	Pi NPU		Orin GPU
Setting	Accuracy	J/correct	Accuracy	J/correct
448 px	44 [24, 64]	116.2 [70, 216]	48 [28, 68]	127.9 [76, 229]
672 px	52 [32, 72]	97.0 [62, 166]	56 [36, 76]	101.6 [63, 178]
1120 px	56 [36, 76]	102.9 [67, 172]	60 [40, 80]	109.1 [71, 181]
32 tok	32 [16, 52]	84.7 [52, 174]	32 [16, 52]	77.8 [47, 162]
64 tok	48 [28, 68]	73.5 [50, 130]	52 [32, 72]	70.1 [47, 119]

TABLE VI : Exploratory sweeps with prompt-level bootstrap 95% intervals (25 gradable prompts resampled with replacement, 2000 draws; single run per prompt and setting). Accuracy in %; energy per correct answer in J. The intervals capture prompt sampling only.

Efficient VLM Inference

Efficient vision-language model inference reduces the compute, memory, latency, and energy needed to process images, video, and text.

See all 485 papers

Seeing is Free, Speaking is Not: Uncovering the True Energy Bottleneck in Edge VLM Inference
Jul 10, 2026Junfei Zhan, Haoxun Shen, Mingang Guo +2Efficient VLM Inference Edge Inference
Department of Electrical and Systems Engineering University of Pennsylvania Philadelphia, PA, USA · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences Shenzhen, China · College of Information Science and Technology Jinan University Guangzhou, China
INAR-VL: Input-Aware Routing for Edge-Cloud Vision-Language Inference
May 13, 2026Ahmed Šabanović, Paul Joe Maliakel, Ivona BrandićEfficient VLM Inference Edge-Cloud Computing
TU Wien
EdgeVLN: Runtime-Aware Deployment Ready Quantized Vision Language Navigation Model
Sep 28, 2026Rithvik Jonna, Man Namgung, Aakash Gurram +1Visual Navigation VLM Quantization
Department of Electrical and Computer Engineering Johns Hopkins University · Laboratory for Computational Sensing and Robotics Johns Hopkins University

Jul 10, 2026J/K move · Enter open · S save

Seeing is Free, Speaking is Not: Uncovering the True Energy Bottleneck in Edge VLM Inference

Junfei Zhan, Haoxun Shen, Mingang Guo, Zixuan Huang+1

Department of Electrical and Systems Engineering University of Pennsylvania Philadelphia, PA, USA · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences Shenzhen, China · College of Information Science and Technology Jinan University Guangzhou, China

Efficient VLM Inference Edge Inference Autoregressive Decoding

Open paper AskPDF arXiv

Vision-Language Models (VLMs) are the perceptual backbone of embodied AI, but their energy footprint on edge hardware remains poorly understood. Existing efficiency efforts focus predominantly on reducing visual tokens, implicitly treating visual processing as the dominant energy cost. We overturn this implicit assumption through the first systematic energy profiling of on-device VLM inference, spanning five models across three architecture families, four input resolutions, and two hardware platforms (NVIDIA RTX 3070 and Jetson Orin NX). Our analysis yields three findings. First, average inference power is a model-intrinsic constant, invariant to input resolution, image complexity, and prompt type, with less than 5% variation across all conditions. This means that all energy variation across inputs must arise from variation in inference time, not from variation in power draw. Second, each output token costs 11 to 39x more wall-clock time than each input token due to the compute-bound and memory-bound asymmetry between prefill and decode, making output token count the dominant driver of both latency and energy. Third, image complexity, measured by the number of objects in an image, induces up to 4.1x energy differences at identical resolution. This variation arises not from increased visual processing cost, but from differences in output length. These findings expose a fundamental limitation of visual token pruning: even removing all visual tokens saves at most 10% of total energy for fixed-token models. Across models spanning 1 billion to 8 billion parameters, controlling output length saves up to 97% of total energy, with the energy dominance of decoding growing stronger at larger model scale. In short, the true energy bottleneck in edge VLM inference is not what the model sees, but how much it says.

Abstract

Efficient VLM Inference

Explore similar work

Seeing is Free, Speaking is Not: Uncovering the True Energy Bottleneck in Edge VLM Inference

INAR-VL: Input-Aware Routing for Edge-Cloud Vision-Language Inference

EdgeVLN: Runtime-Aware Deployment Ready Quantized Vision Language Navigation Model