Edge NPU performance depends not only on hardware capabilities, but also on model transformation and the overall deployment pipeline. We propose a benchmarking methodology that separates these effects by comparing three configurations: Baseline, Vendor Guided NPU Only (VG-NPU), and Vendor Guided End-to-End (VG-E2E), where VG-NPU preserves common model I/O and host-side processing while VG-E2E follows the vendor-native deployment pipeline. We apply this methodology to four edge NPUs and three vision workloads. VG-NPU improves both accuracy and Runtime-call latency in 10 of 12 NPU--workload combinations, whereas VG-E2E reduces System-level latency in all combinations but improves accuracy in only four. Moreover, NPU-level performance rankings do not always match End-to-End rankings, and VG-E2E provides an additional 10.4--70.7% reduction in System-level latency over VG-NPU. These results demonstrate the importance of evaluating vendor-guided optimization separately at the controlled model-transformation and full deployment levels when comparing heterogeneous commercial Edge NPU platforms in practice.
Figures & tables
Figure 1: Execution paths and latency measurement intervals of the benchmark framework. Baseline and VG-NPU use the same execution path, whereas VG-E2E follows the vendor-guided deployment pipeline. Runtime-call latency measures the elapsed time of the runtime API invocation, which includes NPU execution and any overhead incurred within the runtime call, while System-level latency measures the total execution time from image loading to final prediction.
Category
Baseline
VG-E2E
VG-NPU
Source model / weights
Same
Same
Same
Calibration / evaluation
1,024 / 5,000 images
Same
Same
Batch / execution
1 / Sync. / Single-stream
Same
Same
Compiler input
Source model subgraph
Source model
Source model subgraph
Quantization
INT 8-bit PTQ
Vendor options
Vendor options
Compiler optimization
Minimal settings
Vendor options
Vendor options
Table 1: Benchmark Configurations
Model / Task
Input
Dataset / Format
Cal. / Eval.
Metric
FP32 Score
ResNet50 / Classification
RGB 224×224
ImageNet-1K / JPEG + class label
train 1,024 / val 5,000
Top-1 accuracy
79.06
YOLOv5m / Detection
RGB 640×640
COCO 2017 / JPEG + bbox annotation
train2017 1,024 / val2017 5,000
bbox AP@[.50:.95]
45.16
YOLOv8s-pose / Pose estimation
RGB 640×640
COCO 2017 / JPEG + keypoint annotation
train2017 1,024 / val2017 5,000
keypoint AP@[.50:.95]
59.39
Table 2: Evaluation workloads, datasets, and accuracy metrics.
Target
INT8 TOPS
Host / OS
CPU Freq.
Compiler
Runtime
Hailo-8
26
Jetson Orin Nano / Ubuntu 22.04, JetPack 6
1.728 GHz
DFC 3.34
HailoRT 4.24
Hailo-10H
20
Jetson Orin Nano / Ubuntu 22.04, JetPack 6
1.728 GHz
DFC 5.3
HailoRT 5.3
DX-M1
25
Jetson Orin Nano / Ubuntu 22.04, JetPack 6
1.728 GHz
DX-COM 2.3-rc.5
DX-RT 3.4
Regulus
10
Regulus SOM / Vendor Yocto Linux
1.2 GHz
QBCompiler 1.1.2
qbruntime 1.2
Table 3: Evaluation NPU Platforms and Software Stacks
NPU
Model
Baseline
VG-NPU
VG-E2E
Hailo-8
ResNet50
78.38
78.88
78.08
YOLOv5m
41.68
43.33
39.21
YOLOv8s-pose
49.67
51.68
48.29
Hailo-10H
ResNet50
78.32
78.92
77.82
YOLOv5m
42.39
43.46
42.80
YOLOv8s-pose
50.58
52.49
47.84
Table 4: Accuracy Results on a 0–100 Scale
NPU
Model
Baseline
VG-NPU
VG-E2E
Hailo-8
ResNet50
11.100
9.786
8.766
YOLOv5m
59.308
43.106
28.101
YOLOv8s-pose
43.938
38.228
22.167
Hailo-10H
ResNet50
16.145
15.082
9.712
YOLOv5m
124.871
114.839
33.697
YOLOv8s-pose
92.837
89.904
32.683
Table 5: System-level Latency (ms).
NPU
Model
Baseline
VG-NPU
Hailo-8
ResNet50
5.506
4.187
YOLOv5m
39.640
23.271
YOLOv8s-pose
18.656
12.891
Hailo-10H
ResNet50
10.512
9.462
YOLOv5m
105.090
95.024
YOLOv8s-pose
67.445
65.157
Table 6: Runtime-call Latency under the Baseline and VG-NPU (ms).
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Output
Postprocessing
Accuracy
Latency
ResNet50
1,000 class scores
Highest-score class
Top-1 accuracy
Same prediction procedure
YOLOv5m
3 detection heads
BBox decoding, confidence filtering, and NMS
conf. 0.001; IoU 0.60; max det. 300
conf. 0.25; IoU 0.45; max det. 1000
YOLOv8s-pose
9 tensors for bbox, class, and keypoints
BBox/keypoint decoding, confidence filtering, and NMS
conf. 0.001; IoU 0.70; max det. 300
conf. 0.25; IoU 0.70; max det. 300
Appendix
Table 7: Model output and evaluation settings.
NPU
Option / Setting
Baseline vs. VG-NPU
VG-NPU vs. VG-E2E
Hailo-8
optimization_level
−100→2
Same as VG-NPU
compression_level
0→1
Same as VG-NPU
ResNet50
W8A8 → W4 × 3 + bias correction
Same as VG-NPU
YOLOv5m
W8A8 → W4 × 3 + finetuning
Same as VG-NPU
YOLOv8s-pose
W8A8 → keypoint A16/W16; finetuning off
finetuning on
Hailo-10H
optimization_level
−100→2
Same as VG-NPU
Appendix
Table 8: Vendor-specific optimization changes across configurations.
NPU
Item
Baseline / VG-NPU
VG-E2E
Hailo-8 Hailo-10H
Input
Normalized FP32
Raw RGB UINT8
Preprocessing
Host
Vendor-native preprocessing
YOLOv5m output
3 detection heads
NMS output [80,5,80]
YOLOv8s-pose output
9 tensors for bbox, class, and keypoints
9 heads: 3 scales × {64, 1, 51}
Postprocessing
Common postprocessor
Hailo vendor postprocessor
Runtime / pipeline
Benchmark pipeline
TAPPAS (GStreamer)
Appendix
Table 9: Deployment configuration changes from the common Baseline/VG-NPU path to VG-E2E.
Hofstra University, Hempstead, NY, USA · Bridgewater-Raritan High School, Bridgewater, NJ, USA · Norwegian University of Science and Technology, Trondheim, Norway