Benchmarking Vendor-Guided Optimizations on Edge NPUs: From NPU Only to End-to-End Performance. Extended technical report
Organizations: Chonnam National University, Gwangju, 61186, Korea · Electronics and Telecommunications Research Institute, Gwangju, 61012, Korea
Abstract
Edge NPU performance depends not only on hardware capabilities, but also on model transformation and the overall deployment pipeline. We propose a benchmarking methodology that separates these effects by comparing three configurations: Baseline, Vendor Guided NPU Only (VG-NPU), and Vendor Guided End-to-End (VG-E2E), where VG-NPU preserves common model I/O and host-side processing while VG-E2E follows the vendor-native deployment pipeline. We apply this methodology to four edge NPUs and three vision workloads. VG-NPU improves both accuracy and Runtime-call latency in 10 of 12 NPU--workload combinations, whereas VG-E2E reduces System-level latency in all combinations but improves accuracy in only four. Moreover, NPU-level performance rankings do not always match End-to-End rankings, and VG-E2E provides an additional 10.4--70.7% reduction in System-level latency over VG-NPU. These results demonstrate the importance of evaluating vendor-guided optimization separately at the controlled model-transformation and full deployment levels when comparing heterogeneous commercial Edge NPU platforms in practice.
Figures & tables
| Category | Baseline | VG-E2E | VG-NPU |
|---|---|---|---|
| Source model / weights | Same | Same | Same |
| Calibration / evaluation | 1,024 / 5,000 images | Same | Same |
| Batch / execution | 1 / Sync. / Single-stream | Same | Same |
| Compiler input | Source model subgraph | Source model | Source model subgraph |
| Quantization | INT 8-bit PTQ | Vendor options | Vendor options |
| Compiler optimization | Minimal settings | Vendor options | Vendor options |
| Model / Task | Input | Dataset / Format | Cal. / Eval. | Metric | FP32 Score |
| ResNet50 / Classification | RGB | ImageNet-1K / JPEG + class label | train 1,024 / val 5,000 | Top-1 accuracy | 79.06 |
| YOLOv5m / Detection | RGB | COCO 2017 / JPEG + bbox annotation | train2017 1,024 / val2017 5,000 | bbox AP@[.50:.95] | 45.16 |
| YOLOv8s-pose / Pose estimation | RGB | COCO 2017 / JPEG + keypoint annotation | train2017 1,024 / val2017 5,000 | keypoint AP@[.50:.95] | 59.39 |
| Target | INT8 TOPS | Host / OS | CPU Freq. | Compiler | Runtime |
| Hailo-8 | 26 | Jetson Orin Nano / Ubuntu 22.04, JetPack 6 | 1.728 GHz | DFC 3.34 | HailoRT 4.24 |
| Hailo-10H | 20 | Jetson Orin Nano / Ubuntu 22.04, JetPack 6 | 1.728 GHz | DFC 5.3 | HailoRT 5.3 |
| DX-M1 | 25 | Jetson Orin Nano / Ubuntu 22.04, JetPack 6 | 1.728 GHz | DX-COM 2.3-rc.5 | DX-RT 3.4 |
| Regulus | 10 | Regulus SOM / Vendor Yocto Linux | 1.2 GHz | QBCompiler 1.1.2 | qbruntime 1.2 |
| NPU | Model | Baseline | VG-NPU | VG-E2E |
|---|---|---|---|---|
| Hailo-8 | ResNet50 | 78.38 | 78.88 | 78.08 |
| YOLOv5m | 41.68 | 43.33 | 39.21 | |
| YOLOv8s-pose | 49.67 | 51.68 | 48.29 | |
| Hailo-10H | ResNet50 | 78.32 | 78.92 | 77.82 |
| YOLOv5m | 42.39 | 43.46 | 42.80 | |
| YOLOv8s-pose | 50.58 | 52.49 | 47.84 |
| NPU | Model | Baseline | VG-NPU | VG-E2E |
|---|---|---|---|---|
| Hailo-8 | ResNet50 | 11.100 | 9.786 | 8.766 |
| YOLOv5m | 59.308 | 43.106 | 28.101 | |
| YOLOv8s-pose | 43.938 | 38.228 | 22.167 | |
| Hailo-10H | ResNet50 | 16.145 | 15.082 | 9.712 |
| YOLOv5m | 124.871 | 114.839 | 33.697 | |
| YOLOv8s-pose | 92.837 | 89.904 | 32.683 |
| NPU | Model | Baseline | VG-NPU |
|---|---|---|---|
| Hailo-8 | ResNet50 | 5.506 | 4.187 |
| YOLOv5m | 39.640 | 23.271 | |
| YOLOv8s-pose | 18.656 | 12.891 | |
| Hailo-10H | ResNet50 | 10.512 | 9.462 |
| YOLOv5m | 105.090 | 95.024 | |
| YOLOv8s-pose | 67.445 | 65.157 |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Output | Postprocessing | Accuracy | Latency |
|---|---|---|---|---|
| ResNet50 | 1,000 class scores | Highest-score class | Top-1 accuracy | Same prediction procedure |
| YOLOv5m | 3 detection heads | BBox decoding, confidence filtering, and NMS | conf. 0.001; IoU 0.60; max det. 300 | conf. 0.25; IoU 0.45; max det. 1000 |
| YOLOv8s-pose | 9 tensors for bbox, class, and keypoints | BBox/keypoint decoding, confidence filtering, and NMS | conf. 0.001; IoU 0.70; max det. 300 | conf. 0.25; IoU 0.70; max det. 300 |
| NPU | Option / Setting | Baseline vs. VG-NPU | VG-NPU vs. VG-E2E |
|---|---|---|---|
| Hailo-8 | optimization_level | Same as VG-NPU | |
| compression_level | Same as VG-NPU | ||
| ResNet50 | W8A8 W4 3 + bias correction | Same as VG-NPU | |
| YOLOv5m | W8A8 W4 3 + finetuning | Same as VG-NPU | |
| YOLOv8s-pose | W8A8 keypoint A16/W16; finetuning off | finetuning on | |
| Hailo-10H | optimization_level | Same as VG-NPU |
| NPU | Item | Baseline / VG-NPU | VG-E2E |
|---|---|---|---|
| Hailo-8 Hailo-10H | Input | Normalized FP32 | Raw RGB UINT8 |
| Preprocessing | Host | Vendor-native preprocessing | |
| YOLOv5m output | 3 detection heads | NMS output | |
| YOLOv8s-pose output | 9 tensors for bbox, class, and keypoints | 9 heads: 3 scales {64, 1, 51} | |
| Postprocessing | Common postprocessor | Hailo vendor postprocessor | |
| Runtime / pipeline | Benchmark pipeline | TAPPAS (GStreamer) |