Is INT8 Portable? A Cross-Platform Measurement Study of Quantized Inference on Embedded and Automotive Accelerators

Yuyeong Shin

cs.ARSep 14, 2026

Is INT8 Portable? A Cross-Platform Measurement Study of Quantized Inference on Embedded and Automotive Accelerators

Authors: Yuyeong Shin

Organizations: Korea Automotive Technology Institute (KATECH)

Abstract

Eight-bit integer (INT8) post-training quantization is the default recipe for edge deployment, under a widely held assumption: INT8 makes inference faster at a small, predictable accuracy cost, and a model quantized once can be carried to any target. We test that assumption with a controlled measurement study across seven hardware classes -- ARM and x86 CPUs, a discrete GPU, an NVIDIA Jetson AGX Orin iGPU and its NVDLA cores, and two vendor NPUs (Qualcomm Hexagon HTP, DEEPX DX-M1) -- holding the ONNX artifact and the quantization scales fixed so the integer kernel or ISA is the only free variable. Portability fails on three axes. (1) The sign of the INT8 speedup is set by the CPU's dot-product ISA (ARM dotprod/SDOT, x86 VNNI): cores that have it speed up by up to 2.1x, cores that lack it slow down by 1.7x, for the identical model and runtime. (2) INT8 outputs are not portable, and the rule is an invariance rather than a gradient: FP32 predictions are bit-identical for every pair (1000/1000), while INT8 predictions agree 1000/1000 exactly when two targets share an integer kernel and 958-965/1000 whenever they do not -- independent of whether the boundary is CPU<->CPU or CPU<->accelerator, and invisible to top-1 accuracy, which is preserved. (3) Vendor NPUs own quantization: a bring-your-own QDQ graph fails silently on one NPU (external scales ignored, accuracy 0.75 -> 0.005 while it compiles, profiles and runs without error) and loudly on the other (the compiler refuses the graph), so only the vendor's native path yields a correct engine. We further show that edge-NPU latency regimes are set by output/device-to-host transfer size rather than compute, and locate the transition with a fixed-compute sweep. We release the scripts and 32 reports. "Quantize once, deploy anywhere" is unsafe for embedded and automotive deployment, where per-input determinism and redundancy matter.

Figures & tables

Class	Target	Role in the study
ARM CPU (dotprod)	Raspberry Pi 5, Cortex-A76	C1 sign, C2 agreement
ARM CPU (dotprod)	Jetson AGX Orin, Cortex-A78AE	C1 sign, C2 CPU $\leftrightarrow$ accelerator
ARM CPU (no dotprod)	i.MX8M-Nano, Cortex-A53	C1 sign (negative)
x86 CPU (no VNNI)	Core i9-10900K	C1 sign (negative), C2
Discrete GPU	RTX 3080 (Ampere)	precision ladder, transformer INT8
Edge iGPU + NVDLA	Jetson AGX Orin (iGPU, $2\times$ NVDLA v2)	accelerator char., C2 GPU kernel

Table 1: Hardware matrix. Every device is a single unit (

n=1

per class); we therefore make relative claims about representative devices, not population claims about an ISA (see §10).

CPU	Dot-product ISA	FP32 $\rightarrow$ INT8	INT8 effect
Cortex-A76 (Raspberry Pi 5)	ARM dotprod (SDOT)	$144.95\rightarrow 79.08$ ms	1.83 $\times$ faster
Cortex-A78AE (Jetson AGX Orin)	ARM dotprod	$38.47\rightarrow 18.22$ ms	2.11 $\times$ faster
Core i9-10900K (x86)	no AVX-512 VNNI	$9.28\rightarrow 16.34$ ms	1.76 $\times$ slower
Cortex-A53 (i.MX8M-Nano)	no dot-product	$680.20\rightarrow 1123.02$ ms	1.65 $\times$ slower

Table 2: The INT8 speedup sign is set by the CPU’s dot-product ISA, for the identical model and runtime.

Panel of Figure 1 — Figure 1: The INT8 speedup sign is an ISA property. Bar length is the log-scaled speedup (FP32 latency $\div$ INT8 latency) for the identical ResNet-50 INT8 QDQ model on the identical ONNX Runtime CPU (MLAS) path; bars to the right of $1\times$ are faster under INT8, bars to the left are slower. The two cores with an ARM dot-product instruction gain; the two without one lose. Same data as Table 2 .

Comparison	Boundary	Integer kernel	FP32	INT8
Jetson A78AE vs. Pi 5 A76	CPU $\leftrightarrow$ CPU	same (MLAS SDOT)	1000/1000	1000/1000
i.MX8M-Nano A53 vs. Pi 5 A76	CPU $\leftrightarrow$ CPU	different	1000/1000	965/1000
i.MX8M-Nano A53 vs. x86 i9	CPU $\leftrightarrow$ CPU	different	1000/1000	961/1000
Raspberry Pi 5 A76 vs. x86 i9	CPU $\leftrightarrow$ CPU	different	1000/1000	958/1000
Jetson iGPU (TensorRT) vs. A78AE (MLAS)	CPU $\leftrightarrow$ accel.	different	1000/1000	961/1000
DEEPX DX-M1 (NPU) vs. host A76 CPU $\ddagger$	CPU $\leftrightarrow$ vendor-NPU	different, and different scales	—	939/1000

Table 3: INT8 top-1 agreement between targets running the same model artifact. Agreement is bimodal: 1000/1000 when the integer kernel is shared, 958–965/1000 when it is not, independent of where the hardware boundary falls. FP32 is 1000/1000 for every pair measured.

\ddagger

excluded from the invariance claim (scales not held fixed; see text).

Panel of Figure 2 — Figure 2: INT8 agreement is bimodal, and the location of the hardware boundary does not predict it. Each row is one target pair running the same model artifact: $\circ$ is FP32 top-1 agreement, $\bullet$ is INT8. FP32 is 1000/1000 wherever it was measured. INT8 is exactly 1000/1000 for the one pair that shares an integer kernel and falls in a 958–965/1000 band (shaded) for every pair that does not — including the CPU $\leftrightarrow$ accelerator pair, which sits inside the CPU $\leftrightarrow$ CPU range. The DEEPX row ( $\ddagger$ , gray) cannot hold the scales fixed (§6) and is a deployment observation, excluded from the invariance claim. Same data as Table 3 .

Model (on DX-M1)	Output	Regime	Multi-core scaling
ResNet-50	4 KB	compute-bound (compute 2.77 ms $\gg$ D2H 0.11 ms)	$2.19\times$ near-linear on 3 cores
YOLO26n	2.82 MB (raw head)	D2H-bound (D2H 21.81 ms $\gg$ compute 9.0 ms)	$1.00\times$ flat
YOLOv5s	5.48 MB	D2H-bound , worse	$1.02\times$ flat; lightest compute (2.59 ms) yet 26 $\times$ slower than ResNet-50

Table 4: On one PCIe-attached edge NPU, the latency regime and multi-core scaling track output/D2H size, not compute.

Output bytes (fixed 2.70 ms compute)	D2H p50	3-core scaling
3,920	0.141 ms	$2.98\times$ (compute-bound plateau)
256,368	0.776 ms	$1.99\times$ (transition band)
511,952	1.367 ms	$1.59\times$ (transition band)
1,000,384	2.429 ms	$1.21\times$ (transition band)
1,999,984	8.892 ms	$1.00\times$ (D2H-bound)

Table 5: Controlled sweep: with compute held fixed, 3-core scaling descends past the compute-bound plateau as output/D2H size grows, tracing the compute-bound

\to

D2H-bound transition. Output sizes are exact byte counts, not rounded KB/MB, because the two regime thresholds fall between conventional round numbers.

Panel of Figure 3 — Figure 3: The compute-bound $\to$ D2H-bound transition with NPU compute held fixed. All 12 synthetic sweep points on the DEEPX DX-M1: 3-core throughput scaling against output (D2H) size, at a constant 2.70 ms 1-core inference (spread 4.9%). Scaling holds a $2.97$ – $2.99\times$ plateau while the output stays below ${\approx}63$ KB, then descends to $1.00\times$ . The shaded band runs from the link-saturation onset (326,377 B, where D2H $=$ compute $/N$ ) to the single-inference crossover (1,095,949 B, where D2H $=$ compute); its $3.36\times$ width equals the core count $N=3$ , because $N$ cores share one D2H link. Five of these points are listed numerically in Table 5 .

Configuration (ORT, COCO val2017 5,000)	mAP	$\Delta$ vs. FP32
FP32	0.4207	—
Backbone only INT8 (53 Convs)	0.2653	$\mathbf{-36.9}$ %
Transformer only INT8 (137 nodes)	0.2391	$-43.2$ %
All INT8 (190 nodes)	0.2402	$-42.9$ %

Table 6: Two-way ablation: each half of DETR collapses on its own, and the two losses are strongly sub-additive — the damage is distributed, not localized.

	ORT (§8.1, discrete GPU)	DEEPX dx_com 2.4.0 (DX-M1)
INT8 scope	backbone Conv $\times 53$ ( input_projection excluded)	backbone + input_projection + first encoder self-attn through its Softmax
Weights	per-channel QInt8	wbit8
Activations	per-tensor QInt8 , MinMax, 100 images	abit8 , EMA, 100 in-domain COCO images
Calib. input shape	dynamic axes (varies per image)	fixed $800\times 1066$ , bit-identical .npy
Evaluation	COCO val2017, 5,000 images	COCO val2017, first 500 images
FP32 $\rightarrow$ INT8	$0.4207\rightarrow 0.2653$	$0.4377\rightarrow 0.4385$

Table 7: The same nominal recipe (“INT8 the backbone, keep the transformer FP32”) under two vendors’ quantizers. The INT8 scope runs against the outcome — DEEPX quantizes strictly more and loses strictly less — leaving the activation calibrator as the operative difference.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Sec.	Claim	Report ( logs/ )	Scripts and results ( experiments/ )
§4 (C1)	INT8 speedup sign set by the dot-product ISA, four CPUs, one model, one runtime	stage4_arm_cpu_fallback_report.html stage4_imx8mn_a53_report.html stage4_jetson_agx_orin_a78ae_report.html	stage5_infrastructure/cpu_proxy/rpi_bench.py rpi_bench_lowmem.py stage5_infrastructure/cpu_proxy/results/resnet50__*.json
§5 (C2)	FP32 1000/1000 control; CPU $\leftrightarrow$ CPU INT8 agreement 1000 / 965 / 961 / 958	stage4_arm_cpu_fallback_report.html stage4_imx8mn_a53_report.html stage4_jetson_agx_orin_a78ae_report.html	stage5_infrastructure/cpu_proxy/README.md (the agreement matrix) stage5_infrastructure/cpu_proxy/raw/
§5 (C2)	CPU $\leftrightarrow$ accelerator 961/1000 from the same QDQ artifact; iGPU INT8 top-1 0.7620	stage3_jetson_orin_accuracy_report.html	stage3_tensorrt/jetson_ondevice/accuracy/scripts/orin_accuracy.py analyze_accuracy.py
§5 (C2 $\ddagger$ )	Vendor-NPU 939/1000, scales not held fixed (deployment observation only)	stage4_deepx_dxm1_accuracy_report.html	stage4_deepx_dxm1/accuracy/scripts/npu_infer.py cpu_infer.py analyze_dxm1_acc.py
§5, §11	Power-of-two mitigation does not transfer across the x86 $\leftrightarrow$ A76 boundary: 958 $\to$ 869 (ceil) / 919 (nearest), gate NO-GO; cross-kernel $\|\Delta\mathrm{logit}\|$ $\times 4.34$ / $\times 2.35$ , tie-breaking refuted; 30 runs bit-identical	stage5_pot_scales_report.html	stage5_infrastructure/pot_scales/pot_rewrite.py pot_bench.py pot_agree.py pot_mech.py stage5_infrastructure/pot_scales/results/
§6 (C3)	Qualcomm HTP silent BYO-QDQ failure 0.75 $\to$ 0.005; native path recovers 0.735	stage4_qualcomm_aihub_report.html	stage4_qualcomm_aihub/scripts/qaihub_int8.py qaihub_native_quant.py qaihub_acc.py

Appendix

Table 8: Claim-to-artifact map. Each row names the report and the scripts behind one claim, so any number in the paper can be traced to the run that produced it.

AI Accelerator Inference

AI accelerator inference studies hardware and software for running neural networks faster and more efficiently across data-center, edge, and mobile devices.

See all 103 papers

Efficient INT8 Inference of Small NLP Models on Server CPUs with PyTorch Native Stack
Aug 18, 2026Weiwen Xia, Yuxin Cui, E CaoEfficient Transformer Inference Inference-Time Optimization
Intel Corporation
The Integer Alibi: Localizing Cross-Kernel Divergence in INT8-Quantized LLM Inference
Aug 13, 2026Teng-Ruei ChenGPU Kernel Correctness Verification LLM Inference
Krixvon, Taipei 100, Taiwan
Rethinking Small VLM Quantization: From Component-Wise Analysis to Hardware-Aware Edge Deployment
Jul 9, 2026Hyeju Shin, Chorwon Kim, Ryangsoo Kim +2Mixed-Precision Quantization Vision-Language Models
1ETRI, Republic of Korea.

Aug 18, 2026J/K move · Enter open · S save

Efficient INT8 Inference of Small NLP Models on Server CPUs with PyTorch Native Stack

Weiwen Xia, Yuxin Cui, E Cao

Intel Corporation

Efficient Transformer Inference Inference-Time Optimization Post-Training Quantization

Open paper AskPDF arXiv

Small NLP models, especially BERT-family encoders, remain important in industrial workloads such as classification, ranking, and retrieval even in the era of large language models. On server CPUs, INT8 quantization offers an attractive latency-throughput-cost trade-off, but users increasingly expect such acceleration to be available directly in the native PyTorch stack. We integrate SmoothQuant into TorchAO and optimize the resulting inference path for Intel Xeon CPUs through graph-level fusion in TorchInductor and efficient INT8 GEMM kernel selection across oneDNN-, AVX512_VNNI-, and AMX-based implementations. Across BERT, DistilBERT, and XLM-RoBERTa benchmarks, the approach delivers up to 5.8x end-to-end throughput speedup with negligible---and in some cases no measurable---accuracy loss relative to the FP32 baseline. We also validated our work by detailed performance analysis with roofline models. The implementation has been upstreamed to PyTorch and TorchAO, enabling out-of-the-box deployment with native PyTorch tooling

Abstract

AI Accelerator Inference

Explore similar work

Efficient INT8 Inference of Small NLP Models on Server CPUs with PyTorch Native Stack

The Integer Alibi: Localizing Cross-Kernel Divergence in INT8-Quantized LLM Inference

Rethinking Small VLM Quantization: From Component-Wise Analysis to Hardware-Aware Edge Deployment