Quantized Edge Inference of a Self-Supervised SAR-Optical Encoder on Commodity x86 Hardware
Organizations: Software College, Northeastern University, Shenyang 110819, China
Abstract
Self-supervised vision transformers are increasingly used as reusable encoders for Earth observation, but their inference cost on the hardware available in the field is rarely characterized. We deploy a pretrained SAR-optical masked-autoencoder encoder on three commodity x86 nodes, from a low-power CPU without AVX2 to an AVX2+VNNI reference, at FP32, FP16 and INT8 on the CPU and, where present, the integrated GPU, and report latency, energy (RAPL and wall plug) and linear-probe accuracy. Quantization is accuracy-neutral: INT8 changes micro-mAP by only +0.0006, within the seed spread, so precision becomes a pure cost decision. INT8 lowers CPU latency on every node, by 1.45x even on the CPU without AVX2, and CPU energy wherever it is measured, and the VNNI reference (1.53x) suggests a further gain beyond AVX2. The Pareto-optimal device shifts from the integrated GPU, which is 4.38x faster than the CPU on the weakest node, to the CPU as the CPU class improves.
Figures & tables
| Role | Machine | CPU | AVX2 | VNNI | iGPU |
|---|---|---|---|---|---|
| Floor | Moort M1 | Celeron N5105 (Jasper Lake) | No | No | UHD Gen11 |
| Step | Lenovo M720q | i3-9100T (Coffee Lake) | Yes | No | UHD 630 Gen9.5 |
| Ref. | Dev. laptop | i7-14700HX (Raptor Lake) | Yes | Yes | — |
| Precision | micro-mAP | macro-mAP | micro vs FP32 |
|---|---|---|---|
| FP32 | 0.0000 | ||
| FP16 | 0.0000 | ||
| INT8 | +0.0006 |
| RAPL (J) | Wall plug (J) | ||||||
|---|---|---|---|---|---|---|---|
| Node | Device | FP32 | FP16 | INT8 | FP32 | FP16 | INT8 |
| Floor | CPU | 2.117 | 2.187 | 1.249 | 2.938 | 3.014 | 1.650 |
| iGPU | 0.393 | 0.400 | 0.497 | 0.541 | 0.544 | 0.668 | |
| Step | CPU | 1.260 | 1.250 | 0.859 | 1.723 | 1.830 | 1.230 |
| iGPU | 0.865 | 0.861 | 1.009 | 1.268 | 1.263 | 1.480 | |
Efficient Transformer Inference
Efficient Transformer Inference studies methods and systems that reduce the latency, memory, and energy needed to run Transformer models.
See all 121 papers