A predictive, per-engine energy roofline of the Apple Neural Engine
Organizations: The University of Texas at Austin, USA
Abstract
The Apple Neural Engine (ANE) is in over a billion devices, but is characterized indirectly via CoreML. Using a CoreML-free fused-graph dispatch path, we measure the ANE, along with the Apple GPU and CPU on the same device, with per-engine energy accounting. The consumer-available roofline does not predict the ANE in practice. A critical correction is that the ANE sees two effective memory bandwidths: a slow activation path and a fast weight-stream path. Two roofline ridges follow, and the weight-stream ridge is below that of the GPU. The ANE single-operand compute peak decreases instead of plateauing. Its weight stream is byte-bound in fp16 but limited by the dequantizer's element rate in int8. As a result, int8 compression saves energy at unchanged throughput and an int4 lookup-table stream raises throughput and efficiency at a cost of 2.55 points of ImageNet top-1. The corrected roofline identifies the lower-energy ANE paths and bounds it from below. A constructed device map reproduces across four unique ANE generations. Results show that operators should be placed by arithmetic intensity and fused for optimal performance.
Figures & tables
| Primitive | Device | Dtype | Peak GFLOP/s | @ size | relerr there | Peak GFLOP/s/W |
|---|---|---|---|---|---|---|
| GEMM | GPU | fp16 | 30 862 | 3.6e-4 | 603 ( ) | |
| GEMM | ANE | fp16 | 10 191 | 1.2e-2 | 866 ( ) | |
| conv | GPU | fp16 | 31 790 | C=512 | 7.4e-3 | 1 392 (C=512) |
| conv | ANE | fp16 | 18 771 | C=512 | 5.5e-4 | 1 372 (C=512) |
| Archetype (bytes) | ANE GB/s | GPU GB/s | ANE GB/s/W | GPU GB/s/W |
|---|---|---|---|---|
| streaming relu / x 2 (2N) | 24.1 | 229.7 | 1.99 | 27.13 |
| reduction sum (1N) | 22.4 | 193.0 | 2.24 | 27.28 |
| softmax (2N) | 15.1 | 238.4 | 1.78 | 18.54 |
| layer_norm (2N) | 4.4 | 41.9 | 0.90 | 2.43 |
| gelu light-compute (2N) | 23.2 | 48.6 | 2.03 | 3.70 |
| Device | Dtype | Compute roof (GFLOP/s) | Bandwidth roof (GB/s) | Ridge (FLOP/byte) |
|---|---|---|---|---|
| GPU | fp16 | gemm 30862 / conv 31790 | 229.7 | 134.4 |
| ANE (activation path) | fp16 | gemm 10191 / conv 18771 | 24.1 | 423.6 |
| ANE (weight-stream path) | fp16 | gemm 10191 / conv 18771 | 143 | 71 |
| Operator archetype | AI | GPU GFLOP/s (%roof) | ANE GFLOP/s (%roof) |
|---|---|---|---|
| matmul | 171 | 1235 ( ) | 1819 ( ) |
| matmul | 683 | 22345 ( ) | 10191 ( ) |
| matmul | 2731 | 30527 ( ) | 4882 ( ) |
| conv ( ) | 287 | 5343 ( ) | 5885 ( ) |
| conv ( ) | 2020 | 31790 ( ) | 18771 ( ) |
| attention ( ) | 52 | 4685 ( ) | 598 ( ) |
| Class | Workload | ANE lat | GPU lat | ANE W (CV) | GPU W (CV) | ANE perf/W | GPU perf/W |
|---|---|---|---|---|---|---|---|
| GEMM floor | M64 K256 N256 | 0.092 | 0.124 | 2.93 ( ) | 6.54 ( ) | 15.7 GF/W | 7.8 GF/W |
| GEMM bandwidth | M128 K1024 N1024 | 0.086 | 0.187 | 9.22 ( ) | 9.40 ( ) | 274 GF/W | 160 GF/W |
| GEMM compute ∗ | M256 K4096 N4096 | 1.293 | 0.461 | 8.71 ( ) | 33.05 ( ) | 751 GF/W | 499 GF/W |
| conv | single ( ) | 0.103 | 0.173 | 5.99 ( ) | 7.61 ( ) | 24.5 GF/W | 14.7 GF/W |
| conv ∗ | ResNet ( ) | 0.757 | 3.159 | 10.12 ( ) | 32.70 ( ) | 2289 GF/W | 175 GF/W |
| attention ∗ | ViT block | 0.230 | 0.269 | 8.50 ( ) | 16.01 ( ) | 522 GF/W | 197 GF/W |
| Latency (ms) | Energy (mJ/inf) | ANE energy advantage | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Model | ANE | MPS | MLX-c | ANE relerr | ANE | MPS fp32 † | MPS | MLX-c | adv. MPS | adv. MLX-c |
| ResNet-18 | 0.330 | 2.033 | 1.645 | 2.3e-3 | 2.30 | 39.1 | 21.3 | 32.6 | ||
| MiniLM (1 sent) | 0.525 | 1.920 | 0.650 | 5.5e-3 | 2.74 | 28.8 | 17.6 | 11.5 | ||
| ViT-B/16 | 18.34 | 15.92 | 2.959 | 7.8e-3 | 65.7 | 1382.7 | 577.9 | 100.4 | ||
| Workload | Throughput crossover | Throughput/W crossover |
|---|---|---|
| Vision (conv classifier) | never ( – ) | never ( – ) |
| Encoder (transformer block) | 23–31 (replicate ) | never ( – ) |
| Attention (self-attention block) | small batch (replicate ) | never ( – ) |
| Batched GEMM | 5 ( ) | never ( at ; Section 10 ) |
| B | ANE tok/s | int8 tok/s | GPU tok/s | ANE-fp16 tok/s/W | ANE-int8 tok/s/W | GPU tok/s/W |
|---|---|---|---|---|---|---|
| 1 | 115 | 123 | 200 | 35.9 | 44.7 | 19.3 |
| 2 | 204 | 204 | 231 | 50.6 | 68.7 | 9.9 |
| 4 | 341 | 342 | 431 | 68.6 | 64.8 | 18.5 |
| 8 | 517 | 515 | 834 | 88.2 | 100.3 | 36.9 |
| 16 | 704 | 698 | 1444 | 106.2 | 113.7 | 65.1 |
| 32 | 847 | 850 | 2322 | 88.0 | 105.4 | 116.4 |