Orion: Characterizing and Programming Apple's Neural Engine for LLM Training and Inference

Ramchand Kumaresan

cs.LGMar 6, 2026

Orion: Characterizing and Programming Apple's Neural Engine for LLM Training and Inference

Authors: Ramchand Kumaresan

Abstract

Over two billion Apple devices ship with a Neural Processing Unit (NPU) - the Apple Neural Engine (ANE) - yet this accelerator remains largely unused for large language model workloads. CoreML, Apple's public ML framework, imposes opaque abstractions that prevent direct ANE programming and do not support on-device training. We present Orion, to our knowledge the first open end-to-end system that combines direct ANE execution, a compiler pipeline, and stable multi-step training with checkpoint resume in a single native runtime, bypassing CoreML entirely via Apple's private _ANEClient and _ANECompiler APIs. Building on prior characterization work by maderix, we extend public knowledge of ANE constraints to a catalog of 20 restrictions on MIL IR programs, memory layout, compilation limits, and numerical behavior, including 14 previously undocumented constraints discovered during Orion development. Orion includes a compiler that lowers a graph IR through five optimization passes to ANE-native MIL and a runtime that manages IOSurface-backed zero-copy tensor I/O, program caching, and delta compilation for weight updates. Because the ANE bakes weights at compile time, naive training normally requires full recompilation per step (~4.2 s). We show that compiled programs can instead be updated by unloading, patching weight files, and reloading, bypassing ANECCompile() and reducing recompilation from 4,200 ms to 494 ms per step (8.5x), yielding a 3.8x training speedup. On an M4 Max, Orion achieves 170+ tokens/s for GPT-2 124M inference and demonstrates stable training of a 110M-parameter transformer on TinyStories for 1,000 steps in 22 minutes with zero NaN occurrences. We also present LoRA adapter-as-input, enabling hot-swap of adapters via IOSurface inputs without recompilation.

Figures & tables

Property	Value
Generation	H16
Neural Engine cores	16
Peak throughput (INT8, Apple spec)	38 TOPS
Peak throughput (fp16, measured 1 1 footnotemark: 1 )	$\sim$ 19 TFLOPS
On-chip SRAM 2 2 2 Apple specifies 38 TOPS (INT8), but the ANE dequantizes INT8 to fp16 before computation. Actual measured peak is $\sim$ 19 TFLOPS fp16 ( maderix, 2026c ) ; INT8 saves only memory bandwidth, not compute cycles.	32 MB
Evaluation queue depth	127

Table 1: Apple Neural Engine hardware characteristics (M4 Max).

Class	Role
_ANEClient	Singleton connection to ANE daemon
_ANECompiler	MIL $\to$ E5 microcode compilation
_ANEInMemoryModel	In-memory model (no filesystem)
_ANEInMemoryModelDescriptor	Accepts MIL + weight blobs
_ANEModel	Holds compiled program handle
_ANERequest	Evaluation request specification

Table 2: Private ANE API surface used by Orion .

#	Constraint	Symptom	Workaround	Src
1	concat MIL op rejected by ANE compiler	Compile failure	Split into separate programs	O
2	Multi-output buffers must have uniform sizes	0x1d at eval	Pad outputs to max size	O
3	Multi-output surfaces ordered alphabetically	Silent wrong data	Name outputs in sorted order	O
4	Minimum $\sim$ 49 KB IOSurface for eval	0x1d at eval	Pad seq dim $\geq$ 16	O
5	$\sim$ 119 compilations per process limit	Silent fail / crash	exec() restart	P
6	SDPA causal masks silently ignored	Wrong attention	Manual causal masking	P ∗

Table 3: ANE constraint catalog. Source: P = prior work ( maderix, 2026b ; maderix, 2026c ; Hollemans, 2022 ) , O = discovered during Orion development, ∗ confirmed by maderix/ANEgpt codebases. Prior-work constraints were independently confirmed on M4 Max.

Panel of Figure 1 — Figure 1: Orion architecture stack. Each layer communicates only with its immediate neighbors. The compiler and runtime together abstract away ANE constraints from the model layer.

Category	Operations
Data	input , const , identity
Linear	conv1x1 , matmul
Elementwise	add , sub , mul , neg
Activation	relu , tanh , sigmoid
Math	exp , pow , sqrt , rsqrt
Reduction	reduce_sum , reduce_mean , reduce_max

Table 4: Graph IR operation categories (27 total).

Operation	Device	Reason
Transformer fwd/bwd (dx)	ANE	Compute-bound convolutions
Token sampling	CPU	Sequential, branching logic
Adam optimizer	CPU	Weights immutable on ANE
$\nabla W$ accumulation	CPU	cblas_sgemm via GCD
NLL loss + gradient	CPU	gather not in MIL
Classifier backward	CPU	32K channels rejected

Table 5: CPU/ANE work division. Operations are assigned based on ANE hardware constraints and performance characteristics.

Panel of Figure 2 — Figure 2: Weight update paths. v1.0 (left) creates new model descriptors and invokes the ANE compiler for every weight update ( $\sim$ 70 ms/kernel). v2.0 (right) reuses existing model objects: unload, write new weight files, reload ( $\sim$ 9 ms/kernel). The compiler is bypassed entirely.

Metric	v1.0	v2.0	Speedup
Train time (compute)	908 ms	849 ms	$\sim$ 1 $\times$
Recompile / reload time	4,200 ms	494 ms	8.5 $\times$
Total step time	5,108 ms	1,345 ms	3.8 $\times$
Recompile % of step	83.9%	36.8%	$-$ 47.1 pp
1000-step wall time	$\sim$ 85 min	22.4 min	3.8 $\times$
Process model	1 step/process	Single process	—

Table 6: Training step time breakdown: v1.0 (full recompile) vs v2.0 (delta reload). Stories110M on M4 Max, lr=

3\!\times\!10^{-4}

, grad_accum=4.

Panel of Figure 3 — Figure 3: Training step time breakdown. v1.0 spends 83.9% of each step on full ANE recompilation ( $\sim$ 4,200 ms for 60 kernels). v2.0’s delta reload reduces this to 494 ms by bypassing ANECCompile() entirely, yielding a 3.8 $\times$ total speedup.

Panel of Figure 4 — Figure 4: LoRA-fused linear layer. Base weights $W_{\text{base}}$ are baked into the compiled ANE program (blue). Adapter matrices $A$ , $B$ are passed as IOSurface inputs (green) and can be swapped without recompilation. $Y=XW_{\text{base}}+\alpha(XA)B$ .

Panel of Figure 5 — Figure 5: Training loss before and after the three-bug NaN fix. ANEgpt diverges to NaN at step 2 with 100% reproducibility (red, dashed arrow indicates divergence to $\infty$ ). Orion achieves stable, monotonically decreasing loss across 5 steps with checkpoint resume (green).

Metric	Value
Resume chains tested	5
Steps per chain	5 (each in a fresh process)
Total training steps	25
NaN / Inf occurrences	0 / 25
Loss monotonically decreasing	5 / 5 chains
Step 1 loss (mean $\pm$ std across chains)	$13.975\pm 0.003$

Table 7: Training stability stress test results (Stories110M, lr=

10^{-5}

, grad_accum=4, M4 Max). Each chain is an independent 5-step resume sequence.

Step	Chain 1	Chain 2	Chain 3	Chain 4	Chain 5
1	13.979	13.978	13.975	13.974	13.971
2	13.968	13.965	13.962	13.961	13.958
3	13.946	13.949	13.945	13.941	13.935
4	13.932	13.927	13.924	13.926	13.920
5	13.923	13.916	13.911	13.911	13.904

Table 8: Per-chain loss trajectories from the stability stress test. All chains decrease monotonically across all 5 steps with zero NaN.

Configuration	Throughput	Latency (p50)
CPU decode	283 tok/s	3.48 ms/tok
ANE full forward (decode)	170 tok/s	5.76 ms/tok
ANE prefill (first call)	12 tok/s	—
ANE prefill (cached)	165 tok/s	—

Table 9: GPT-2 124M inference performance (M4 Max). CPU uses cblas_sgemm ; ANE uses compiled MIL programs via private APIs.

Panel of Figure 6 — Figure 6: GPT-2 124M inference throughput on M4 Max. First-call ANE prefill includes $\sim$ 1015 ms compilation time. CPU decode is faster due to ANE’s $\sim$ 2.3 ms IOSurface round-trip overhead per dispatch.

Metric	v1.0	v2.0
Compute time (mean)	908 ms/step	849 ms/step
Recompile / reload time	4,200 ms/step	494 ms/step
Total step time	5,108 ms/step	1,345 ms/step
Throughput	0.612 TFLOPS	0.656 TFLOPS
1000-step wall time	$\sim$ 85 min	22.4 min
NaN occurrences	0 / 1,000	0 / 1,000

Table 10: Stories110M training performance (M4 Max). v1.0 uses full recompile + exec() restart per step; v2.0 uses delta reload in a single process.

Panel of Figure 7 — Figure 7: Stories110M training loss on TinyStories over 1,000 steps (lr= $3\!\times\!10^{-4}$ , grad_accum=4). v1.0 (blue): each step in a separate process via exec() restart, $\sim$ 85 min total, loss 12.3 $\to$ 6.2. v2.0 (green): single process with delta reload, 22 min total, loss 12.3 $\to$ 9.6, zero NaN in both. The v2.0 curve plateaus higher because the single-process data loader sees a different sample ordering than v1.0’s per-process restarts; the training loop itself is equally stable. The speedup is purely mechanical: 3.8 $\times$ less wall time for the same step count.

Panel of Figure 8 — Figure 8: Per-step time breakdown: v1.0 vs v2.0. Compute time is nearly identical ( $\sim$ 850–900 ms); the 3.8 $\times$ total speedup comes entirely from replacing full ANE recompilation (4,200 ms) with delta reload (494 ms).

Panel of Figure 9 — Figure 9: 1,000-step training wall time comparison. v2.0 (delta reload) completes in 22 minutes vs $\sim$ 85 minutes for v1.0 (full recompile), a 3.8 $\times$ speedup. Both runs: Stories110M, TinyStories, lr= $3\!\times\!10^{-4}$ , grad_accum=4, zero NaN.

Panel of Figure 10 — Figure 10: ANE kernel latencies (M4 Max, log scale). Single-token dispatch shows the bare XPC+IOKit overhead ( $\sim$ 0.03 ms). The gap between dispatch and decode per-token ( $\sim$ 5.78 ms) reflects IOSurface round-trip costs across 12 transformer layers.

Operation	CPU	ANE	Speedup
Classifier fwd (embed $\times$ $x$ )	10.77 ms	1.06 ms	10.2 $\times$
Softmax (vocab=32000)	81.11 ms	2.40 ms	33.8 $\times$
RMSNorm backward	0.18 ms	0.21 ms	$\sim$ 1 $\times$

Table 11: ANE vs CPU latency for individual operations (Stories110M).

Framework	Hardware	Training	ANE Direct	Compiler	Weight Update
MLX ( Apple Machine Learning Research, 2023 )	GPU	Yes	No	Yes	In-memory
llama.cpp ( Gerganov and others, 2023 )	CPU+GPU	No	No	No	N/A
MLC-LLM ( Chen and others, 2023 )	GPU	No	No	Yes	N/A
CoreML ( Apple Inc., 2023 )	CPU/GPU/ANE	No	No	Yes	Recompile
Orion	ANE	Yes	Yes	Yes	Delta reload

Table 12: LLM frameworks on Apple silicon. Orion is unique in targeting the ANE directly for both training and inference, with delta compilation for weight updates and LoRA hot-swap.

Neural Network Hardware Acceleration

See all 37 papers

Jun 12, 2026

ANEForge: Python for direct computation on the Apple Neural Engine

ANEForge is a Python package that programs the Apple Neural Engine (ANE), the fixed-function neural accelerator on every recent Apple device, directly and without CoreML. In production the engine is reachable only through CoreML, which treats it as a scheduling option: no configuration requires the ANE, and a model can silently run on the CPU or GPU instead. ANEForge compiles a lazy tensor graph, built from 58 fused operators and 19 native bridge operators, into a single ANE program. The program is dispatched through the same ANE daemon and kernel-driver stack as Apple's internal framework. Beyond inference, the package reaches the engine's native fused attention, streams int8, int4, and sparse weights, keeps decoder and optimizer state resident across steps, and runs the forward pass, backward pass, and optimizer update of training on the engine. A small fused program completes a call in about 90us, near the engine's 70us per-program dispatch floor, and a pretrained ResNet-18 forward runs end-to-end in 0.33ms. ResNet-18, a sentence encoder, and a Vision Transformer run end-to-end against framework references, and a Stable Diffusion U-Net validates its forward pass. ANEForge targets Apple Silicon under macOS 14 and later. Each release is verified against a recorded macOS and ANE-compiler version.

Spencer H. Bryngelson

School of Computational Science & Engineering, Georgia Institute of Technology, Atlanta, GA 30332, USA · Daniel Guggenheim School of Aerospace Engineering, Georgia Institute of Technology, Atlanta, GA 30332, USA · George W. Woodruff School of Mechanical Engineering, Georgia Institute of Technology, Atlanta, GA 30332, USA

Efficient Neural Network Inference AI Accelerator Inference Neural Network Hardware Acceleration

Ask PDF

Jul 21, 2026

BaseRT: Advancing Best-in-Class LLM Inference with Apple M5 Neural Accelerators

Apple's M5 generation introduces a redesigned GPU architecture in which every core carries a dedicated Neural Accelerator: on-die matrix units exposed through the Metal4 tensor API. We show that BaseRT, our native Metal inference runtime for large language models on Apple Silicon, exploits these units to push inference throughput on Apple hardware substantially beyond both llama.cpp and MLX. Building on BaseRT's framework-free design, we add a family of hand-written Metal4 tensor-core kernels (including dense and mixture-of-experts GEMM and flash-attention prefill kernels) that route the compute-bound matrix multiplications of inference through the M5 Neural Accelerators while leaving the memory-bound decode path on our existing specialised kernels. On an Apple M5 Pro, across fifteen model configurations spanning the Qwen3, Qwen3.5/3.6, Llama~~3.2, and Gemma~~4 families from sub-1B to 35B parameters, BaseRT delivers up to

6.4\times

higher prompt-processing throughput than llama.cpp and

3.9\times

higher than MLX, with the largest margins on the mixture-of-experts models where matrix multiplication dominates, while maintaining its lead on decode of up to

1.75\times

over llama.cpp and

1.33\times

over MLX. These results establish a new performance ceiling for on-device LLM inference and show that the M5's tensor cores are the decisive lever for prompt processing on Apple Silicon. BaseRT is publicly available at https://github.com/basecompute/baseRT.

Fabian Waschkowski, Prabod Rathnayaka, Lukas Wesemann

Base Compute, Melbourne, Australia

AI Accelerator Inference On-Device Language Model Inference LLM Inference Acceleration Neural Network Hardware Acceleration

Ask PDF

Oct 17, 2026

First-Order LoRA Training with Phone NPUs

On-device fine-tuning lets a large language model (LLM) learn from its user's private data without uploading it. Low-rank adaptation (LoRA) reduces the trainable parameters, but each first-order training step still multiplies by the frozen base-model weights during the forward pass, activation recomputation, and input-gradient computation. A phone's neural processing unit (NPU) already holds these weights for inference; using it for training also requires computing input gradients, which its inference interface does not support. We present a system that runs all three frozen-weight products on the NPU while keeping the training graph, adapters, and optimizer on the CPU. Its input-gradient kernels reuse the resident weight tiles and the forward dequantizer, avoiding a transposed weight copy and ensuring forward and backward use the same effective weights. To preserve usable gradients, the system selects precision by gradient sensitivity and applies a per-row range guard for half-precision (FP16) outputs. Its runtime fuses the remaining CPU operators and overlaps NPU execution with adapter computation, while resumable execution and thermal pacing let training proceed as a background job. We evaluate it on two Qualcomm Snapdragon platforms using four LLMs from the Qwen and Llama families. Our workloads cover nightly personalization with labels supplied on the phone and the public LaMP-2 benchmark; comparisons include an optimized CPU trainer, a phone-GPU trainer, forward-only training, and GPU-trained references.

Xu Zheng, Wei Cheng, Zhuomin Chen +4

Florida International University, Miami, FL, USA · NEC Laboratories America, Princeton, NJ, USA · North Carolina State University, Raleigh, NC, USA +2

On-Device Training LLM Fine-Tuning Neural Network Hardware Acceleration Low-Rank Adaptation

Ask PDF

Abstract

Neural Network Hardware Acceleration

Explore similar work

ANEForge: Python for direct computation on the Apple Neural Engine

BaseRT: Advancing Best-in-Class LLM Inference with Apple M5 Neural Accelerators

First-Order LoRA Training with Phone NPUs