te-comp: A Complete MLIR Compiler Generating Self-Contained C Inference Engines for MCUs
Organizations: TeComp AI, CA, US.
Abstract
Deploying deep learning on microcontrollers (MCUs) requires fitting both the model weights and runtime activations within tens to hundreds of kilobytes of SRAM and Flash. Existing approaches either use an interpreter with a pre-written kernel library or combine a Python code generator with hand-written C kernels. This incurs dispatch overhead and requires framework dependencies. None produces self-contained, dependency-free C code specialized to a given model. In this paper, we present te-comp, the first complete MLIR-based compiler that takes a standard model in .tflite or .safetensors format and emits all C source files needed for deployment: the inference engine (kernels, weight arrays, memory layout) and the RTOS integration layer with zero runtime dependencies. On 10 MCUNet CNN benchmarks, te-comp outperforms MCUNet TinyEngine by using 38% less SRAM (up to 5.9) and 15% less Flash, while generating complete C code rather than linking a pre-written kernel library. To demonstrate, we use te-comp to compile an 8.3M-parameter GPT-Neo transformer (TinyStories-v2) and deploy it on a single-core Cortex-M7 at 160MHz. The full firmware uses 249KB of 512KB SRAM and 3.62MB of 4MB Flash and achieves approximately 1 token/s interactive text generation via Q4_0 weight quantization, a stateful int4 KV cache, and vocabulary pruning -- a capability no existing MCU inference framework supports.
Figures & tables
| Category | Operations |
|---|---|
| CNN | conv2d , depthwise_conv2d , |
| (22 ops) | transpose_conv2d , fully_connected , |
| avg_pool2d , max_pool2d , | |
| add , mul , max , min , | |
| clamp , rescale , softmax , | |
| sigmoid , tanh , hard_swish , |
| Model | MACs | #Params | MCU SRAM (KB) | te SRAM (KB) | SRAM ratio | MCU Flash (KB) | te Flash (KB) | Flash ratio |
|---|---|---|---|---|---|---|---|---|
| mcunet-in0 | 6.4M | 0.75M | 266 | 45 | 0.17 | 889 | 823 | 0.93 |
| mcunet-in1 | 12.8M | 0.64M | 307 | 90 | 0.29 | 992 | 704 | 0.71 |
| mcunet-in2 | 67.3M | 0.73M | 242 | 200 | 0.83 | 878 | 816 | 0.93 |
| mcunet-in3 | 81.8M | 0.74M | 293 | 242 | 0.83 | 897 | 816 | 0.91 |
| mcunet-in4 | 125.9M | 1.73M | 456 | 400 | 0.88 | 1876 | 1823 | 0.97 |
| mbv2-w0.35 | 23.5M | 0.75M | 308 | 283 | 0.92 | 862 | 820 | 0.95 |
| Model | Out | Tens. | Max LSB | RMSE | Elem. err | TFL GT | Eng GT |
| Deep mobile architectures (50–70+ layers) | |||||||
| mcunet-320kb-vww | 2 | 353 | 2 | 1.118 | 65.0% | 9/10 | 9/10 |
| mcunet-10fps-vww | 2 | 225 | 3 | 1.449 | 75.0% | 9/10 | 9/10 |
| mcunet-5fps-vww | 2 | 241 | 3 | 1.449 | 85.0% | 9/10 | 9/10 |
| mbv2-w0.35-r144 | 1000 | 278 | 7 | 1.403 | 70.5% | 10/10 | 10/10 |
| mcunet-256kb-in | 1000 | 289 | 7 | 1.608 | 74.0% | 10/10 | 10/10 |
| Component | Size | Section |
|---|---|---|
| Q4_0 weights + pruned embedding | 3.02 MB | .rodata |
| BPE tokenizer blob | 230 KB | .rodata |
| Generated engine + firmware code | 382 KB | .text |
| Total Flash ( .text ) | 3.62 MB | Fits 4 MB |
| Int4 KV cache + scales | 108 KB | .bss |
| Activation arena | 58 KB | .bss |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| # | Step / Flag | Summary |
|---|---|---|
| 1 | Input conversion | .tflite / .safetensors TOSA MLIR (Python converters) |
| 2 | --te-flash-weights | Mark large constants as flash-resident (zero SRAM) |
| 3 | --te-kv-cache L | Insert stateful KV cache; must precede weight quantization |
| 4 | --te-quantize-weights L | Q4_0 or int8 weight quantization |
| 5 | --te-parallel-matmul L | Multi-core row splitting |
| 6 | --convert-tosa-to-te | TOSA te with op fusion |
| Category | # | Operation | Description |
| CNN (22 ops) | 1 | conv2d | Quantized 2D conv with fused bias, per-channel requantize (mult+shift), and activation clamp. |
| 2 | depthwise_conv2d | Depthwise 2D conv (channel-wise filter). Supports in_place flag for output-aliased execution. | |
| 3 | transpose_conv2d | Quantized transposed (deconvolution) 2D conv with fused requantize and clamp. | |
| 4 | fully_connected | Quantized dense layer: . | |
| 5 | avg_pool2d | Quantized average pooling with optional fused rescale. | |
| 6 | max_pool2d | Max pooling (operates directly on int8 values; no rescale). |
| Dialect | Operation | Description / HW Instruction |
| arm_dsp | smlad | Dual 16 16 signed MAC (SMLAD) |
| sxtb16 | Sign-extend bytes 0,2 to halfwords (SXTB16) | |
| sadd16 | SIMD add of two halfword pairs (SADD16) | |
| pkhbt | Pack bottom/top halfwords (PKHBT) | |
| requantize | Per-channel requant (SMULL + shift) | |
| xtensa | ee_vmulas_s8 | 16 int8 vector MAC (PIE) |
| # | Pass flag | Category | Description |
| LLM pre-processing passes (run before TOSA te conversion) | |||
| 1 | --te-flash-weights | LLM / CNN | Marks constants above a size threshold with te.flash_resident ; memory planner skips them (zero SRAM), C emitter places them in .rodata . |
| 2 | --te-kv-cache | LLM | Detects attention patterns (reshape ) and inserts te.kv_cache_write / te.kv_cache_attention ops for stateful autoregressive generation. Replaces static wpe with te.pos_embed . |
| 3 | --te-quantize-weights | LLM | Quantizes fp32 weight matrices to Q4_0 (4-bit, bf16 scales, 7 compression) or int8 ( 4 ). Reports per-layer RMSE/SNR at compile time. |
| 4 | --te-parallel-matmul | LLM | Annotates large matmuls with row-split attributes ( te.parallel_split , te.core_ranges ) for multi-core execution. Supports equal and asymmetric (core-weights) splitting. |
| Dialect conversion | |||
Efficient Neural Network Inference
Efficient neural network inference reduces the time, memory, and energy needed to run trained models across hardware and deployment settings.
See all 507 papers