Deploying deep learning on microcontrollers (MCUs) requires fitting both the model weights and runtime activations within tens to hundreds of kilobytes of SRAM and Flash. Existing approaches either use an interpreter with a pre-written kernel library or combine a Python code generator with hand-written C kernels. This incurs dispatch overhead and requires framework dependencies. None produces self-contained, dependency-free C code specialized to a given model. In this paper, we present te-comp, the first complete MLIR-based compiler that takes a standard model in .tflite or .safetensors format and emits all C source files needed for deployment: the inference engine (kernels, weight arrays, memory layout) and the RTOS integration layer with zero runtime dependencies. On 10 MCUNet CNN benchmarks, te-comp outperforms MCUNet TinyEngine by using 38% less SRAM (up to 5.9×) and 15% less Flash, while generating complete C code rather than linking a pre-written kernel library. To demonstrate, we use te-comp to compile an 8.3M-parameter GPT-Neo transformer (TinyStories-v2) and deploy it on a single-core Cortex-M7 at 160MHz. The full firmware uses 249KB of 512KB SRAM and 3.62MB of 4MB Flash and achieves approximately 1 token/s interactive text generation via Q4_0 weight quantization, a stateful int4 KV cache, and vocabulary pruning -- a capability no existing MCU inference framework supports.
Figures & tables
Figure 1: A model file is converted to TOSA MLIR, lowered to the te dialect with op fusion, optimized by composable passes, lowered to loops with DSP intrinsics, and emitted as self-contained C source files.
Category
Operations
CNN
conv2d , depthwise_conv2d ,
(22 ops)
transpose_conv2d , fully_connected ,
avg_pool2d , max_pool2d ,
add , mul , max , min ,
clamp , rescale , softmax ,
sigmoid , tanh , hard_swish ,
Table 1: The 34 operations in the te dialect.
Model
MACs
#Params
MCU SRAM (KB)
te SRAM (KB)
SRAM ratio
MCU Flash (KB)
te Flash (KB)
Flash ratio
mcunet-in0
6.4M
0.75M
266
45
0.17 ×
889
823
0.93 ×
mcunet-in1
12.8M
0.64M
307
90
0.29 ×
992
704
0.71 ×
mcunet-in2
67.3M
0.73M
242
200
0.83 ×
878
816
0.93 ×
mcunet-in3
81.8M
0.74M
293
242
0.83 ×
897
816
0.91 ×
mcunet-in4
125.9M
1.73M
456
400
0.88 ×
1876
1823
0.97 ×
mbv2-w0.35
23.5M
0.75M
308
283
0.92 ×
862
820
0.95 ×
Table 2: Peak SRAM and Flash of te-comp vs. MCUNet TinyEngine on 10 benchmark models. MCUNet numbers from Lin et al. (2020) .
Model
Out
Tens.
Max LSB
RMSE
Elem. err
TFL → GT
Eng → GT
Deep mobile architectures (50–70+ layers)
mcunet-320kb-vww
2
353
2
1.118
65.0%
9/10
9/10
mcunet-10fps-vww
2
225
3
1.449
75.0%
9/10
9/10
mcunet-5fps-vww
2
241
3
1.449
85.0%
9/10
9/10
mbv2-w0.35-r144
1000
278
7
1.403
70.5%
10/10
10/10
mcunet-256kb-in
1000
289
7
1.608
74.0%
10/10
10/10
Table 3: Numerical accuracy of te-comp vs. TFLite reference interpreter on 10 int8 models (10 real test samples per model). Max LSB = largest element-wise absolute difference. Elem. err = fraction of output elements differing by ≥1 LSB. TFL → GT and Eng → GT = number of inputs (out of 10) where TFLite / te-comp predict the correct ground-truth label.
Component
Size
Section
Q4_0 weights + pruned embedding
3.02 MB
.rodata
BPE tokenizer blob
230 KB
.rodata
Generated engine + firmware code
382 KB
.text
Total Flash ( .text )
3.62 MB
Fits 4 MB
Int4 KV cache + scales
108 KB
.bss
Activation arena
58 KB
.bss
Table 4: Memory usage of TinyStories-v2 (pruned 14.6K vocabulary) on S32K344 MCU.
Mark large constants as flash-resident (zero SRAM)
3
--te-kv-cache L
Insert stateful KV cache; must precede weight quantization
4
--te-quantize-weights L
Q4_0 or int8 weight quantization
5
--te-parallel-matmul L
Multi-core row splitting
6
--convert-tosa-to-te
TOSA → te with op fusion
Appendix
Table 5: End-to-end compilation pipeline in recommended order. Steps marked L are LLM-only; C are CNN-only; unmarked steps apply to both.
Category
#
Operation
Description
CNN (22 ops)
1
conv2d
Quantized 2D conv with fused bias, per-channel requantize (mult+shift), and activation clamp.
2
depthwise_conv2d
Depthwise 2D conv (channel-wise filter). Supports in_place flag for output-aliased execution.
3
transpose_conv2d
Quantized transposed (deconvolution) 2D conv with fused requantize and clamp.
4
fully_connected
Quantized dense layer: clamp(requant(xWT+b)) .
5
avg_pool2d
Quantized average pooling with optional fused rescale.
6
max_pool2d
Max pooling (operates directly on int8 values; no rescale).
Appendix
Table 6: Complete reference of the 34 te dialect operations. All operations are prefixed with te. in MLIR assembly (e.g., te.conv2d ).
Dialect
Operation
Description / HW Instruction
arm_dsp
smlad
Dual 16 × 16 signed MAC (SMLAD)
sxtb16
Sign-extend bytes 0,2 to halfwords (SXTB16)
sadd16
SIMD add of two halfword pairs (SADD16)
pkhbt
Pack bottom/top halfwords (PKHBT)
requantize
Per-channel requant (SMULL + shift)
xtensa
ee_vmulas_s8
16 × int8 vector MAC (PIE)
Appendix
Table 7: Per-target intrinsic dialect operations.
#
Pass flag
Category
Description
LLM pre-processing passes (run before TOSA → te conversion)
1
--te-flash-weights
LLM / CNN
Marks constants above a size threshold with te.flash_resident ; memory planner skips them (zero SRAM), C emitter places them in .rodata .
2
--te-kv-cache
LLM
Detects attention patterns (reshape [B,S,D]→[B,S,nh,dh] ) and inserts te.kv_cache_write / te.kv_cache_attention ops for stateful autoregressive generation. Replaces static wpe with te.pos_embed .
3
--te-quantize-weights
LLM
Quantizes fp32 weight matrices to Q4_0 (4-bit, bf16 scales, ∼ 7 × compression) or int8 ( ∼ 4 × ). Reports per-layer RMSE/SNR at compile time.
4
--te-parallel-matmul
LLM
Annotates large matmuls with row-split attributes ( te.parallel_split , te.core_ranges ) for multi-core execution. Supports equal and asymmetric (core-weights) splitting.
Dialect conversion
Appendix
Table 8: Complete reference of all compilation and optimization passes in te-comp , listed in recommended pipeline order.