MATCHA: Efficient Deployment of Deep Neural Networks on Multi-Accelerator Heterogeneous Edge SoCs

Enrico Russo; Mohamed Amine Hamdi; Alessandro Ottaviano; Francesco Conti; Angelo Garofalo; Daniele Jahier Pagliari; Maurizio Palesi; Luca Benini

cs.DCApr 10, 2026

MATCHA: Efficient Deployment of Deep Neural Networks on Multi-Accelerator Heterogeneous Edge SoCs

Authors: Enrico Russo, Mohamed Amine Hamdi, Alessandro Ottaviano, Francesco Conti, Angelo Garofalo, Daniele Jahier Pagliari, Maurizio Palesi, Luca Benini,

Organizations: University of Catania, Italy · Politecnico di Torino, Italy · ETH Zurich, Switzerland · University of Bologna, Italy

Abstract

Deploying DNNs on System-on-Chips (SoC) with multiple heterogeneous acceleration engines is challenging, and the majority of deployment frameworks cannot fully exploit heterogeneity. We present MATCHA, a unified DNN deployment framework that generates highly concurrent schedules for parallel, heterogeneous accelerators and uses constraint programming to optimize L3/L2 memory allocation and scheduling. Using pattern matching, tiling, and mapping across individual HW units enables parallel execution and high accelerator utilization. On the MLPerf Tiny benchmark, using a SoC with two heterogeneous accelerators, MATCHA improves accelerator utilization and reduces inference latency by up to 35% with respect to the the state-of-the-art MATCH compiler.

Figures & tables

Work	Open Source	HSoC Support	Operator Extension	Device Extension	Multi ISA	Async. Execution	Pattern Tiling
TFLite ( David et al., 2021 )					$\bullet$		$\bullet$
TelaMalloc ( Maas et al., 2022 )					$\bullet$	$\bullet$
Bolt ( Xing et al., 2022 )					$\bullet$
COMB ( Zheng et al., 2023 )
HTVM ( Van Delm et al., 2023 )
DORY ( Burrello et al., 2021 )

Table 1. Comparison of state-of-the-art DNN deployment tools.

Figure 1. MATCHA deployment framework: inputs (left) and pipeline stages (right).

Figure 2. Conventional pattern matching in a heterogeneous system resulting in sequential execution across devices.

Figure 3. MATCHA’s tile-centric pattern matching and tiling with asynchronous execution across heterogeneous devices.

Figure 4. Scheduling and memory plan example with different types of tensor allocation.

Figure 5. Use case Carfield HSoC considered for evaluation.

Model	MACs	Params	Cycles	Runtime	FLOPS	Cycles	Runtime	FLOPS	Cycles	Runtime	FLOPS	Cycles	Runtime	FLOPS
			TVM			MATCH			MATCHA (No Tiling)			MATCHA (Ours)
AutoEncoder	0.27M	268k	5.03M	100.2 ms	5.36M	1.01M	20.1 ms	26.7M	1.01M	20.1 ms	26.7M	0.673M	13.4 ms	40.1M
DS-CNN	2.8M	22.6k	30.3M	604.6 ms	9.29M	6.58M	131.1 ms	42.9M	6.58M	131.1 ms	42.9M	6.58M	131.1 ms	42.9M
MobileNet	7.9M	210k	157.4M	3137.8 ms	5.07M	24.4M	486.7 ms	32.7M	24.4M	486.7 ms	32.7M	24.4M	486.7 ms	32.7M
ResNet18	12.8M	78k	200.2M	3991.7 ms	6.43M	22.9M	456.6 ms	56.2M	19.9M	395.9 ms	64.9M	16.3M	325.1 ms	79.0M

Table 2. MLPerf Benchmark results. MATCH employs always the best accelerator for each layer (Spatz / Pulp).

Figure 6. ResNet inference profiling timeline (left) and the execution time breakdown (right) across different devices.

Figure 7. FLOPS comparison for DNN benchmark blocks.

Efficient Neural Network Inference

Efficient neural network inference reduces the time, memory, and energy needed to run trained models across hardware and deployment settings.

See all 465 papers

Multi-DNN Inference of Sparse Models on Edge SoCs
Mar 10, 2026Jiawei Luo, Di Wu, Simon Dobson +1Efficient Neural Network Inference Edge Inference
School of Computer Science, University of St Andrews, Scotland UK
Terastal: Layer-Variant-based Scheduling for Real-Time Multi-DNN Workloads on Heterogeneous Accelerators
Jun 5, 2026Sing-Yao Wu, Fengshuo Song, Eli BozorgzadehAI Accelerator Inference Real-Time Scheduling
Department of Computer Science, University of California, Irvine, CA, USA
Para-Pipe: Exploiting Hierarchical Operator Parallelism of ML Computational Graphs on SoCs
Sep 3, 2026Yujie Zhang, Huiying Lan, Ehsan Aghapour +5Efficient Neural Network Inference Energy-Efficient ML
School of Computing, National University of Singapore, Singapore · Parallel Computer Systems, the University of Amsterdam, 1098 XH Amsterdam, The Netherlands · Information Technology Department, Black Sesame Technologies, San Jose, CA 95131, USA

Mar 10, 2026J/K move · Enter open · S save

Multi-DNN Inference of Sparse Models on Edge SoCs

Jiawei Luo, Di Wu, Simon Dobson, Blesson Varghese

School of Computer Science, University of St Andrews, Scotland UK

Efficient Neural Network Inference Edge Inference Sparse Neural Networks

Open paper AskPDF arXiv

Modern edge applications increasingly require multi-DNN inference systems to execute tasks on heterogeneous processors, gaining performance from both concurrent execution and from matching each model to the most suited accelerator. However, existing systems support only a single model (or a few sparse variants) per task, which impedes the efficiency of this matching and results in high Service Level Objective violation rates. We introduce model stitching for multi-DNN inference systems, which creates model variants by recombining subgraphs from sparse models without re-training. We present a demonstrator system, SparseLoom, that shows model stitching can be deployed to SoCs. We show experimentally that SparseLoom reduces SLO violation rates by up to 74%, improves throughput by up to 2.31x, and lowers memory overhead by an average of 28% compared to state-of-the-art multi-DNN inference systems.

Abstract

Efficient Neural Network Inference

Explore similar work

Multi-DNN Inference of Sparse Models on Edge SoCs

Terastal: Layer-Variant-based Scheduling for Real-Time Multi-DNN Workloads on Heterogeneous Accelerators

Para-Pipe: Exploiting Hierarchical Operator Parallelism of ML Computational Graphs on SoCs