Multi-DNN Inference of Sparse Models on Edge SoCs

Jiawei Luo; Di Wu; Simon Dobson; Blesson Varghese

cs.DCMar 10, 2026

Multi-DNN Inference of Sparse Models on Edge SoCs

Authors: Jiawei Luo, Di Wu, Simon Dobson, Blesson Varghese

Organizations: School of Computer Science, University of St Andrews, Scotland UK

Abstract

Modern edge applications increasingly require multi-DNN inference systems to execute tasks on heterogeneous processors, gaining performance from both concurrent execution and from matching each model to the most suited accelerator. However, existing systems support only a single model (or a few sparse variants) per task, which impedes the efficiency of this matching and results in high Service Level Objective violation rates. We introduce model stitching for multi-DNN inference systems, which creates model variants by recombining subgraphs from sparse models without re-training. We present a demonstrator system, SparseLoom, that shows model stitching can be deployed to SoCs. We show experimentally that SparseLoom reduces SLO violation rates by up to 74%, improves throughput by up to 2.31x, and lowers memory overhead by an average of 28% compared to state-of-the-art multi-DNN inference systems.

Figures & tables

Figure 1. An AR use case of multi-DNN inference on a heterogeneous edge SoC. The application executes four tasks in parallel on the CPU, GPU, and NPU. Each task is provisioned with a sparse model zoo to meet varying SLO requirements.

Figure 2. Model stitching: given three sparse models (dense, pruned, quantized) each split into subgraphs S1–S3, Stitched Variant 1 combines S1 from the Dense Model (blue), S2 from the Pruned Variant (purple), and S3 from the Quantized Variant (orange).

Figure 3. SLO violations with vs. without stitching. Stitching substantially reduces SLO violation rate. The x-axis represents different SLO configurations, where larger index (e.g., C8) corresponds to more challenging SLO with stricter accuracy and latency requirements.

Panel of Figure 4 — Figure 4. Histogram of ResNet101 variants in the accuracy–latency space; cell counts show density, and red-edged cells indicate the Pareto frontier. Model stitching expand the variant space and achieve a better accuracy–latency frontier than original sparse models.

Method	Without Stitching	With Stitching
Processor placement orders	$P!$	$P!$
Total variants	$T\cdot V$	$T\cdot V^{S}$
Accuracy profiling runs	$T\cdot V$	$T\cdot V^{S}$
Latency profiling runs	$T\cdot V\cdot P!$	$T\cdot V^{S}\cdot P!$
Total profiling runs	$T\cdot V\cdot(P!+1)$	$T\cdot V^{S}\cdot(P!+1)$

Table 1. Profiling complexity with and without model stitching. Notation:

T

: #tasks;

V

: #variants per task;

S

: #subgraphs per variant;

P

: #processors.

Panel of Table 2 — Table 2. Latency of stitched ResNet101 variants with different processor placement orders. The lowest latency is highlighted in blue. P: Pruned, Q: Quantized, D: Dense; C: CPU, G: GPU, N: NPU.

Panel of Figure 5 — Figure 5. (a) Latency breakdown, including compilation, loading, and inference. (b) Memory breakdown, including active variants, preloaded variants, others.

Figure 6. Three phases of multi-DNN inference in SparseLoom .

Panel of Figure 7 — Figure 7. Evaluation of the proposed estimators. (a) Accuracy estimator: Top-K recall on highest-accuracy variants. (b) Latency estimator vs. ground-truth.

Panel of Figure 8 — Figure 8. Profiling runs of SparseLoom with and without estimators for varying $T$ and $V$ .

Figure 9. Hotness scores of all subgraphs at the third position, with top four subgraphs are dominant.

Platform	CPU	GPU	NPU
Desktop (Intel Core Ultra 7 265K)	x86-64; 20-core	4- $\text{X}^{e}$ core	Intel AI Boost
Laptop (Intel Core Ultra 5 135U)	x86-64; 12-core	4- $\text{X}^{e}$ core	Intel AI Boost
NVIDIA Jetson AGX Orin (MAXN mode)	ARM Cortex 12-core	2048-core Ampere; 64 Tensor cores	N/A

Table 3. Platforms used in the evaluation.

Task type	Dataset	Base Model
Image classification	ImageNet-1K ( Deng et al., 2009 )	ResNet-101 ( He et al., 2016 )
Sentiment classification	SST-2 ( Socher et al., 2013 )	BERT-Base ( Devlin et al., 2019 )
Human activity recognition	HAR ( Meet Nagadia, 2021 )	ViT-Small ( Dosovitskiy et al., 2021 )
Speech recognition	LibriSpeech ASR ( Panayotov et al., 2015 )	Wav2vec2 ( Baevski et al., 2020 )

Table 4. Tasks, datasets, and models for evaluation. The selected tasks and models are representative of the AR use case discussed in Section 1 .

Panel of Figure 10 — Figure 10. SLO violation rates of SparseLoom and baselines across three SoCs. SparseLoom consistently achieves lower violation rates than all baselines, with reductions of up to 74%.

Panel of Figure 11 — Figure 11. Inference throughput of SparseLoom and baselines across three SoCs. SparseLoom consistently achieves higher throughput than all baselines, with improvements of up to 2.31 $\times$ .

Panel of Figure 12 — Figure 12. Profiling time (minutes) for SparseLoom with and without estimators. SparseLoom reduces profiling cost to under 8 minutes (up to 99% reduction) on all SoCs.

Panel of Figure 13 — Figure 13. Impact of processor placement order on inference throughput (C = CPU, G = GPU, N = NPU). SparseLoom automatically selects the best placement, achieving up to $2\times$ higher throughput.

Panel of Figure 14 — Figure 14. Impact of memory budget (percentage of memory required for fully preloading all subgraphs) on SLO violation. SparseLoom cuts memory usage by average of 28% without increasing the violation rate.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Platform	Variant Type	Sparse Pattern
Intel SoCs (Desktop / Laptop)	Base model	Dense (FP32)
	Quantized model	Dense (INT8)
	Unstructured pruning	90%, FP32
		85%, FP32
		80%, FP32
		75%, FP32

Appendix

Table 5. Sparse variants on each platform. Sparse Pattern indicates sparsity level and numerical precision of the model parameters. Note:

x

% sparsity means

x

% weights are pruned.

Notation	Description
$t,T$	Task index and total number of tasks
$i,V$	Variant index and variants per task
$j,S$	Subgraph index and subgraphs per variant
$k$	Stitched variant index (total $V^{S}$ variants)
$v^{t,i}$	Variant $i$ in task $t$
$s_{j}^{t,i}$	Subgraph $j$ of variant $i$ in task $t$

Appendix

Table 6. Summary of notation.

Features	Class 1 ( Han et al., 2024 ; Wang et al., 2019 ; Han et al., 2022 )	Class 2 ( Shen et al., 2025 ; Jeong et al., 2022 ; Karatzas and Anagnostopoulos, 2023 )	Class 3 ( Taufique et al., 2024 ; Sen and Shen, 2024 ; Fang et al., 2018 )	Variants Generation Methods ( Sanh et al., 2020 ; Ro and Choi, 2021 )	SparseLoom
Data- and training-free variant generation	✗	✗	✗	✗	✓
Multiple variants selection	✗	✗	✓	✗	✓
# model variants supported	1	1	Scale of 10	Scale of 10	Scale of 1000
Heterogeneous subgraph placement	✗	✓	✗	✗	✓
Optimized sparsity-aware placement	✗	✗	✗	✗	✓
Memory-efficient preloading	✗	✗	✗	✗	✓

Appendix

Table 7. Key features of different classes of multi-DNN inference systems, variants generation methods and SparseLoom . Class 1 : systems that optimize task-level scheduling on a single processor, Class 2 : systems that optimize subgraph placement on heterogeneous processors, and Class 3 : systems that support the selection of multiple variants for diverse SLOs. Variant-generation methods generates fine-grained sparse variants.

Panel of Figure 15 — Figure 15. SLO violation rates of SparseLoom and baselines across three SoCs under accuracy-guaranteed SLOs. SparseLoom consistently achieves lower violation rates than all baselines, with reductions of up to 73.6%.

Panel of Figure 16 — Figure 16. SLO violation rates of SparseLoom and baselines across three SoCs under latency-guaranteed SLOs. SparseLoom consistently achieves lower violation rates than all baselines, with reductions of up to 68.2%..

Efficient Neural Network Inference

Efficient neural network inference reduces the time, memory, and energy needed to run trained models across hardware and deployment settings.

See all 465 papers

MATCHA: Efficient Deployment of Deep Neural Networks on Multi-Accelerator Heterogeneous Edge SoCs
Apr 10, 2026Enrico Russo, Mohamed Amine Hamdi, Alessandro Ottaviano +6Efficient Neural Network Inference Neural Network Hardware Acceleration
University of Catania, Italy · Politecnico di Torino, Italy · ETH Zurich, Switzerland +1
MANE: A Multi-Path Adaptive Network for Edge Onloading of Deep Neural Networks
Sep 13, 2026Sokratis Nikolaidis, Stylianos I. Venieris, Leonidas Malachias +1Edge Inference Distributed Inference
National Technical University of Athens, Athens, Greece · Samsung AI Center, Cambridge, UK
Memory-Efficient Partitioned DNN Inference on Resource-Constrained Android Crowds
May 20, 2026Lakshani Manamperi, Disumi Pathirana, Thiwanka Pathirana +2Efficient Neural Network Inference Edge Computing
Department of Computer Science and Engineering, University of Moratuwa, Moratuwa, Sri Lanka.

Apr 10, 2026J/K move · Enter open · S save

MATCHA: Efficient Deployment of Deep Neural Networks on Multi-Accelerator Heterogeneous Edge SoCs

Enrico Russo, Mohamed Amine Hamdi, Alessandro Ottaviano, Francesco Conti+5

University of Catania, Italy · Politecnico di Torino, Italy · ETH Zurich, Switzerland+1

Efficient Neural Network Inference Neural Network Hardware Acceleration Heterogeneous Computing Edge Computing

Open paper AskPDF arXiv

Deploying DNNs on System-on-Chips (SoC) with multiple heterogeneous acceleration engines is challenging, and the majority of deployment frameworks cannot fully exploit heterogeneity. We present MATCHA, a unified DNN deployment framework that generates highly concurrent schedules for parallel, heterogeneous accelerators and uses constraint programming to optimize L3/L2 memory allocation and scheduling. Using pattern matching, tiling, and mapping across individual HW units enables parallel execution and high accelerator utilization. On the MLPerf Tiny benchmark, using a SoC with two heterogeneous accelerators, MATCHA improves accelerator utilization and reduces inference latency by up to 35% with respect to the the state-of-the-art MATCH compiler.

Abstract

Efficient Neural Network Inference

Explore similar work

MATCHA: Efficient Deployment of Deep Neural Networks on Multi-Accelerator Heterogeneous Edge SoCs

MANE: A Multi-Path Adaptive Network for Edge Onloading of Deep Neural Networks

Memory-Efficient Partitioned DNN Inference on Resource-Constrained Android Crowds