Modern edge applications increasingly require multi-DNN inference systems to execute tasks on heterogeneous processors, gaining performance from both concurrent execution and from matching each model to the most suited accelerator. However, existing systems support only a single model (or a few sparse variants) per task, which impedes the efficiency of this matching and results in high Service Level Objective violation rates. We introduce model stitching for multi-DNN inference systems, which creates model variants by recombining subgraphs from sparse models without re-training. We present a demonstrator system, SparseLoom, that shows model stitching can be deployed to SoCs. We show experimentally that SparseLoom reduces SLO violation rates by up to 74%, improves throughput by up to 2.31x, and lowers memory overhead by an average of 28% compared to state-of-the-art multi-DNN inference systems.
Figures & tables
Figure 1. An AR use case of multi-DNN inference on a heterogeneous edge SoC. The application executes four tasks in parallel on the CPU, GPU, and NPU. Each task is provisioned with a sparse model zoo to meet varying SLO requirements.
Figure 2. Model stitching: given three sparse models (dense, pruned, quantized) each split into subgraphs S1–S3, Stitched Variant 1 combines S1 from the Dense Model (blue), S2 from the Pruned Variant (purple), and S3 from the Quantized Variant (orange).
Figure 3. SLO violations with vs. without stitching. Stitching substantially reduces SLO violation rate. The x-axis represents different SLO configurations, where larger index (e.g., C8) corresponds to more challenging SLO with stricter accuracy and latency requirements.
Figure 4. Histogram of ResNet101 variants in the accuracy–latency space; cell counts show density, and red-edged cells indicate the Pareto frontier. Model stitching expand the variant space and achieve a better accuracy–latency frontier than original sparse models.
Method
Without Stitching
With Stitching
Processor placement orders
P!
P!
Total variants
T⋅V
T⋅VS
Accuracy profiling runs
T⋅V
T⋅VS
Latency profiling runs
T⋅V⋅P!
T⋅VS⋅P!
Total profiling runs
T⋅V⋅(P!+1)
T⋅VS⋅(P!+1)
Table 1. Profiling complexity with and without model stitching. Notation: T : #tasks; V : #variants per task; S : #subgraphs per variant; P : #processors.
Table 2. Latency of stitched ResNet101 variants with different processor placement orders. The lowest latency is highlighted in blue. P: Pruned, Q: Quantized, D: Dense; C: CPU, G: GPU, N: NPU.
Figure 5. (a) Latency breakdown, including compilation, loading, and inference. (b) Memory breakdown, including active variants, preloaded variants, others.
Figure 6. Three phases of multi-DNN inference in SparseLoom .
Figure 7. Evaluation of the proposed estimators. (a) Accuracy estimator: Top-K recall on highest-accuracy variants. (b) Latency estimator vs. ground-truth.
Figure 8. Profiling runs of SparseLoom with and without estimators for varying T and V .
Figure 9. Hotness scores of all subgraphs at the third position, with top four subgraphs are dominant.
Platform
CPU
GPU
NPU
Desktop (Intel Core Ultra 7 265K)
x86-64; 20-core
4- Xe core
Intel AI Boost
Laptop (Intel Core Ultra 5 135U)
x86-64; 12-core
4- Xe core
Intel AI Boost
NVIDIA Jetson AGX Orin (MAXN mode)
ARM Cortex 12-core
2048-core Ampere; 64 Tensor cores
N/A
Table 3. Platforms used in the evaluation.
Task type
Dataset
Base Model
Image classification
ImageNet-1K ( Deng et al., 2009 )
ResNet-101 ( He et al., 2016 )
Sentiment classification
SST-2 ( Socher et al., 2013 )
BERT-Base ( Devlin et al., 2019 )
Human activity recognition
HAR ( Meet Nagadia, 2021 )
ViT-Small ( Dosovitskiy et al., 2021 )
Speech recognition
LibriSpeech ASR ( Panayotov et al., 2015 )
Wav2vec2 ( Baevski et al., 2020 )
Table 4. Tasks, datasets, and models for evaluation. The selected tasks and models are representative of the AR use case discussed in Section 1 .
Figure 10. SLO violation rates of SparseLoom and baselines across three SoCs. SparseLoom consistently achieves lower violation rates than all baselines, with reductions of up to 74%.
Figure 11. Inference throughput of SparseLoom and baselines across three SoCs. SparseLoom consistently achieves higher throughput than all baselines, with improvements of up to 2.31 × .
Figure 12. Profiling time (minutes) for SparseLoom with and without estimators. SparseLoom reduces profiling cost to under 8 minutes (up to 99% reduction) on all SoCs.
Figure 13. Impact of processor placement order on inference throughput (C = CPU, G = GPU, N = NPU). SparseLoom automatically selects the best placement, achieving up to 2× higher throughput.
Figure 14. Impact of memory budget (percentage of memory required for fully preloading all subgraphs) on SLO violation. SparseLoom cuts memory usage by average of 28% without increasing the violation rate.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Platform
Variant Type
Sparse Pattern
Intel SoCs (Desktop / Laptop)
Base model
Dense (FP32)
Quantized model
Dense (INT8)
Unstructured pruning
90%, FP32
85%, FP32
80%, FP32
75%, FP32
Appendix
Table 5. Sparse variants on each platform. Sparse Pattern indicates sparsity level and numerical precision of the model parameters. Note: x % sparsity means x % weights are pruned.
Notation
Description
t,T
Task index and total number of tasks
i,V
Variant index and variants per task
j,S
Subgraph index and subgraphs per variant
k
Stitched variant index (total VS variants)
vt,i
Variant i in task t
sjt,i
Subgraph j of variant i in task t
Appendix
Table 6. Summary of notation.
Features
Class 1 ( Han et al., 2024 ; Wang et al., 2019 ; Han et al., 2022 )
Class 2 ( Shen et al., 2025 ; Jeong et al., 2022 ; Karatzas and Anagnostopoulos, 2023 )
Class 3 ( Taufique et al., 2024 ; Sen and Shen, 2024 ; Fang et al., 2018 )
Variants Generation Methods ( Sanh et al., 2020 ; Ro and Choi, 2021 )
SparseLoom
Data- and training-free variant generation
✗
✗
✗
✗
✓
Multiple variants selection
✗
✗
✓
✗
✓
# model variants supported
1
1
Scale of 10
Scale of 10
Scale of 1000
Heterogeneous subgraph placement
✗
✓
✗
✗
✓
Optimized sparsity-aware placement
✗
✗
✗
✗
✓
Memory-efficient preloading
✗
✗
✗
✗
✓
Appendix
Table 7. Key features of different classes of multi-DNN inference systems, variants generation methods and SparseLoom . Class 1 : systems that optimize task-level scheduling on a single processor, Class 2 : systems that optimize subgraph placement on heterogeneous processors, and Class 3 : systems that support the selection of multiple variants for diverse SLOs. Variant-generation methods generates fine-grained sparse variants.
Figure 15. SLO violation rates of SparseLoom and baselines across three SoCs under accuracy-guaranteed SLOs. SparseLoom consistently achieves lower violation rates than all baselines, with reductions of up to 73.6%.
Figure 16. SLO violation rates of SparseLoom and baselines across three SoCs under latency-guaranteed SLOs. SparseLoom consistently achieves lower violation rates than all baselines, with reductions of up to 68.2%..
Deploying DNNs on System-on-Chips (SoC) with multiple heterogeneous acceleration engines is challenging, and the majority of deployment frameworks cannot fully exploit heterogeneity. We present MATCHA, a unified DNN deployment framework that generates highly concurrent schedules for parallel, heterogeneous accelerators and uses constraint programming to optimize L3/L2 memory allocation and scheduling. Using pattern matching, tiling, and mapping across individual HW units enables parallel execution and high accelerator utilization. On the MLPerf Tiny benchmark, using a SoC with two heterogeneous accelerators, MATCHA improves accelerator utilization and reduces inference latency by up to 35% with respect to the the state-of-the-art MATCH compiler.
Split computing constitutes a widely used distributed inference approach, where a lightweight head model is onloaded onto the device and a heavier tail model resides on an edge server, leveraging the growing computational capabilities of modern System-on-Chips while alleviating server load. As intelligent indoor environments such as smart offices grow increasingly populated with diverse IoT devices, a single edge server must simultaneously assist multiple devices, each competing for the same shared inference resources. Without a principled mechanism to manage this shared load, the server is quickly overwhelmed, causing latency SLO violations and rendering server-assisted inference ineffective. In this work, we present MANE, a distributed inference framework that equips the server with a multi-path tail architecture, enabling a dynamic accuracy--throughput trade-off at runtime. By introducing a novel multi-path model architecture, a three-stage training scheme featuring a Joint Head Network Distillation loss and a hysteresis-based scheduler with an equitable device-fallback policy, MANE maintains over 80% SLO satisfaction rate where state-of-the-art onloading methods fail completely, while preserving accuracy 6pp higher than on-device alternatives, across up to 40 concurrent devices.
Sokratis Nikolaidis, Stylianos I. Venieris, Leonidas Malachias +1
National Technical University of Athens, Athens, Greece · Samsung AI Center, Cambridge, UK
Deploying large deep neural networks on memory-constrained mobile devices is a central challenge in edge ML. While compression, pruning, and quantization reduce per-parameter cost, transformer-based models remain too large for the 3.3-7.4 GB RAM envelope of commodity Android handsets. We present the DNN pipeline scheduling subsystem of CROWDio, which achieves practical ONNX inference across resource-constrained Android workers without model modification, by distributing memory pressure across devices via five mechanisms: JIT deferred partition loading, a single-partition-resident constraint, a 4-tier affinity scheduler, a zlib-compressed tensor transport, and a streaming 1:1 dependency model. Evaluated on DistilBERT (Sanh et al., 2019) (approximately 67 M parameters, SST-2) across five Android handsets over ten runs, our system holds peak per-device RSS to 43+-2 MB and limits battery draw to 50+-3 mAh per run, while streaming concurrency cuts batch latency 34% below barrier synchronisation.