Replica-Aware Split Inference for Edge IoT Systems

Zied Jenhani; Nasrin Razmi; Admela Jukan

cs.DCNov 3, 2026

Replica-Aware Split Inference for Edge IoT Systems

Authors: Zied Jenhani, Nasrin Razmi, Admela Jukan

Organizations: Technische Universität Braunschweig, Germany

Abstract

Split Inference (SI) addresses the hardware resource limitations of Internet of Things (IoT) devices by distributing tiny Machine Learning (TinyML) model segments across multiple IoT devices. However, a device failure can interrupt the inference pipeline. This paper proposes a replica-aware recovery approach that deploys replicas of TinyML model segments originally hosted on failed devices onto surviving devices. We formulate the joint selection of replica hosts and communication protocols as a mixed-integer linear programming (MILP) problem that minimizes post-reconfiguration inference latency, accounting for computation and transmission latencies, under flash memory and runtime memory constraints. Simulations using MobileNetV2 on heterogeneous IoT networks compare the proposed approach with four baselines. We implement the model in a network of 15 IoT devices with a maximum link-rate reduction of 75%, and show that MILP approach achieves an average modeled end-to-end inference latency that is 6.8% lower than that of Computation Greedy and 18.3% lower than that of Random. Index Terms TinyML, split inference, Internet of Things, fault tolerance, replica placement.

Figures & tables

Fig. 1 : Architecture of the replica-aware split-inference framework.

Device	SRAM	PSRAM	Internal Flash	External Flash	OTA slot
ESP32-S3-WROOM-1-N16R8	512	8192	–	16384	7680
Arduino Nano 33 BLE Sense Rev2	256	–	1024	–	448
Raspberry Pi Pico 2 W	520	–	–	4096	1744
STM32 B-U585I-IOT02A	786	8192	2048	65536	832

TABLE I : Device memory and OTA slot sizes (KiB)

Panel of Figure 3 — Fig. 2 : Average E2E inference latency at the original link rates.

Panel of Figure 4 — Fig. 3 : Average E2E inference latency under link-rate reductions for 15 devices.

Fig. 4 : Median algorithm computation time versus failure count, with interquartile bands.

Split Inference

Split inference divides model computation across devices or services to balance latency, memory, communication, and privacy.

See all 15 papers

Split CNN Inference on Networked Microcontrollers
May 10, 2026Junyu Lu, Shashwath Suresh, Hao Liu +2Efficient Neural Network Inference Edge Inference
Delft University of Technology, Delft, The Netherlands
Reliable Federated TinyML Deployment for IoT Security
Sep 23, 2026Younsoo Park, Seokhyoen Bae, Shasi Kumar Ramachandran Prabhu +2IoT Intrusion Detection Tiny ML
The Pennsylvania State University, University Park, Pennsylvania, USA · Elizabethtown College, Elizabethtown, Pennsylvania, USA
MANE: A Multi-Path Adaptive Network for Edge Onloading of Deep Neural Networks
Sep 13, 2026Sokratis Nikolaidis, Stylianos I. Venieris, Leonidas Malachias +1Edge Inference Distributed Inference
National Technical University of Athens, Athens, Greece · Samsung AI Center, Cambridge, UK

May 10, 2026J/K move · Enter open · S save

Split CNN Inference on Networked Microcontrollers

Junyu Lu, Shashwath Suresh, Hao Liu, Qi Hong+1

Delft University of Technology, Delft, The Netherlands

Efficient Neural Network Inference Edge Inference Split Inference Convolutional Neural Networks On-Device Inference

Open paper AskPDF arXiv

Running deep neural networks on microcontroller units (MCUs) is severely constrained by limited memory resources. While TinyML techniques reduce model size and computation, they often fail in practice due to excessive peak Random Access Memory (RAM) usage during inference, dominated by intermediate activations. As a result, many models remain infeasible on standalone MCUs. In this work, we present a fine-grained split inference system for networked MCUs that enables collaborative inference of Convolutional Neural Networks (CNN) models across multiple devices. Our key insight is that breaking the memory bottleneck requires splitting inference at sub-layer granularity rather than at layer boundaries. We reinterpret pre-trained models to enable kernel-wise and neuron-wise partitioning, and distribute both model parameters and intermediate activations across multiple MCUs. A lightweight, resource-aware coordinator orchestrates the inference across MCU devices with heterogeneous resources. We implement the proposed system on a real testbed and evaluate it on up to 8 MCUs using MobileNetV2, a representative CNN model. Our experimental results show that CNN models infeasible on a single MCU can be executed across networked MCUs, reducing the per-MCU peak RAM usage while maintaining the practical end-to-end inference latency. All the source code of this work can be found here: https://github.com/shashsuresh/split-inference-on-MCUs.

Abstract

Split Inference

Explore similar work

Split CNN Inference on Networked Microcontrollers

Reliable Federated TinyML Deployment for IoT Security

MANE: A Multi-Path Adaptive Network for Edge Onloading of Deep Neural Networks