Memory-Efficient Partitioned DNN Inference on Resource-Constrained Android Crowds

Lakshani Manamperi; Disumi Pathirana; Thiwanka Pathirana; Nipun Premarathna; Kutila Gunasekera

cs.LGMay 20, 2026

Memory-Efficient Partitioned DNN Inference on Resource-Constrained Android Crowds

Authors: Lakshani Manamperi, Disumi Pathirana, Thiwanka Pathirana, Nipun Premarathna, Kutila Gunasekera

Abstract

Deploying large deep neural networks on memory-constrained mobile devices is a central challenge in edge ML. While compression, pruning, and quantization reduce per-parameter cost, transformer-based models remain too large for the 3.3-7.4 GB RAM envelope of commodity Android handsets. We present the DNN pipeline scheduling subsystem of CROWDio, which achieves practical ONNX inference across resource-constrained Android workers without model modification, by distributing memory pressure across devices via five mechanisms: JIT deferred partition loading, a single-partition-resident constraint, a 4-tier affinity scheduler, a zlib-compressed tensor transport, and a streaming 1:1 dependency model. Evaluated on DistilBERT (Sanh et al., 2019) (approximately 67 M parameters, SST-2) across five Android handsets over ten runs, our system holds peak per-device RSS to 43+-2 MB and limits battery draw to 50+-3 mAh per run, while streaming concurrency cuts batch latency 34% below barrier synchronisation.

Explore similar work

Split CNN Inference on Networked Microcontrollers
May 10, 2026Junyu Lu, Shashwath Suresh, Hao Liu +2Graph Neural Network Inference Cortex
Adaptive DNN Partitioning and Offloading in Heterogeneous Edge-Cloud Continuum
May 10, 2026Akuen Akoi Deng, Eimantas Butkus, Alfreds Lapkovskis +1Edge Computing Mobilenetv2
MobileMoE: Scaling On-Device Mixture of Experts
May 26, 2026Yanbei Chen, Hanxian Huang, Ernie Chang +5Mixture-Of-Expert Language Models Mixture-Of-Expert Inference

May 10, 2026 · cs.DCJ/K move · Enter open · S save

Split CNN Inference on Networked Microcontrollers

Junyu Lu, Shashwath Suresh, Hao Liu, Qi Hong+1

Delft University of Technology, Delft, The Netherlands

Running deep neural networks on microcontroller units (MCUs) is severely constrained by limited memory resources. While TinyML techniques reduce model size and computation, they often fail in practice due to excessive peak Random Access Memory (RAM) usage during inference, dominated by intermediate activations. As a result, many models remain infeasible on standalone MCUs. In this work, we present a fine-grained split inference system for networked MCUs that enables collaborative inference of Convolutional Neural Networks (CNN) models across multiple devices. Our key insight is that breaking the memory bottleneck requires splitting inference at sub-layer granularity rather than at layer boundaries. We reinterpret pre-trained models to enable kernel-wise and neuron-wise partitioning, and distribute both model parameters and intermediate activations across multiple MCUs. A lightweight, resource-aware coordinator orchestrates the inference across MCU devices with heterogeneous resources. We implement the proposed system on a real testbed and evaluate it on up to 8 MCUs using MobileNetV2, a representative CNN model. Our experimental results show that CNN models infeasible on a single MCU can be executed across networked MCUs, reducing the per-MCU peak RAM usage while maintaining the practical end-to-end inference latency. All the source code of this work can be found here: https://github.com/shashsuresh/split-inference-on-MCUs.

Graph Neural Network Inference Cortex Convolutional Neural Networks

Open paper Ask about it PDF

Abstract

Explore similar work

Split CNN Inference on Networked Microcontrollers

Adaptive DNN Partitioning and Offloading in Heterogeneous Edge-Cloud Continuum

MobileMoE: Scaling On-Device Mixture of Experts