MANE: A Multi-Path Adaptive Network for Edge Onloading of Deep Neural Networks

Sokratis Nikolaidis; Stylianos I. Venieris; Leonidas Malachias; Iakovos S. Venieris

cs.DCSep 13, 2026

MANE: A Multi-Path Adaptive Network for Edge Onloading of Deep Neural Networks

Authors: Sokratis Nikolaidis, Stylianos I. Venieris, Leonidas Malachias, Iakovos S. Venieris

Organizations: National Technical University of Athens, Athens, Greece · Samsung AI Center, Cambridge, UK

Abstract

Split computing constitutes a widely used distributed inference approach, where a lightweight head model is onloaded onto the device and a heavier tail model resides on an edge server, leveraging the growing computational capabilities of modern System-on-Chips while alleviating server load. As intelligent indoor environments such as smart offices grow increasingly populated with diverse IoT devices, a single edge server must simultaneously assist multiple devices, each competing for the same shared inference resources. Without a principled mechanism to manage this shared load, the server is quickly overwhelmed, causing latency SLO violations and rendering server-assisted inference ineffective. In this work, we present MANE, a distributed inference framework that equips the server with a multi-path tail architecture, enabling a dynamic accuracy--throughput trade-off at runtime. By introducing a novel multi-path model architecture, a three-stage training scheme featuring a Joint Head Network Distillation loss and a hysteresis-based scheduler with an equitable device-fallback policy, MANE maintains over 80% SLO satisfaction rate where state-of-the-art onloading methods fail completely, while preserving accuracy 6pp higher than on-device alternatives, across up to 40 concurrent devices.

Figures & tables

Fig. 1: Multi-path adaptive architecture for edge onloading.

Fig. 2: MANE’s three-stage training process.

Model	Path	Param. Reduction	Feature Map Reduction
ResNet152	Accurate	$\phantom{1}213\times$	$42.7\times$
	Balanced	$1550\times$	$21.3\times$
	Fast	$1922\times$	$21.3\times$
ConvNeXt Small	Accurate	$\phantom{1}102\times$	$16\times$
	Balanced	$1329\times$	$\phantom{1}8\times$
	Fast	$1930\times$	$\phantom{1}8\times$

TABLE I: Compression Analysis

Model	Path	Accuracy	Accel.
ResNet152	Accurate	83.98%	$1.23\times$
	Balanced	80.79%	$1.85\times$
	Fast	78.03%	$2.11\times$
ConvNeXt Small	Accurate	89.39%	$1.35\times$
	Balanced	85.43%	$2.12\times$
	Fast	80.39%	$2.43\times$

TABLE II: Path Specifications

Model	Loc.	Accuracy	Latency
MANE Shared Head	Device	—	1 5ms
MobileNetV2	Device	73.13%	25ms
ResNet152	Server	82.64%	15ms
MANE ResNet152	Server	83.98%	13ms
ConvNeXt Small	Server	89.35%	13ms
MANE ConvNeXt Small	Server	89.39%	11ms

TABLE III: End-to-end Model Performance

Fig. 3: SLO Satisfaction Rate and Accuracy for ConvNeXt Small.

Fig. 4: SLO Satisfaction Rate and Accuracy for ResNet152.

Fig. 5: Throughput for ResNet152 and ConvNeXt Small.

Fig. 6: SLO Satisfaction Rate and Accuracy with 200ms time target.

Edge Inference

Edge inference runs trained AI models on nearby devices such as phones, cameras, robots, and embedded computers under tight resource and latency limits.

See all 219 papers

CANS: Accelerating Multiuser Collaborative Edge Inference via Cooperative Autodidactic NeuroSurgeon
Jun 8, 2026Zheshun Wu, Ziyang Zhang, Changyao Lin +2Edge Inference Contextual Bandits
School of Computer Science and Technology, Harbin Institute of Technology Shenzhen, Shenzhen 518055, China · NESLab, Politecnico di Milano, 39-20133 Milano, Italy · School of Computer Science and Technology, Harbin Institute of Technology, Harbin, Heilongjiang 150001, China +2
DART-FL: Burst-Aware Multitask Federated Learning under Dynamic Inference Demand at the Edge
Aug 27, 2026Yiming Xie, Pinrui Yu, Geng Yuan +2Federated Learning Edge AI
Department of Electrical and Computer Engineering, Northeastern University, Boston, MA, USA · School of Computing, University of Georgia, Athens, GA, USA
HetRoute Heterogeneous and Cost-aware Collaborative Routing Framework for Distributed Edge MoE Inference
Aug 1, 2026Xin Yuan, Ning Li, Wenchao Xu +2Edge Inference Cost-Aware Inference
School of Ocean Engineering, Harbin Institute of Technology, Heilongjiang, China · School of Artificial Intelligence, University of Science and Technology Beijing, China · Division of Integrative Systems and Design, Hong Kong University of Science and Technology, Hong Kong +1

Jun 8, 2026J/K move · Enter open · S save

CANS: Accelerating Multiuser Collaborative Edge Inference via Cooperative Autodidactic NeuroSurgeon

Zheshun Wu, Ziyang Zhang, Changyao Lin, Zenglin Xu+1

School of Computer Science and Technology, Harbin Institute of Technology Shenzhen, Shenzhen 518055, China · NESLab, Politecnico di Milano, 39-20133 Milano, Italy · School of Computer Science and Technology, Harbin Institute of Technology, Harbin, Heilongjiang 150001, China+2

Edge Inference Contextual Bandits

Open paper AskPDF arXiv

Recently, mobile edge computing (MEC)-enabled collaborative deep neural network (DNN) inference has emerged as a promising approach for delivering intelligent services to resource-constrained mobile devices. A representative scenario is multi-user collaborative edge inference, where distinct devices independently partition their DNN models and offload backend computation to a common edge server over wireless networks. However, determining the optimal DNN partition for each device is challenging due to unknown and time-varying system conditions, including fluctuating wireless links and diverse device capabilities. To address this problem, we propose Cooperative Autodidactic NeuroSurgeon (CANS), a collaborative edge inference framework that enables devices to adaptively learn optimal DNN partitions by sharing informative feedback during online inference. To handle the challenge of device heterogeneity and better leverage offline inference experience, we integrate a novel FedLinUCB-DW algorithm that groups devices of the same type and warm-starts online exploration using local offline early-exit inference experience. Furthermore, we provide theoretical guarantees for FedLinUCB-DW by deriving the regret upper bound. We also validate our method on both a simulated environment and a hardware prototype system. Empirical evaluations demonstrate that CANS achieves lower inference latency compared to state-of-the-art baselines. Especially, in prototype experiments on two edge devices, the proposed CANS reduced average inference latency by up to 50% compared to the non-cooperative baseline.

Abstract

Edge Inference

Explore similar work

CANS: Accelerating Multiuser Collaborative Edge Inference via Cooperative Autodidactic NeuroSurgeon

DART-FL: Burst-Aware Multitask Federated Learning under Dynamic Inference Demand at the Edge

HetRoute Heterogeneous and Cost-aware Collaborative Routing Framework for Distributed Edge MoE Inference