Orchestrating Multimodal DNN Workloads in Wireless Neural Processing

Sai Xu; Kai-Kit Wong; Yanan Du; Hyundong Shin

eess.SPMar 2, 2026

Orchestrating Multimodal DNN Workloads in Wireless Neural Processing

Authors: Sai Xu, Kai-Kit Wong, Yanan Du, Hyundong Shin

Organizations: Department of Electronic and Electrical Engineering, University College London, WC1E 7JE, London, UK · Department of Electronic Engineering, Kyung Hee University, Yongin-si, Gyeonggi-do 17104, Republic of Korea · School of Electrical and Electronic Engineering, the University of Sheffield, Sheffield S10 2TN, UK · Department of Electronics and Information Convergence Engineering, Kyung Hee University, Yongin-si, Gyeonggi-do 17104, Republic of Korea

Abstract

In edge inference, wireless resource allocation and accelerator-level deep neural network (DNN) scheduling have yet to be co-optimized in an end-to-end manner. The lack of coordination between wireless transmission and accelerator-level DNN execution prevents efficient overlap, leading to higher end-to-end inference latency. To address this issue, this paper investigates multimodal DNN workload orchestration in wireless neural processing (WNP), a paradigm that integrates wireless transmission and multi-core accelerator execution into a unified end-to-end pipeline. First, we develop a unified communication-computation model for multimodal DNN execution and formulate the corresponding optimization problem. Second, we propose O-WiN, a framework that orchestrates DNN workloads in WNP through two tightly coupled stages: simulation-based optimization and runtime execution. Third, we develop two algorithms, RTFS and PACS. RTFS schedules communication and computation sequentially, whereas PACS interleaves them to enable pipeline parallelism by overlapping wireless data transfer with accelerator-level DNN execution. Simulation results demonstrate that PACS significantly outperforms RTFS under high modality heterogeneity by better masking wireless latency through communication-computation overlap, thereby highlighting the effectiveness of communication-computation pipelining in accelerating multimodal DNN execution in WNP.

Figures & tables

Symbol	Description	Symbol	Description
$\mathcal{G}=(\mathcal{V},\mathcal{E})$	Computation DAG of the multimodal DNN	$\mathcal{V},\ \mathcal{E}$	Vertex set and edge set of $\mathcal{G}$
$\mathrm{pred}(v),\ \mathrm{succ}(v)$	Predecessor/successor sets of job $v$	$(u\!\to\!v)\in\mathcal{E}$	Precedence constraint: $v$ can start only after $u$ finishes
$\mathcal{K}=\{1,\ldots,K\}$	Set of modalities (sensor nodes)	$\mathcal{G}_{k}=(\mathcal{V}_{k},\mathcal{E}_{k})$	Subgraph for modality $k$
$\mathcal{C}=\{1,\ldots,C\}$	Set of accelerator cores	$s_{v},\ f_{v}$	Start/finish time of job $v$
$y_{v,c}\in\{0,1\}$	Job-to-core assignment indicator	$p_{v,c}$	Processing time of job $v$ on core $c$
$\mathcal{F}_{c}$	Feasible schedules on core $c$	$B_{\mathrm{tot}}$	Total NoC bandwidth budget

TABLE I: Main Notation

Fig. 1: A system architecture for multimodal DNN execution in WNP.

Fig. 2: O-WiN: A modular and scalable orchestration framework that decomposes the unified communication–computation pipeline into interface-defined functional modules for decoupled integration and systematic cross-module coordination.

Fig. 3: Network architecture of multimodal DNN workload.

Parameter	Symbol	Value / Range
Number of nodes/modalities	$K$	6
Slice token sizes	—	$[512,128,256,128,192,128]$
Payload model	—	$D{=}512$ , fp16 ( $2$ bytes), overhead $1.05$
Compression factors	—	$[0.25,0.25,0.80,0.60,0.60,0.60]$
Slice sizes	$D_{k}$	$[134.4,33.6,215.0,80.6,121.0,80.6]$ kB
Accelerator cores	$C$	4

TABLE II: Simulation Parameters

Fig. 4: The end-to-end execution timelines of RTFS and PACS.

Fig. 5: NoC bandwidth sharing on the multi-core accelerator of RTFS and PACS.

Fig. 6: The effects of four system-scaling dimensions—(a) the number of accelerator cores, (b) the number of OFDMA subcarriers, (c) the wireless-latency scaling factor, and (d) the NoC bandwidth budget—on the end-to-end makespan of RTFS and PACS.

Slice token sizes	Compression factors $[0.5,0.5,0.50,$ $0.50,0.50,0.5]$			Compression factors $[0.25,0.25,0.50,$ $0.50,0.50,0.5]$			Compression factors $[0.25,0.25,0.80,$ $0.60,0.60,0.6]$			Compression factors $[0.25,0.25,0.50,$ $0.50,0.80,0.8]$			Compression factors $[0.5,0.5,0.50,$ $0.50,0.80,0.8]$
	RTFS	PACS	Gain	RTFS	PACS	Gain	RTFS	PACS	Gain	RTFS	PACS	Gain	RTFS	PACS	Gain
$[128,128,128,128,128,128]$	113.92	113.92	0.00%	105.92	104.92	+0.94%	114.92	116.92	-1.74%	126.92	126.94	-0.02%	134.92	132.43	+1.85%
$[64,64,128,128,256,256]$	154.87	143.83	+7.13%	150.87	141.47	+6.23%	168.87	161.41	+4.42%	191.87	181.47	+5.42%	196.87	184.59	+6.24%
$[64,128,256,128,192,128]$	139.15	126.34	+9.21%	132.15	132.15	0.00%	145.15	136.53	+5.94%	157.15	148.54	+5.48%	168.15	155.52	+7.51%
$[512,128,256,128,192,128]$	205.60	166.39	+19.08%	189.60	153.39	+19.10%	204.60	168.39	+17.70%	217.60	180.39	+17.10%	231.60	191.39	+17.36%
$[512,128,256,384,192,64]$	230.33	193.66	+15.92%	213.33	179.66	+15.78%	230.33	192.66	+16.36%	236.33	199.66	+15.52%	236.33	199.66	+15.52%

TABLE III: The end-to-end makespan

T_{\mathrm{makespan}}

(ms) under different token lengths and compression settings.

Efficient Multimodal Inference

Efficient multimodal inference reduces the compute, memory, latency, and energy needed to process image, video, audio, text, and sensor inputs.

See all 227 papers

Metasurfaces-Integrated Wireless Neural Networks for Lightweight Over-The-Air Edge Inference
Feb 22, 2026Kyriakos Stylianopoulos, Mario Edoardo Pandolfo, Paolo Di Lorenzo +1Wireless Communications Edge Inference
Department of Informatics and Telecommunications, National and Kapodistrian University of Athens, 16122 Athens, Greece · National Inter-University Consortium for Telecommunications (CNIT), Parma, Italy · DIAG Department, Sapienza University of Rome, via Ariosto 25, Rome, Italy +1
Joint Optimization of Memory and Computing Frequency for Energy-Efficient DNN Inference
Aug 14, 2026Yunchu Han, Zhaojun Nan, Sheng Zhou +1Efficient Neural Network Inference Energy-Efficient ML
Beijing National Research Center for Information Science and Technology Department of Electronic Engineering, Tsinghua University, Beijing 100084, China · School of Electronics and Internet of Things Chongqing Polytechnic University of Electronic Technology, Chongqing 401331, China
MATCHA: Efficient Deployment of Deep Neural Networks on Multi-Accelerator Heterogeneous Edge SoCs
Apr 10, 2026Enrico Russo, Mohamed Amine Hamdi, Alessandro Ottaviano +6Efficient Neural Network Inference Neural Network Hardware Acceleration
University of Catania, Italy · Politecnico di Torino, Italy · ETH Zurich, Switzerland +1

Feb 22, 2026J/K move · Enter open · S save

Metasurfaces-Integrated Wireless Neural Networks for Lightweight Over-The-Air Edge Inference

Kyriakos Stylianopoulos, Mario Edoardo Pandolfo, Paolo Di Lorenzo, George C. Alexandropoulos

Department of Informatics and Telecommunications, National and Kapodistrian University of Athens, 16122 Athens, Greece · National Inter-University Consortium for Telecommunications (CNIT), Parma, Italy · DIAG Department, Sapienza University of Rome, via Ariosto 25, Rome, Italy+1

Wireless Communications Edge Inference Energy-Efficient ML Reconfigurable Intelligent Surfaces

Open paper AskPDF arXiv

The upcoming sixth Generation (6G) of wireless networks envisions ultra-low latency and energy efficient Edge Inference (EI) for diverse Internet of Things (IoT) applications. However, traditional digital hardware for machine learning is power intensive, motivating the need for alternative computation paradigms. Over-The-Air (OTA) computation is regarded as an emerging transformative approach assigning the wireless channel to actively perform computational tasks. This article introduces the concept of Metasurfaces-Integrated Neural Networks (MINNs), a physical-layer-enabled deep learning framework that leverages programmable multi-layer metasurface structures and Multiple-Input Multiple-Output (MIMO) channels to realize computational layers in the wave propagation domain. The MINN system is conceptualized as three modules: Encoder, Channel (uncontrollable propagation features and metasurfaces), and Decoder. The first and last modules, realized respectively at the multi-antenna transmitter and receiver, consist of conventional digital or purposely designed analog Deep Neural Network (DNN) layers, and the metasurfaces responses of the Channel module are optimized alongside all modules as trainable weights. This architecture enables computation offloading into the end-to-end physical layer, flexibly among its constituent modules, achieving performance comparable to fully digital DNNs while significantly reducing power consumption. The training of the MINN framework, two representative variations, and performance results for indicative applications are presented, highlighting the potential of MINNs as a lightweight and sustainable solution for future EI-enabled wireless systems. The article is concluded with a list of open challenges and promising research directions.

Abstract

Efficient Multimodal Inference

Explore similar work

Metasurfaces-Integrated Wireless Neural Networks for Lightweight Over-The-Air Edge Inference

Joint Optimization of Memory and Computing Frequency for Energy-Efficient DNN Inference

MATCHA: Efficient Deployment of Deep Neural Networks on Multi-Accelerator Heterogeneous Edge SoCs