Where Should Models Run? A Collaborative Inference Framework for Embodied Intelligence in 6G
Authors: Wanlu Sun, Zhenhui Yuan, Xiang Ji, Yi Han, Qing Li, Gan Zheng
Organizations: School of Engineering, University of Warwick, UK. · School of Information Engineering, Nanchang University, Nanchang, China. · School of Information Engineering, Wuhan University of Technology, Wuhan, China. · Pengcheng Laboratory, Shenzhen, China
Embodied intelligence systems rely on fast and reliable inference to enable real-time perception, reasoning, and control. Recent advances in large-scale artificial intelligence (AI) models, including Large Language Models (LLMs) and Vision-Language-Action (VLA) models, have significantly enhanced the capabilities of embodied agents in complex environments. However, deploying these models remains challenging due to the limited computational resources and energy budgets of robotic platforms. A fundamental question is where inference should be executed. Collaborative inference, which partitions models across end devices, edge servers, and the cloud, has emerged as a promising solution. Yet determining the optimal partitioning point remains an open problem, as it depends on wireless network conditions, model structure, and privacy requirements. This article reviews the state of the art in collaborative inference for embodied intelligence and proposes an edge-centric framework in which the edge coordinates model partitioning and execution. Through both simulation and real-world experiments over WiFi and 5G networks, we demonstrate that the optimal partitioning strategy varies with network conditions and model structure. We further identify latency-minimizing partition points for ResNet-101, Qwen2.5-VL-7B, and OpenVLA-7B in collaborative inference between NVIDIA Jetson Nano or AGX Xavier devices and an AGX Thor edge server under varying wireless conditions. Finally, we discuss the future directions for collaborative inference in 6G-empowered embodied intelligence systems.
Figures & tables
Model
Model type
Organization
Depth a
Params.
Model size (MB)
Deployment
Comm. b latency (ms)
Download latency (ms)
Application
T5
Transformer
Google
E:6–24 D:6–24
60M–11B
240–44,000
Edge/Cloud
–
1000
Language processing
Qwen2.5-VL-7B
Transformer
Alibaba
ViT: 32 LLM: 28
8.29B
33,169
Device/Edge
–
1000
Multimodal reasoning
NoMaD
Diffusion policy
UC Berkeley
4 (Trans.)
19M
76
Device/Edge
12 (DL)
1000
Navigation and exploration
MobileNetV3
CNN
Google
11/15 (blocks)
2.54M–5.48M
10.2–21.9
Device/Edge
15 (UL)
1000
Object recognition
ResNet-101
CNN
Microsoft Research
101 (conv./FC)
44.5M
178
Edge/Cloud
15 (UL)
1000
Image recognition
YOLO26n
CNN
Ultralytics
260 (modules)
2.57M
10.3
Device
–
1000
Object detection
Table 1: Representative DNN models for embodied intelligence and their communication KPIs and place of deployment in alignment with 3GPP TS 22.261 [ 2 ] .
Figure 1: Illustrative architecture of collaborative inference across device, edge and cloud aligned with ITU-T Y.4509.
Figure 2: Functional architecture for edge-centric device–edge collaborative inference, informed by ITU-T Y.4509 and 3GPP TR 37.817. The edge server determines the model partition point(s) based on device, network, model structure and edge-server conditions. The architecture supports both two-way splitting (device–edge) and three-way splitting (device–edge–device), with the prediction head retained on the device in the latter.
Model
SNR
Partitioning point
Communication latency (ms)
Inference latency at device (ms)
Inference latency at edge (ms)
E2E latency (ms)
ResNet-101
Excellent
k=4
27.69
40.12
3.52
71.33
Good
k=4
66.23
40.12
3.52
109.87
Fair
k=8
162.56
74.37
3.00
239.93
Poor
k=31
207.85
246.92
0.41
455.18
Bad
k=31
307.84
246.92
0.41
555.17
Qwen2.5-VL-7B
Excellent
k=S1
22.97
63.87
433.93
520.77
Table 2: Simulated collaborative inference under five wireless channel conditions. For each model and condition, the table reports the latency-minimizing partition among the evaluated candidates and its latency components.
Figure 3: Collaborative inference testbed comprising a camera-equipped robotic rover with a Jetson AGX Xavier, a private 5G link, and a Jetson AGX Thor edge server. The perception task is to identify a person wearing a black top and carrying a white backpack.
Channel condition
Partitioning point
Communication latency (ms)
Inference latency at device (ms)
Inference latency at edge (ms)
E2E latency (ms)
Inference energy at edge (J)
Target identified
C1
k=S1
344.96
78.02
1910.42
2426.73
30.92
Yes
k=S2
344.65
130.60
1872.93
2447.24
30.74
Yes
k=S3
357.99
187.06
1859.31
2501.45
30.52
Yes
k=S4
340.24
244.65
1841.87
2522.61
30.33
Yes
k=SM
283.66
247.26
1835.07
2461.97
30.34
Yes
C2
k=S1
714.65
79.80
2177.91
3062.52
32.94
Yes
Table 3: Hardware performance of Qwen2.5-VL-7B collaborative inference under three measured wireless channel conditions. Bold values identify the lowest reported E2E latency within each condition.
Large language models (LLMs) are transforming society, powering applications from smartphone assistants to autonomous driving. Yet cloud-based LLM services alone cannot serve a growing class of applications, including those operating under intermittent connectivity, sub-second latency budgets, data-residency constraints, or sustained high-volume inference. On-device deployment is in turn constrained by limited computation and memory. No single endpoint can deliver high-quality service across this spectrum. This article focuses on collaborative intelligence, a paradigm in which multiple independent LLMs distributed across device and cloud endpoints collaborate at the task level through natural language or structured messages. Such collaboration strives for superior response quality under heterogeneous resource constraints spanning computation, memory, communication, and cost across network tiers. We present collaborative inference along two complementary and composable dimensions: vertical device-cloud collaboration and horizontal multi-agent collaboration, which can be combined into hybrid topologies in practice. We then examine learning to collaborate, addressing the training of routing policies and the development of cooperative capabilities among LLMs. Finally, we identify open research challenges including scaling under resource heterogeneity and trustworthy collaborative intelligence.
Liangqi Yuan, Wenzhi Fang, Shiqiang Wang +2
Purdue University, IN, USA. · University of Exeter, UK. · Princeton University, USA.
Generative AI (GenAI) has transformed applications in natural language processing and content creation, yet centralized inference remains hindered by high latency, limited customizability, and privacy concerns. Deploying large models (LMs) in mobile edge networks emerges as a promising solution. However, it also poses new challenges, including heterogeneous multi-modal LMs with diverse resource demands and inference speeds, varied prompt/output modalities that complicate orchestration, and resource-limited infrastructure ill-suited for concurrent LM execution. In response, we propose a Multi-Agentic AI framework for latency- and fairness-aware multi-modal LM inference in mobile edge networks. Our solution includes a long-term Global Planning Agent, a short-term Prompt Scheduling Agent, and multiple on-node LM Deployment Agents, all powered by foundation language models. These agents cooperatively optimize prompt routing and LM deployment through natural language reasoning over runtime telemetry and historical experience. To evaluate its performance, we further develop a city-wide testbed that supports network monitoring, containerized LM deployment, intra-server resource management, and inter-server communications. Experiments demonstrate that our solution reduces average end-to-end latency by 42.9% compared with a state-of-the-art online-learning baseline and by 80.6% compared with the full-activation baseline, while maintaining a mean service-type Jain's index of 0.925. Moreover, our solution adapts quickly without fine-tuning, offering a generalizable solution for optimizing GenAI services in edge environments.
Haiyuan Li, Hari Madhukumar, Shuangyi Yan +2
Smart Internet Lab, Department of Electrical and Electronic Engineering, University of Bristol, BS8 1QU, U.K.
Embodied AI requires sub-second inference near the Radio Access Network (RAN), but deployments span heterogeneous tiers (on-device, RAN-edge, cloud) and must not disrupt real-time baseband processing. We report measurements from a 5G Standalone (SA) AI-RAN testbed using a fixed baseline policy for repeatability. The setup includes an on-device tier, a three-node RAN-edge cluster co-hosting a containerized 5G RAN, and a cloud tier. We find that on-device execution remains multi-second and fails to meet sub-second budgets. At the RAN edge, SLA feasibility is primarily determined by model variant choice: quantized models concentrate below 0.5,s, while unquantized and some larger quantized models incur deadline misses due to stalls and queuing. In the cloud tier, meeting a 0.5,s deadline is challenging on the measured WAN path (up to 32.9% of requests complete within 0.5,s), but all evaluated variants meet a 1.0,s deadline (100% within 1.0,s). Under saturated downlink traffic and up to N=20 concurrent inference clients, Multi-Instance GPU (MIG) isolation preserves baseband timing-health proxies, supporting safe co-location under fixed partitioning.
Hariz Yet, Nguyen Thanh Tam, Mao V. Ngo +5
Singapore University of Technology and Design (SUTD)