SLA-Aware Distributed LLM Inference Across Device-RAN-Cloud

Hariz Yet; Nguyen Thanh Tam; Mao V. Ngo; Lim Yi Shen; Lin Wei; Jihong Park; Binbin Chen; Tony Q. S. Quek

cs.NIFeb 27, 2026

SLA-Aware Distributed LLM Inference Across Device-RAN-Cloud

Authors: Hariz Yet, Nguyen Thanh Tam, Mao V. Ngo, Lim Yi Shen, Lin Wei, Jihong Park, Binbin Chen, Tony Q. S. Quek

Organizations: Singapore University of Technology and Design (SUTD)

Abstract

Embodied AI requires sub-second inference near the Radio Access Network (RAN), but deployments span heterogeneous tiers (on-device, RAN-edge, cloud) and must not disrupt real-time baseband processing. We report measurements from a 5G Standalone (SA) AI-RAN testbed using a fixed baseline policy for repeatability. The setup includes an on-device tier, a three-node RAN-edge cluster co-hosting a containerized 5G RAN, and a cloud tier. We find that on-device execution remains multi-second and fails to meet sub-second budgets. At the RAN edge, SLA feasibility is primarily determined by model variant choice: quantized models concentrate below 0.5,s, while unquantized and some larger quantized models incur deadline misses due to stalls and queuing. In the cloud tier, meeting a 0.5,s deadline is challenging on the measured WAN path (up to 32.9% of requests complete within 0.5,s), but all evaluated variants meet a 1.0,s deadline (100% within 1.0,s). Under saturated downlink traffic and up to $N{=}20$ concurrent inference clients, Multi-Instance GPU (MIG) isolation preserves baseband timing-health proxies, supporting safe co-location under fixed partitioning.

Figures & tables

Fig. 1: AI-RAN testbed architecture for SLA-tiered distributed LLM inference across Device–Edge–Cloud.

Tier	Latency Budget( $L$ )	Guarantees and behavior
Premium	$L_{P}{=}0.5$ s	Reserved MIG slice; strict tail-latency target; may preempt lower tiers
Medium	$L_{M}{=}1.0$ s	Opportunistic access (quantized variants); served when capacity available
Basic	$L_{B}\geq 1.0$ s	Best-effort; fallback to on-device/cloud; no exclusivity

TABLE I: SLA tiers for AI-on-RAN inference services used in this study.

Item	Setting (this study)
UE device	Jetson Orin NX (client); on-device tier runs execute vLLM locally on Jetson (MAXN)
Edge tier	3 $\times$ GH200 (Red Hat OpenShift), NVIDIA H100 MIG enabled; two nodes: 2 $\times$ 1g.12GB + 1 $\times$ 2g.24GB + 1 $\times$ 3g.48GB; one node: 2 $\times$ 3g.48GB with 1 $\times$ 3g.48GB reserved for Aerial
RAN stack	OAI O-CU/O-DU + NVIDIA Aerial 5G low O-DU
Cloud tier	AWS Mumbai EC2 (p5.4xlarge, NVIDIA H100) running same serving stack
Request cadence	0.5 s (trace replay), 3 runs per condition
Concurrency stress	$N\in\{1,5,10,15,20\}$ parallel clients + saturated downlink ( iperf )

TABLE II: Experimental configuration (summary).

On-Device	CPU (W)	GPU (W)
3B-FP16	8.05	16.14
3B-AWQ	6.00	11.29
3B-W4A16	6.00	11.61

TABLE III: On Device-tier power performance on Jetson Orin NX.

Variant	Platform	Request E2E Mean $\pm$ Std (ms)	TTFT Mean $\pm$ Std (ms)	RTT Mean $\pm$ Std (ms)	Hit (%) @0.5/@1.0
3B-FP16	On-Device	4651 $\pm$ 519	353 $\pm$ 447	–	0.0/0.0
	Edge	490 $\pm$ 35	159 $\pm$ 30	20.022 $\pm$ 6.138	73.9/100.0
	Cloud	559 $\pm$ 36	300 $\pm$ 35	84.312 $\pm$ 5.702	0.4/100.0
3B-AWQ	On-Device	5195 $\pm$ 178	352 $\pm$ 15	–	0.0/0.0
	Edge	391 $\pm$ 29	154 $\pm$ 27	19.963 $\pm$ 6.235	98.3/100.0
	Cloud	529 $\pm$ 35	298 $\pm$ 35	83.898 $\pm$ 5.517	18.3/100.0

TABLE IV: E2E, TTFT, RTT, and [email protected]/1.0 across tiers for Qwen2.5-VL variants (pooled over 3 runs;

N{=}903

requests/entry).

$N$	SlotInd rate (s -1 )			U-plane On-time (%)
	Median	P01	Min	Median	P05
0	1999.9	1998.9	1997.9	99.984	99.970
1	2000.0	1999.0	1997.5	99.984	99.965
5	1999.9	1998.9	1997.9	99.984	99.967
10	2000.0	1999.0	1998.7	99.984	99.964
15	1999.9	1998.9	1997.9	99.984	99.964

TABLE V: Timing-oriented baseband health proxies (shared-node, MIG-isolated) under increasing AI contention.

Panel of Figure 7 — Fig. 2: Shared-node RAN KPIs vs. concurrent inference clients $N$ under saturated downlink (MIG-isolated).

	Shared-node			Different-node
$N$	Mean	BLER 95	HARQ	Mean	BLER 95	HARQ
	(Mb/s)	(%)	(%)	(Mb/s)	(%)	(%)
0	199.9	2.79	98.40	199.9	2.79	98.40
1	199.3	5.56	99.70	199.9	1.18	100.00
5	199.4	6.69	91.28	199.8	7.01	92.00
10	198.1	3.09	99.46	199.2	2.75	98.78

TABLE VI: RAN stability summary under AI contention with saturated downlink.

Panel of Figure 9 — Fig. 3: MAC-layer tail BLER (left) and HARQ health (right) under increasing AI contention with saturated downlink traffic. $N{=}0$ denotes the no-inference baseline.

LLM Inference Serving

Large language model inference serving studies how to run model requests efficiently while meeting latency, throughput, cost, and energy goals.

See all 70 papers

LLM Inference at the Edge: Mobile, NPU, and GPU Performance Efficiency Trade-offs Under Sustained Load
Mar 24, 2026Pranay Tummalapalli, Sahil Arayakandy, Ritam Pal +1On-Device Language Model Inference LLM Inference Efficiency
Conscious Engines
Exposing AI Inference as a Telco Network API over the 5G User Plane
Oct 10, 2026Rodrigo Moreira, Larissa Ferreira Rodrigues Moreira, Joberto S. B. Martins +2Edge Inference Split Inference
Federal University of Viçosa (UFV), Minas Gerais, Brazil · University of Salvador (UNIFACS), Salvador, Brazil · University of São Paulo (USP), São Paulo, Brazil +1
Cloud to Edge: Benchmarking LLM Inference On Hardware-Accelerated Single-Board Computers
Apr 24, 2026Harri Renney, Fouad Trad, Michael Mattarock +1AI Accelerator Inference On-Device Language Model Inference
Kaze Technologies Kaze Consulting Bath, BA1 2HN, UK · Electrical and Computer Engineering Lebanese American University Byblos, Lebanon · Carnegie Mellon University Pittsburgh Pennsylvania USA +1

Mar 24, 2026J/K move · Enter open · S save

LLM Inference at the Edge: Mobile, NPU, and GPU Performance Efficiency Trade-offs Under Sustained Load

Pranay Tummalapalli, Sahil Arayakandy, Ritam Pal, Kautuk Kundan

Conscious Engines

On-Device Language Model Inference LLM Inference Efficiency

Open paper AskPDF arXiv

Deploying large language models on-device for always-on personal agents demands sustained inference from hardware tightly constrained in power, thermal envelope, and memory. We benchmark Qwen 2.5 1.5B (4-bit quantised) across four platforms: a Raspberry Pi 5 with Hailo-10H NPU, a Samsung Galaxy S24 Ultra, an iPhone 16 Pro, and a laptop NVIDIA RTX 4050 GPU. Using a fixed 258-token prompt over 20 warm-condition iterations per device, we measure throughput, latency, power, and thermal behaviour. For mobile platforms, thermal management supersedes peak compute as the primary constraint: the iPhone 16 Pro loses nearly half its throughput within two iterations, and the S24 Ultra suffers a hard OS-enforced GPU frequency floor that terminates inference entirely. On dedicated hardware, distinct constraints dominate: the RTX 4050 is bounded by its battery power ceiling, while the Hailo-10H is limited by on-module memory bandwidth. The RTX 4050 sustains 131.7 tok/s at 34.1 W; the Hailo-10H sustains 6.9 tok/s at under 2 W with near-zero variance, matching the RTX 4050 in energy proportionality at 19x lower throughput. Results should be interpreted as platform-level deployment characterisations for a single model and prompt type, reflecting hardware and software combined, rather than general claims about hardware capability alone.

Abstract

LLM Inference Serving

Explore similar work

LLM Inference at the Edge: Mobile, NPU, and GPU Performance Efficiency Trade-offs Under Sustained Load

Exposing AI Inference as a Telco Network API over the 5G User Plane

Cloud to Edge: Benchmarking LLM Inference On Hardware-Accelerated Single-Board Computers