Goodput Maximization for Large Language Model Edge Inference: A Two-Phase Maskable PPO Approach

Xiaojing Chen; Qi Zhang; Wei Ni; Shunqing Zhang; Yanzan Sun

eess.SYAug 26, 2026

Goodput Maximization for Large Language Model Edge Inference: A Two-Phase Maskable PPO Approach

Authors: Xiaojing Chen, Qi Zhang, Wei Ni, Shunqing Zhang, Yanzan Sun

Organizations: Key Laboratory of Specialty Fiber Optics and Optical Access Networks, Shanghai University, Shanghai 200444, China · School of Engineering, Edith Cowan University, Perth, WA 6027, Australia

Abstract

This paper presents a novel two-phase maskable proximal policy optimization (TP-MPPO) algorithm, which maximizes the system goodput counting request throughput with strict service level objective (SLO) compliance for large language model (LLM) inference services in wireless edge networks. In the first phase of TP-MPPO, we optimize the task offloading decisions by MPPO with action masking mechanism, effectively avoiding exploring invalid actions and reducing the action space. In the second phase, closed-form solutions are derived for uplink bandwidth allocation; a greedy algorithm is designed for downlink bandwidth allocation to provide immediate rewards for the MPPO in the next round. The two stages alternate till convergence. Simulation results demonstrate that TP-MPPO can improve the system reward by 33.3%--87.5% compared to its benchmarks and achieve the highest goodput.

Figures & tables

Fig. 1: Illustration of the system supporting LLM inference services.

Fig. 2: Workflow of LLM inference system: Requests R1–R6 arrive in the first slot; the AP assigns R1–R4 to Node 1 and R5–R6 to Node 2. Subsequently, R7–R9 are allocated to Node 3, since Nodes 1 and 2 remain occupied with ongoing inference.

Fig. 3: Reward as the number of episode grows.

Fig. 4: Comparisons of throughput, goodput, TTFT, and TPOT.

Panel of Figure 5 — Fig. 5: Goodput ratio with different numbers of nodes and users, and different edge-node GPU provisioning levels (Llama-30B).

LLM Inference Scheduling

Large language model inference scheduling allocates requests, batches, compute, and KV-cache memory to balance latency, throughput, and service-level objectives.

See all 166 papers

BALANCE: Hybrid Autoregressive-Speculative LLM Inference in Wireless Edge Networks
Aug 6, 2026Guanqiao Qu, Shuo Chen, Qian Chen +2Edge Computing LLM Inference Scheduling
Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong SAR, China · Electrical and Electronic Engineering Department, Imperial College, London SW7 2BT, U.K. · Electrical and Electronic Engineering, and Computing Departments, Imperial College, London SW7 2BT, U.K.
AIR-LLM: Broadcasting AI Weights over Radio for Memory-Free Edge LLM Inference via RF Computing
Sep 30, 2026Zhihui Gao, Tingjun Chen, Dirk EnglundOn-Device Language Model Inference LLM Inference Acceleration
Massachusetts Institute of Technology Cambridge, MA, USA · Duke University Durham, NC, USA
Scalable Joint Resource Allocation for SLO-Constrained LLM Inference in Heterogeneous GPU Clouds
Apr 8, 2026Jiaming Cheng, Duong Tung NguyenLLM Serving LLM Inference Scheduling
The Ira A. Fulton Schools of Engineering, Arizona State University, Tempe, AZ, USA.

Aug 6, 2026J/K move · Enter open · S save

BALANCE: Hybrid Autoregressive-Speculative LLM Inference in Wireless Edge Networks

Guanqiao Qu, Shuo Chen, Qian Chen, Kin K. Leung+1

Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong SAR, China · Electrical and Electronic Engineering Department, Imperial College, London SW7 2BT, U.K. · Electrical and Electronic Engineering, and Computing Departments, Imperial College, London SW7 2BT, U.K.

Edge Computing LLM Inference Scheduling LLM Inference Acceleration

Open paper AskPDF arXiv

Edge inference is a promising paradigm to provide large language model (LLM) inference services in next-generation mobile networks. LLM inference mainly relies on two approaches: Autoregressive decoding (AD) generates output tokens sequentially, resulting in long latency; Speculative decoding (SD) accelerates inference by using a small language model (SLM) to generate multiple draft tokens for LLM verification, but incurs extra memory costs. Due to this latency-memory tradeoff, neither approach alone can efficiently serve users with heterogeneous demands under limited edge computing resources. To address this challenge, we propose a hybrid autoregressive-speculative inference (BALANCE) framework for edge LLM inference. In BALANCE, an edge server hosts both an SLM and an LLM, assigns each user to AD or SD, and performs the two modes simultaneously. To maximize the number of served users, we formulate a task throughput maximization problem to jointly determine user scheduling and computing resource allocation between AD and SD under user latency requirements and server memory constraints. Since the problem is NP-hard, we develop a polynomial-time algorithm that transforms the original problem into two sub-problems and obtains a sub-optimal solution with a constant approximation guarantee. Experiments demonstrate that BALANCE consistently outperforms conventional AD and SD and significantly improves task throughput.

Abstract

LLM Inference Scheduling

Explore similar work

BALANCE: Hybrid Autoregressive-Speculative LLM Inference in Wireless Edge Networks

AIR-LLM: Broadcasting AI Weights over Radio for Memory-Free Edge LLM Inference via RF Computing

Scalable Joint Resource Allocation for SLO-Constrained LLM Inference in Heterogeneous GPU Clouds