ConfigSpec: Profiling-Based Configuration Selection for Distributed Edge--Cloud Speculative LLM Serving

Xiangchen Li; Saeid Ghafouri; Jiakun Fan; Babar Ali; Hans Vandierendonck; Dimitrios S. Nikolopoulos

cs.DCApr 8, 2026

ConfigSpec: Profiling-Based Configuration Selection for Distributed Edge--Cloud Speculative LLM Serving

Authors: Xiangchen Li, Saeid Ghafouri, Jiakun Fan, Babar Ali, Hans Vandierendonck, Dimitrios S. Nikolopoulos

Organizations: Virginia Tech, Blacksburg, Virginia, USA · Queen’s University Belfast, Belfast, Northern Ireland, UK

Abstract

Speculative decoding enables collaborative Large Language Model (LLM) inference across cloud and edge by separating lightweight token drafting from heavyweight verification. While prior systems show performance and cost benefits, practical deployment requires navigating a large configuration space spanning draft model variants, quantisation levels, speculative lengths, and heterogeneous edge devices. This paper presents ConfigSpec, a configurationselection framework for distributed speculative LLM serving. ConfigSpec profiles edge devices and draft-target alignment, and models drafting throughput, acceptance rate, and power to evaluate goodput, verification cost efficiency, and energy efficiency across the joint configuration space. Our analysis across three edge platforms and two LLM families reveals structurally conflicting optima. Firstly, goodput is maximised by the smallest, fastest draft model at device-dependent speculative lengths (K*=2-10). Secondly, both cost and energy efficiency converge to K=2 due to a dominant bonus-token effect-with cost favouring the largest drafter for its high acceptance rate and energy favouring the smallest for its low power draw. These conflicts confirm that no single fixed configuration can simultaneously optimise all objectives, underscoring the need for profiling-based configuration selection in disaggregated edge-cloud LLM inference.

Figures & tables

Figure 1. System abstraction used by ConfigSpec. Heterogeneous edge devices generate speculative tokens using local draft models and interact with a centralized cloud verifier.

Target: Llama-3.1-70B		Target: Qwen3-32B
Draft Model	$\boldsymbol{\alpha}$	Draft Model	$\boldsymbol{\alpha}$
Llama-3.2-1B	0.462	Qwen3-0.6B	0.378
Llama-3.2-1B-Instruct	0.546	Qwen3-1.7B	0.466
Llama-3.2-3B-Instruct	0.572	Qwen3-4B	0.487
Llama-3.1-8B	0.622	Qwen3-8B	0.522

Table 1. Acceptance rate (

\alpha

) of draft models against each target model with speculative length of 5. Higher values indicate stronger alignment between draft and target distributions.

Figure 2. Goodput vs. speculative length across devices and models.

Figure 3. Verified token generation speed.

Figure 4. Cost efficiency of draft models.

Figure 5. Energy efficiency of draft models.

Figure 6. Energy efficiency and speed (goodput) comparison.

Target	Device	Objective	Configuration	$K$	$G$	$\eta_{\mathrm{cost}}$	$E$
Llama-3.1-70B	RPi 4B	Max Goodput	Llama-3.2-1B-Inst Q4	2	2.44	1334K	—
		Min Cost/tok	Llama-3.1-8B-Inst Q4	2	0.77	1401K	—
		Min Energy	no power data
	RPi 5	Max Goodput	Llama-3.2-1B-Inst Q4	6	4.50	763K	0.84
		Min Cost/tok	Llama-3.1-8B-Inst Q4	2	1.55	1401K	3.75
		Min Energy	Llama-3.2-1B-Inst Q4	2	3.76	1334K	0.48

Table 2. Recommended configurations under different optimisation objectives.

G

: goodput (tok/s),

\eta_{\mathrm{cost}}

: cost efficiency (K tokens/

),

E$ : energy per accepted token (J/tok). Best values per (target, device) group are bolded .

LLM Inference Efficiency

Large language model inference efficiency reduces the time, memory, energy, and cost of generating useful outputs.

See all 351 papers

Delay-Adaptive Speculation Control for Low-Latency Edge-Cloud LLM Inference
May 17, 2026Kangkang Sun, Jianhua Li, Xiuzhen Chen +2Sequential Decision Making Optimal Stopping
Shanghai Key Laboratory of Integrated Administration Technologies for Information Security, School of Computer Science, Shanghai Jiao Tong University, Shanghai 200240, China · School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen, Guangdong, China
SPADE: Speculative Decoding for Precise and Low Cost Distributed Edge Cloud Inference
Aug 13, 2026Divya Jyoti Bajpai, Kishan Kumar Upadhyay, Manjesh Kumar HanawalEdge-Cloud Collaborative Inference Edge Computing
MLiONS, Department of IEOR, IIT Bombay Mumbai, Maharashtra-400076, India
MemSpec: Memory-Aware Runtime for Adaptive Draft Scheduling in Speculative Decoding on Edge Devices
Aug 11, 2026Eunjeong Kim, Yeong Jun Jeon, Myeonggyun HanOn-Device Language Model Inference LLM Inference Scheduling
Kyungpook National University Daegu, Republic of Korea

May 17, 2026J/K move · Enter open · S save

Delay-Adaptive Speculation Control for Low-Latency Edge-Cloud LLM Inference

Kangkang Sun, Jianhua Li, Xiuzhen Chen, Junyi He+1

Shanghai Key Laboratory of Integrated Administration Technologies for Information Security, School of Computer Science, Shanghai Jiao Tong University, Shanghai 200240, China · School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen, Guangdong, China

Sequential Decision Making Optimal Stopping Speculative Decoding Bandit Learning LLM Inference Acceleration Edge-Cloud Computing

Open paper AskPDF arXiv

Speculative decoding accelerates large language model (LLM) inference by using a lightweight draft model to propose tokens and a larger target model to verify them in parallel. In distributed edge-cloud inference, however, draft length must be controlled online: longer drafts amortize communication delay but reduce token acceptance, whereas shorter drafts preserve acceptance but trigger more communication rounds. We formulate this tradeoff as a ratio-type optimal stopping problem and prove that the optimal draft length is a finite delay-monotone threshold. The analysis identifies a critical delay below which single-token speculation is optimal and shows that the optimal length grows only logarithmically with communication delay. For time-varying networks, we extend the model to Markov-modulated channels and establish, under a bounded horizon and monotone stopping-region conditions, a state-dependent threshold policy. For unknown environments, we propose UCB-SpecStop, an online control algorithm with gap-free and gap-dependent expected regret bounds of

O(L_{\max}\sqrt{K_{\max}T\log(K_{\max}T)})

and

O(\sum_{k:Δ_k>0}L_{\max}^2\log(K_{\max}T)/Δ_k)

. We implement the method on a real edge-cloud testbed with a Jetson Orin Nano Super edge node and an RTX~~3090 Ti cloud node, using Qwen and Llama draft--target pairs. Experiments validate the predicted phase transition, with transition points near 83~~ms and 111~ms. Qwen matches the geometric prediction, while Llama requires empirical-prefix calibration due to heavy-head acceptance. Across the tested delay grid, UCB-SpecStop reduces per-token latency over SpecDec++ by up to 22.4%, approaches an offline oracle within 0.2--2.4% in communication-dominated regimes, improves over naive UCB by up to 7.5%, removes the 14.0--18.7% gap caused by static tuning under delay drift, and gains 3.0--6.8% with contextual channel-state information.

Abstract

LLM Inference Efficiency

Explore similar work

Delay-Adaptive Speculation Control for Low-Latency Edge-Cloud LLM Inference

SPADE: Speculative Decoding for Precise and Low Cost Distributed Edge Cloud Inference

MemSpec: Memory-Aware Runtime for Adaptive Draft Scheduling in Speculative Decoding on Edge Devices