Distributed LLM Inference via Communication-Efficient Speculative Decoding

Joohyun Shin; Byeonghyeon Park; Keumseo Ryum; Joonhyuk Kang

cs.LGOct 25, 2026

Distributed LLM Inference via Communication-Efficient Speculative Decoding

Authors: Joohyun Shin, Byeonghyeon Park, Keumseo Ryum, Joonhyuk Kang

Organizations: School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea

Abstract

Distributed speculative decoding has emerged as a promising approach for device--server collaborative large language model (LLM) inference, but its decoding speedup is often limited by the vocabulary-scale communication overhead. In existing distributed speculative decoding methods, exact rejection sampling requires transmitting full token distributions over the LLM vocabulary between the device and the server, which becomes costly in bandwidth-constrained environments. To address this issue, we propose DISCO, a communication-efficient framework for distributed speculative decoding, that eliminates vocabulary-scale distribution transfer while preserving the exact target distribution. DISCO achieves this by introducing a server-side replica of the draft small language model (SLM), which enables the server to locally obtain the draft-side distribution required for rejection sampling. As a result, DISCO exchanges only drafted token indices and scalar probability values between the device and the server. Experimental results on multiple datasets show that DISCO consistently achieves higher decoding speedup than existing distributed speculative decoding methods, particularly under stringent bandwidth constraints.

Figures & tables

Fig. 1 : Distributed speculative decoding process. The device generates draft tokens using a SLM and sends them to the server. The server-side LLM verifies the received tokens and updates the prefix with the accepted tokens.

Fig. 2 : Framework of DISCO. The device generates draft tokens and sends only token indices and scalar probabilities to the server. The server performs parallel verification using the LLM, and upon rejection, conducts server-side resampling with the server SLM. The accepted prefix and the resampled token are then fed back to the device.

Method	Uplink payload	Expected downlink payload
DSD	$\gamma\!\left(\|\mathcal{V}\|b_{\mathrm{prob}}+b_{\mathrm{tok}}\right)$	$b_{\mathrm{tok}}+b_{\mathrm{pos}}$
DSSD	$\gamma\!\left(b_{\mathrm{tok}}+b_{\mathrm{prob}}\right)$	$b_{\mathrm{pos}}+A_{\gamma}b_{\mathrm{tok}}+(1-A_{\gamma})\|\mathcal{V}\|b_{\mathrm{prob}}$
DISCO	$\gamma\!\left(b_{\mathrm{tok}}+b_{\mathrm{prob}}\right)$	$b_{\mathrm{tok}}+b_{\mathrm{pos}}$

TABLE I : Communication payload per speculative decoding round.

Panel of Figure 4 — Fig. 3 : Speedup of DISCO, DSSD, and DSD under varying bandwidth across different datasets with $RTT$ fixed to 20 ms. DISCO consistently outperforms the baselines across all datasets and bandwidth settings, with especially clear benefits in the low-bandwidth regime.

		$\gamma=2$			$\gamma=4$			$\gamma=6$			$\gamma=8$
Model	BW	DSD	DSSD	DISCO	DSD	DSSD	DISCO	DSD	DSSD	DISCO	DSD	DSSD	DISCO
OPT-125m/30B	1	0.08	0.57	1.41	0.05	0.53	1.67	0.04	0.56	1.81	0.04	0.56	1.85
	5	0.31	1.01	1.40	0.24	1.09	1.67	0.20	1.17	1.81	0.18	1.19	1.86
	10	0.51	1.18	1.41	0.42	1.33	1.69	0.37	1.43	1.84	0.32	1.46	1.89
	50	1.06	1.38	1.42	1.06	1.62	1.68	1.02	1.74	1.82	0.94	1.74	1.84
Qwen2.5-0.5B/32B	1	0.04	0.44	1.34	0.03	0.40	1.55	0.02	0.39	1.62	0.02	0.39	1.63

TABLE II: Speedup on GSM8K with varying draft length under different bandwidth conditions over 3 seeds. BW is measured in Mbps. For each model, bandwidth, and

\gamma

, the best speedup among DSD, DSSD, and DISCO is in bold.

Fig. 4 : Latency breakdown per token under $BW=10$ Mbps and $RTT=20$ ms on GSM8K. DISCO significantly reduces communication latency compared to DSD and DSSD, while computation latency remains similar across methods.

Config.	Method	Dataset
Config.	Method	GSM8K	CNN/DM	HumanEval
RTX3090–A100	DISCO	1.75	1.65	1.64
	DSSD	1.57	1.52	1.44
	DSD	0.92	0.92	0.86
RTX4090–A100	DISCO	1.92	1.78	1.78
	DSSD	1.72	1.63	1.57

TABLE III : Average decoding speedup across datasets under different device–server GPU configurations.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 314 papers

DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification
Apr 8, 2026Ziyi Wang, Siva Rajesh Kasa, Ankith M S +6Speculative Decoding LLM Inference Acceleration
Purdue University · Amazon Inc. · University of Illinois Urbana-Champaign +1
Beyond the Target: From Imitation to Collaboration in Speculative Decoding
May 24, 2026Jinze Li, Yixing Xu, Guanchen Li +7Speculative Decoding LLM Inference Acceleration
Advanced Micro Devices, Inc., Beijing, China · The University of Hong Kong · University of North Texas
Unlocking Parallelism in Autoregressive Language Models via Speculative Decoding with Progressive Tree Drafting
Jul 12, 2026Zipeng Gao, Zhi Zheng, Qingrong Xia +5Speculative Decoding LLM Inference Acceleration
1State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · 2Huawei Technologies Co., Ltd.

Apr 8, 2026J/K move · Enter open · S save

DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification

Ziyi Wang, Siva Rajesh Kasa, Ankith M S, Santhosh Kumar Kasa+5

Purdue University · Amazon Inc. · University of Illinois Urbana-Champaign+1

Speculative Decoding LLM Inference Acceleration Parallel Speculative Decoding

Open paper AskPDF arXiv

Speculative decoding is an effective technique for accelerating large language model inference by drafting multiple tokens in parallel. In practice, its speedup is often bottlenecked by a rigid verification step that strictly enforces the accepted token distribution to exactly match the target model. This constraint leads to the rejection of many plausible tokens, lowering the acceptance rate and limiting overall time speedup. To overcome this limitation, we propose Dynamic Verification Relaxed Speculative Decoding (DIVERSED), a relaxed verification framework that improves time efficiency while preserving generation quality. DIVERSED learns an ensemble-based verifier that blends the draft and target model distributions with a task-dependent and context-dependent weight. We provide theoretical justification for our approach and demonstrate empirically that DIVERSED achieves substantially higher inference efficiency compared to standard speculative decoding methods. Code is available at: https://github.com/comeusr/diversed.

Abstract

Speculative Decoding

Explore similar work

DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification

Beyond the Target: From Imitation to Collaboration in Speculative Decoding

Unlocking Parallelism in Autoregressive Language Models via Speculative Decoding with Progressive Tree Drafting