SpecRelay: Amortize or Avoid the Round Trip in Edge-Cloud Speculative Decoding
Authors: Jai Jain, Chihyo Ahn, Yong Liu, Charles Chen, Duc N. M Hoang, Ajay Kumar Jaiswal, Fatih Ilhan, Han-Byul Kim, Lauren Hannah, Richa Dixit, Minsik Cho, Yingyan Celine Lin, Hyesoon Kim
Organizations: Georgia Institute of Technology · Apple
Edge-cloud speculative decoding moves autoregressive drafting to an edge device while a larger cloud model verifies draft tokens in parallel, so that edge drafting and cloud verification overlap under a synchronous draft-verify schedule. Beyond low-latency local networks, however, the next verification request of that schedule can be sent only after the previous reply arrives, so every round still waits out a full round trip, and throughput falls as the network RTT grows. To recover this loss, we present SpecRelay, an edge-cloud LLM inference system that addresses this cost at two levels: it amortizes round trips when cloud verification is necessary and avoids them when local reasoning suffices. For cloud-bound queries, an asynchronous verification pipeline keeps fixed-budget draft trees in flight while earlier replies are still outstanding, and the cloud rejects stale requests before the target-model forward pass (the asynchronous schedule). At the query level, an RTT-conditioned router (the Latency-Aware Router) estimates quality and completion latency for three on-device modes (direct generation, Chain-of-Draft, and full reasoning) and cloud speculative decoding, selecting among them under a configurable accuracy-latency objective without adding latency during inference, since the router runs concurrently with the draft-model prefill. The asynchronous schedule alone improves decoding throughput by up to 50% over a reproduced synchronous baseline across multiple benchmarks. At matched target-model accuracy and 250 ms of added round-trip latency, the full system with the router delivers 1.95x and 1.55x the throughput of the same baseline on SPROUT and OlympiadBench, respectively, while sending only 47% and 39% of the queries to the cloud.
Figures & tables
Figure 1
Figure 3: Anchor-hedged asynchronous verification pipeline. (a) Edge and server timelines under the synchronous and asynchronous schedules. (b–d) Edge KV state of Async-K3: (b) Tree1 drafted under three anchors of Tree0 and in flight; (c) reply to Tree0 ( Verify0 ) received; (d) reply to Tree1 ( Verify1 ) received, lineage-preserving rebuild at Tree2 .
Table 1: Experimental setup: hardware, models and the variants that appear in the experiments.
SPROUT (1024-tok)
SpecBench (256-tok)
OlympiadBench (1024-tok)
Added
Eff. RTT
dmax
Up
Ref
S1
S3
A1
A3
R
Up
Ref
S1
S3
A1
A3
R
Up
Ref
S1
S3
A1
A3
R
(ms)
(ms)
(sync)
tok/s
× Ref
tok/s
× Ref
tok/s
× Ref
0
12
3
38.0
51.9
1.13
1.16
1.12
1.18
1.24
39.5
47.5
1.06
1.10
0.95
1.02
1.12
39.7
65.4
1.06
1.09
1.05
1.15
1.13
25
38
5
30.0
44.8
1.17
1.20
1.09
1.25
1.31
30.0
37.8
1.10
1.10
0.97
1.12
1.14
32.0
58.7
1.08
1.13
1.03
1.17
1.14
50
64
6
23.4
38.9
1.12
1.16
1.10
1.28
1.38
24.0
30.9
1.07
1.09
1.02
1.21
1.25
26.0
51.2
1.06
1.10
1.08
1.24
1.22
100
115
9
18.8
29.9
1.13
1.15
1.18
1.39
1.59
17.4
22.2
1.09
1.09
1.11
1.33
1.33
21.8
41.7
1.08
1.11
1.13
1.32
1.30
Table 2: Decoding throughput and speedup across added RTT on the RTX 4090 edge. Up and Ref in tok/s; the other columns are speedups over Ref ; bold = best per benchmark and RTT.
Figure 5
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
depth cap
SPROUT
SpecBench
OlympiadBench
Added
Tab. 2
S1 fill
S3 fill
S1
S1 fill
S3
S3 fill
A1
A3
S1
S1 fill
S3
S3 fill
A1
A3
S1
S1 fill
S3
S3 fill
A1
A3
(ms)
dmax
proactive
proactive
tok/s
tok/s
tok/s
0
3
5
5
58.6
64.9 +
60.2
66.5 +
58.1
61.2
50.4
51.0 +
52.3
50.3
45.1
48.5
69.3
78.6 +
71.3
78.6 +
68.7
75.2
50
6
9
9
43.6
45.5 +
45.1
47.7 +
42.8
49.8
33.1
33.6 +
33.7
35.6 +
31.5
37.4
54.3
59.4 +
56.3
61.6 +
55.3
63.5
100
9
15
14
33.8
35.5 +
34.4
37.1 +
35.3
41.6
24.2
24.9 +
24.2
25.8 +
24.6
29.5
45.0
47.9 +
46.3
49.8 +
47.1
55.0
150
12
21
20
28.2
28.1
28.4
30.2 +
30.4
36.1
18.9
19.4 +
19.1
20.2 +
21.0
24.4
38.8
39.0 +
39.5
40.9 +
42.0
48.5
Appendix
Table 3: Synchronous variants re-tuned to fill the wait (RTX 4090, 32 queries, tok/s): main tree at depth 3 , proactive cap ≈ wait / draft-forward time (listed; one level lower for S3 ), 32 -node budget unchanged. S1 , S3 , A1 and A3 = the Table 2 rows.
SPROUT
SpecBench
OlympiadBench
Edge
Budget
50
100
200
300
50
100
200
300
50
100
200
300
tok/s (bold = best budget per column)
RTX 4090
32
45.0
34.4
24.8
19.7
33.7
24.3
16.2
12.5
56.3
46.3
35.4
28.1
64
44.6
37.0
26.4
20.6
34.6
26.1
17.0
12.7
55.1
47.8
37.6
30.8
96
43.9
37.9
27.2
21.7
34.0
27.2
17.4
13.3
53.8
48.4
38.5
31.5
RTX 3090
32
43.6
34.0
24.5
19.7
33.2
23.9
15.8
12.4
55.4
45.9
35.2
28.3
Appendix
Table 4: Edge draft-budget sweep of S3 (1 client, tok/s; budget 32= the S3 rows of Tables 2 and 11 ; bold = best budget per column). acc/verify = accepted tokens per verify forward; verify ms = mean server-side verify time.
Figure 6: Edge time per committed token at 100 ms added RTT (RTX 3090). (a) Mean draft-forward time over the three benchmarks. (b–d) Edge seconds per 1k committed tokens.
Variant
SPROUT
SpecBench
OlympiadBench
useful
rejected
useful
rejected
useful
rejected
Ref
0.88
1.38
0.83
2.39
0.90
0.81
S1 fill
0.88
1.85
0.83
3.13
0.91
1.08
S3 fill
0.88
1.66
0.83
2.98
0.91
0.98
A1
0.81
1.62
0.79
2.66
0.82
1.08
A3
0.81
1.19
0.78
2.06
0.82
0.72
Appendix
Table 5: Draft forwards per committed token at 100 ms added RTT (RTX 3090): useful (level on the accepted path) and rejected.
tok/s
SPROUT
SpecBench
OlympiadBench
Edge
Added (ms)
K=1
K=2
K=3
K=4
K=5
K=1
K=2
K=3
K=4
K=5
K=1
K=2
K=3
K=4
K=5
RTX 4090
0
58.2
62.0
61.2
57.9
56.3
45.0
48.5
48.3
45.7
46.5
68.6
74.6
74.9
69.9
67.3
50
42.6
49.5
49.9
47.8
46.9
31.5
35.5
37.4
35.9
36.0
55.1
63.2
63.6
59.9
58.5
100
35.3
40.2
41.6
39.0
36.6
24.7
28.2
29.6
29.4
28.6
47.1
55.2
55.0
52.4
51.5
200
26.2
30.3
32.1
31.0
30.7
17.7
20.0
21.0
21.1
21.1
36.8
43.3
44.3
42.1
41.1
300
21.3
24.0
25.4
25.3
25.3
13.9
15.6
16.2
16.4
16.4
30.0
35.3
36.2
35.7
35.2
Appendix
Table 6: Sweep over the number of anchors K of the asynchronous schedule.
SPROUT
SpecBench
OlympiadBench
Jitter (added RTT 0)
S1
S3
S3 until
A1
A3
A3 / S1
S1
S3
S3 until
A1
A3
A3 / S1
S1
S3
S3 until
A1
A3
A3 / S1
RTX 4090 edge
none (control)
58.9
60.0
63.7
57.4
61.7
1.05
50.5
51.2
48.6
46.0
49.4
0.98
70.1
71.3
78.2
70.4
74.3
1.06
Pareto-AR(1), s=10
49.8
50.7
55.7
49.1
57.0
1.14
39.8
41.2
40.3
38.5
41.9
1.05
59.3
60.5
69.7
62.6
69.8
1.18
Pareto-AR(1), s=30
38.7
39.2
43.7
43.7
49.5
1.28
29.2
29.6
29.3
30.0
34.3
1.17
46.3
47.0
56.8
53.4
59.5
1.29
Pareto-AR(1), s=60
29.0
29.4
33.4
36.9
41.1
1.42
20.4
20.7
21.0
24.4
26.4
1.29
34.8
35.3
43.2
44.0
46.3
1.33
Appendix
Table 7: Jitter resilience at added RTT 0 (1 client, tok/s): server-side de-meaned Pareto-AR(1) delay (scale s ) and Gilbert–Elliott bursts (mean bad-state delay f ms). S1 / S3 at the nominal depth 3 ; the blank cell hit a deterministic K=1 livelock in both attempts.
Throughput (tok/s)
Added RTT 0 ms
Added RTT 300 ms
Edge
Variant
T=0
T=0.7
T=0
T=0.7
RTX 4090
Ref
47.5
44.9
11.4
11.0
S3
52.1
49.9
12.5
11.1
A3
48.3
45.7
16.2
15.3
RTX 3090
Ref
41.4
40.4
10.9
10.5
S3
50.7
48.3
12.4
11.4
Appendix
Table 8: Verifier temperature 0 vs. 0.7 on SpecBench (1 client, client-span tok/s).
Llama-3.2-1B → 3.1-8B (tok/s)
SPROUT
SpecBench
OlympiadBench
Added (ms)
dmax
Ref
S1
S3
A1
A3
A3 / S1
Ref
S1
S3
A1
A3
A3 / S1
Ref
S1
S3
A1
A3
A3 / S1
0
5
75.7
88.4
91.3
75.4
87.7
0.99
68.2
73.4
74.5
62.0
70.0
0.95
95.6
101.9
104.5
88.5
99.1
0.97
50
10
55.6
63.6
66.7
53.8
66.0
1.04
46.1
48.7
49.3
42.2
52.0
1.07
72.4
78.3
79.1
65.8
78.4
1.00
100
15
40.8
46.1
46.8
45.1
53.9
1.17
33.6
34.8
35.6
33.0
40.7
1.17
53.2
56.3
58.5
54.6
64.7
1.15
200
21
28.6
31.2
32.4
31.8
39.1
1.25
22.6
23.3
23.6
22.8
27.4
1.18
35.7
37.5
38.5
40.2
48.9
1.30
300
27
22.1
24.0
24.6
25.3
30.1
1.25
17.1
17.6
17.8
17.8
21.3
1.21
27.4
28.7
28.8
32.3
38.2
1.33
Appendix
Table 9: Llama-3.2-1B-Instruct (edge draft) with Llama-3.1-8B-Instruct (cloud target); RTX 4090 edge, A100 server, 1 client, tok/s (Llama tokenizer). dmax= synchronous depth cap re-tuned for the pair.
Qwen3-1.7B → 32B (tok/s)
SPROUT
SpecBench
OlympiadBench
Added (ms)
dmax
Ref
S1
S3
A1
A3
A3 / S1
Ref
S1
S3
A1
A3
A3 / S1
Ref
S1
S3
A1
A3
A3 / S1
0
5
42.8
51.8
53.0
46.9
51.0
0.98
37.3
41.6
41.7
36.8
40.6
0.98
60.9
67.6
68.1
61.5
68.0
1.01
50
8
32.3
36.8
37.7
37.5
41.6
1.13
24.8
26.8
26.9
26.0
30.2
1.13
47.8
51.3
53.1
49.6
58.1
1.13
100
13
24.3
28.6
30.1
30.3
35.3
1.23
17.6
19.5
19.6
21.0
24.6
1.26
40.6
44.6
45.2
42.1
51.0
1.14
200
20
18.7
21.6
22.2
23.2
26.8
1.24
12.6
13.7
13.8
15.4
18.1
1.32
30.4
33.1
34.0
34.0
40.4
1.22
300
27
15.5
17.1
17.2
18.9
21.6
1.26
10.1
10.9
11.0
12.4
14.1
1.29
24.8
26.3
27.1
28.1
33.6
1.28
Appendix
Table 10: Qwen3-1.7B (edge draft) with Qwen3-32B (cloud target) on an H100 server; RTX 4090 edge, 1 client, tok/s. dmax= synchronous depth cap re-tuned for the pair.
SPROUT
SpecBench
OlympiadBench
Added RTT (ms)
0
50
100
200
300
400
500
0
50
100
200
300
400
500
0
50
100
200
300
400
500
1 client
Ref
42.5
32.4
26.3
19.5
16.3
13.9
12.2
41.4
28.5
20.8
13.7
10.9
9.2
8.0
55.3
46.6
39.7
29.6
24.7
20.6
17.8
S1
58.0
42.6
33.6
24.1
19.1
15.9
13.7
50.2
32.6
23.9
15.5
12.2
10.2
8.7
67.7
53.8
44.7
33.5
27.4
22.7
19.3
S3
59.0
43.6
34.0
24.5
19.7
16.4
14.0
50.7
33.2
23.9
15.8
12.4
10.1
8.7
68.9
55.4
45.9
35.2
28.3
23.6
20.1
A1
54.0
43.2
36.0
26.7
21.2
18.0
15.8
43.8
32.2
25.4
18.2
14.1
11.6
9.8
63.7
54.9
46.5
36.7
30.1
25.9
22.4
A3
55.0
45.7
38.8
29.8
24.5
20.7
17.9
44.9
34.9
28.0
20.2
15.8
13.1
11.0
65.0
57.3
50.8
41.0
34.3
29.2
26.1
Appendix
Table 11: RTX 3090 edge , 1 and 2 clients, with the server of Table 2 . 1 client: tok/s; 2 clients: sum of both clients’ tok/s (two edge GPUs sharing one A100).
Figure 7: Routing shares vs. added RTT on the three benchmarks ( 50 ms steps).
Acc. (%)
Share (%)
Group
n
Ref
R
Spd
Cl
Dir
CoD
Th
GPQA
94
42.6
41.5
1.54 ×
74
2
6
17
MuSR
123
59.3
61.8
2.09 ×
49
2
15
35
MMLU-Pro
1772
69.4
67.6
1.58 ×
78
3
9
10
MATH
1462
88.1
93.2
1.57 ×
11
3
61
25
OpenHermes
2912
82.9
83.5
1.89 ×
57
12
5
27
Appendix
Table 12: Per-group routing on SPROUT at 300 ms added latency, max-accuracy operating point (router on A3 ; overall 80.5% , +0.9 pp over the always- 14 B target; 1.72× Ref throughput). Acc: Ref (the 14 B target) vs. R ; Spd: speedup over Ref (as in Table 2 ); Share: routing fractions over Spec-Cloud , Edge-Direct , Edge-CoD , Edge-Think .
Dataset
Operating point
Added RTT (ms)
0
100
200
300
400
500
SPROUT
Iso-acc ( × Ref )
1.24
1.59
1.84
1.97
2.13
2.21
Max-acc ( × Ref )
0.92
1.31
1.56
1.72
1.86
1.96
Δ acc (pp)
+1.1
+1.0
+0.9
+0.9
+0.9
+0.9
SpecBench
Iso-acc ( × Ref )
1.12
1.33
1.50
1.50
1.47
1.47
Max-acc ( × Ref )
1.13
1.33
1.50
1.50
1.47
1.47
Appendix
Table 13: Router speedup over Ref at the iso-accuracy and max-accuracy operating points (router on the fastest Spec-Cloud variant of each cell: A3 , or S3 for SpecBench at added 0 ), with the max-accuracy point’s accuracy change over the 14 B target ( Δ acc, pp). The iso-accuracy row is the R column of Table 2 ; on SpecBench no operating point reaches the 14 B accuracy, so both rows are the max-accuracy point.
Metric
Added
Eff. RTT
Ref
S1
S3
A1
A3
R
Server compute (s)
100
115
107
105
103
201
191
149
200
218
92
90
90
207
195
111
300
321
88
88
89
212
198
61
Rental ($/1M tok)
100
115
29
27
26
26
22
17
200
218
39
36
35
33
28
16
300
321
48
45
44
41
34
10
Appendix
Table 14: Server compute and cost on OlympiadBench (RTX 4090 edge, one client): verifier-forward time per 32 -query run, session rental cost per 1 M tokens (A100 4.05/h+RTX40900.35/h by wall clock) and the forward-only lower bound; R = A3 scaled by the router’s cloud token share ( 78 / 57 / 31% at 100 / 200 / 300 ms); bold = cheapest.
Figure 8: Decoding throughput (bars) and server runtime per 1 k output tokens ( ⧫ ). Srv = server-only speculative decoding, whose draft forwards also run on the server; R ’s server time ( ◊ ) = A3 ’s per-token server time × the router’s cloud token share.
Speculative decoding accelerates large language model (LLM) inference by using a lightweight draft model to propose tokens and a larger target model to verify them in parallel. In distributed edge-cloud inference, however, draft length must be controlled online: longer drafts amortize communication delay but reduce token acceptance, whereas shorter drafts preserve acceptance but trigger more communication rounds. We formulate this tradeoff as a ratio-type optimal stopping problem and prove that the optimal draft length is a finite delay-monotone threshold. The analysis identifies a critical delay below which single-token speculation is optimal and shows that the optimal length grows only logarithmically with communication delay. For time-varying networks, we extend the model to Markov-modulated channels and establish, under a bounded horizon and monotone stopping-region conditions, a state-dependent threshold policy. For unknown environments, we propose UCB-SpecStop, an online control algorithm with gap-free and gap-dependent expected regret bounds of O(LmaxKmaxTlog(KmaxT)) and O(∑k:Δk>0Lmax2log(KmaxT)/Δk). We implement the method on a real edge-cloud testbed with a Jetson Orin Nano Super edge node and an RTX3090 Ti cloud node, using Qwen and Llama draft--target pairs. Experiments validate the predicted phase transition, with transition points near 83ms and 111~ms. Qwen matches the geometric prediction, while Llama requires empirical-prefix calibration due to heavy-head acceptance. Across the tested delay grid, UCB-SpecStop reduces per-token latency over SpecDec++ by up to 22.4%, approaches an offline oracle within 0.2--2.4% in communication-dominated regimes, improves over naive UCB by up to 7.5%, removes the 14.0--18.7% gap caused by static tuning under delay drift, and gains 3.0--6.8% with contextual channel-state information.
Kangkang Sun, Jianhua Li, Xiuzhen Chen +2
Shanghai Key Laboratory of Integrated Administration Technologies for Information Security, School of Computer Science, Shanghai Jiao Tong University, Shanghai 200240, China · School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen, Guangdong, China
Autoregressive decoding is bottlenecked by its sequential nature. Speculative decoding has become a standard way to accelerate inference by using a fast draft model to predict upcoming tokens from a slower target model, and then verifying them in parallel with a single target model forward pass. However, speculative decoding itself relies on a sequential dependence between speculation and verification. We introduce speculative speculative decoding (SSD) to parallelize these operations. While a verification is ongoing, the draft model predicts likely verification outcomes and prepares speculations pre-emptively for them. If the actual verification outcome is then in the predicted set, a speculation can be returned immediately, eliminating drafting overhead entirely. We identify three key challenges presented by speculative speculative decoding, and suggest principled methods to solve each. The result is Saguaro, an optimized SSD algorithm. Our implementation is on average 30% faster than optimized speculative decoding baselines and up to 5x faster than autoregressive decoding with open source inference engines.
Tanishq Kumar, Tri Dao, Avner May
Department of Computer Science, Stanford University · Department of Computer Science, Princeton University · Together AI
While distributed device-edge speculative decoding enhances resource utilization across heterogeneous nodes, its performance is often bottlenecked by conventional token-level verification strategies. Such rigid alignment leads to excessive rejections, significantly diminishing the accepted sequence length and increasing interaction rounds under fluctuating wireless conditions. In this paper, we propose WISV (Wireless-Informed Semantic Verification), a novel distributed speculative decoding framework that goes beyond strict token-level matching via a channel-aware semantic acceptance policy. WISV integrates a lightweight decision head into the edge-side target LLM to dynamically evaluate speculative tokens by synthesizing high-dimensional hidden representations with instantaneous channel state information (CSI). To optimize the trade-off between verification fidelity and communication overhead, we further design two tailored communication protocols: full-hidden upload and mismatch-first selective-hidden upload. Extensive simulations using a 1B drafter and an 8B target model demonstrate that WISV achieves up to a 60.8% increase in accepted length, a 37.3% reduction in interaction rounds, and a 31.4% improvement in end-to-end latency compared to vanilla speculative decoding across tested settings, while maintaining a negligible task accuracy drop (<1%). Finally, we validate WISV on a hardware testbed comprising an NVIDIA Jetson AGX Orin and an A40-equipped server, confirming its real-world efficacy in accelerating edge-deployed LLM inference.
Zixuan Liu, Zhiyong Chen, Nan Xue +4
Cooperative Medianet Innovation Center and the Department of Electronic Engineering, Shanghai Jiao Tong University, Shanghai 200240, China