SpecRelay: Amortize or Avoid the Round Trip in Edge-Cloud Speculative Decoding
Authors: Jai Jain, Chihyo Ahn, Yong Liu, Charles Chen, Duc N. M Hoang, Ajay Kumar Jaiswal, Fatih Ilhan, Han-Byul Kim, Lauren Hannah, Richa Dixit, Minsik Cho, Yingyan Celine Lin, Hyesoon Kim
Organizations: Georgia Institute of Technology · Apple
Edge-cloud speculative decoding moves autoregressive drafting to an edge device while a larger cloud model verifies draft tokens in parallel, so that edge drafting and cloud verification overlap under a synchronous draft-verify schedule. Beyond low-latency local networks, however, the next verification request of that schedule can be sent only after the previous reply arrives, so every round still waits out a full round trip, and throughput falls as the network RTT grows. To recover this loss, we present SpecRelay, an edge-cloud LLM inference system that addresses this cost at two levels: it amortizes round trips when cloud verification is necessary and avoids them when local reasoning suffices. For cloud-bound queries, an asynchronous verification pipeline keeps fixed-budget draft trees in flight while earlier replies are still outstanding, and the cloud rejects stale requests before the target-model forward pass (the asynchronous schedule). At the query level, an RTT-conditioned router (the Latency-Aware Router) estimates quality and completion latency for three on-device modes (direct generation, Chain-of-Draft, and full reasoning) and cloud speculative decoding, selecting among them under a configurable accuracy-latency objective without adding latency during inference, since the router runs concurrently with the draft-model prefill. The asynchronous schedule alone improves decoding throughput by up to 50% over a reproduced synchronous baseline across multiple benchmarks. At matched target-model accuracy and 250 ms of added round-trip latency, the full system with the router delivers 1.95x and 1.55x the throughput of the same baseline on SPROUT and OlympiadBench, respectively, while sending only 47% and 39% of the queries to the cloud.
Figures & tables
Figure 1
Figure 3: Anchor-hedged asynchronous verification pipeline. (a) Edge and server timelines under the synchronous and asynchronous schedules. (b–d) Edge KV state of Async-K3: (b) Tree1 drafted under three anchors of Tree0 and in flight; (c) reply to Tree0 ( Verify0 ) received; (d) reply to Tree1 ( Verify1 ) received, lineage-preserving rebuild at Tree2 .
Table 1: Experimental setup: hardware, models and the variants that appear in the experiments.
SPROUT (1024-tok)
SpecBench (256-tok)
OlympiadBench (1024-tok)
Added
Eff. RTT
dmax
Up
Ref
S1
S3
A1
A3
R
Up
Ref
S1
S3
A1
A3
R
Up
Ref
S1
S3
A1
A3
R
(ms)
(ms)
(sync)
tok/s
× Ref
tok/s
× Ref
tok/s
× Ref
0
12
3
38.0
51.9
1.13
1.16
1.12
1.18
1.24
39.5
47.5
1.06
1.10
0.95
1.02
1.12
39.7
65.4
1.06
1.09
1.05
1.15
1.13
25
38
5
30.0
44.8
1.17
1.20
1.09
1.25
1.31
30.0
37.8
1.10
1.10
0.97
1.12
1.14
32.0
58.7
1.08
1.13
1.03
1.17
1.14
50
64
6
23.4
38.9
1.12
1.16
1.10
1.28
1.38
24.0
30.9
1.07
1.09
1.02
1.21
1.25
26.0
51.2
1.06
1.10
1.08
1.24
1.22
100
115
9
18.8
29.9
1.13
1.15
1.18
1.39
1.59
17.4
22.2
1.09
1.09
1.11
1.33
1.33
21.8
41.7
1.08
1.11
1.13
1.32
1.30
Table 2: Decoding throughput and speedup across added RTT on the RTX 4090 edge. Up and Ref in tok/s; the other columns are speedups over Ref ; bold = best per benchmark and RTT.
Figure 5
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
depth cap
SPROUT
SpecBench
OlympiadBench
Added
Tab. 2
S1 fill
S3 fill
S1
S1 fill
S3
S3 fill
A1
A3
S1
S1 fill
S3
S3 fill
A1
A3
S1
S1 fill
S3
S3 fill
A1
A3
(ms)
dmax
proactive
proactive
tok/s
tok/s
tok/s
0
3
5
5
58.6
64.9 +
60.2
66.5 +
58.1
61.2
50.4
51.0 +
52.3
50.3
45.1
48.5
69.3
78.6 +
71.3
78.6 +
68.7
75.2
50
6
9
9
43.6
45.5 +
45.1
47.7 +
42.8
49.8
33.1
33.6 +
33.7
35.6 +
31.5
37.4
54.3
59.4 +
56.3
61.6 +
55.3
63.5
100
9
15
14
33.8
35.5 +
34.4
37.1 +
35.3
41.6
24.2
24.9 +
24.2
25.8 +
24.6
29.5
45.0
47.9 +
46.3
49.8 +
47.1
55.0
150
12
21
20
28.2
28.1
28.4
30.2 +
30.4
36.1
18.9
19.4 +
19.1
20.2 +
21.0
24.4
38.8
39.0 +
39.5
40.9 +
42.0
48.5
Appendix
Table 3: Synchronous variants re-tuned to fill the wait (RTX 4090, 32 queries, tok/s): main tree at depth 3 , proactive cap ≈ wait / draft-forward time (listed; one level lower for S3 ), 32 -node budget unchanged. S1 , S3 , A1 and A3 = the Table 2 rows.
SPROUT
SpecBench
OlympiadBench
Edge
Budget
50
100
200
300
50
100
200
300
50
100
200
300
tok/s (bold = best budget per column)
RTX 4090
32
45.0
34.4
24.8
19.7
33.7
24.3
16.2
12.5
56.3
46.3
35.4
28.1
64
44.6
37.0
26.4
20.6
34.6
26.1
17.0
12.7
55.1
47.8
37.6
30.8
96
43.9
37.9
27.2
21.7
34.0
27.2
17.4
13.3
53.8
48.4
38.5
31.5
RTX 3090
32
43.6
34.0
24.5
19.7
33.2
23.9
15.8
12.4
55.4
45.9
35.2
28.3
Appendix
Table 4: Edge draft-budget sweep of S3 (1 client, tok/s; budget 32= the S3 rows of Tables 2 and 11 ; bold = best budget per column). acc/verify = accepted tokens per verify forward; verify ms = mean server-side verify time.
Figure 6: Edge time per committed token at 100 ms added RTT (RTX 3090). (a) Mean draft-forward time over the three benchmarks. (b–d) Edge seconds per 1k committed tokens.
Variant
SPROUT
SpecBench
OlympiadBench
useful
rejected
useful
rejected
useful
rejected
Ref
0.88
1.38
0.83
2.39
0.90
0.81
S1 fill
0.88
1.85
0.83
3.13
0.91
1.08
S3 fill
0.88
1.66
0.83
2.98
0.91
0.98
A1
0.81
1.62
0.79
2.66
0.82
1.08
A3
0.81
1.19
0.78
2.06
0.82
0.72
Appendix
Table 5: Draft forwards per committed token at 100 ms added RTT (RTX 3090): useful (level on the accepted path) and rejected.
tok/s
SPROUT
SpecBench
OlympiadBench
Edge
Added (ms)
K=1
K=2
K=3
K=4
K=5
K=1
K=2
K=3
K=4
K=5
K=1
K=2
K=3
K=4
K=5
RTX 4090
0
58.2
62.0
61.2
57.9
56.3
45.0
48.5
48.3
45.7
46.5
68.6
74.6
74.9
69.9
67.3
50
42.6
49.5
49.9
47.8
46.9
31.5
35.5
37.4
35.9
36.0
55.1
63.2
63.6
59.9
58.5
100
35.3
40.2
41.6
39.0
36.6
24.7
28.2
29.6
29.4
28.6
47.1
55.2
55.0
52.4
51.5
200
26.2
30.3
32.1
31.0
30.7
17.7
20.0
21.0
21.1
21.1
36.8
43.3
44.3
42.1
41.1
300
21.3
24.0
25.4
25.3
25.3
13.9
15.6
16.2
16.4
16.4
30.0
35.3
36.2
35.7
35.2
Appendix
Table 6: Sweep over the number of anchors K of the asynchronous schedule.
SPROUT
SpecBench
OlympiadBench
Jitter (added RTT 0)
S1
S3
S3 until
A1
A3
A3 / S1
S1
S3
S3 until
A1
A3
A3 / S1
S1
S3
S3 until
A1
A3
A3 / S1
RTX 4090 edge
none (control)
58.9
60.0
63.7
57.4
61.7
1.05
50.5
51.2
48.6
46.0
49.4
0.98
70.1
71.3
78.2
70.4
74.3
1.06
Pareto-AR(1), s=10
49.8
50.7
55.7
49.1
57.0
1.14
39.8
41.2
40.3
38.5
41.9
1.05
59.3
60.5
69.7
62.6
69.8
1.18
Pareto-AR(1), s=30
38.7
39.2
43.7
43.7
49.5
1.28
29.2
29.6
29.3
30.0
34.3
1.17
46.3
47.0
56.8
53.4
59.5
1.29
Pareto-AR(1), s=60
29.0
29.4
33.4
36.9
41.1
1.42
20.4
20.7
21.0
24.4
26.4
1.29
34.8
35.3
43.2
44.0
46.3
1.33
Appendix
Table 7: Jitter resilience at added RTT 0 (1 client, tok/s): server-side de-meaned Pareto-AR(1) delay (scale s ) and Gilbert–Elliott bursts (mean bad-state delay f ms). S1 / S3 at the nominal depth 3 ; the blank cell hit a deterministic K=1 livelock in both attempts.
Throughput (tok/s)
Added RTT 0 ms
Added RTT 300 ms
Edge
Variant
T=0
T=0.7
T=0
T=0.7
RTX 4090
Ref
47.5
44.9
11.4
11.0
S3
52.1
49.9
12.5
11.1
A3
48.3
45.7
16.2
15.3
RTX 3090
Ref
41.4
40.4
10.9
10.5
S3
50.7
48.3
12.4
11.4
Appendix
Table 8: Verifier temperature 0 vs. 0.7 on SpecBench (1 client, client-span tok/s).
Llama-3.2-1B → 3.1-8B (tok/s)
SPROUT
SpecBench
OlympiadBench
Added (ms)
dmax
Ref
S1
S3
A1
A3
A3 / S1
Ref
S1
S3
A1
A3
A3 / S1
Ref
S1
S3
A1
A3
A3 / S1
0
5
75.7
88.4
91.3
75.4
87.7
0.99
68.2
73.4
74.5
62.0
70.0
0.95
95.6
101.9
104.5
88.5
99.1
0.97
50
10
55.6
63.6
66.7
53.8
66.0
1.04
46.1
48.7
49.3
42.2
52.0
1.07
72.4
78.3
79.1
65.8
78.4
1.00
100
15
40.8
46.1
46.8
45.1
53.9
1.17
33.6
34.8
35.6
33.0
40.7
1.17
53.2
56.3
58.5
54.6
64.7
1.15
200
21
28.6
31.2
32.4
31.8
39.1
1.25
22.6
23.3
23.6
22.8
27.4
1.18
35.7
37.5
38.5
40.2
48.9
1.30
300
27
22.1
24.0
24.6
25.3
30.1
1.25
17.1
17.6
17.8
17.8
21.3
1.21
27.4
28.7
28.8
32.3
38.2
1.33
Appendix
Table 9: Llama-3.2-1B-Instruct (edge draft) with Llama-3.1-8B-Instruct (cloud target); RTX 4090 edge, A100 server, 1 client, tok/s (Llama tokenizer). dmax= synchronous depth cap re-tuned for the pair.
Qwen3-1.7B → 32B (tok/s)
SPROUT
SpecBench
OlympiadBench
Added (ms)
dmax
Ref
S1
S3
A1
A3
A3 / S1
Ref
S1
S3
A1
A3
A3 / S1
Ref
S1
S3
A1
A3
A3 / S1
0
5
42.8
51.8
53.0
46.9
51.0
0.98
37.3
41.6
41.7
36.8
40.6
0.98
60.9
67.6
68.1
61.5
68.0
1.01
50
8
32.3
36.8
37.7
37.5
41.6
1.13
24.8
26.8
26.9
26.0
30.2
1.13
47.8
51.3
53.1
49.6
58.1
1.13
100
13
24.3
28.6
30.1
30.3
35.3
1.23
17.6
19.5
19.6
21.0
24.6
1.26
40.6
44.6
45.2
42.1
51.0
1.14
200
20
18.7
21.6
22.2
23.2
26.8
1.24
12.6
13.7
13.8
15.4
18.1
1.32
30.4
33.1
34.0
34.0
40.4
1.22
300
27
15.5
17.1
17.2
18.9
21.6
1.26
10.1
10.9
11.0
12.4
14.1
1.29
24.8
26.3
27.1
28.1
33.6
1.28
Appendix
Table 10: Qwen3-1.7B (edge draft) with Qwen3-32B (cloud target) on an H100 server; RTX 4090 edge, 1 client, tok/s. dmax= synchronous depth cap re-tuned for the pair.
SPROUT
SpecBench
OlympiadBench
Added RTT (ms)
0
50
100
200
300
400
500
0
50
100
200
300
400
500
0
50
100
200
300
400
500
1 client
Ref
42.5
32.4
26.3
19.5
16.3
13.9
12.2
41.4
28.5
20.8
13.7
10.9
9.2
8.0
55.3
46.6
39.7
29.6
24.7
20.6
17.8
S1
58.0
42.6
33.6
24.1
19.1
15.9
13.7
50.2
32.6
23.9
15.5
12.2
10.2
8.7
67.7
53.8
44.7
33.5
27.4
22.7
19.3
S3
59.0
43.6
34.0
24.5
19.7
16.4
14.0
50.7
33.2
23.9
15.8
12.4
10.1
8.7
68.9
55.4
45.9
35.2
28.3
23.6
20.1
A1
54.0
43.2
36.0
26.7
21.2
18.0
15.8
43.8
32.2
25.4
18.2
14.1
11.6
9.8
63.7
54.9
46.5
36.7
30.1
25.9
22.4
A3
55.0
45.7
38.8
29.8
24.5
20.7
17.9
44.9
34.9
28.0
20.2
15.8
13.1
11.0
65.0
57.3
50.8
41.0
34.3
29.2
26.1
Appendix
Table 11: RTX 3090 edge , 1 and 2 clients, with the server of Table 2 . 1 client: tok/s; 2 clients: sum of both clients’ tok/s (two edge GPUs sharing one A100).
Figure 7: Routing shares vs. added RTT on the three benchmarks ( 50 ms steps).
Acc. (%)
Share (%)
Group
n
Ref
R
Spd
Cl
Dir
CoD
Th
GPQA
94
42.6
41.5
1.54 ×
74
2
6
17
MuSR
123
59.3
61.8
2.09 ×
49
2
15
35
MMLU-Pro
1772
69.4
67.6
1.58 ×
78
3
9
10
MATH
1462
88.1
93.2
1.57 ×
11
3
61
25
OpenHermes
2912
82.9
83.5
1.89 ×
57
12
5
27
Appendix
Table 12: Per-group routing on SPROUT at 300 ms added latency, max-accuracy operating point (router on A3 ; overall 80.5% , +0.9 pp over the always- 14 B target; 1.72× Ref throughput). Acc: Ref (the 14 B target) vs. R ; Spd: speedup over Ref (as in Table 2 ); Share: routing fractions over Spec-Cloud , Edge-Direct , Edge-CoD , Edge-Think .
Dataset
Operating point
Added RTT (ms)
0
100
200
300
400
500
SPROUT
Iso-acc ( × Ref )
1.24
1.59
1.84
1.97
2.13
2.21
Max-acc ( × Ref )
0.92
1.31
1.56
1.72
1.86
1.96
Δ acc (pp)
+1.1
+1.0
+0.9
+0.9
+0.9
+0.9
SpecBench
Iso-acc ( × Ref )
1.12
1.33
1.50
1.50
1.47
1.47
Max-acc ( × Ref )
1.13
1.33
1.50
1.50
1.47
1.47
Appendix
Table 13: Router speedup over Ref at the iso-accuracy and max-accuracy operating points (router on the fastest Spec-Cloud variant of each cell: A3 , or S3 for SpecBench at added 0 ), with the max-accuracy point’s accuracy change over the 14 B target ( Δ acc, pp). The iso-accuracy row is the R column of Table 2 ; on SpecBench no operating point reaches the 14 B accuracy, so both rows are the max-accuracy point.
Metric
Added
Eff. RTT
Ref
S1
S3
A1
A3
R
Server compute (s)
100
115
107
105
103
201
191
149
200
218
92
90
90
207
195
111
300
321
88
88
89
212
198
61
Rental ($/1M tok)
100
115
29
27
26
26
22
17
200
218
39
36
35
33
28
16
300
321
48
45
44
41
34
10
Appendix
Table 14: Server compute and cost on OlympiadBench (RTX 4090 edge, one client): verifier-forward time per 32 -query run, session rental cost per 1 M tokens (A100 4.05/h+RTX40900.35/h by wall clock) and the forward-only lower bound; R = A3 scaled by the router’s cloud token share ( 78 / 57 / 31% at 100 / 200 / 300 ms); bold = cheapest.
Figure 8: Decoding throughput (bars) and server runtime per 1 k output tokens ( ⧫ ). Srv = server-only speculative decoding, whose draft forwards also run on the server; R ’s server time ( ◊ ) = A3 ’s per-token server time × the router’s cloud token share.
Shanghai Key Laboratory of Integrated Administration Technologies for Information Security, School of Computer Science, Shanghai Jiao Tong University, Shanghai 200240, China · School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen, Guangdong, China