Early-exit (EE) LLMs let confident tokens skip a model's deep layers, but batched serving loses most of this gain. Current systems either force tokens to exit before their own ramp is confident, an error the ramp does not bound, or save almost nothing when decoding is weight-bound, because any continuing token makes a step read the deep weights. We present DREX, a batched EE serving system that never forces a token to exit. DREX introduces cross-iteration batching: tokens that continue past the exit layer wait there, and every few steps one deep pass over all waiting tokens reads the deep weights once for several steps. Its two policies differ in what such a token's request does: holding pauses it until the deep pass, and speculative holding lets it keep decoding from the exit layer's own guess, which the next deep pass verifies. Exited tokens are trusted and never verified. Using a calibrated cost model, a controller picks speculative holding where reading weights dominates a step and guesses are often right, and holding elsewhere. DREX keeps vLLM's asynchronous scheduling by planning each deep pass from exit counts of earlier steps. With synthetic exits where 90% of tokens exit at a quarter of the depth and 40% of guesses are correct, DREX achieves 2.22-2.24× decode speedup over non-EE across batch sizes on Llama-2-70B, while per-row skipping (vSkipper's design) gains at most 7.0%. On LayerSkip models, DREX adds no statistically resolved quality loss to the EE model's own. On CNN/DM at full memory, DREX with LayerSkip achieves 1.58-1.91× decode speedup while remaining within 1.4 ROUGE-2 points of the non-EE model. These speedups are 9-63% larger than EAGLE's across the evaluated model sizes, with each method measured against its own non-EE target.
Figures & tables
Figure 1: Conceptual diagram of an LLM with EE. Each token is a separate step. The sequence exits early in the 2nd and 4th steps (when outputting the tokens “is” and “EOS.”)
Figure 2: Three faithful policies for the same requests and exit decisions (numbers are positions). Blue cells are the tokens a deep read serves: (a) wastes most of each read, (b) fills it by pausing requests, and (c) fills it further by letting held requests keep decoding from the exit layer’s token, which the read verifies.
Policy
Exits (%)
Invol. exit (%)
Invol. stay (%)
Accuracy (%)
GSM8K
BBH
GSM8K
BBH
GSM8K
BBH
GSM8K
BBH
Non-EE
–
–
–
–
–
–
26.4
46.8
Batch-1 EE
50.7
50.1
–
–
–
–
21.6
43.0
Consensus
0.5
0.1
0
0
49.1
49.6
25.2
46.2
Majority
83.5
91.9
17.0
28.4
7.3
3.6
3.2
13.2
Greedy
100
100
20.0
29.8
0
0
2.0
8.8
Table 1: Exits, involuntary exits and stays, and accuracy (exact match) on LayerSkip Llama-2-13B (exit after layer 15 of 40, 50% exit-rate target, N=64 , held-out documents).
Figure 3: DREX ’s components on the flush step for the window of Figure 2 c, in compact mode. 1 When the EE ramp says continue, the exit layer’s token at the held position is its guess, and 2 the guess is fed as the next input, so later positions are tentative. 3 Plan-ahead flushing plans the flush and its buckets from counters two steps old, with k and m from the period controller and the gap clock’s bound. 4 The compacted deep pass gathers held positions into deep rows and exits whose deep K and V are pending into K/V-only riding rows. 5 Verification accepts a guess if it equals the deep token, and each request commits up to its first rejection and discards the rest.
Figure 4: Decode throughput over non-EE (dashed) at full memory. (a) The synthetic ramp, by exit pattern (E/L,p) with p≥E/L . (b) CNN/DM on LayerSkip against ROUGE-2: DREX from τ=∞ to the 1% and 3% risk targets, the baselines at both targets, and EAGLE at its best k , each over its own target’s non-EE.
Figure 5: Llama-2-70B at α=.4 after the controller’s warm-up against the optima in W ( Theorems 1 and 2 ): the best full-flush policy and the holding optimum SN∗ , with the calibrated model (thin lines).
Figure 6: Quality at N=64 on LayerSkip: the EE model’s own loss (B − A, B = batch-1 EE, A = the non-EE model) and DREX − B at three targets per task, at one exit layer per model (13B: layer 15, 70B: 20, 8B: 8). Table 11 adds a second exit layer.
Figure 7: Llama-3.1-8B, sampled PG-19. (a) EAGLE-3’s speedup falls with N while DREX ’s grows (synthetic ramp at α=.4 , LayerSkip-8B at ϵ∗ 3%). (b) The break-even acceptance β∗ a draft head needs to match DREX rises past EAGLE-3’s acceptance (purple band).
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
SN∗ at N=
fixed / SN∗
(E/L,p)
G1
maxkS
1
4
16
32
64
256
16
64
(.25, .9)
3.08
2.09
3.08
2.33
2.15
2.12
2.10
2.09
.972
.993
(.375, .7)
1.78
1.25
1.78
1.35
1.28
1.27
1.26
1.25
.975
.992
(.25, .5)
1.60
1.20
1.60
1.30
1.22
1.21
1.20
1.20
.983
.996
(.5, .5)
1.33
1.00
1.33
1.09
1.04
1.03
1.02
1.01
.960
.980
Appendix
Table 2: The holding optimum SN∗ in W against the batch-1 gain and a fixed flush period.
Policy information
N=16
32
64
Exits observed, α≥.2
.978–.995
.989–.998
.994–.999
Guess outcomes revealed, α≥.2
≥ .972
≥ .985
≥ .994
Holding only ( α=0 )
.960–.984
.972–.991
.980–.996
Appendix
Table 3: A fixed window’s share of the best full-flush policy in W ( § 3.3 ).
N
16
64
128
192
280
Non-EE (tokens/s)
1134
3829
6335
7746
8650
Within-iteration
0.98
1.01
1.07
1.13
1.14
Per-row skipping
0.98
1.05
1.17
1.38
1.63
DREX (holding only)
1.93
1.96
2.02
2.09
2.08
DREX , α=0
1.94
1.99
2.05
2.09
2.24
DREX , α=.4
2.12
2.19
2.16
2.24
2.35
Appendix
Table 4: Llama-2-70B with tensor parallelism 2 at (.25, .9): decode throughput over non-EE at the same N (280: full memory), and DREX over per-row skipping.
Llama-2-70B (weight-bound)
Llama-2-13B (attention-bound)
(E/L,p)
holding only
DREX , α 0
α .2
α .4
α .6
holding only
DREX , α 0
α .2
α .4
α .6
(.25, .9)
2.01 / 2.05
2.01 / 2.07
2.06 / 2.10
2.22 / 2.24
2.41 / 2.45
2.02 / 2.38
2.02 † / 2.38 †
2.01 / 2.38 †
2.13 / 2.38 †
2.24 / 2.53
(.375, .7)
1.22 / 1.25
1.22 / 1.24
1.30 / 1.30
1.38 / 1.38
1.50 / 1.51
1.29 / 1.49
1.29 / 1.50
1.29 / 1.50
1.39 / 1.50
1.48 / 1.55
(.25, .5)
1.18 / 1.20
1.18 / 1.18
1.25 / 1.24
1.39 / 1.39
1.57 / 1.62
1.22 / 1.37
1.22 / 1.38
1.22 / 1.39
1.34 / 1.39
1.47 / 1.51
(.5, .5)
1.00 / 1.02
1.00 / 1.00
1.05 / 1.05
1.12 / 1.12
1.19 / 1.19
1.04 / 1.17
1.04 / 1.16
1.04 / 1.16
1.11 / 1.16
1.17 / 1.21
(.75, .3)
1.00 / 1.00
1.00 † / 1.00 †
1.00 † / 1.00 †
1.00 † / 1.00 †
1.00 † / 1.00 †
1.00 / 1.04
1.00 † / 1.03
1.00 † / 1.03
1.00 † / 1.03
1.00 † / 1.03
Appendix
Table 5: Decode throughput of the deployed systems over non-EE with the synthetic ramp at N=16 / full memory (47 on 70B, 80 on 13B), sweeping the guess acceptance α . † : the deployment runs in place, the calibrated model’s choice at the cell’s N , p , and α , where DREX holds, so the cell repeats the number of DREX (holding only) (compact mode’s in Appendix C ). ‡ : the full-depth fallback ran ( Appendix H ). Bold: α=.4 where p≥E/L . Ratios in the text use unrounded values.
N=16 / full memory
N=384
(E/L,p)
holding only
DREX , α 0
α .2
α .4
α .6
hold
α 0
.2
.4
.6
(.25, .9)
1.37 / 1.86
1.37 † / 1.95
1.37 † / 1.97
1.33 / 1.97
1.37 / 1.97
1.79
1.85
1.86
1.87
1.89
(.375, .7)
1.13 / 1.39
1.13 † / 1.42
1.13 / 1.43
1.16 / 1.43
1.23 / 1.43
1.37
1.41
1.40
1.40
1.41
(.25, .5)
0.97 ‡ / 1.21
0.97 †‡ / 1.32
0.97 †‡ / 1.32
1.13 / 1.32
1.15 / 1.32
1.19
1.27
1.27
1.27
1.27
(.5, .5)
0.98 ‡ / 1.12
0.98 †‡ / 1.17
0.98 †‡ / 1.18
1.03 / 1.18
1.09 / 1.17
1.11
1.16
1.15
1.15
1.15
(.75, .3)
0.98 ‡ / 1.01
0.98 †‡ / 1.00
0.98 †‡ / 1.00
0.98 †‡ / 1.00
0.98 †‡ / 1.00
1.01
1.01
1.00
1.00
1.00
Appendix
Table 6: Llama-3.1-8B (compute-bound) with the synthetic ramp: decode throughput of the deployed systems over non-EE at N=16 / full memory (527) and at the middle concurrency N=384 ( † , ‡ , and bold as in Table 5 ).
Llama-2-70B (weight-bound)
Llama-2-13B (attention-bound)
Llama-3.1-8B (compute-bound)
(E/L,p)
within-iteration
per-row skipping
within-iteration
per-row skipping
within-iteration
per-row skipping
(.25, .9)
1.000 / 1.038
1.011 / 1.070
1.187 / 1.683
1.177 / 1.720
0.974 / 1.521
0.932 / 1.939
(.375, .7)
1.006 / 1.031
1.004 / 1.042
1.108 / 1.355
1.095 / 1.348
0.950 / 1.270
0.913 / 1.438
(.25, .5)
0.992 / 1.017
0.993 / 1.032
1.074 / 1.281
1.070 / 1.283
0.954 / 1.216
0.913 / 1.330
(.5, .5)
1.004 / 1.019
0.990 / 1.015
1.045 / 1.167
1.040 / 1.167
0.953 / 1.125
0.917 / 1.184
(.75, .3)
0.999 / 1.003
0.995 / 1.000
1.001 / 1.038
0.997 / 1.036
0.948 / 1.014
0.921 / 1.012
Appendix
Table 7: Within-iteration batching and per-row skipping with the synthetic ramp: decode throughput over non-EE at N=16 / full memory (47 on 70B, 80 on 13B, 527 on 8B), to three decimals because the gains are small.
Figure 8: Gap bounds, speedup over non-EE: each line runs from unbounded to 2 × and 1.5 × non-EE’s TPOT (dotted, 2 × right, 1.5 × left). 8B at N≥384 is not drawn: one 0.26–0.28 s pause per run, non-EE’s included, exceeds any such bound there.
13B
70B
8B
Arm
TPOT
p99 gap
TPOT
p99 gap
TPOT
p99 gap
Within-iter.
0.59–0.93
0.56–0.96 (0.80)
0.96–1.01
0.93–1.00 (0.98)
0.66–1.05
0.98–1.07 (1.04)
Holding only
0.42–0.82
1.08–3.80 (1.29)
0.49–0.85
1.23–2.45 (1.72)
0.54–1.03
1.00–2.33 (1.06)
Per-row skip.
0.58–0.93
0.72–1.47 (0.92)
0.93–1.01
0.95–1.95 (1.83)
0.52–1.10
0.98–1.09 (1.03)
Comp., holding only
0.41–0.82
1.07–1.98 (1.29)
0.48–0.85
1.35–2.95 (1.72)
0.51–1.09
1.01–2.63 (1.02)
DREX , α=0
0.41–0.82
1.12–4.60 (1.69)
0.48–0.85
1.38–2.72 (1.75)
0.51–1.09
1.00–3.23 (1.04)
Appendix
Table 8: Latency over non-EE where EE pays (median p99 largest gap in parentheses). On 13B, four cells include one 78–95 ms GPU step that every request in flight waits on. On 8B, gaps at N≥384 include one batch-wide pause of 0.26–0.28 s per run that non-EE has too.
Figure 9: Open loop on Llama-2-70B ( α=.4 ): requests/s that meet TTFT ≤ 2 s and TPOT ≤ 50 ms.
Element
13B
70B
8B
Speculation, α=.4
1.08–1.17
0.97–1.17
Speculation, α=.6
1.18–1.36
1.00–1.19
Speculation, α=0
0.989–1.003
0.975–1.000
No host sync (plan-ahead)
1.37–1.43
1.08–1.11
1.34–1.69
non-EE’s own overlap
1.11–1.14
1.05
1.28–1.36
Controller vs. best fixed (k,m)
0.975–1.007
0.993–1.002
0.920–1.077
Appendix
Table 9: Ablations: throughput with the element over without it.
Model
Policy
Measured
Zero-overhead
13B
Greedy
1.11 / 1.57 / 1.71
3.53 / 3.83 / 3.51
13B
Majority
1.07 / 1.53 / 1.68
3.40 / 3.78 / 3.46
70B
Greedy
2.69 / 2.70 / 2.75
7.59 / 7.67 / 7.54
70B
Majority
2.53 / 2.61 / 2.60
6.14 / 6.32 / 6.02
Appendix
Table 10: Grouped policies on greedy PG-19 at ϵ∗ 3%, exit after layer 10, N = 16 / mid / full memory: measured and re-costed zero-overhead speedup over non-EE. Greedy continuations of PG-19 fall into repetition loops (distinct-4 of .04–.05) and LayerSkip’s exit rate climbs toward .98, so these cells are a high- p stress test that we do not count for early exit.
Task (model)
A
B − A
DREX − B
0 in CI
GSM8K (13B)
.264
− .18 to − .05
− .020 to +.012
6/6
BBH (13B)
.468
− .26 to − .04
− .006 to +.018
5/6 †
MBPP (13B)
.281
− .18 to − .08
− .004 to +.008
6/6
HumanEval (13B)
.098
− .06 to +.01
.000
6/6
CNN/DM (13B, pts)
16.4
− 0.4 to +0.8
− 0.4 to +0.6
5/6 †
GSM8K (70B)
.490
− .29 to − .14
.00 to +.04
6/6
Appendix
Table 11: Quality at N=64 , the ranges behind Figure 6 : non-EE model (A), ranges over two depths and three targets of the EE model’s loss (B − A, B = batch-1 EE) and of DREX − B, and its 95% intervals that contain 0 ( † : the other favours DREX ).
System
CNN/DM
13B
70B
8B
DREX , τ=∞
1.27 / 1.30 / 1.27
1.42 / 1.43 / 1.43
0.95 / 0.91 / 0.92
DREX , ϵ∗ 1%
1.70 / 1.91 / 1.86
1.58 / 1.60 / 1.58
0.93 / 0.86 / 1.91
DREX , ϵ∗ 3%
1.53 / 1.78 / 1.78
1.57 / 1.64 / 1.59
1.06 / 1.11 / 1.96
Holding only, ϵ∗ 1%
1.63 / 1.82 / 1.85
1.25 / 1.29 / 1.26
0.91 / 0.92 / 1.71
Holding only, ϵ∗ 3%
1.54 / 1.79 / 1.79
1.50 / 1.50 / 1.50
1.06 / 1.24 / 1.88
Appendix
Table 12: CNN/DM summarization at N = 16 / 64 / full memory: decode speedup over non-EE over each batch’s whole decode phase, and the realized exit rate at full memory ( ϵ∗ 1% / 3%). 13B exits at layer 15, 70B at 20, and 8B at 8. EAGLE at its best k≤4 over its chat target, on 8B with the better of two EAGLE-3 heads. Ratios in the text use unrounded values.
System
13B ( N 104–105)
70B ( N 61)
8B ( N 795–813)
Non-EE, LayerSkip
3.79 / 16.5
1.75 / 19.1
26.6 / 17.6
DREX , τ=∞
4.81 / 16.5
2.50 / 19.1
24.3 / 17.7
DREX , ϵ∗ 1%
7.07 / 17.5
2.75 / 18.5
50.8 / 16.2
Holding only, ϵ∗ 1%
6.99 / 17.5
2.21 / 18.6
45.4 / 16.2
Non-EE, chat target
3.56 / 16.6
1.54 / 19.3
26.9 / 16.4
EAGLE, best k
5.64 / 16.4
2.24 / 19.3
31.5 / 16.4
Appendix
Table 13: Absolute values behind Figure 4 b (CNN/DM, full memory, ϵ∗ 1%, 13B at layer 15): decode throughput (thousand tokens/s) and ROUGE-2 (points).
Model
Target
p
Holding only
DREX
α
13B
ϵ∗ 1%
.28–.29
1.02 / 1.11 / 1.13
1.16 / 1.11 / 1.17
.39 / .39 / .42
13B
ϵ∗ 3%
.49–.51
1.20 / 1.40 / 1.51
1.32 / 1.42 / 1.49
.41 / .40 / .37
13B
p∗ .5
.05–.06
0.97 ‡ / 1.00 / 1.01
1.01 / 0.97 / 0.99
.41 / .39 / .40
70B
ϵ∗ 1%
.31–.34
1.02 / 1.05 / 1.05
1.20 / 1.16 / 1.18
.28 / .31 / .31
70B
ϵ∗ 3%
.51–.56
1.21 / 1.17 / 1.18
1.32 / 1.28 / 1.32
.29 / .28 / .27
70B
p∗ .5
.06–.07
0.99 / 0.99 / 0.99
1.08 / 1.07 / 1.07
.38 / .38 / .38
Appendix
Table 14: LayerSkip on sampled PG-19 at N = 16 / mid / full memory (mid: 64 on 13B, 32 on 70B, 256 on 8B, full: 86, 47, 528), means of three repeats: exit rate p of DREX (holding only), DREX (holding only) and DREX over non-EE, and DREX ’s live acceptance ( ‡ : the full-depth fallback ran, Appendix H ).
Self-speculation, measured
Estimate from W
DREX over self-speculation
Model
N
ctl / k 2 / 3 / 4
α
1−TV
coupled
rejection
measured
rejection estimate
13B
16
0.97 / 1.03 / 0.92 / 0.80
0.43
0.50
1.05
1.09
1.10 / 1.35 / 0.98
1.05 / 1.28 / 0.93
13B
64
0.97 / 1.08 / 0.84 / 0.73
0.43
0.48
1.07
1.11
1.05 / 1.31 / 0.90
1.02 / 1.27 / 0.87
13B
full
0.97 / 0.96 / 0.86 / 0.68
0.42
0.49
0.97
1.00
1.21 / 1.54 / 1.02
1.17 / 1.49 / 0.99
70B
16
1.08 / 1.09 / 0.98 / 0.89
0.41
0.44
1.09
1.12
1.10 / 1.23 / 0.99
1.07 / 1.20 / 0.96
70B
32
1.10 / 1.10 / 1.01 / 0.89
0.42
0.46
1.11
1.14
1.05 / 1.16 / 0.98
1.01 / 1.12 / 0.95
Appendix
Table 15: Self-speculation at τ=∞ in DREX ’s engine on sampled PG-19 ( N = 16, 64 or 32, full memory): speed over non-EE of the controller and fixed windows k=m=2,3,4 , the engine’s coupled acceptance (best configuration), the measured 1−TV , the estimate from W with the coupled acceptance (which reproduces the measured best) and with 1−TV (rejection sampling, a model-based estimate), and DREX over measured self-speculation and over the rejection-sampling estimate at ϵ∗ 1% / 3% / p∗ .5.
DREX at ϵ∗
Model
N
EAGLE
best k
1% / 3%
EAGLE over DREX
13B
16
1.50
3
1.14 / 1.39
1.32 / 1.08
13B
64
1.52
3
1.13 / 1.41
1.34 / 1.08
13B
86
1.60
2
1.18 / 1.49
1.36 / 1.07
70B
16
1.50
3
1.19 / 1.34
1.26 / 1.12
70B
32
1.49
2
1.15 / 1.28
1.30 / 1.17
Appendix
Table 16: EAGLE on sampled PG-19 (13B and 70B): EAGLE-1 heads on the Llama-2 chat targets at their best k , over non-EE on LayerSkip’s checkpoint of the same architecture and lengths, with DREX from the job of that non-EE run (one run, exit after layer 10 on 13B and 20 on 70B, Table 14 gives three-repeat means), and EAGLE over DREX .
System
16
64
128
256
384
528
Sampled
EAGLE-3, best k
1.25
1.17
1.17
1.12
1.07
0.99
(.25, .9) compact
1.18
1.28
1.47
1.64
1.77
1.79
(.25, .9) in place
1.25
1.40
1.53
1.62
1.68
1.74
(.25, .9) per-row skip.
0.92
1.08
1.27
1.45
1.59
1.70
(.25, .9) within-iter.
0.97
1.08
1.20
1.35
1.40
1.43
Appendix
Table 17: Llama-3.1-8B on PG-19 (prompt 201, 2000 tokens), speedup over non-EE at N=16 / 64 / 128 / 256 / 384 / 528 (full memory). Synthetic ramp at α=.4 in both modes of DREX , the calibrated model’s pick underlined, and per-row skipping and within-iteration batching, each over its own job’s non-EE. LayerSkip-8B: exit layer 8, risk targets 3% and 1%. ‡ : the full-depth fallback ran ( Appendix H ). –: not run.
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.
Large Language Models (LLMs) have demonstrated exceptional performance across diverse domains but are increasingly constrained by high inference latency. Early Exit has emerged as a promising solution to accelerate inference by dynamically bypassing redundant layers. However, in decoder-only architectures, the efficiency of Early Exit is severely bottlenecked by the KV Cache Absence problem, where skipped layers fail to provide the necessary historical states for subsequent tokens. Existing solutions, such as recomputation or masking, either introduce significant latency overhead or incur severe precision loss, failing to bridge the gap between theoretical layer reduction and practical wall-clock speedup. In this paper, we propose River-LLM, a training-free framework that enables seamless token-level Early Exit. River-LLM introduces a lightweight KV-Shared Exit River that allows the backbone's missing KV cache to be naturally generated and preserved during the exit process, eliminating the need for costly recovery operations. Furthermore, we utilize state transition similarity within decoder blocks to predict cumulative KV errors and guide precise exit decisions. Extensive experiments on mathematical reasoning and code generation tasks demonstrate that River-LLM achieves 1.53 to 2.16 times of practical speedup while maintaining high generation quality.
Large language model (LLM) inference is often constrained by both computation and memory, especially in offloading-based deployments where model weights are transferred across memory hierarchies during autoregressive decoding. In this setting, reducing the number of executed layers can lower per-token latency while also avoiding costly weight movement. Motivated by this observation, we present FlexEE, an early exiting framework for resource-constrained and offloading-based LLM inference. FlexEE makes early exiting practical for LLM decoding through layer-wise exit supervision for reliable intermediate-layer prediction, self-speculative decoding over a Top-K local vocabulary for low-cost exit decisions, and dynamic hidden state management for KV-cache-correct and memory-aware execution. Across generative and downstream tasks, FlexEE enables efficient early exit with minimal accuracy degradation, delivering up to 1.27×/3.16× and 1.25×/2.83× end-to-end speedups on Llama2-7B and Llama3-8B under 0%/50% weight offloading, respectively.
Qihu Xie, Ziwei Li, Yi Kang
University of Science and Technology of China, Hefei, China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Hefei, China
Autoregressive LLM decoding evaluates every generated token through the full layer stack, even though many tokens become predictable at intermediate depths. Existing lossless depth-adaptive methods exploit this redundancy by choosing a single non-final exit depth and verifying its prediction with the final-depth model. However, our measurements show that this selection-based strategy leaves substantial headroom: choosing an exit too late wastes computation, while choosing one too early triggers fallback and discards dependent drafts. We propose Depth Exploration Decoding (DEX), a lossless decoding algorithm that replaces single-depth selection with parallel exploration over multiple candidate depths. At each commit position, DEX validates candidates against the final-depth reference, commits exactly the final-depth token, and collapses the exploration lattice to retain only reusable branch states. This expand--commit--collapse procedure preserves equivalence to standard autoregressive decoding while reducing the cost of committing each token. Across early-exit-trained and standard LLMs, DEX outperforms representative depth-selection baselines and achieves competitive end-to-end throughput against speculative and distributed decoding methods. Moreover, DEX improves as the explored depths become finer, showing that parallel depth exploration provides a scalable way to exploit the underused depth axis of LLM decoding.
Weisi Yang, Zipeng Sun, Stephen Xia
Northwestern University · MILA – Quebec AI Institute · McGill University