ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarios

Xinyi Hu; Yuhao Shen; Baolin Zhang; Hengxin Zhang; Jun Dai; Shuang Ge; Lei Chen; Yue Li

cs.DCMar 10, 2026

ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarios

Authors: Xinyi Hu, Yuhao Shen, Baolin Zhang, Hengxin Zhang, Jun Dai, Shuang Ge, Lei Chen, Yue Li,

Organizations: Qwen Applications Business Group of Alibaba · Zhejiang University

Abstract

Speculative Decoding promises to accelerate the inference of Large Language Models, yet its efficacy often degrades in production-grade serving. Existing evaluations typically overlook the compute-bound nature of high-concurrency regimes, where verification compute becomes the dominant bottleneck. Consequently, prior methods face a dilemma: static trees incur massive verification waste, while dynamic trees suffer from cumulative misjudgments and kernel incompatibility. To bridge this gap, we introduce ECHO, a high concurrency-oriented framework integrated into SGLang that reformulates speculative execution as a budgeted scheduling problem. Crucially, ECHO employs sparse confidence gating to manage the batch as a unified super-tree, elastically pivoting budget between depth and width to co-optimize the trade-off between reducing global verification steps and maximizing per-step efficiency. Extensive evaluations across diverse model scales-particularly the industrial-grade Qwen3-235B-demonstrate that ECHO consistently outperforms SOTA methods in both low-load and high-load scenarios, achieving up to 5.35x walltime speedup and delivering over 20% relative speedup gain.

Figures & tables

Figure 1: Performance degradation in high-concurrency scenarios on MT-Bench. Left: Latency breakdown for LLaMA-3.3-70B. Verification cost (

T_{t}

) scales linearly, becoming the dominant bottleneck over AR cost (

T_{AR}

) as batch size increases. Right: Throughput for Qwen3-235B. Due to this bottleneck, the speedup of EAGLE-3 diminishes and eventually underperforms vanilla AR at high concurrency (BS=128).

Figure 2: Visualization of confidence distributions across draft depths (LLaMA3.1-8B on MT-Bench). The two views show the probability of accepted and rejected draft tokens at different tree depths. Low overlap indicates better discriminability of the confidence scores, so these layers are identified as “Sweet Spots”.

Figure 3: Overview of the ECHO Framework. (1) Super-Tree Construction: Draft trees are evaluated (truncated or extended) only at sparse gates. (2) Unified Elastic Budget Scheduler: Under a global verification cap ( $K_{\max}$ ), the scheduler dynamically adapts resource allocation. In Low-Load scenarios, the budget saved by truncation is reused locally to widen the current tree. In High-Load scenarios, budget saved from truncated low-confidence requests is reallocated to extend the depth of high-confidence requests. (3) Flatten & Pack: Finally, the ragged batch formed by requests with varying token counts is packed into a dense, kernel-compatible layout for efficient verification.

Models

Methods

HumanEval

GSM8K

CNN/DM

Alpaca

MT-Bench

Avg.

MAT

Speedup

MAT

Speedup

MAT

Speedup

MAT

Speedup

MAT

Speedup

Vicuna-13B

Lookahead

1.73

1.69

\times

1.88

1.79

\times

1.48

1.44

\times

1.49

1.44

\times

1.67

1.63

\times

1.60

\times

Sps

2.55

1.81

\times

1.99

1.75

\times

2.31

1.71

\times

2.01

1.74

\times

2.25

1.81

\times

1.76

\times

Medusa

2.78

2.25

\times

2.63

2.12

\times

2.09

1.65

\times

2.44

1.96

\times

2.58

2.08

\times

2.01

\times

Hydra

3.87

2.75

\times

3.66

2.60

\times

2.82

1.95

\times

3.51

2.48

\times

3.64

2.53

\times

2.46

\times

OPT-Tree

8.21

3.85

\times

6.77

3.25

\times

6.91

2.75

\times

6.56

3.20

\times

6.95

3.30

\times

3.27

\times

Table 1: Main results on five benchmarks under the low-load setting (BS = 1) . Performance comparison between ECHO and existing baselines across diverse model configurations. Bold numbers denote the best speedup.

Figure 4: Request-level Draft Utilization (

u

) across datasets and models. We evaluate performance on LLaMA3.3-70B (a) and Qwen3-235B (b) across HumanEval, GSM8K, and MT-Bench. Each box shows the distribution of per-request Draft Utilization, spanning the 25th–75th percentiles, with whiskers covering the 5th–95th percentiles. ECHO consistently surpasses EAGLE-3 and DDD in average utilization.

Figure 5: Main results on High-Load Case (

\text{BS}>1

). We evaluate ECHO against EAGLE3 and two ECHO variants on three benchmarks using four model configurations. The maximum improvement percentage below each column is against EAGLE3.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Models	Layers	dim	FFN dim	Vocabulary size
Vicuna 68M	2	768	3072	32000
Vicuna 13B	40	5120	13824	32000
LLaMA-3.1 8B	32	4096	14336	128256
LLaMA-3.3 70B	80	8192	28672	128256
Qwen-3 8B	36	4096	12288	151936
Qwen-3 32B	64	5120	25600	151936

Appendix

Table 2: Model configurations.

Figure 6: Visualization of confidence distributions across draft depths (LLaMA3.1-8B on MT-Bench).

Figure 7: Visualization of confidence distributions across draft depths (Qwen3-8B on MT-Bench).

Figure 8: Visualization of confidence distributions across draft depths (Qwen3-32B on MT-Bench).

Figure 9: Visualization of confidence distributions across draft depths (LLaMA3.3-70B on MT-Bench).

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 309 papers

ECHO: Early-layer Collaborative Hierarchical Orchestration with Bonus Logits in Speculative Decoding
Sep 15, 2026Ziyang Ma, Zihong Zhang, Zuchao Li +4Speculative Decoding LLM Inference Acceleration
School of Computer Science, Wuhan University, Wuhan, China · School of Artificial Intelligence, Wuhan University, Wuhan, China · 3Xiaomi, Beijing, China
SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding
Mar 19, 2026Shenggui Li, Chao Wang, Yikai Zhu +14Speculative Decoding LLM Inference Acceleration
Nanyang Technological University · SpecForge Team · Meituan +2
DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification
Apr 8, 2026Ziyi Wang, Siva Rajesh Kasa, Ankith M S +6Speculative Decoding LLM Inference Acceleration
Purdue University · Amazon Inc. · University of Illinois Urbana-Champaign +1

Sep 15, 2026J/K move · Enter open · S save

ECHO: Early-layer Collaborative Hierarchical Orchestration with Bonus Logits in Speculative Decoding

Ziyang Ma, Zihong Zhang, Zuchao Li, Lefei Zhang+3

School of Computer Science, Wuhan University, Wuhan, China · School of Artificial Intelligence, Wuhan University, Wuhan, China · 3Xiaomi, Beijing, China

Speculative Decoding LLM Inference Acceleration

Open paper AskPDF arXiv

While draft-model-free speculative decoding offers a promising path to efficient LLM inference, it is frequently constrained by stale draft candidates and the high computational cost of the verification. To address these challenges, we propose ECHO, a hierarchical dual-loop framework that exploits the functional asymmetry between LLM layers. Leveraging the high discriminative efficiency of early layers and the authoritative distribution of final layers, ECHO bifurcates inference into a high-frequency inner loop and a low-frequency outer loop. Within the inner loop, early-layer bonus logits drive rapid, multi-step draft-tree exploration at a minimal cost. Simultaneously, the outer loop performs authoritative full-model verification through a state-reuse mechanism. Crucially, the outer loop also utilizes final-layer bonus logits to correct existing paths and supplement the tree with high-confidence candidates for subsequent cycles. Experimental results across diverse benchmarks demonstrate that ECHO significantly boosts mean accepted tokens and achieves a 2.4

\times

to 2.9

\times

speedup, outperforming existing state-of-the-art baselines with negligible engineering overhead and no extra deployment parameters, albeit with a one-shot fine-tuning dependency for optimal acceleration. The code is available at https://github.com/whucs21Mzy/ECHO.

Abstract

Speculative Decoding

Explore similar work

ECHO: Early-layer Collaborative Hierarchical Orchestration with Bonus Logits in Speculative Decoding

SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding

DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification