Autoregressive decoding remains a primary bottleneck in large language model (LLM) serving, motivating speculative decoding methods that reduce expensive teacher-model invocations by verifying multiple candidate tokens per step. Tree-structured speculation further increases parallelism, but is often brittle when ported across heterogeneous backends and accelerator stacks, where attention masking, KV-cache layouts, and indexing semantics are not interchangeable. We present EAGLE-Pangu, a reproducible system that ports EAGLE-3-style tree speculative decoding to a Pangu teacher backend on Ascend NPUs. EAGLE-Pangu contributes (i) an explicit branch/commit cache manager built on the Cache API, (ii) accelerator-safe tree tensorization that removes undefined negative indices by construction and validates structural invariants, and (iii) a fused-kernel-compatible teacher verification path with a debuggable eager fallback. On 240 turns from MT-Bench and HumanEval-style prompts, EAGLE-Pangu improves end-to-end decoding throughput by 1.27x on average, up to 2.46x at p99, over teacher-only greedy decoding in the fused-kernel performance path. We also provide a fused-kernel-free reference path with structured traces and invariant checks to support reproducible debugging and ablation across execution modes and tree budgets.
Figures & tables
Figure 1: Input/output length distributions in our evaluation set (from structured traces).
Metric
mean
p50
p90
p99
Baseline Tok/s
17.65
17.72
18.69
19.37
EA Tok/s
22.42
21.72
31.65
42.09
Speedup ( × )
1.27
1.21
1.84
2.46
accept_L ( Lk )
3.17
3
6
8
Table 1: Throughput microbenchmark (240 turns, fused attention enabled). Tok/s and speedup are summarized across turns; accept_L summarizes flattened Lk samples across all EA verification steps.
Figure 3: Position-wise acceptance ( accept_pos ) aggregated over the evaluation set. Later draft positions are harder to accept, explaining the long-tail behavior of Lk .
Sweep
Setting
EA Tok/s (mean)
Speedup (mean)
Scan M (fixed Dmax=10 )
M=16
27.35
1.48
M=32
19.54
1.06
M=64
21.44
1.17
M=128
22.74
1.24
M=256
22.95
1.25
Scan Dmax (fixed M=64 )
Dmax=4
19.58
1.06
Table 2: Budget sweep summary (humaneval-only, max_new_tokens=256 , fused on). We report mean Tok/s and mean speedup.
Figure 4: Budget sweeps show non-monotonic throughput behavior and configuration-dependent sweet spots.
Figure 5: Overhead breakdown (instrumented; analysis-only). Tree tensorization and mask construction are millisecond-scale, while verification and commit are comparable in magnitude. Prefill exhibits a pronounced long tail.
Window W
EA Tok/s (mean)
Speedup (mean)
accept_L mean
accept_L p90
none
19.93
1.15
3.17
6
128
11.97
0.69
1.48
3
256
13.22
0.76
1.68
4
512
15.14
0.87
1.99
4
Table 3: Drafter-only fixed-window truncation ( max_new_tokens=1024 , fused on). Truncation reduces acceptance and can negate throughput gains.
Figure 6: Throughput speedup under drafter-only fixed-window truncation. Smaller windows reduce Lk and degrade end-to-end speed.
Figure 7: Draft attention evidence (instrumented; analysis-only): the top-1 attention location frequently lies in far history ( 256_plus bucket), consistent with truncation harming draft quality and acceptance.
Large language models incur high inference latency due to sequential autoregressive decoding. Speculative decoding alleviates this bottleneck by using a lightweight draft model to propose multiple tokens for batched verification. However, its adoption has been limited by the lack of high-quality draft models and scalable training infrastructure. We introduce SpecForge, an open-source, production-oriented framework for training speculative decoding models with full support for EAGLE-3. SpecForge incorporates target-draft decoupling, hybrid parallelism, optimized training kernels, and integration with production-grade inference engines, enabling up to 9.9x faster EAGLE-3 training for Qwen3-235B-A22B. In addition, we release SpecBundle, a suite of production-grade EAGLE-3 draft models trained with SpecForge for mainstream open-source LLMs. Through a systematic study of speculative decoding training recipes, SpecBundle addresses the scarcity of high-quality drafts in the community, and our draft models achieve up to 4.48x end-to-end inference speedup on SGLang, establishing SpecForge as a practical foundation for real-world speculative decoding deployment.
Shenggui Li, Chao Wang, Yikai Zhu +14
Nanyang Technological University · SpecForge Team · Meituan +2
We evaluate speculative decoding with EAGLE3 as an inference-time optimization for PayPal's Commerce Agent, powered by a fine-tuned llama3.1-nemotron-nano-8B-v1 model. Building on prior work (NEMO-4-PAYPAL) that reduced latency and cost through domain-specific fine-tuning, we benchmark EAGLE3 via vLLM against NVIDIA NIM on identical 2xH100 hardware across 40 configurations spanning speculative token counts (gamma=3, gamma=5), concurrency levels (1-32), and sampling temperatures (0, 0.5). Key findings: (1) gamma=3 achieves 22-49% throughput improvement and 18-33% latency reduction at zero additional hardware cost; (2) acceptance rates remain stable at approximately 35.5% for gamma=3 across all conditions; (3) gamma=5 yields diminishing returns (approximately 25% acceptance rate); (4) LLM-as-Judge evaluation confirms fully preserved output quality; and (5) speculative decoding on a single H100 matches or exceeds NIM on two H100s, enabling 50% GPU cost reduction.
Speculative decoding accelerates Large Language Models (LLMs) inference by using a lightweight draft model to propose candidate tokens that are verified in parallel by the target model. However, existing draft model training objectives are not directly aligned with the inference-time goal of maximizing consecutive token acceptance. To address this issue, we reformulate the draft model optimization objective, shifting the focus from token prediction accuracy to the overall acceptance length. In this paper, we build upon PARD to propose PARD-2, a dual-mode speculative decoding framework with Confidence-Adaptive Token (CAT) optimization. This approach adaptively reweights each token to better align with the verification process. Notably, PARD-2 enables a single draft model to support both target-dependent and target-independent modes. Experiments across diverse models and tasks demonstrate that PARD-2 achieves up to 6.94× lossless acceleration, surpassing EAGLE-3 by 1.9× and PARD by 1.3× on Llama3.1-8B. Our code is available at https://github.com/AMD-AGI/PARD.
Zihao An, Taichi Liu, Ziqiong Liu +3
Advanced Micro Devices, Inc. · Rutgers University · Michigan State University