EAGLE-Pangu: Accelerator-Safe Tree Speculative Decoding on Ascend NPUs

Chang Han; Yijie Hu; Jingling Liu

cs.LGMar 9, 2026

EAGLE-Pangu: Accelerator-Safe Tree Speculative Decoding on Ascend NPUs

Authors: Chang Han, Yijie Hu, Jingling Liu

Organizations: College of Computer Science, Chongqing University

Abstract

Autoregressive decoding remains a primary bottleneck in large language model (LLM) serving, motivating speculative decoding methods that reduce expensive teacher-model invocations by verifying multiple candidate tokens per step. Tree-structured speculation further increases parallelism, but is often brittle when ported across heterogeneous backends and accelerator stacks, where attention masking, KV-cache layouts, and indexing semantics are not interchangeable. We present EAGLE-Pangu, a reproducible system that ports EAGLE-3-style tree speculative decoding to a Pangu teacher backend on Ascend NPUs. EAGLE-Pangu contributes (i) an explicit branch/commit cache manager built on the Cache API, (ii) accelerator-safe tree tensorization that removes undefined negative indices by construction and validates structural invariants, and (iii) a fused-kernel-compatible teacher verification path with a debuggable eager fallback. On 240 turns from MT-Bench and HumanEval-style prompts, EAGLE-Pangu improves end-to-end decoding throughput by 1.27x on average, up to 2.46x at p99, over teacher-only greedy decoding in the fused-kernel performance path. We also provide a fused-kernel-free reference path with structured traces and invariant checks to support reproducible debugging and ablation across execution modes and tree budgets.

Figures & tables

Panel of Figure 1 — Figure 1: Input/output length distributions in our evaluation set (from structured traces).

Metric	mean	p50	p90	p99
Baseline Tok/s	17.65	17.72	18.69	19.37
EA Tok/s	22.42	21.72	31.65	42.09
Speedup ( $\times$ )	1.27	1.21	1.84	2.46
accept_L ( $L_{k}$ )	3.17	3	6	8

Table 1: Throughput microbenchmark (240 turns, fused attention enabled). Tok/s and speedup are summarized across turns; accept_L summarizes flattened

L_{k}

samples across all EA verification steps.

Panel of Figure 2 — Figure 2: Throughput microbenchmark diagnostics (fused on). Speedup correlates with accepted draft length.

Figure 3: Position-wise acceptance ( accept_pos ) aggregated over the evaluation set. Later draft positions are harder to accept, explaining the long-tail behavior of $L_{k}$ .

Sweep	Setting	EA Tok/s (mean)	Speedup (mean)
Scan $M$ (fixed $D_{\max}=10$ )	$M=16$	27.35	1.48
	$M=32$	19.54	1.06
	$M=64$	21.44	1.17
	$M=128$	22.74	1.24
	$M=256$	22.95	1.25
Scan $D_{\max}$ (fixed $M=64$ )	$D_{\max}=4$	19.58	1.06

Table 2: Budget sweep summary (humaneval-only, max_new_tokens=256 , fused on). We report mean Tok/s and mean speedup.

Panel of Figure 4 — Figure 4: Budget sweeps show non-monotonic throughput behavior and configuration-dependent sweet spots.

Panel of Figure 5 — Figure 5: Overhead breakdown (instrumented; analysis-only). Tree tensorization and mask construction are millisecond-scale, while verification and commit are comparable in magnitude. Prefill exhibits a pronounced long tail.

Window $W$	EA Tok/s (mean)	Speedup (mean)	accept_L mean	accept_L p90
none	19.93	1.15	3.17	6
128	11.97	0.69	1.48	3
256	13.22	0.76	1.68	4
512	15.14	0.87	1.99	4

Table 3: Drafter-only fixed-window truncation ( max_new_tokens=1024 , fused on). Truncation reduces acceptance and can negate throughput gains.

Figure 6: Throughput speedup under drafter-only fixed-window truncation. Smaller windows reduce $L_{k}$ and degrade end-to-end speed.

Figure 7: Draft attention evidence (instrumented; analysis-only): the top-1 attention location frequently lies in far history ( 256_plus bucket), consistent with truncation harming draft quality and acceptance.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 291 papers

SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding
Mar 19, 2026Shenggui Li, Chao Wang, Yikai Zhu +14Speculative Decoding LLM Inference Acceleration
Nanyang Technological University · SpecForge Team · Meituan +2
Accelerating PayPal's Commerce Agent with Speculative Decoding: An Empirical Study on EAGLE3 with Fine-Tuned Nemotron Models
Mar 27, 2026Ally Qin, Jian Wan, Sarat Mudunuri +1LLM Inference LLM Inference Acceleration
PARD-2: Target-Aligned Parallel Draft Model for Dual-Mode Speculative Decoding
May 9, 2026Zihao An, Taichi Liu, Ziqiong Liu +3Speculative Decoding LLM Inference Acceleration
Advanced Micro Devices, Inc. · Rutgers University · Michigan State University

Mar 19, 2026J/K move · Enter open · S save

SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding

Shenggui Li, Chao Wang, Yikai Zhu, Yubo Wang+13

Nanyang Technological University · SpecForge Team · Meituan+2

Speculative Decoding LLM Inference Acceleration Efficient Language Model Training

Open paper AskPDF arXiv

Large language models incur high inference latency due to sequential autoregressive decoding. Speculative decoding alleviates this bottleneck by using a lightweight draft model to propose multiple tokens for batched verification. However, its adoption has been limited by the lack of high-quality draft models and scalable training infrastructure. We introduce SpecForge, an open-source, production-oriented framework for training speculative decoding models with full support for EAGLE-3. SpecForge incorporates target-draft decoupling, hybrid parallelism, optimized training kernels, and integration with production-grade inference engines, enabling up to 9.9x faster EAGLE-3 training for Qwen3-235B-A22B. In addition, we release SpecBundle, a suite of production-grade EAGLE-3 draft models trained with SpecForge for mainstream open-source LLMs. Through a systematic study of speculative decoding training recipes, SpecBundle addresses the scarcity of high-quality drafts in the community, and our draft models achieve up to 4.48x end-to-end inference speedup on SGLang, establishing SpecForge as a practical foundation for real-world speculative decoding deployment.

Abstract

Speculative Decoding

Explore similar work

SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding

Accelerating PayPal's Commerce Agent with Speculative Decoding: An Empirical Study on EAGLE3 with Fine-Tuned Nemotron Models

PARD-2: Target-Aligned Parallel Draft Model for Dual-Mode Speculative Decoding