Lossless Inference Acceleration for ASR via EAGLE-Style Speculative Decoding
Authors: Sen Zhang, Xianghu Yue, Yiming Chen, Wenhuan Lu, Jianguo Wei, Haizhou Li
Organizations: College of Intelligence and Computing, Tianjin University, Tianjin 300350, China · ASUS Intelligent Cloud Services, Singapore 486030 · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, Guangdong 518172, China · Department of Electrical and Computer Engineering, National University of Singapore, Singapore 117583
Autoregressive Transformer and Conformer models have achieved state-of-the-art performance in automatic speech recognition (ASR). However, their sequential token generation mechanism imposes a significant bottleneck during decoding, resulting in substantial inference latency. While speculative decoding has emerged as a promising lossless acceleration technique, existing ASR-oriented approaches either suffer from low draft acceptance rates due to suboptimal draft quality or rely on specially designed external draft models, which limit their generality and practicality. To overcome these limitations, we propose EAGLE-ASR, a novel framework that accelerates ASR inference via EAGLE-style speculative decoding while strictly preserving lossless output quality. Specifically, we introduce Speech-Draft, a lightweight autoregressive module that serves as a universal draft model by jointly leveraging encoder acoustic features and hierarchical decoder representations from the target ASR model. Furthermore, we adopt a multi-step generation strategy with self-distilled data to train the Speech-Draft, enabling it to better approximate the target model's output distribution and maintain causal consistency during multi-token generation. Extensive experiments on representative Whisper and Conformer AED models demonstrate that EAGLE-ASR achieves significant lossless speedups of up to 1.9× and 2.4×, respectively, consistently outperforming existing speculative decoding baselines for ASR. Our proposed framework further generalizes effectively to multilingual ASR and speech translation tasks, establishing EAGLE-ASR as a practical and universally applicable solution for accelerating modern ASR systems without compromising recognition accuracy.
Figures & tables
Fig. 1: Pipeline of EAGLE-ASR. Training phase (upper): The Speech-Draft model is trained using a multi-token generation strategy together with a weighted cross-entropy loss with exponential decay. During training, only the FC layers and the Transformer decoder block of the Speech-Draft are fine-tuned, while remaining parameters of the Speech-Draft, as well as all parameters of the target model, are kept frozen. Inference phase (lower): The target model predicts the next token, which feeds into the Speech-Draft serving as the draft model, thereby recursively generating draft tokens for subsequent parallel verification.
Fig. 2: The overall architecture of the Speech-Draft, which consists of two fully connected (FC) layers, an embedding layer, a Transformer decoder block, and an LM head.
Fig. 3: An illustration of self-distilled training data generation, designed to mitigate the distributional discrepancy between the original transcripts and the target model outputs.
Target model
Method
Draft Model
Draft Params
Test Clean
Test Other
Speedup
τ
Δ WER
Speedup
τ
Δ WER
Whisper (1550 M)
SpD [ 12 ]
Distil-large-v3
756 M
1.72 ×
4.14
0.00
1.52 ×
3.76
0.00
SpD [ 12 ]
Whisper-tiny
39 M
1.76 ×
3.04
0.00
1.58 ×
2.56
0.00
Medusa [ 25 ]
Medusa-Linear
68 M
1.67 ×
3.51
+1.88
1.59 ×
3.32
+3.61
Medusa [ 25 ]
Medusa-Block
92 M
1.65 ×
3.16
+1.17
1.54 ×
3.01
+1.50
EAGLE-ASR
Speech-Draft
117 M
1.91 ×
3.79
0.00
1.79 ×
3.55
0.00
TABLE I: Comparison of acceleration performance across different speculative decoding approaches on the LibriSpeech dataset.
Fig. 4: Speedup comparison across different decoding methods. EAGLE-ASR consistently outperforms all baseline methods for both target models while guaranteeing lossless decoding.
Subset
Speedup
τ
Δ WER
VoxPopuli English
1.96 ×
4.06
0.00
VoxPopuli German
2.14 ×
4.50
0.00
VoxPopuli Spanish
2.10 ×
4.46
0.00
VoxPopuli French
2.15 ×
4.64
0.00
VoxPopuli Italian
2.10 ×
4.46
0.00
Average
2.09 ×
4.42
0.00
TABLE II: Acceleration performance of EAGLE-ASR across diverse languages for ASR.
ST task
Speedup
τ
Δ BLEU
German-to-English
1.49 ×
3.44
0.00
Spanish-to-English
1.73 ×
3.75
0.00
French-to-English
1.66 ×
3.60
0.00
Italian-to-English
1.59 ×
3.49
0.00
Average
1.62 ×
3.57
0.00
TABLE III: Acceleration performance of EAGLE-ASR across various speech translation tasks.
Fig. 5: Effect of the draft length K on the acceleration performance of EAGLE-ASR. Left: Speedup ratio. Right: Mean accepted length τ .
Fig. 6: Effect of top-k on the acceleration performance of EAGLE-ASR. Left: Speedup ratio. Right: Mean accepted length τ .
Fig. 7: Effect of top-k on the acceleration performance of EAGLE-ASR on the ST (German-to-English) task. Left: Speedup ratio. Right: Mean accepted length τ .
Method
Speedup
τ
Δ WER
w/o encoder feature
0.89 ×
0.08
0.00
w/o fused decoder features
1.35 ×
1.84
0.00
w/o self-distilled data
1.43 ×
2.02
0.00
w/o multi-step generation
1.17 ×
1.00
0.00
w/o Lmtp
1.66 ×
3.28
0.00
EAGLE-ASR
1.85 ×
3.67
0.00
TABLE IV: Ablation study results of EAGLE-ASR on the Whisper model.
Speculative decoding accelerates autoregressive automatic speech recognition (ASR), but the acceptance length of alignment-aware drafters can saturate as the draft horizon increases. We propose a progress-aware speculative drafter that recurrently propagates an acoustic progress state across draft steps and feeds it back into audio cross-attention to guide token generation. We jointly train the drafter and progress predictor over variable draft horizons. On five ASR test sets, our method achieves lossless, macro-averaged end-to-end speedups of 1.657x and 1.227x over target-only autoregressive decoding with Qwen3-ASR-0.6B and Qwen3-ASR-1.7B, respectively. Relative to AnchorDraft, our method improves the macro-averaged speedup by 34.3% and 9.0%, respectively. Horizon sweeps show continued growth in acceptance length beyond the baselines' saturation. Code is available at https://github.com/yuanyuanjia71-spec/ProgDraft.
Speculative decoding speeds up generation by letting a cheap draft propose several tokens that a target model checks in one pass. In the single-model form, the draft is a lightweight module attached to the target rather than a separate model. Applying this design to Automatic Speech Recognition (ASR) introduces an extra problem. The draft can read the whole audio at every step, yet its proposals get worse as it runs on its own. Access is not localization. The accepted text keeps the transcript position explicit, but the draft must also track the changing audio position. In the primary matched comparison, per-step audio access changes the first proposal modestly but roughly doubles later-proposal acceptance. Fixed-width windows show that the audio position explains part of this gap. A correctly placed window recovers continuation, while an equally narrow window at the wrong position reduces it. Late-draft median error reaches 21 frames in the hardest reported condition, while target attention during verification stays within a 2-frame median. We test two ways to reduce this drift. The first reads the audio position from verification attention and uses it to guide the next draft round. It saves time only when the extra accepted tokens offset the readout cost. The second is AnchorDraft, which teaches the draft to track the audio position during training without changing the inference graph. The trained draft improves end-to-end speed at both tested target scales. These results show that ASR self-speculation depends on token prediction, audio-position tracking, and draft cost.
Large language models incur high inference latency due to sequential autoregressive decoding. Speculative decoding alleviates this bottleneck by using a lightweight draft model to propose multiple tokens for batched verification. However, its adoption has been limited by the lack of high-quality draft models and scalable training infrastructure. We introduce SpecForge, an open-source, production-oriented framework for training speculative decoding models with full support for EAGLE-3. SpecForge incorporates target-draft decoupling, hybrid parallelism, optimized training kernels, and integration with production-grade inference engines, enabling up to 9.9x faster EAGLE-3 training for Qwen3-235B-A22B. In addition, we release SpecBundle, a suite of production-grade EAGLE-3 draft models trained with SpecForge for mainstream open-source LLMs. Through a systematic study of speculative decoding training recipes, SpecBundle addresses the scarcity of high-quality drafts in the community, and our draft models achieve up to 4.48x end-to-end inference speedup on SGLang, establishing SpecForge as a practical foundation for real-world speculative decoding deployment.
Shenggui Li, Chao Wang, Yikai Zhu +14
Nanyang Technological University · SpecForge Team · Meituan +2