Large language models incur high inference latency due to sequential autoregressive decoding. Speculative decoding alleviates this bottleneck by using a lightweight draft model to propose multiple tokens for batched verification. However, its adoption has been limited by the lack of high-quality draft models and scalable training infrastructure. We introduce SpecForge, an open-source, production-oriented framework for training speculative decoding models with full support for EAGLE-3. SpecForge incorporates target-draft decoupling, hybrid parallelism, optimized training kernels, and integration with production-grade inference engines, enabling up to 9.9x faster EAGLE-3 training for Qwen3-235B-A22B. In addition, we release SpecBundle, a suite of production-grade EAGLE-3 draft models trained with SpecForge for mainstream open-source LLMs. Through a systematic study of speculative decoding training recipes, SpecBundle addresses the scarcity of high-quality drafts in the community, and our draft models achieve up to 4.48x end-to-end inference speedup on SGLang, establishing SpecForge as a practical foundation for real-world speculative decoding deployment.
Figures & tables
Figure 1: EAGLE3 attention mask used in Training-Time Testing.
Figure 2: Architecture comparisons
Target Model
Framework
Target Model
Draft Model
Max Batch Size
Seq Length
Step Time (s)
Throughput (tokens/s)
speedup
Llama3.1-8B
EAGLE
ZeRO 2
ZeRO 2
16
4096
1.04
63015.4
1
SpecForge
TP=1
ZeRO 2
64
2.07
126639.6
2.01
Llama3.3-70B
EAGLE
ZeRO 2
ZeRO 2
16
4096
OOM
-
-
EAGLE
ZeRO 3
ZeRO 3
8
2.21
14827.1
1
SpecForge
TP=4
ZeRO 2
16
3.18
20608.8
1.39
Qwen3-30B-A3B
EAGLE
ZeRO 2
ZeRO 2
8
4096
1.12
29257.1
1
Table 1: End-to-end performance on various models.
Figure 3: Training time with different execution backends
Figure 4: Comparison of execution time and memory usage between naive EAGLE3 attention and our optimized kernel.
Target Model
Draft Model
#GPUs
MTBench
GPQA
FinanceQA
Throughput
Speedup
Throughput
Speedup
Throughput
Speedup
Llama-3.1-8B
-
1
190.0
1
190.5
1
185.7
1
Existing
454.7
2.39
438.1
2.30
237.2
1.27
SpecBundle
450.0
2.37
514.2
2.70
258.6
1.39
Llama-3.3-70B
-
4
540.5
1
575.7
1
512.6
1
Existing
1272.7
2.35
1049.0
1.82
981.7
1.92
Table 2: Performance of various models on general benchmarks
Target Model
Draft Model
#GPUs
LiveCodeBench
HumanEval
GSM8K
Math500
Throughput
Speedup
Throughput
Speedup
Throughput
Speedup
Throughput
Speedup
Llama-3.1-8B
-
1
189.7
1
190.9
1
181.8
1
191.0
1
Existing
398.4
2.10
480.3
2.52
228.6
1.26
422.4
2.21
SpecBundle
516.9
2.72
571.5
2.99
329.7
1.81
638.0
3.34
Llama-3.3-70B
-
4
560.9
1
561.0
1
453.2
1
567.4
1
Existing
1303.4
2.32
1282.8
2.29
521.5
1.15
1122.2
1.98
Table 3: Performance of various models on math and coding benchmarks
Figure 5: Inference performance of Llama3.1-8B with EAGLE3 trained on datasets with and without regenerating the responses. The experiment was conducted on 1 H200 GPU with batch size 8.
Figure 6: Scaling TTT for the Llama3.1-8B model on the perfect-blend dataset.
Configuration
#Experts
MTBench
GPQA
FinanceQA
Math500
GSM8K
HumanEval
LiveCodeBench
Same Params
2
1.16
1.05
1.07
1.17
1.05
1.2
1.14
Same FLOPS
2
1.61
1.56
1.34
1.67
1.59
1.64
1.46
3
1.62
1.69
1.35
1.77
1.7
1.67
1.57
4
1.63
1.7
1.36
1.79
1.72
1.69
1.59
With Shared Experts
2
1.52
1.69
1.5
1.75
1.61
1.53
1.52
3
1.52
1.7
1.46
1.75
1.61
1.53
1.52
Table 4: Acceptance rate of MoE models with different settings. The results are obtained with configurations MoE top-k = 1, EAGLE3 number of steps = 3, EAGLE3 top-k = 1 and EAGLE3 number of draft tokens = 4.