Accelerating PayPal's Commerce Agent with Speculative Decoding: An Empirical Study on EAGLE3 with Fine-Tuned Nemotron Models

Ally Qin; Jian Wan; Sarat Mudunuri; Srinivasan Manoharan

cs.LGMar 27, 2026

Accelerating PayPal's Commerce Agent with Speculative Decoding: An Empirical Study on EAGLE3 with Fine-Tuned Nemotron Models

Authors: Ally Qin, Jian Wan, Sarat Mudunuri, Srinivasan Manoharan

Abstract

We evaluate speculative decoding with EAGLE3 as an inference-time optimization for PayPal's Commerce Agent, powered by a fine-tuned llama3.1-nemotron-nano-8B-v1 model. Building on prior work (NEMO-4-PAYPAL) that reduced latency and cost through domain-specific fine-tuning, we benchmark EAGLE3 via vLLM against NVIDIA NIM on identical 2xH100 hardware across 40 configurations spanning speculative token counts (gamma=3, gamma=5), concurrency levels (1-32), and sampling temperatures (0, 0.5). Key findings: (1) gamma=3 achieves 22-49% throughput improvement and 18-33% latency reduction at zero additional hardware cost; (2) acceptance rates remain stable at approximately 35.5% for gamma=3 across all conditions; (3) gamma=5 yields diminishing returns (approximately 25% acceptance rate); (4) LLM-as-Judge evaluation confirms fully preserved output quality; and (5) speculative decoding on a single H100 matches or exceeds NIM on two H100s, enabling 50% GPU cost reduction.

Figures & tables

Conc.	Normal	Spec-3	$\Delta$	Spec-5	$\Delta$
1	0.45	0.67	+49%	0.60	+34%
4	1.70	2.51	+48%	2.07	+22%
8	3.12	4.02	+29%	3.25	+4%
16	5.68	6.90	+22%	5.90	+4%
32	9.54	11.9	+25%	10.86	+14%

Table 1: Throughput comparison (req/s).

\Delta

is relative to the Normal (NIM) baseline.

Conc.	Normal	Spec-3	$\Delta$	Spec-5	$\Delta$
1	2,222	1,495	$-$ 33%	1,661	$-$ 25%
4	2,345	1,586	$-$ 32%	1,925	$-$ 18%
8	2,529	1,978	$-$ 22%	2,443	$-$ 3%
16	2,787	2,294	$-$ 18%	2,684	$-$ 4%
32	3,323	2,665	$-$ 20%	2,917	$-$ 12%

Table 2: Mean latency comparison (ms). Lower is better.

Conc.	S3 (t=0)	S3 (t=0.5)	S5 (t=0)	S5 (t=0.5)
1	35.4%	36.2%	25.7%	25.8%
4	35.0%	35.8%	25.1%	25.8%
8	35.5%	36.0%	25.6%	25.7%
16	35.5%	36.2%	25.4%	25.5%
32	35.3%	36.0%	25.3%	25.4%

Table 3: Acceptance rates across concurrency and temperature.

C.	Normal	S3 t=0	S3 t=.5	S5 t=0	S5 t=.5
1	55.2%	50.0%	52.0%	50.3%	58.5%
4	81.5%	60.5%	51.8%	58.5%	76.2%
8	78.0%	77.8%	73.1%	73.6%	72.4%
16	74.4%	71.2%	61.5%	69.6%	70.0%
32	67.8%	63.9%	49.9%	68.9%	52.1%

Table 4: GPU utilization comparison.

Config	Accepted	Draft	Rate
spec3-c1-t0	20,914	59,057	35.4%
spec3-c16-t0	150,998	425,509	35.5%
spec3-c32-t0.5	219,920	610,623	36.0%
spec5-c1-t0	22,103	86,160	25.7%
spec5-c16-t0	167,742	661,332	25.4%
spec5-c32-t0.5	350,999	1,380,759	25.4%

Table 5: Raw token counts from Prometheus metrics (50 requests per test).

Config	N Sc.	S Sc.	N Win	S Win	Ties
s3-c1-t0	5.00	5.00	0	0	20
s3-c1-t0.5	5.00	5.00	6	10	4
s3-c32-t0	5.00	5.00	2	0	18
s3-c32-t0.5	5.00	5.00	9	3	8
s5-c1-t0	5.00	5.00	0	0	20
s5-c1-t0.5	5.00	5.00	13	6	1

Table 6: LLM-as-Judge quality evaluation. 20 pairwise comparisons per config with randomized A/B ordering.

Metric	Spec (1 $\times$ H100)	NIM (2 $\times$ H100)	$\Delta$
Throughput	3.19 req/s	3.14 req/s	+6.0%
Mean Latency	2,292 ms	2,529 ms	$-$ 4.4%
P50 Latency	2,279 ms	2,485 ms	$-$ 4.5%
GPU Cost	1 $\times$ H100	2 $\times$ H100	$-$ 50%

Table 7: Hardware efficiency comparison at concurrency = 8.

LLM Inference

Large language model inference studies how models generate outputs efficiently, reliably, and securely across hardware and workloads.

See all 268 papers

SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding
Mar 19, 2026Shenggui Li, Chao Wang, Yikai Zhu +14Speculative Decoding LLM Inference Acceleration
Nanyang Technological University · SpecForge Team · Meituan +2
AgentSpec: Speculative Decoding for Batch Inference of LLM Agents
Aug 25, 2026Xin Wang, Ziming Miao, Yi Zhu +4Speculative Decoding LLM Inference Acceleration
The Ohio State University · Microsoft Research · University of Michigan
EAGLE-Pangu: Accelerator-Safe Tree Speculative Decoding on Ascend NPUs
Mar 9, 2026Chang Han, Yijie Hu, Jingling LiuSpeculative Decoding LLM Inference Acceleration
College of Computer Science, Chongqing University

Mar 19, 2026J/K move · Enter open · S save

SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding

Shenggui Li, Chao Wang, Yikai Zhu, Yubo Wang+13

Nanyang Technological University · SpecForge Team · Meituan+2

Speculative Decoding LLM Inference Acceleration Efficient Language Model Training

Open paper AskPDF arXiv

Large language models incur high inference latency due to sequential autoregressive decoding. Speculative decoding alleviates this bottleneck by using a lightweight draft model to propose multiple tokens for batched verification. However, its adoption has been limited by the lack of high-quality draft models and scalable training infrastructure. We introduce SpecForge, an open-source, production-oriented framework for training speculative decoding models with full support for EAGLE-3. SpecForge incorporates target-draft decoupling, hybrid parallelism, optimized training kernels, and integration with production-grade inference engines, enabling up to 9.9x faster EAGLE-3 training for Qwen3-235B-A22B. In addition, we release SpecBundle, a suite of production-grade EAGLE-3 draft models trained with SpecForge for mainstream open-source LLMs. Through a systematic study of speculative decoding training recipes, SpecBundle addresses the scarcity of high-quality drafts in the community, and our draft models achieve up to 4.48x end-to-end inference speedup on SGLang, establishing SpecForge as a practical foundation for real-world speculative decoding deployment.

Abstract

LLM Inference

Explore similar work

SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding

AgentSpec: Speculative Decoding for Batch Inference of LLM Agents

EAGLE-Pangu: Accelerator-Safe Tree Speculative Decoding on Ascend NPUs