Mixture-of-Experts (MoE) architectures have become essential for scaling large language models, driving the development of specialized device-initiated communication libraries such as DeepEP, Hybrid-EP, and others. These libraries demonstrate the performance benefits of GPU-initiated RDMA for MoE dispatch and combine operations. This paper presents NCCL EP (Expert Parallelism), a ground-up MoE communication library built entirely on NCCL's Device API. NCCL EP provides unified ncclEpDispatch and ncclEpCombine primitives with both C and Python interfaces, supporting Low-Latency (LL) mode for inference decoding and High-Throughput (HT) mode for training and inference prefill. LL targets small batch sizes (1-128 tokens) using direct all-to-all RDMA+NVLink mesh connectivity with double-buffered communication for overlapping dispatch and combine phases. HT targets large batches (4096+ tokens) using hierarchical communication that aggregates tokens within NVLink domains before inter-node RDMA transmission. Both modes leverage Device API for both intra- and inter-node communications, taking advantage of its topology awareness and optimized GPU-initiated implementation. We evaluate NCCL EP on an H100-based cluster across multi-node configurations, demonstrating competitive LL kernel performance and presenting end-to-end results with vLLM integration. By building MoE communication natively within NCCL, NCCL EP provides a supported path for expert parallelism on current and emerging NVIDIA platforms.
Figures & tables
Fig. 1: NCCL EP architecture: low-latency kernels adapted from DeepEP [ 8 ] and high-throughput kernels adapted from Hybrid-EP [ 10 ] use NCCL GIN (GPU-Initiated Networking) for inter-node RDMA, while retaining their native NVLink implementations for intra-node communication.
Library
Unified API
LL Mode
HT Mode
C API
Python
Inter-node
Target Workload
DeepEP [ 8 ]
No (separate)
Yes
Yes
No
Yes
NVSHMEM/IBGDA
Training + Inference
Hybrid-EP [ 10 ]
No (HT only)
No
Yes
Yes
Yes
IBGDA
Training (Megatron)
pplx-kernels [ 9 ]
No (separate)
Yes
Yes
Rust/C++
Yes
RDMA (CX7, EFA)
Inference
UCCL-EP [ 15 ]
No (separate)
Yes
Yes
Yes
Yes
CPU proxy
Multi-vendor portable
NCCLX [ 16 ]
Yes
Yes
Yes
Yes
Yes
CTran (IB/RoCE)
Meta-scale (100k+ GPUs)
MORI [ 17 ]
Modular
Yes
Yes
Yes
Yes
rocSHMEM
AMD GPUs
TABLE I: MoE Communication Libraries: Feature Comparison
Gather expert outputs and reduce to original token order.
TABLE II: NCCL EP C API: Resource management functions create and destroy groups (long-lived) and handles (per-forward-pass). Communication functions perform dispatch and combine operations with optional staged execution for overlap.
Fig. 2: NCCL EP execution flow: Group creation (once), followed by repeated Handle → Dispatch → Expert FFN → Combine cycles. Staged mode splits dispatch and combine into send and receive phases for overlap.
Aspect
Low-Latency
High-Throughput
Target workload
Decode (1–128 tokens)
Train/Prefill (4096+)
Communication
All-to-all mesh
Hierarchical NVLink+RDMA
Output layout
3D expert-major
2D with token counts
Reduction
Per-token at receiver
Hierarchical
TABLE III: Comparison of HT and LL kernel modes
// token data (input or output)
NCCL_EP_TENSOR_TAG_TOKENS
// top-k expert indices
NCCL_EP_TENSOR_TAG_TOPK_IDX
// top-k router weights
NCCL_EP_TENSOR_TAG_TOPK_WEIGHTS
// FP8 quantization scales
TABLE IV: Tensor tags identifying the role of each ncclNDTensor_t in ncclEpTensorCreate , ncclEpDispatch , and ncclEpCombine .
Fig. 3: LL mode: 2D input to 3D expert-major output.
Fig. 4: HT mode: 2D input to 2D concatenated output.
Fig. 5: MoE training flow: handles are created before dispatch and shared between forward and backward passes to maintain routing state.
Operation
Call signature in Python notation
Dispatch
recv_x, recv_i, recv_w, h, ev = buf.dispatch(x, topk_i, topk_w, handle)
TABLE V: Buffer operations: framework-facing API mapping to NCCL EP primitives.
Fig. 6: NCCL EP in Megatron-LM. The Flex dispatcher supports three device-initiated backends; NCCL EP (green) is a drop-in alternative to DeepEP (gray) via compatible APIs at each layer.
Specification
EOS
Number of Nodes
576
GPU Model
H100 80GB HBM3
GPUs per Node
8 (640 GB total)
GPU Memory BW
3.35 TB/s
NVLink
4th Gen, 900 GB/s, 18 links/GPU
CPU Model
Xeon 8480CL
TABLE VI: Hardware specifications of the EOS evaluation cluster.
Fig. 7: LL dispatch throughput: NCCL EP vs DeepEP.
Fig. 8: LL combine throughput: NCCL EP vs DeepEP.
Scale
Metric
NCCL EP
DeepEP
1 node (8 GPUs)
Output throughput (tok/s)
755.1
811.2
Total throughput (tok/s)
1,509.7
1,621.8
TTFT mean (ms)
258.5
255.6
TTFT p99 (ms)
326.5
325.5
ITL mean (ms)
39.7
36.8
ITL p99 (ms)
42.6
40.6
TABLE VII: vLLM 0.10 serving metrics: NCCL EP vs DeepEP v1.2.1 with NVSHMEM 3.2.5/IBGDA (TP=1, EP=DP={8, 16, 32} (equal to total number of GPUs), 1000 requests, max concurrency 32). TTFT = time to first token, ITL = inter-token latency, TPOT = time per output token.
Dept. of Computer Science and Engineering Seoul National University · Dept. of Computer Science and Engineering Graduate School of Data Science Seoul National University