Mixture-of-Experts (MoE) architectures have become essential for scaling large language models, driving the development of specialized device-initiated communication libraries such as DeepEP, Hybrid-EP, and others. These libraries demonstrate the performance benefits of GPU-initiated RDMA for MoE dispatch and combine operations. This paper presents NCCL EP (Expert Parallelism), a ground-up MoE communication library built entirely on NCCL's Device API. NCCL EP provides unified ncclEpDispatch and ncclEpCombine primitives with both C and Python interfaces, supporting Low-Latency (LL) mode for inference decoding and High-Throughput (HT) mode for training and inference prefill. LL targets small batch sizes (1-128 tokens) using direct all-to-all RDMA+NVLink mesh connectivity with double-buffered communication for overlapping dispatch and combine phases. HT targets large batches (4096+ tokens) using hierarchical communication that aggregates tokens within NVLink domains before inter-node RDMA transmission. Both modes leverage Device API for both intra- and inter-node communications, taking advantage of its topology awareness and optimized GPU-initiated implementation. We evaluate NCCL EP on an H100-based cluster across multi-node configurations, demonstrating competitive LL kernel performance and presenting end-to-end results with vLLM integration. By building MoE communication natively within NCCL, NCCL EP provides a supported path for expert parallelism on current and emerging NVIDIA platforms.
Figures & tables
Fig. 1: NCCL EP architecture: low-latency kernels adapted from DeepEP [ 8 ] and high-throughput kernels adapted from Hybrid-EP [ 10 ] use NCCL GIN (GPU-Initiated Networking) for inter-node RDMA, while retaining their native NVLink implementations for intra-node communication.
Library
Unified API
LL Mode
HT Mode
C API
Python
Inter-node
Target Workload
DeepEP [ 8 ]
No (separate)
Yes
Yes
No
Yes
NVSHMEM/IBGDA
Training + Inference
Hybrid-EP [ 10 ]
No (HT only)
No
Yes
Yes
Yes
IBGDA
Training (Megatron)
pplx-kernels [ 9 ]
No (separate)
Yes
Yes
Rust/C++
Yes
RDMA (CX7, EFA)
Inference
UCCL-EP [ 15 ]
No (separate)
Yes
Yes
Yes
Yes
CPU proxy
Multi-vendor portable
NCCLX [ 16 ]
Yes
Yes
Yes
Yes
Yes
CTran (IB/RoCE)
Meta-scale (100k+ GPUs)
MORI [ 17 ]
Modular
Yes
Yes
Yes
Yes
rocSHMEM
AMD GPUs
TABLE I: MoE Communication Libraries: Feature Comparison
Gather expert outputs and reduce to original token order.
TABLE II: NCCL EP C API: Resource management functions create and destroy groups (long-lived) and handles (per-forward-pass). Communication functions perform dispatch and combine operations with optional staged execution for overlap.
Fig. 2: NCCL EP execution flow: Group creation (once), followed by repeated Handle → Dispatch → Expert FFN → Combine cycles. Staged mode splits dispatch and combine into send and receive phases for overlap.
Aspect
Low-Latency
High-Throughput
Target workload
Decode (1–128 tokens)
Train/Prefill (4096+)
Communication
All-to-all mesh
Hierarchical NVLink+RDMA
Output layout
3D expert-major
2D with token counts
Reduction
Per-token at receiver
Hierarchical
TABLE III: Comparison of HT and LL kernel modes
// token data (input or output)
NCCL_EP_TENSOR_TAG_TOKENS
// top-k expert indices
NCCL_EP_TENSOR_TAG_TOPK_IDX
// top-k router weights
NCCL_EP_TENSOR_TAG_TOPK_WEIGHTS
// FP8 quantization scales
TABLE IV: Tensor tags identifying the role of each ncclNDTensor_t in ncclEpTensorCreate , ncclEpDispatch , and ncclEpCombine .
Fig. 3: LL mode: 2D input to 3D expert-major output.
Fig. 4: HT mode: 2D input to 2D concatenated output.
Fig. 5: MoE training flow: handles are created before dispatch and shared between forward and backward passes to maintain routing state.
Operation
Call signature in Python notation
Dispatch
recv_x, recv_i, recv_w, h, ev = buf.dispatch(x, topk_i, topk_w, handle)
TABLE V: Buffer operations: framework-facing API mapping to NCCL EP primitives.
Fig. 6: NCCL EP in Megatron-LM. The Flex dispatcher supports three device-initiated backends; NCCL EP (green) is a drop-in alternative to DeepEP (gray) via compatible APIs at each layer.
Specification
EOS
Number of Nodes
576
GPU Model
H100 80GB HBM3
GPUs per Node
8 (640 GB total)
GPU Memory BW
3.35 TB/s
NVLink
4th Gen, 900 GB/s, 18 links/GPU
CPU Model
Xeon 8480CL
TABLE VI: Hardware specifications of the EOS evaluation cluster.
Fig. 7: LL dispatch throughput: NCCL EP vs DeepEP.
Fig. 8: LL combine throughput: NCCL EP vs DeepEP.
Scale
Metric
NCCL EP
DeepEP
1 node (8 GPUs)
Output throughput (tok/s)
755.1
811.2
Total throughput (tok/s)
1,509.7
1,621.8
TTFT mean (ms)
258.5
255.6
TTFT p99 (ms)
326.5
325.5
ITL mean (ms)
39.7
36.8
ITL p99 (ms)
42.6
40.6
TABLE VII: vLLM 0.10 serving metrics: NCCL EP vs DeepEP v1.2.1 with NVSHMEM 3.2.5/IBGDA (TP=1, EP=DP={8, 16, 32} (equal to total number of GPUs), 1000 requests, max concurrency 32). TTFT = time to first token, ITL = inter-token latency, TPOT = time per output token.
Expert parallelism (EP) enables inference of large Mixture-of-Experts (MoE) models by placing their experts across multiple GPUs, but requires substantial communication between GPUs at every MoE layer. As contemporary MoE models activate more experts per token, this communication accounts for a growing fraction of inference time. The cost becomes particularly pronounced on PCIe-based consumer GPU systems, where all inter-GPU transfers traverse CPU memory. However, existing MoE-specialized EP communication libraries assume that direct GPU-to-GPU access is available, largely overlooking consumer GPUs. Therefore, most LLM frameworks instead rely on NCCL, whose CPU-staged communication incurs redundant PCIe transfers and competes with expert computation for GPU resources, limiting their overlap. We present ThunderEP, a novel communication design for such systems that removes the relay hops of traditional ring algorithm, moves data through DMA engines to avoid compute resource contention, and minimizes synchronization latency by reducing the polling overhead of completion flags in CPU memory. We integrate the proposed design into vLLM and evaluate it on three widely used MoE models. Experiments on two PCIe systems equipped with RTX 4090 and RTX 5090 GPUs show that ThunderEP achieves average speedups of 2.00× and 1.53× over NCCL for dispatch and combine, respectively, and up to 1.66× end-to-end speedup over state-of-the-art MoE inference frameworks.
Jaehwan Lee, Sangmin Lee, Chaewon Kim +2
Dept. of Computer Science and Engineering Seoul National University · Dept. of Computer Science and Engineering Graduate School of Data Science Seoul National University
Mixture-of-experts (MoE) architectures enable trillion-parameter LLMs with sparsely activated experts. Expert parallelism (EP) is a widely adopted MoE training strategy, but it suffers from severe all-to-all communication bottlenecks, which is exaggerated by the limited inter-node network bandwidth as the growing model size requires distributing experts across GPU nodes. Prior work focused on overlapping these all-to-all communications with feed-forward network (FFN) and self-attention computations, which often leaves residual network-bound stalls due to inherent imbalance in attention and FFN layers' computation-communication ratios. We present DisagMoE, a disaggregated MoE training system that jointly optimizes model placement and scheduling for maximal efficiency. DisagMoE separates attention and FFN layers into disjoint GPU groups, introduces a multi-stage pipeline with uni-directional, many-to-many communications, and employs a computation-communication roofline model to balance GPU and network bandwidth allocation among the attention and FFN groups. DisagMoE is implemented on Megatron-LM, and evaluation shows that DisagMoE improves training efficiency across multiple MoE models with up to 1.8x speedup on 16-node 8xH800 clusters.
Zhichen Zeng, Chi-Chih Chang, Jiayi Wang +10
ByteDance Seed · University of Washington · Cornell University
Most recent state-of-the-art (SOTA) large language models (LLMs) use Mixture-of-Experts (MoE) architectures to scale model capacity without proportional per-token compute, enabling higher-quality outputs at manageable serving costs. However, MoE inference at scale is fundamentally bottlenecked by expert load imbalance and inefficient token routing, especially in multi-node deployments where tokens are not guaranteed to be routed to local experts, resulting in significant inter-node all-to-all communication overhead. To systematically characterize these challenges, we profile SOTA open-source MoE models, including Llama 4 Maverick, DeepSeek V3-671B, and Qwen3-230B-A22B, on various datasets and collected over 100k real expert activation traces. Upon studying the expert activation patterns, we uncover various persistent properties across all the frontier MoE models: variable expert load imbalance, domain-specific expert activation where expert popularity shifts across task families (code, math, chat, general), and a strong correlation between prefill and decode expert activations. Motivated by these findings, we propose workload-aware micro-batch grouping and an expert placement strategy to maximize token locality to the destination expert, thereby reducing inter-node communication. Across models and datasets, these optimizations help reduce all2all communication data up to 20, resulting in lower MoE decode latency and better accelerator utilization.
Abhimanyu Bambhaniya, Geonhwa Jeong, Jason Park +6
1Georgia Institute of Technology, USA · 2Meta Platforms, Inc.