Expert Parallelism

Follow this topic Ask about this topic

Expert parallelism places mixture-of-experts (MoE) layers across GPUs or nodes, sending each token to the experts selected by the model’s router and combining their outputs. Systems commonly optimize the dispatch-and-combine communication, including all-to-all transfers, while load-balancing methods use expert replication, token rerouting, or dynamic placement to reduce slowdowns from uneven demand. Other approaches plan parallel layouts around hardware topology and heterogeneity, or overlap communication with computation during training and serving. The central tension is that balancing and communication optimizations add their own costs, so their benefits depend on routing patterns, workload phase, and the cluster’s interconnect.

Latest papers

17 papers3 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Efficient Expert-Parallel Communication on PCIe-Connected Consumer GPUs
Sep 30, 2026Jaehwan Lee, Sangmin Lee, Chaewon Kim +2Expert Parallelism Mixture-of-Experts Inference
Dept. of Computer Science and Engineering Seoul National University · Dept. of Computer Science and Engineering Graduate School of Data Science Seoul National University
HAPMoE: Heterogeneity-Aware Automatic Parallelism Planning for Mixture-of-Experts Models Training
Sep 30, 2026Mengyuan Fan, Peizhuang Cong, Zixiao Huang +7Expert Parallelism Heterogeneous Computing
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Infinigence AI · Tsinghua University
TopoEP: Topology-Aware Load Balancing for Expert-Parallel MoE Training
Sep 28, 2026Jiacheng Zhu, Xie Zhao, Gongming Zhao +3Expert Parallelism Expert Load Balancing
University of Science and Technology of China
TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimes
Aug 13, 2026Jie Li, Chenxin Jia, Jinliang Shen +5Expert Parallelism Expert Load Balancing
KlingAI Research
The Evolution of Mixture-of-Experts Architectures in Large Language Models: Routing, Topology, Load Balancing, and Expert Parallelism
Aug 9, 2026Jiguo LiExpert Parallelism Expert Load Balancing
This report was completed with the assistance of Codex.
Tevatron Meets Megatron: Expert-Parallel LLM Reranker Training on an Academic Budget
Aug 2, 2026Zhichao Xu, Xueguang Ma, Shengyao Zhuang +5Expert Parallelism Cross-Encoder Reranking
University of Utah · University of Waterloo · The University of Queensland +2
UBEP: Re-architecting Expert Parallelism Communication Library for Production Superpods
Jul 7, 2026Yipeng Liu, Chang Liu, Si Shen +16Expert Parallelism High-Performance Computing
State Key Laboratory for Novel Software Technology, Nanjing University · Huawei Technologies Co., Ltd.
FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs
Jun 17, 2026Lorenzo Sani, Zeyu Cao, Meghdad Kurmanji +5Expert Parallelism Communication-Efficient Distributed Training
Cambridge ML Systems Lab (CaMLSys), University of Cambridge · Flower Labs
Director: Accelerating Distributed MoE Serving via Online Proactive Expert Placement
Jun 13, 2026Qianli Liu, Kaibin Guo, Zicong Hong +5Expert Parallelism LLM Serving
Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong · School of Software Engineering, Sun Yat-Sen University, China · School of Cyber Science and Engineering, Xi’an Jiaotong University, China +1
Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Design
Jun 9, 2026Wenxin Wang, Yule Hou, Yu Ji +2Expert Parallelism On-Device Language Model Inference
1Tsinghua University · 2Xingyun Integrated Circuits Co., Ltd. · 3Beijing National Research Center for Information Science and Technology
UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancing
Jun 2, 2026Xinming Wei, Chao Jin, Tuo Dai +10Expert Parallelism Expert Load Balancing
School of Computer Science, Peking University · Xiaohongshu Inc. · Shanghai AI Laboratory +1
Diagnosing Overhead in Dispatch Operations: Cross-architecture Observatory
May 20, 2026Bole Ma, Jan Eitzinger, Harald Koestler +1Expert Parallelism Expert Load Balancing
DisagMoE: Computation-Communication overlapped MoE Training via Disaggregated AF-Pipe Parallelism
May 10, 2026Zhichen Zeng, Chi-Chih Chang, Jiayi Wang +10Expert Parallelism Communication-Efficient Distributed Training
ByteDance Seed · University of Washington · Cornell University
Federation of Experts: Communication Efficient Distributed Inference for Large Language Models
May 7, 2026Muhammad Shahir Abdurrahman, Chun Deng, Azalia Mirhoseini +1Expert Parallelism Mixture-of-Experts Inference
Stanford University
ZeRO-Prefill: Zero Redundancy Overheads in MoE Prefill Serving
May 3, 2026Zhaoyuan Su, Olatunji Ruwase, Karthik Ganesan +5Expert Parallelism LLM Serving
University of Virginia · Snowflake AI Research · Harvard University
Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Scheduling
Mar 29, 2026Songchen Ma, Hongyi Li, Weihao Zhang +8Mixture-of-Experts Inference Efficient Neural Network Inference
AI Chip Center for Emerging Smart Systems, Hong Kong SAR, China · The Hong Kong University of Science and Technology, Hong Kong SAR, China · Shanghai UniVista Industrial Software Group Co., Ltd., Shanghai, China
NCCL EP: Towards a Unified Expert Parallel Communication API for NCCL
Mar 13, 2026Amos Goldman, Nimrod Boker, Maayan Sheraizin +15Communication-Efficient Mixture-of-Experts Expert Parallelism
NVIDIA Corporation

Sep 30, 2026J/K move · Enter open · S save

Efficient Expert-Parallel Communication on PCIe-Connected Consumer GPUs

Jaehwan Lee, Sangmin Lee, Chaewon Kim, Junsik Shin+1

Dept. of Computer Science and Engineering Seoul National University · Dept. of Computer Science and Engineering Graduate School of Data Science Seoul National University

Expert Parallelism Mixture-of-Experts Inference LLM Inference Acceleration

Open paper AskPDF arXiv

Expert parallelism (EP) enables inference of large Mixture-of-Experts (MoE) models by placing their experts across multiple GPUs, but requires substantial communication between GPUs at every MoE layer. As contemporary MoE models activate more experts per token, this communication accounts for a growing fraction of inference time. The cost becomes particularly pronounced on PCIe-based consumer GPU systems, where all inter-GPU transfers traverse CPU memory. However, existing MoE-specialized EP communication libraries assume that direct GPU-to-GPU access is available, largely overlooking consumer GPUs. Therefore, most LLM frameworks instead rely on NCCL, whose CPU-staged communication incurs redundant PCIe transfers and competes with expert computation for GPU resources, limiting their overlap. We present ThunderEP, a novel communication design for such systems that removes the relay hops of traditional ring algorithm, moves data through DMA engines to avoid compute resource contention, and minimizes synchronization latency by reducing the polling overhead of completion flags in CPU memory. We integrate the proposed design into vLLM and evaluate it on three widely used MoE models. Experiments on two PCIe systems equipped with RTX 4090 and RTX 5090 GPUs show that ThunderEP achieves average speedups of 2.00

\times

and 1.53

\times

over NCCL for dispatch and combine, respectively, and up to 1.66

\times

end-to-end speedup over state-of-the-art MoE inference frameworks.