Overview

Expert parallelism places mixture-of-experts (MoE) layers across GPUs or nodes, sending each token to the experts selected by the model’s router and combining their outputs. Systems commonly optimize the dispatch-and-combine communication, including all-to-all transfers, while load-balancing methods use expert replication, token rerouting, or dynamic placement to reduce slowdowns from uneven demand. Other approaches plan parallel layouts around hardware topology and heterogeneity, or overlap communication with computation during training and serving. The central tension is that balancing and communication optimizations add their own costs, so their benefits depend on routing patterns, workload phase, and the cluster’s interconnect.

Latest papers

CardsList

25 papers7 papers in the last four weeks, up 133% on the four weeks before. 0.1% of all new papers.

  1. Communicate the Change: Efficient Expert Parallelism via Activation Compression

    Nov 17, 2026Fengyuan Liu, Jay Gala, Siva Reddy +1Communication-Efficient Mixture-of-ExpertsMixture-of-Experts Inference

  2. Supercharging Olmo-core for Efficient and Scalable MoE Training

    Nov 13, 2026Tianhua Tao, Akshita Bhagia, Dirk Groeneveld +8Mixture-of-Experts ModelsCommunication-Efficient Mixture-of-Experts

  3. Fuse-All MoE Inference Kernel with Load Balancing on TPUs

    Nov 13, 2026Rupeng Liu, Bangsheng Tang, Zongwei Zhou +2Mixture-of-Experts InferenceLLM Inference Acceleration

  4. KOSMOS: Self-scheduling Persistent Kernels for Communication-Compute Overlap

    Nov 5, 2026Alexander Magro, Aristotle Martin, Ye Wang +1GPU Kernel OptimizationDistributed Training

  5. AdaFloor: Adaptive Floor Planning and Rank-Time Reuse for Long-Tail MoE RL Rollouts

    Nov 4, 2026Jia Wei, Ziyi Qiu, Yi Xu +1RL Post-TrainingReinforcement Learning

  6. Efficient Expert-Parallel Communication on PCIe-Connected Consumer GPUs

    Sep 30, 2026Jaehwan Lee, Sangmin Lee, Chaewon Kim +2Expert ParallelismMixture-of-Experts Inference

  7. HAPMoE: Heterogeneity-Aware Automatic Parallelism Planning for Mixture-of-Experts Models Training

    Sep 30, 2026Mengyuan Fan, Peizhuang Cong, Zixiao Huang +7Expert ParallelismHeterogeneous Computing

  8. TopoEP: Topology-Aware Load Balancing for Expert-Parallel MoE Training

    Sep 28, 2026Jiacheng Zhu, Xie Zhao, Gongming Zhao +3Expert ParallelismExpert Load Balancing

  9. Tevatron Meets Megatron: Expert-Parallel LLM Reranker Training on an Academic Budget

    Aug 2, 2026Zhichao Xu, Xueguang Ma, Shengyao Zhuang +5Expert ParallelismCross-Encoder Reranking

  10. UBEP: Re-architecting Expert Parallelism Communication Library for Production Superpods

    Jul 7, 2026Yipeng Liu, Chang Liu, Si Shen +16Expert ParallelismHigh-Performance Computing

  11. FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs

    Jun 17, 2026Lorenzo Sani, Zeyu Cao, Meghdad Kurmanji +5Expert ParallelismCommunication-Efficient Distributed Training

  12. Director: Accelerating Distributed MoE Serving via Online Proactive Expert Placement

    Jun 13, 2026Qianli Liu, Kaibin Guo, Zicong Hong +5Expert ParallelismLLM Serving

  13. Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Design

    Jun 9, 2026Wenxin Wang, Yule Hou, Yu Ji +2Expert ParallelismOn-Device Language Model Inference

  14. UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancing

    Jun 2, 2026Xinming Wei, Chao Jin, Tuo Dai +10Expert ParallelismExpert Load Balancing

  15. DisagMoE: Computation-Communication overlapped MoE Training via Disaggregated AF-Pipe Parallelism

    May 10, 2026Zhichen Zeng, Chi-Chih Chang, Jiayi Wang +10Expert ParallelismCommunication-Efficient Distributed Training

  16. ZeRO-Prefill: Zero Redundancy Overheads in MoE Prefill Serving

    May 3, 2026Zhaoyuan Su, Olatunji Ruwase, Karthik Ganesan +5Expert ParallelismLLM Serving

  17. Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Scheduling

    Mar 29, 2026Songchen Ma, Hongyi Li, Weihao Zhang +8Mixture-of-Experts InferenceEfficient Neural Network Inference