Expert Parallelism
Overview
Expert parallelism places mixture-of-experts (MoE) layers across GPUs or nodes, sending each token to the experts selected by the model’s router and combining their outputs. Systems commonly optimize the dispatch-and-combine communication, including all-to-all transfers, while load-balancing methods use expert replication, token rerouting, or dynamic placement to reduce slowdowns from uneven demand. Other approaches plan parallel layouts around hardware topology and heterogeneity, or overlap communication with computation during training and serving. The central tension is that balancing and communication optimizations add their own costs, so their benefits depend on routing patterns, workload phase, and the cluster’s interconnect.
Latest papers
25 papers7 papers in the last four weeks, up 133% on the four weeks before. 0.1% of all new papers.