Server-Side Sparse Assignment for Federated Mixture-of-Experts
Organizations: Department of Electrical and Computer Engineering, The Ohio State University, Columbus, OH, USA
Abstract
Large language models (LLMs) are increasingly recognized for their remarkable capabilities; federated learning (FL) offers a promising framework for collaboratively fine-tuning LLMs on edge devices while achieving data privacy. However, edge devices are inherently heterogeneous: clients may exhibit diverse data distributions and resource constraints, including limited memory and computation budgets. Recent federated Mixture-of-Experts (MoE) approaches address these challenges by using sparse expert activation or partial expert loading. However, existing partial-expert methods rely on heuristic full-pool observations or prioritize system-wide load balance, without guarantees for recovering client-specific relevant experts from partial observations. We propose -MoE, a server-side sparse expert assignment framework that estimates client-specific expert relevance from partial observations and refines assignments through an exploit--explore mechanism. We also establish relevance-estimation error and exact Top- recovery guarantees. Experiments using a CNN-based MoE on CIFAR-100 show that -MoE recovers task-dependent client--expert preferences and reduces conflicts caused by correlated expert selection across clients. Compared with Full-20 FedAvg, which gives each client access to all 20 experts for local Top-5 routing, -MoE achieves 1.31 percentage points higher mean final accuracy while loading only experts per client.
Figures & tables
| Experts | Method | Router | Synchronized scope | Acc. (%) | Client/round (MiB) | Total (GiB) |
|---|---|---|---|---|---|---|
| 5 | -MoE | Local | 5 experts | 5.99 | 5.85 | |
| -MoE-Greedy | Local | 5 experts | 5.99 | 5.85 | ||
| Random-5 | Local | 5 experts | 5.99 | 5.85 | ||
| FLEX-MoE-ACC (personalized) | Local | 5 experts | 5.99 | 5.85 | ||
| FLEX-MoE-ACC | Local | Backbone + 5 experts | 72.83 | 71.12 | ||
| 20 | S-MoE (personalized) | Local | 20 experts | 23.96 | 23.40 |
Sparse Mixture-of-Experts
Sparse mixture-of-experts models route each input to a small subset of specialized networks to increase capacity while limiting computation.
See all 128 papers