cs.LGNov 4, 2026

Server-Side Sparse Assignment for Federated Mixture-of-Experts

Authors: Xue Zheng, Aylin Yener

Organizations: Department of Electrical and Computer Engineering, The Ohio State University, Columbus, OH, USA

Abstract

Large language models (LLMs) are increasingly recognized for their remarkable capabilities; federated learning (FL) offers a promising framework for collaboratively fine-tuning LLMs on edge devices while achieving data privacy. However, edge devices are inherently heterogeneous: clients may exhibit diverse data distributions and resource constraints, including limited memory and computation budgets. Recent federated Mixture-of-Experts (MoE) approaches address these challenges by using sparse expert activation or partial expert loading. However, existing partial-expert methods rely on heuristic full-pool observations or prioritize system-wide load balance, without guarantees for recovering client-specific relevant experts from partial observations. We propose S3S^3-MoE, a server-side sparse expert assignment framework that estimates client-specific expert relevance from partial observations and refines assignments through an exploit--explore mechanism. We also establish relevance-estimation error and exact Top-KK recovery guarantees. Experiments using a CNN-based MoE on CIFAR-100 show that S3S^3-MoE recovers task-dependent client--expert preferences and reduces conflicts caused by correlated expert selection across clients. Compared with Full-20 FedAvg, which gives each client access to all 20 experts for local Top-5 routing, S3S^3-MoE achieves 1.31 percentage points higher mean final accuracy while loading only 55 experts per client.

Figures & tables

Explore similar work

CardsList
  1. Conflict-Aware Federated Fine-Tuning of Large Language Models with Mixture-of-Experts

    Jun 14, 2026Yijun Lu, Zihan Fang, Pengpeng Qiao +6Federated Learning AggregationLLM Fine-Tuning

  2. Federated Mixture-of-Experts Alignment on Mobile Edge Networks under Data Heterogeneity

    Mar 22, 2026Zihan Fang, Qianru Wang, Haonan An +4Mixture-of-Experts Language ModelsFederated Learning Aggregation

  3. UB-SMoE: Universally Balanced Sparse Mixture-of-Experts for Resource-adaptive Federated Fine-tuning of Foundation Models

    May 15, 2026Van-Tuan Tran, Hong-Hanh Nguyen-Le, Marco Ruffini +1Expert Load BalancingSparse Mixture-of-Experts