Organizations: Arizona State University · Member, IEEE · California State University, Dominguez Hills · University of California, Irvine · Johns Hopkins University
Federated instruction fine-tuning enables Large Language Models (LLMs) to adapt to decentralized, privacy-sensitive data without requiring data sharing. Recent Mixture-of-Experts (MoE) LLMs are particularly attractive for federated learning because their sparse activation reduces computation and communication while scaling model capacity. However, existing federated MoE methods primarily focus on parameter aggregation and personalization, overlooking the routing behavior of MoE models as a source of information for client collaboration. Under heterogeneous instruction distributions, indiscriminate aggregation can lead to negative transfer, highlighting the need to identify which clients should collaborate during federated optimization. We propose ClientMorpher, a routing-aware, personalized federated instruction fine-tuning framework that leverages routing signatures from pretrained MoE models to organize client collaboration prior to aggregation. We investigate two complementary clustering strategies: ClientMorpher-C, which directly clusters clients using expert activation profiles, and ClientMorpher-E, which first clusters experts based on their cross-client usage signatures and then derives client collaboration groups. We evaluate ClientMorpher for federated instruction fine-tuning on the Databricks Dolly-15K dataset, using pathological and Dirichlet-based heterogeneous client distributions across multiple instruction-following tasks. Experimental results show that routing-aware collaboration consistently improves personalized performance compared to conventional federated averaging and local training, while maintaining the same communication cost. Furthermore, our study shows that client-centric and expert-centric clustering provides an effective and scalable approach for personalized federated instruction fine-tuning of sparse MoE LLMs.
Figures & tables
Fig. 1: Motivation for ClientMorpher. (a) Local training provides personalization but no collaboration. (b) FedAvg aggregates all MoE experts across clients, ignoring expert usage patterns. (c) ClientMorpher clusters clients using sparse expert activation profiles and performs cluster-wise aggregation of shared parameters.
Fig. 2: The end-to-end workflow of the ClientMorpher Framework.
Fig. 3: The depiction of router-guided client clustering for MoE-LLMs.
CC
EC
K-Means input
Client profiles
Expert signatures
Input shape
N×E
E×N
Output
Client labels ai
Expert labels ze
Assignment
Direct ( ai )
argmaxk∑eui,eMe,k
Aggregation
Cluster-wise FedAvg
Cluster-wise FedAvg
TABLE I: Comparison of the two clustering strategies.
Method
CLF
CQA
IE
SUM
Avg.
Comm.%
Local
0.6036
0.4096
0.5087
0.4075
0.4824
0.00
MoE-FedAvg (LoRA)
0.6299
0.4232
0.5085
0.4090
0.4927
0.0878
ClientMorpher-C
0.5937
0.3785
0.5243
0.3960
0.4731
0.0878
ClientMorpher-E
0.6411
0.4254
0.5099
0.4181
0.4986
0.0878
TABLE II: 4-client pathological split. All federated methods exchange identical LoRA parameters (0.0878% of model/round).
α=0.1
α=1.0
Client
Task
Local
MoE-FedAvg
Client Cluster
Expert Cluster
Task
Client Cluster
Expert Cluster
C0
QA
0.4679
0.5175
0.4732
0.4702
CLS
0.4833
0.4748
C1
IE
0.4660
0.5215
0.5085
0.5129
Summ
0.5174
0.4953
C2
Summ
0.3179
0.3552
0.3849
0.3798
Summ
0.4253
0.4269
C3
CLS
0.5383
0.5529
0.5472
0.5327
CLS
0.5648
0.5835
C4
QA
0.4013
0.4294
0.4767
0.4875
IE
0.4781
0.4766
TABLE III: Comparison of Local Training, MoE-FedAvg, Client Clustering, and Expert Clustering under different Dirichlet concentration parameters ( α ). Client IDs remain fixed across experiments, while task assignments may differ under different α values. ROUGE-L is reported for each client (higher is better).
Mixture-of-Experts (MoE) has become a widely adopted architecture for Large Language Models (LLMs), as it improves model capacity while limiting computational overhead through sparse expert activation. This property makes MoE-based LLMs particularly attractive for resource-constrained distributed environments. However, federated fine-tuning of MoE-based LLMs remains challenging under heterogeneous client data. Since clients often correspond to different task preferences, directly aggregating their local updates may weaken expert specialization and introduce conflicting update directions on shared experts. To address these challenges, we propose FedTAR, a task-aware federated fine-tuning method for MoE-based LLMs. FedTAR establishes the association between local updates and task preference via routing outputs. Specifically, we apply Singular Value Decomposition (SVD) to both routing features and local updates to extract low-dimensional task coordinates and update directions. Based on the task coordinates, FedTAR performs intra-cluster aggregation among clients with similar task preferences and inter-cluster aggregation across different task groups. The aggregated update is then reconstructed through the learned task-to-update mapping, ensuring that the final update remains aligned with task-specific optimization directions. In this way, FedTAR preserves expert specialization and mitigates destructive interference among heterogeneous clients. We evaluate FedTAR on four benchmark tasks under different non-IID settings. Experimental results demonstrate that FedTAR consistently outperforms strong federated fine-tuning baselines and achieves state-of-the-art performance.
Tingqi Wang, Hongyu Ke, Haoxin Wang +2
Department of Computer Science, Georgia State University Atlanta, GA, United States
Federated PEFT enables LLMs to collaboratively adapt to decentralized private data without sharing raw examples. However, task heterogeneity across clients can cause cross-task interference and gradient conflicts during aggregation. Federated MoE-LoRA addresses this challenge through specialized LoRA experts and conditional routing. Yet existing methods typically specialize at client granularity, implicitly assuming task-coherent clients. Our core insight is that experts need purity, namely pattern-coherent updates that preserve specialization, whereas routers need contrast, namely mixed-task observations that support expert comparison. We propose FedWeave, a framework that adopts asymmetric aggregation, separating expert aggregation from router optimization to meet these two requirements. FedWeave uses unsupervised prototype discovery to form local buckets and align them across clients, enabling prototype-level expert aggregation while retaining mixed-task client trajectories for router training. At inference, FedWeave performs sparse inference with one active expert while preserving nearly all soft-routing performance. Our theoretical analysis explains why asymmetric aggregation is advantageous: it controls expert convergence in stationarity through off-pattern contamination, identifies the consensus error induced by fragmented router trajectories, and bounds sparse-inference risk. On a heterogeneous multi-task benchmark with mainstream LLM backbones, FedWeave consistently outperforms strong baselines, while ablations verify the effectiveness of our design.
Donghang Duan, Xu Zheng, Lizong Zhang +2
University of Electronic Science and Technology of China · 2Zhejiang University
The continuous scaling of large language models (LLMs) incurs prohibitive computational costs, making Mixture-of-Experts (MoE) a scalable alternative for efficient fine-tuning via sparse activation. While federated learning (FL) emerges as the paradigm for privacy-preserving collaborative optimization, integrating MoE into FL under data heterogeneity may trigger conflicting expert optimizations. Client-specific data distributions force same-indexed experts to optimize under inconsistent or even conflicting feature-label correlations. This mismatch induces destructive interference during aggregation, thus destabilizing the optimization trajectory and degrading model performance. To address this issue, we propose FC-MoE, a federated conflict-aware framework for MoE fine-tuning. It employs an importance aware weighting scheme to prioritize reliable local updates and utilizes gradient consensus projection to suppress conflicting updates, ensuring a stable global optimization path. Moreover, a local knowledge retention mechanism further preserves specialized client expertise by re-anchoring domain-specific residuals. Extensive experiments demonstrate that FC-MoE accelerates convergence and enhances both global and local model performance in non-IID federated environments.
Yijun Lu, Zihan Fang, Pengpeng Qiao +6
Waseda University, Tokyo, Japan · City University of Hong Kong, China · Institute of Science Tokyo, Tokyo, Japan +4