Mixture-of-Experts (MoE) has become a widely adopted architecture for Large Language Models (LLMs), as it improves model capacity while limiting computational overhead through sparse expert activation. This property makes MoE-based LLMs particularly attractive for resource-constrained distributed environments. However, federated fine-tuning of MoE-based LLMs remains challenging under heterogeneous client data. Since clients often correspond to different task preferences, directly aggregating their local updates may weaken expert specialization and introduce conflicting update directions on shared experts. To address these challenges, we propose FedTAR, a task-aware federated fine-tuning method for MoE-based LLMs. FedTAR establishes the association between local updates and task preference via routing outputs. Specifically, we apply Singular Value Decomposition (SVD) to both routing features and local updates to extract low-dimensional task coordinates and update directions. Based on the task coordinates, FedTAR performs intra-cluster aggregation among clients with similar task preferences and inter-cluster aggregation across different task groups. The aggregated update is then reconstructed through the learned task-to-update mapping, ensuring that the final update remains aligned with task-specific optimization directions. In this way, FedTAR preserves expert specialization and mitigates destructive interference among heterogeneous clients. We evaluate FedTAR on four benchmark tasks under different non-IID settings. Experimental results demonstrate that FedTAR consistently outperforms strong federated fine-tuning baselines and achieves state-of-the-art performance.
Figures & tables
Fig. 1 : Illustration of challenges in Federated MoE-based LLMs Fine-Tuning.
Fig. 2 : Overview of FedTAR. Client Side: Each client performs local LoRA fine-tuning and extracts routing features along with local updates. Server Side: The server clusters clients based on the task coordinates, aggregates updates within clusters, and reconstructs a global update via task-aligned subspace projection.
Fig. 3 : Task Structure vs. Update Directions (dolly)
Fig. 4 : KL-divergence heatmap
Method
GSM8K (EM)
MMLU (ACC)
BBH (EM)
AVG.
α
0.1
0.5
0.9
0.1
0.5
0.9
0.1
0.5
0.9
0.1
0.5
0.9
Base Model
61.50%
61.50%
61.50%
62.50%
62.50%
62.50%
36.46%
36.46%
36.46%
53.49%
53.49%
53.49%
Centralized
68.00%
68.00%
68.00%
70.17%
70.17%
70.17%
42.57%
42.57%
42.57%
60.25%
60.25%
60.25%
Local
54.23%
62.01%
64.50%
57.85%
62.10%
62.90%
33.23%
36.60%
37.22%
48.44%
53.57%
54.87%
FedAvg ((AISTATS 2017)) [ 14 ]
52.16%
63.87%
66.59%
56.94%
65.00%
64.30%
30.13%
36.72%
39.87%
46.41%
55.20%
56.92%
FedProx (MLSys 2020) [ 15 ]
55.38%
63.95%
66.31%
58.26%
63.40%
64.00%
31.76%
37.36%
39.38%
48.47%
54.90%
56.56%
TABLE I : Performance comparison on reasoning and knowledge benchmarks under different task heterogeneity levels. All reported results in the following tables are averaged over three random seeds, with standard deviation below 0.5%.
Fig. 5 : Training dynamics under federated settings. We report loss (left) and accuracy (right) across communication rounds on four datasets. Our method achieves faster convergence and consistently better performance.
Method
Dolly-15K
α
0.1
0.5
0.9
Base Model
24.54%
24.54%
24.54%
Centralized
34.17%
34.17%
34.17%
Local
25.89%
27.03%
33.23%
FedAvg
23.56%
25.57%
34.05%
FedProx
24.45%
27.65%
33.17%
TABLE II : Results on dolly (ROUGE-L).
Variant
GSM8K
Dolly
MMLU
BBH
Avg.
Full model
66.34%
32.12%
68.95%
41.76%
52.29%
w/o SVD
64.17%
26.56%
66.24%
37.31%
48.57%
w/o clustering
63.65%
27.01%
67.87%
39.14%
49.42%
w/o reconstruction
64.83%
27.75%
67.43%
39.77%
49.95%
mean-only router
65.26%
31.24%
68.32%
41.34%
51.54%
coact-only router
65.67%
31.58%
68.51%
41.56%
51.83%
TABLE III: Ablation study of the proposed method across four datasets ( α=0.5 ).
Fig. 6 : Impact of number of clients in FedTAR ( α=0.5 ).
Fig. 7 : Impact of number of rank in FedTAR ( α=0.5 ).
Fig. 8 : Impact of number of cluster in FedTAR ( α=0.5 ).