Federated Learning (FL) of Large Language Models (LLMs) in multilingual environments presents significant challenges stemming from heterogeneous language distributions across clients and disparities in language resource availability. To address these challenges, we extended the FederatedScope-LLM framework to support multilingual instruction-tuning experiments with LLMs. We also introduced a novel client-specific early stopping mechanism, Local Dynamic Early Stopping (LDES-FL), which allows clients to pause and resume local training based on client-side validation performance, enhancing training efficiency and sustainability. Through a series of experiments, we studied how client language composition - from fully monolingual to increasingly multilingual clients - affects multilingual quality, fairness and training cost. Monolingual local fine-tuning remains the most effective for single-language specialization, whereas federated training is better suited to learning a single balanced multilingual model. In FL, increasing within-client multilinguality leads to stronger and fairer global models, narrows the gap to centralized multilingual fine-tuning, and yields the largest gains for lower-resource languages, albeit at the cost of more optimization steps. Overall, our results identify client language composition as a key design variable in multilingual FL, shaping performance, fairness and efficiency.
Figures & tables
Figure 1: Illustration of a multilingual FL setup. Each client primarily contains data from a single dominant language (represented by the tallest bar) along with smaller, equal portions of data from the remaining languages. Each color corresponds to a different language. As in our experiments, there are eight clients in total.
Figure 2: Training evolution of clients using LDES-FL with FedAvg, where each client holds data in a different language (100% mono).
Figure 3: Validation loss across clients under standard FedAvg (100% mono). Low-resource languages show higher validation loss, whereas high-resource languages achieve lower loss. All clients improve at a similar rate during federated training. Shaded regions mark rounds where a client is stopped.
Client Composition
Heterogeneity Level
100% mono - 0% multi
⚫ ⚫ ⚫ ⚫ ⚫ ⚫
85% mono - 15% multi
⚫ ⚫ ⚫ ⚫ ⚫ ❍
70% mono - 30% multi
⚫ ⚫ ⚫ ⚫ ❍ ❍
50% mono - 50% multi
⚫ ⚫ ⚫ ❍ ❍ ❍
30% mono - 70% multi
⚫ ⚫ ❍ ❍ ❍ ❍
15% mono - 85% multi
⚫ ❍ ❍ ❍ ❍ ❍
Table 1: Client language compositions used to control cross-client language heterogeneity in our FL experiments. More filled circles indicate higher heterogeneity and stronger cross-client non-IIDness (i.e., larger monolingual share), whereas fewer filled circles indicate a more IID setting in which clients have more similar language distributions.
Mean ↑
σ↓
Optim. [-0.25ex] steps
R
F b
R
F b
100% mono
Standard
0.202
0.877
6.75e-2
1.29e-2
1.44e+04
LDES
0.203
0.877
6.47e-2
1.26e-2
1.12e+04
15% mono
Standard
0.224
0.880
6.09e-2
1.22e-2
1.11e+05
LDES
0.221
0.880
6.09e-2
1.23e-2
7.57e+04
Table 2: Performance and optimization steps obtained with two early stopping methods (standard federated early stopping and LDES-FL) under two client language compositions. Federated fine-tuned models are evaluated on the multilingual test set described in Section 5.1 . R and F b denote ROUGE-L and F bert , respectively.
EN
ES
DE
CA
DA
SR
HR
EU
Mean ↑
σ↓
R
F b
R
F b
R
F b
R
F b
R
F b
R
F b
R
F b
R
F b
R
F b
R
F b
Base Model
0.307
0.893
0.215
0.877
0.170
0.873
0.207
0.871
0.160
0.869
0.074
0.861
0.126
0.856
0.080
0.840
0.167
0.867
7.70e-2
1.57e-2
Local FT (EN)
0.351
0.903
0.232
0.882
0.187
0.876
0.233
0.878
0.186
0.875
0.131
0.869
0.147
0.863
0.105
0.852
0.197
0.875
7.73e-2
1.49e-2
Local FT (ES)
0.295
0.897
0.248
0.883
0.188
0.877
0.196
0.876
0.187
0.875
0.168
0.868
0.159
0.866
0.102
0.850
0.193
0.874
5.80e-2
1.47e-2
Local FT (DE)
0.322
0.898
0.237
0.881
0.212
0.883
0.233
0.880
0.189
0.877
0.181
0.871
0.165
0.870
0.105
0.851
0.206
0.876
6.32e-2
1.34e-2
Local FT (CA)
0.296
0.893
0.131
0.866
0.168
0.871
0.253
0.884
0.146
0.865
0.145
0.860
0.104
0.849
0.065
0.834
0.163
0.865
7.61e-2
1.86e-2
Table 3: Results for the multilingual test set of the server. R and F b stand for ROUGE-L Lin (2004) and F bert Zhang et al. (2020) respectively. The language in columns corresponds to the monolingual part of the test set. Mean reports the average score across all languages, whereas σ denotes the corresponding standard deviation, used here as an indicator of multilingual fairness. A bootstrap test with 100 sets (p-value < 0.05) confirms that all federated models achieve statistically significant improvements over the Base Model on all evaluated metrics and languages, as indicated by an asterisk (*).
Training Setting
Norm. Optim. Steps
Local FT (EN)
0.106
Local FT (ES)
0.312
Local FT (DE)
0.345
Local FT (CA)
0.362
Local FT (DA)
0.351
Local FT (SR)
0.429
Table 4: Training cost normalized by the highest-step configuration, Local FT (multilingual) , which requires 1.83×105 total optimization steps. Each value is reported as a fraction of this reference.
ROUGE-L
F bert
Base
Avg. Local
Local FT
FedAvg
FedAvg
Δ
Base
Avg. Local
Local FT
FedAvg
FedAvg
Δ
Model
FT (mono)
(multi)
100% mono
15% mono
Model
FT (mono)
(multi)
100% mono
15% mono
H
0.231
0.232
0.278
0.251
0.263
+0.012
0.881
0.881
0.892
0.886
0.888
+0.002
M
0.184
0.198
0.246
0.213
0.229
+0.016
0.870
0.874
0.882
0.879
0.883
+0.004
L
0.093
0.128
0.189
0.149
0.173
+0.024
0.852
0.859
0.868
0.865
0.869
+0.004
Table 5: Average performance by resource group. H, M and L stand for high-resource, mid-resource and low-resource respectively. Avg. Local FT (mono) is the average of the eight monolingual Local FT models within each resource group. Δ denotes the absolute improvement from 100% mono to 15% mono.
Figure 4: Training evolution of clients using LDES-FL with FedAvg in the 50% mono setting. Note that clients are not labeled by language as in Figure 2 , as here each client contains a mix of languages.