MoE--LoRA Methods for Multilingual Adaptation: A Systematic Empirical Study
Organizations: Data Science Group, Heinz Nixdorf Institute, Paderborn University, Germany
Abstract
Routed low-rank adapters offer a parameter-efficient means to improve the performance of large language models (LLMs) across tasks. Hence, it is assumed that they can also be used to improve the support of low-resource languages in LLMs, which currently serve English far better than the thousands of other languages spoken worldwide. However, this assumption has never been verified empirically under a shared multilingual evaluation protocol. We address exactly this gap by comparing eight low-rank adaptation (LoRA) methods on a frozen base model using 200 unique languages in more than 9{,}000 evaluation settings. Our results suggest that a reduced-scale search can identify viable configurations of LoRA methods but does not reliably predict their relative performance, as the top-ranking configuration is never best at scale. LoRA reliably improves language-model fit on translation data, consistently degrades reading comprehension, and benefits low-resource languages most. These findings suggest that LoRA methods should be regarded as targeted tools rather than general-purpose means for improving the multilingual capabilities of LLMs.
Figures & tables
| Method | Bele. | FLO. | SIB | Taxi | Overall |
|---|---|---|---|---|---|
| MOELoRA | 2.04 | 3.32 | 2.81 | 2.56 | 2.68 |
| AdaMoE | 2.62 | 2.80 | 2.63 | 2.83 | 2.72 |
| HydraLoRA | 3.36 | 3.12 | 2.74 | 3.79 | 3.25 |
| MTL-LoRA | 3.78 | 2.97 | 4.18 | 3.75 | 3.67 |
| LoRA | 3.35 | 2.88 | 4.76 | 3.93 | 3.73 |
| HMoRA | 5.85 | 5.91 | 3.88 | 4.14 | 4.94 |
| System | Belebele | FLORES | SIB-200 | Taxi1500 | Overall | ||||
| 96 training languages | |||||||||
| MOELoRA r2/32 k8 | 4.25 | 3.35 | 1.49 | 5.36 | 3.61 | ||||
| LoRA r2 | 4.08 | 4.38 | 4.19 | 2.86 | 3.88 | ||||
| MOELoRA r2/16 k8 | 4.28 | 4.11 | 4.14 | 3.04 | 3.89 | ||||
| MTL-LoRA r8/16 | 5.22 | 3.76 | 5.92 | 3.27 | 4.54 | ||||
| Base model | 2.08 | 6.87 | 4.00 | 5.24 | 4.55 | ||||
| Configuration | 10 | 96 | 200 |
|---|---|---|---|
| MTL-LoRA r8/16 | 1 | 3 | 3 |
| MOELoRA r2/16 k8 | 2 | 2 | 5 |
| HydraLoRA r2 | 3 | 4 | 1 |
| MOELoRA r2/32 k8 | 4 | 1 | 2 |
| AdaMoE r2/8 k6 | 5 | 5 | 4 |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Stage | Languages | Hours/run | Runs | Purpose |
|---|---|---|---|---|
| One-axis rank sweep | 10 | 3.6 | 47 | Effect of rank |
| One-axis expert-count sweep | 10 | 3.6 | 63 | Effect of expert count |
| Mixed follow-up | 10 | 7.2 | 22 | Joint configurations and shortlist |
| Broader validation (96 languages) | 96 | 34.5 | 7 | Test the shortlist at scale |
| Broader validation (200 languages) | 200 | 72 | 7 | Test the shortlist at scale |
| Total | – | – | 146 | 129 unique configurations; 10,842 evaluation runs |
| Method | Rank search | Runs | Expert-count search ( : top- ) | Runs | Panel |
|---|---|---|---|---|---|
| AdaMoE | at , | 7 | : 1; : 1, 2; : 1, 2, 4; : 1, 2, 4, 8; : 1, 2, 4, 8, 16 | 15 | yes |
| AdaMoLE | – | – | , , , | 4 | no |
| HMoRA | at , | 7 | : 1; : 1, 2; : 1, 2, 4; : 1, 2, 4, 8; : 1, 2, 4, 8, 16 | 15 | yes |
| HydraLoRA | 8 | , , , , | 5 | yes | |
| LoRA | 10 | – | – | yes | |
| MOELoRA | at , | 6 | : 1; : 1, 2; : 1, 2, 4; : 1, 2, 4, 8; : 1, 2, 4, 8 | 14 | yes |
| Configuration | Train. | Active | Steps | Tokens | Tok./s | Corpus | GPU | ||||
| (M) | (M) | (k) | (B) | (k) | (%) | (h) | |||||
| 96 training languages | |||||||||||
| MTL-LoRA r8/16 | 16 | dense | 8 | 16 | 188.0 | 186.1 | 20.0 | 0.98 | 8.0 | 63 | 136 |
| MOELoRA r2/16 k8 | 16 | 8 | 32 | 64 | 27.3 | 13.6 | 7.6 | 0.37 | 3.0 | 24 | 136 |
| HydraLoRA r2 | 8 | dense | 2 | 4 | 34.5 | 34.5 | 10.3 | 0.51 | 4.1 | 33 | 136 |
| MOELoRA r2/32 k8 | 32 | 8 | 64 | 128 | 54.5 | 13.6 | 6.7 | 0.33 | 2.7 | 21 | 136 |
| Training set | Labels | Ex. | Top-1 | Top-5 | 50% | 90% | Min |
|---|---|---|---|---|---|---|---|
| (M) | (%) | (%) | (%) | ||||
| 10 (search) | 13 | 0.10 | 35.5 | 85.9 | 2 | 7 | 0.0040 |
| 10 (follow-up) | 13 | 0.40 | 35.5 | 85.9 | 2 | 7 | 0.0040 |
| 96 | 102 | 1.51 | 23.3 | 71.8 | 3 | 16 | 0.0001 |
| 200 | 215 | 3.88 | 18.2 | 57.4 | 5 | 26 | 0.0001 |
| Factor | Selection analysis | Follow-up values |
|---|---|---|
| Rank | 2 (next: 8) | 2, 8 |
| Expert count | 32 (next: 16) | 32, 16 |
| Setting | Belebele | FLORES | SIB-200 | Taxi1500 | Overall |
|---|---|---|---|---|---|
| Rank | |||||
| 2 | 2.76 | 2.00 | 2.89 | 2.31 | 2.49 |
| 4 | 2.85 | 2.37 | 3.41 | 2.62 | 2.81 |
| 8 | 2.41 | 2.36 | 3.32 | 2.79 | 2.72 |
| 16 | 3.01 | 3.85 | 3.63 | 3.46 | 3.49 |
| 32 | 4.94 | 4.83 | 3.60 | 4.60 | 4.49 |
| Benchmark | ||||
|---|---|---|---|---|
| Rank | ||||
| Belebele | 54 | 106.24 | 0.393 | *** |
| FLORES | 108 | 341.43 | 0.632 | *** |
| SIB-200 | 54 | 13.60 | 0.050 | * |
| Taxi1500 | 36 | 74.52 | 0.414 | *** |
| Expert count | ||||
| Method | Rank | Experts | Top- | Belebele | FLORES | SIB-200 | Taxi1500 | Overall |
|---|---|---|---|---|---|---|---|---|
| MTL-LoRA | 8 | 16 | – | 7.11 | 10.28 | 4.33 | 7.42 | 7.28 |
| MOELoRA | 2 | 16 | 8 | 6.44 | 4.94 | 11.61 | 6.92 | 7.48 |
| HydraLoRA | 2 | – | – | 10.83 | 9.89 | 4.11 | 5.17 | 7.50 |
| MOELoRA | 2 | 32 | 8 | 4.17 | 9.22 | 14.50 | 4.83 | 8.18 |
| AdaMoE | 2 | 8 | 6 | 10.83 | 4.33 | 8.94 | 12.83 | 9.24 |
| # | Run ID | Method | Rank | Experts | Top- | Belebele | FLORES | SIB-200 | Taxi1500 | Overall |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | mtllora_r8_nup16 | MTL-LoRA | 8 | 16 | – | 7.11 | 10.28 | 4.33 | 7.42 | 7.28 |
| 2 | moelora_tr32_e16_k8 | MOELoRA | 2 | 16 | 8 | 6.44 | 4.94 | 11.61 | 6.92 | 7.48 |
| 3 | hydralora_r2 | HydraLoRA | 2 | – | – | 10.83 | 9.89 | 4.11 | 5.17 | 7.50 |
| 4 | moelora_tr64_e32_k8 | MOELoRA | 2 | 32 | 8 | 4.17 | 9.22 | 14.50 | 4.83 | 8.18 |
| 5 | mola_tr16_e8_k6 | AdaMoE | 2 | 8 | 6 | 10.83 | 4.33 | 8.94 | 12.83 | 9.24 |
| 6 | moelora_tr64_e32_k4 | MOELoRA | 2 | 32 | 4 | 9.00 | 8.33 | 11.28 | 8.92 | 9.38 |
| Method | Rank | Preserved parent | Best mixed descendant | Belebele | FLORES | SIB-200 | Taxi1500 | Overall |
|---|---|---|---|---|---|---|---|---|
| MOELoRA | 2 | moelora_tr16_e8_k2 | moelora_tr32_e16_k8 | +1.44 | +9.44 | +9.72 | +4.67 | +6.32 |
| MOELoRA | 8 | moelora_tr64_e8_k2 | moelora_tr256_e32_k8 | +11.39 | +2.22 | +5.67 | -10.08 | +2.30 |
| AdaMoE | 2 | mola_tr16_e8_k6 | mola_tr64_e32_k4 | -4.17 | -5.22 | -5.94 | +5.75 | -2.40 |
| AdaMoE | 8 | mola_tr64_e8_k6 | mola_tr256_e32_k4 | -1.78 | +4.44 | +7.50 | +13.58 | +5.94 |
| HydraLoRA | 2 | hydralora_r2 | hydralora_r2_heads32 | +1.22 | -1.61 | -5.39 | -11.08 | -4.22 |
| MTL-LoRA | 8 | mtllora_r8 | mtllora_r8_nup16 | +1.06 | +9.56 | +1.89 | +1.17 | +3.42 |
| Benchmark | Kendall’s | Omnibus | MTL-LoRA –MOELoRA | MTL-LoRA –HydraLoRA | MOELoRA –HydraLoRA | |
|---|---|---|---|---|---|---|
| Belebele | 9 | 0.410 | *** | |||
| FLORES | 18 | 0.511 | *** | * | ** | |
| SIB-200 | 9 | 0.612 | *** | |||
| Taxi1500 | 6 | 0.604 | *** |
| 96 training languages | 200 training languages | |||||
| Configuration | Median diff. | Effect | Median diff. | Effect | ||
| Belebele | ||||||
| MTL-LoRA r8/16 | -0.0250 | -0.87 | *** | -0.0222 | -0.90 | *** |
| MOELoRA r2/16 k8 | -0.0200 | -0.85 | *** | -0.0244 | -0.98 | *** |
| HydraLoRA r2 | -0.0167 | -0.82 | *** | -0.0233 | -0.92 | *** |
| MOELoRA r2/32 k8 | -0.0228 | -0.82 | *** | -0.0178 | -0.86 | *** |
| Benchmark | ||||
|---|---|---|---|---|
| 96 training languages | ||||
| Belebele | 40 | 79.70 | 0.285 | *** |
| FLORES | 100 | 142.71 | 0.204 | *** |
| SIB-200 | 51 | 170.87 | 0.479 | *** |
| Taxi1500 | 58 | 118.12 | 0.291 | *** |
| 200 training languages | ||||
| Configuration | High | Medium | Low |
|---|---|---|---|
| 96 training languages (17/8/55 languages) | |||
| MTL-LoRA r8/16 | 40 | 40 | 70 |
| MOELoRA r2/16 k8 | 49 | 60 | 77 |
| HydraLoRA r2 | 32 | 51 | 62 |
| MOELoRA r2/32 k8 | 58 | 66 | 68 |
| AdaMoE r2/8 k6 | 33 | 40 | 66 |
| Gain on | Base-model score taken from | |||
|---|---|---|---|---|
| Belebele | FLORES | SIB-200 | Taxi1500 | |
| 96 training languages | ||||
| Belebele | – | -0.52*** | -0.53*** | -0.67*** |
| FLORES | -0.58*** | – | -0.48*** | -0.56** |
| SIB-200 | -0.69*** | -0.76*** | – | -0.50** |
| Taxi1500 | +0.01 | -0.03 | +0.10 | – |
| Name in this paper | Released run identifier |
|---|---|
| MTL-LoRA r8/16 | mtllora_lr5e5_r8_nup16 |
| MOELoRA r2/16 k8 | moelora_tr32_e16_k8 |
| HydraLoRA r2 | hydralora_r2 |
| MOELoRA r2/32 k8 | moelora_tr64_e32_k8 |
| AdaMoE r2/8 k6 | mola_tr16_e8_k6 |
| LoRA r2 | lora_r2 |
Low-Rank Adaptation
Low-rank adaptation (LoRA) fine-tunes pretrained models with compact weight updates, reducing trainable parameters and deployment costs.
See all 585 papers