We investigate whether Mixture-of-Experts (MoE) language models develop linguistically structured expert routing during bilingual language acquisition. Inspired by the Declarative-Procedural framework, we analyze lexical, grammatical, and syntactic processing in a decoder-only English-German MoE Transformer trained under sequential language exposure. We construct a probe-based validation set and extract token-level routing distributions to quantify category-dependent specialisation using mutual information, routing entropy, and Jensen-Shannon distance. The curriculum-trained model exhibits a peak mutual information of 0.1148 at layer 5, indicating category-dependent differences in routing distributions across linguistic categories. Surprisingly, a no-curriculum baseline trained on mixed English-German data shows stronger aggregate specialisation, reaching a peak mutual information of 0.2599 at the same layer. These results suggest that interpretable linguistic organization emerges within MoE routing patterns even without sequential language exposure. A replication at a second training seed shows that the no-curriculum condition's specialisation concentrates on a single language whose identity is seed-dependent, whereas the curriculum consistently yields a stable, language-balanced routing profile; rather than uniformly increasing specialisation, staged bilingual exposure reduces single-language dominance. The official Github repository: https://github.com/Amrit828/DP-Theory-MOE-Interpretability-Research
Figures & tables
Language
Lexical
Grammatical
Syntactic
English
460,250
650,000
650,000
German
138,127
175,000
175,000
Table 1: Dataset size by language and linguistic category.
Condition
Perplexity
Accuracy (%)
Forward Curriculum
111.19
28.99
No Curriculum
117.01
29.65
Table 2: Validation performance on the held-out diagnostic probe set.
Figure 1: Layer-wise mutual information between expert-routing distributions and linguistic categories.
Layer
Observed MI
Null Mean
Null Std
p -value
1
0.0520
0.0004
0.0002
0.000999
3
0.0384
0.0003
0.0001
0.000999
5
0.1148
0.0003
0.0001
0.000999
7
0.0207
0.0001
0.0000
0.000999
Table 3: Permutation test results for routing-category mutual information. All routed layers exhibit statistically significant routing-category associations. All reported p-values correspond to the minimum attainable empirical value under 1000 permutations, indicating that no permuted sample exceeded the observed MI.
Table 9: Routing-control and sparsity ablations evaluated on the held-out validation set (MI@L5). Random routing collapses routing-category dependence toward near-null values, indicating that specialisation does not arise from architectural sparsity alone. Freezing router parameters preserves weaker but non-zero specialisation. Lower load balancing increases routing-category dependence, while the routing-sparsity ablations show that top- k=2 yields a similar but slightly lower pooled MI than the main model, whereas top- k=4 produces substantially stronger category-conditioned routing, especially for English probes.
Forward
No-Curr
MI@L5
seed 42
seed 50
seed 42
seed 50
Combined
0.115
0.089
0.260
0.113
English
0.103
0.148
0.646
0.078
German
0.144
0.063
0.112
0.213
∣ EN − DE ∣
0.042
0.084
0.534
0.135
Appendix
Table 10: Layer-5 routing-category MI at both training seeds.
Frequency Bin
Forward MI@L5
No-Curr MI@L5
Frequent
0.0149
0.0055
Medium
0.0206
0.0976
Rare
0.2261
0.2494
Appendix
Table 11: English lexical irregular-versus-regular routing mutual information at layer 5 after binning probe lemmas by curriculum-weighted frequency.
Layer
Forward I(R;C∣L)
No-Curr I(R;C∣L)
1
0.0935
0.2584
3
0.0639
0.1084
5
0.1235
0.3787
7
0.0312
0.0895
Appendix
Table 12: Conditional mutual information between routing and linguistic category given language, I(R;C∣L) , across routed layers. In both the forward-curriculum and no-curriculum conditions, conditional MI remains clearly non-zero and permutation-significant at all routed layers ( p=0.000999 ), indicating that category-sensitive routing is not reducible to language separation alone.
Figure 5: Evolution of routing-category mutual information across training epochs for the routed layers.
Figure 6: Evolution of routing entropy across training epochs for the routed layers.
Mixture-of-Experts language models use input-dependent routing to activate a small subset of expert networks, increasing capacity without proportional compute.
Mixture-of-Experts (MoE) architectures enable efficient model scaling, yet expert routing behavior across underrepresented languages remains poorly understood. We analyze routing dynamics in two architecturally distinct MoE models -- a pure Transformer (Qwen3-30B-A3B) and a hybrid Mamba-Transformer (Nemotron-3-Nano-30B-A3B) -- using Hebrew as a morphologically rich, low-resource testbed. Both pre-trained models exhibit deep-layer routing collapse: usage entropy drops sharply in final layers and tokens concentrate on a narrow expert subset, a pattern largely absent for English. Continual pre-training (CPT) on balanced bilingual data substantially corrects this imbalance, increasing entropy and shifting routing toward shared, language-agnostic experts; supervised fine-tuning (SFT) alone achieves less complete correction. Extending the analysis to Japanese reveals quantitatively consistent collapse signatures, providing cross-linguistic evidence that the phenomenon is a systematic consequence of pre-training underrepresentation rather than any language-intrinsic property. Routing improvements correlate with consistent downstream benchmark gains, positioning routing entropy and expert specialization as principled diagnostics for multilingual capacity in MoE systems.
Mixture-of-Experts (MoE) models are widely used to scale language models, yet their expert routing behavior and adaptation in a multilingual setting remain underexplored. In this work, we study multilingual routing dynamics during continual pre-training of an English-centric MoE model on a multilingual corpus, analyzing how expert usage varies across languages. We find that continual multilingual pre-training leads to diffused, language-agnostic routing in early and middle layers, with language specialization primarily emerging in the final layers. We also show that token-level vocabulary overlap between languages plays an important role in how languages are routed. Motivated by these findings, we propose a parameter-efficient adaptation strategy that updates language-specific and shared experts in the final MoE layers. Experiments on MultiBLiMP and Belebele show that our method achieves a strong performance-efficiency trade-off, attaining competitive performance relative to fine-tuning complete final layers, while updating less than 2% of the parameters. Overall, our findings provide insights into where and how language specialization emerges in MoEs during continual pre-training and provide practical insights for low-resource multilingual adaptation. Our code is available at https://github.com/aditi184/moe-routing-adaptation.
Aditi Khandelwal, Marius Mosbach, Verna Dankers +2
Mixture-of-Experts (MoE) models exhibit striking performance disparities across languages, yet the internal mechanisms driving these gaps remain poorly understood. In this work, we conduct a systematic analysis of expert routing patterns in MoE models, revealing a phenomenon we term Language Routing Isolation, in which high- and low-resource languages tend to activate largely disjoint expert sets. Through layer-stratified analysis, we further show that routing patterns exhibit a layer-wise convergence-divergence pattern across model depth. Building on these findings, we propose RISE (Routing Isolation-guided Subnetwork Enhancement), a framework that exploits routing isolation to identify and adapt language-specific expert subnetworks. RISE applies a tripartite selection strategy, using specificity scores to identify language-specific experts in shallow and deep layers and overlap scores to select universal experts in middle layers. By training only the selected subnetwork while freezing all other parameters, RISE substantially improves low-resource language performance while preserving capabilities in other languages. Experiments on 10 languages demonstrate that RISE achieves target-language F1 gains of up to 10.85% with minimal cross-lingual degradation.
Kening Zheng, Wei-Chieh Huang, Jiahao Huo +9
University of Illinois Chicago · University of Maryland · HKUST (Guangzhou)