A Declarative-Procedural Perspective on Expert Routing in Bilingual Mixture-of-Experts Language Models

Amrit Gopinath; Raghul; Durairaj Thenmozhi

cs.CLAug 15, 2026

A Declarative-Procedural Perspective on Expert Routing in Bilingual Mixture-of-Experts Language Models

Authors: Amrit Gopinath, Raghul, Durairaj Thenmozhi

Organizations: Sri Sivasubramaniya Nadar College of Engineering, Chennai, India · Shiv Nadar University Chennai, India

Abstract

We investigate whether Mixture-of-Experts (MoE) language models develop linguistically structured expert routing during bilingual language acquisition. Inspired by the Declarative-Procedural framework, we analyze lexical, grammatical, and syntactic processing in a decoder-only English-German MoE Transformer trained under sequential language exposure. We construct a probe-based validation set and extract token-level routing distributions to quantify category-dependent specialisation using mutual information, routing entropy, and Jensen-Shannon distance. The curriculum-trained model exhibits a peak mutual information of 0.1148 at layer 5, indicating category-dependent differences in routing distributions across linguistic categories. Surprisingly, a no-curriculum baseline trained on mixed English-German data shows stronger aggregate specialisation, reaching a peak mutual information of 0.2599 at the same layer. These results suggest that interpretable linguistic organization emerges within MoE routing patterns even without sequential language exposure. A replication at a second training seed shows that the no-curriculum condition's specialisation concentrates on a single language whose identity is seed-dependent, whereas the curriculum consistently yields a stable, language-balanced routing profile; rather than uniformly increasing specialisation, staged bilingual exposure reduces single-language dominance. The official Github repository: https://github.com/Amrit828/DP-Theory-MOE-Interpretability-Research

Figures & tables

Language	Lexical	Grammatical	Syntactic
English	460,250	650,000	650,000
German	138,127	175,000	175,000

Table 1: Dataset size by language and linguistic category.

Condition	Perplexity	Accuracy (%)
Forward Curriculum	111.19	28.99
No Curriculum	117.01	29.65

Table 2: Validation performance on the held-out diagnostic probe set.

Figure 1: Layer-wise mutual information between expert-routing distributions and linguistic categories.

Layer	Observed MI	Null Mean	Null Std	$p$ -value
1	0.0520	0.0004	0.0002	0.000999
3	0.0384	0.0003	0.0001	0.000999
5	0.1148	0.0003	0.0001	0.000999
7	0.0207	0.0001	0.0000	0.000999

Table 3: Permutation test results for routing-category mutual information. All routed layers exhibit statistically significant routing-category associations. All reported p-values correspond to the minimum attainable empirical value under 1000 permutations, indicating that no permuted sample exceeded the observed MI.

Figure 2: Category-conditioned expert allocation patterns.

Figure 3: Layer-wise expert utilisation across routed layers.

Figure 4: Routing entropy across routed layers and linguistic categories. Lower entropy indicates more concentrated expert utilisation.

Left Category	Right Category	$d_{JS}$
Grammatical	Lexical	0.0788
Grammatical	Syntactic	0.1147
Lexical	Syntactic	0.1218

Table 4: Pairwise Jensen–Shannon distance between category-conditioned routing distributions.

Layer	No-Curr MI
1	0.1945
3	0.0674
5	0.2599
7	0.0739

Table 5: Pooled bilingual routing-category MI under the no-curriculum condition.

Layer	Fwd EN	Fwd DE	No-Curr EN	No-Curr DE
1	0.0508	0.1361	0.3810	0.1357
3	0.0366	0.0911	0.1364	0.0804
5	0.1025	0.1444	0.6457	0.1116
7	0.0571	0.0053	0.1141	0.0649

Table 6: Language-wise routing-category mutual information for the forward curriculum and no-curriculum settings.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Epochs	$\lambda_{\text{DE}}$	Description
1–8	0.00	EN only
9	0.20	L2 introduction
10	0.25
11	0.30
12	0.35
13	0.40

Appendix

Table 7: Forward curriculum L2 exposure schedule.

Probe Set	Records	Tokens	Uniqueness
EN Irregular Verb	1500	46	100%
EN Irregular Plural	1500	30	100%
DE Strong Verb	500	35	100%
EN SCONJ	3000	21	100%
EN Aux Agreement	1500	17	100%

Appendix

Table 8: Illustrative diagnostic probe diversity statistics.

Ablation	Combined	EN	DE
Random routing	0.0018	0.0029	0.0057
Frozen routing	0.0274	0.0647	0.0140
Load balancing ( $\alpha=0.005$ )	0.1482	0.3288	0.1042
Forward top- $k=2$	0.1100	0.1767	0.1349
Forward top- $k=4$	0.2382	0.4107	0.1144

Appendix

Table 9: Routing-control and sparsity ablations evaluated on the held-out validation set (MI@L5). Random routing collapses routing-category dependence toward near-null values, indicating that specialisation does not arise from architectural sparsity alone. Freezing router parameters preserves weaker but non-zero specialisation. Lower load balancing increases routing-category dependence, while the routing-sparsity ablations show that top-

k=2

yields a similar but slightly lower pooled MI than the main model, whereas top-

k=4

produces substantially stronger category-conditioned routing, especially for English probes.

	Forward		No-Curr
MI@L5	seed 42	seed 50	seed 42	seed 50
Combined	0.115	0.089	0.260	0.113
English	0.103	0.148	0.646	0.078
German	0.144	0.063	0.112	0.213
$\|$ EN $-$ DE $\|$	0.042	0.084	0.534	0.135

Appendix

Table 10: Layer-5 routing-category MI at both training seeds.

Frequency Bin	Forward MI@L5	No-Curr MI@L5
Frequent	0.0149	0.0055
Medium	0.0206	0.0976
Rare	0.2261	0.2494

Appendix

Table 11: English lexical irregular-versus-regular routing mutual information at layer 5 after binning probe lemmas by curriculum-weighted frequency.

Layer	Forward $I(R;C\mid L)$	No-Curr $I(R;C\mid L)$
1	0.0935	0.2584
3	0.0639	0.1084
5	0.1235	0.3787
7	0.0312	0.0895

Appendix

Table 12: Conditional mutual information between routing and linguistic category given language,

I(R;C\mid L)

, across routed layers. In both the forward-curriculum and no-curriculum conditions, conditional MI remains clearly non-zero and permutation-significant at all routed layers (

p=0.000999

), indicating that category-sensitive routing is not reducible to language separation alone.

Figure 5: Evolution of routing-category mutual information across training epochs for the routed layers.

Figure 6: Evolution of routing entropy across training epochs for the routed layers.

Mixture-of-Experts Language Models

Mixture-of-Experts language models use input-dependent routing to activate a small subset of expert networks, increasing capacity without proportional compute.

See all 225 papers

Mixture of Experts for Low-Resource LLMs
May 17, 2026Ori Bar Joseph, Smadar Arvatz, Noam Kayzer +2Mixture-of-Experts Language Models Expert Routing
Leveraging Routing Dynamics in Mixture-of-Experts Models for Efficient Language Adaptation
May 28, 2026Aditi Khandelwal, Marius Mosbach, Verna Dankers +2Multilingual Language Models Continual Learning for LLMs
Mila – Quebec AI Institute & McGill University
Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptation
Apr 4, 2026Kening Zheng, Wei-Chieh Huang, Jiahao Huo +9Mixture-of-Experts Language Models Mixture of Experts
University of Illinois Chicago · University of Maryland · HKUST (Guangzhou)

May 17, 2026J/K move · Enter open · S save

Mixture of Experts for Low-Resource LLMs

Ori Bar Joseph, Smadar Arvatz, Noam Kayzer, Dan Revital+1

Mixture-of-Experts Language Models Expert Routing LLM Routing Low-Resource Language Processing

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) architectures enable efficient model scaling, yet expert routing behavior across underrepresented languages remains poorly understood. We analyze routing dynamics in two architecturally distinct MoE models -- a pure Transformer (Qwen3-30B-A3B) and a hybrid Mamba-Transformer (Nemotron-3-Nano-30B-A3B) -- using Hebrew as a morphologically rich, low-resource testbed. Both pre-trained models exhibit deep-layer routing collapse: usage entropy drops sharply in final layers and tokens concentrate on a narrow expert subset, a pattern largely absent for English. Continual pre-training (CPT) on balanced bilingual data substantially corrects this imbalance, increasing entropy and shifting routing toward shared, language-agnostic experts; supervised fine-tuning (SFT) alone achieves less complete correction. Extending the analysis to Japanese reveals quantitatively consistent collapse signatures, providing cross-linguistic evidence that the phenomenon is a systematic consequence of pre-training underrepresentation rather than any language-intrinsic property. Routing improvements correlate with consistent downstream benchmark gains, positioning routing entropy and expert specialization as principled diagnostics for multilingual capacity in MoE systems.

Abstract

Mixture-of-Experts Language Models

Explore similar work

Mixture of Experts for Low-Resource LLMs

Leveraging Routing Dynamics in Mixture-of-Experts Models for Efficient Language Adaptation

Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptation