NeuronMoE: Neuron-Guided Mixture-of-Experts for Efficient Multilingual LLM Extension

Rongzhi Li; Hitomi Yanaka

cs.CLMar 5, 2026

NeuronMoE: Neuron-Guided Mixture-of-Experts for Efficient Multilingual LLM Extension

Authors: Rongzhi Li, Hitomi Yanaka

Organizations: The University of Tokyo · Riken · Tohoku University

Abstract

Extending large language models to low-resource languages is essential for global accessibility, but training separate models per language is prohibitively expensive. Mixture-of-Experts (MoE) architectures address this by adding sparse language-specific parameters, but determining how many experts each layer needs remains an open question. Current approaches allocate experts based on layer-level similarity, yet language processing exhibits fine-grained specialization at individual neurons. We propose $\textbf{NeuronMoE}$ , a method that analyzes language-specific neurons across all transformer components to guide expert allocation per layer based on empirically measured cross-lingual neuron diversity. Applied to Llama-3.2-3B for low-resource languages (Greek, Turkish, and Hungarian), this approach achieves approximately 40% average parameter reduction while matching the performance of the LayerMoE baseline. We find that low-resource language experts independently develop neuron specialization patterns mirroring the high-resource language, which are concentrated in early and late layers. This reveals potential universal architectural principles in how multilingual models organize linguistic knowledge.

Figures & tables

Figure 1: Expert allocation comparison on Llama-3.2-3B for Greek extension. Cross-lingual neuron diversity (red line) exhibits heterogeneous distribution across layers. NeuronMoE allocates experts following this distribution, achieving 41.7% parameter reduction (49 vs 84 experts) compared to LayerMoE’s uniform allocation strategy.

Figure 2: Overview of NeuronMoE’s core insight and allocation strategy. Top: Cross-lingual neuron diversity exhibits dramatic heterogeneity across layers. For example, Layer 0 contains 342 unique language-specific neurons while Layer 10 contains only 7, demonstrating that different layers have vastly different language-specific processing requirements. Bottom: NeuronMoE allocates experts following this empirical distribution (49 experts total), concentrating capacity where specialization occurs. In contrast, LayerMoE’s similarity-based approach over-allocates in middle layers (84 experts total). This neuron-guided strategy achieves 41.7% parameter reduction while maintaining comparable performance.

Llama-3.2-3B (28 layers):
		ARC		Belebele		HellaSwag		MMLU
Model	#Exp	EN	EL	EN	EL	EN	EL	EN	EL
Dense	-	51.11	31.93	74.11	65.22	76.33	43.38	56.45	41.17
Stage 1 (Expert Initialization):
LayerMoE-S1	84	47.27	35.19	73.89	66.33	75.84	52.58	55.40	43.41
NeuronMoE-EN	37 (-56%)	49.15	33.13	73.56	63.67	76.45	49.03	56.89	43.26

Table 1: Performance on Greek extension across Llama-3.2-3B and Qwen-1.5-1.8B. #Exp indicates the number of experts. NeuronMoE achieves 41.7-50.0% parameter reduction with comparable performance, demonstrating cross-architecture generalization.

Layers	LayerMoE	NeuronMoE
0-2 (Early)	5,5,5	6,2,1
3-10 (Early-Mid)	4,3,3,2,2,2,2,2	1,1,1,1,1,1,1,1
11-18 (Mid-Late)	2,2,2,2,2,2,2,3	1,1,1,1,1,1,1,2
19-27 (Late)	3,3,3,3,4,4,4,4,4	1,1,4,1,4,2,2,4,4
Total	84	49

Table 2: Layer-wise expert allocation comparison (Llama-3.2-3B, Greek extension). NeuronMoE achieves 41.7% parameter reduction by concentrating experts in high-diversity layers while minimizing allocation in middle layers.

			ARC		Belebele		HellaSwag		MMLU
Language	Model	#Exp	EN	Target	EN	Target	EN	Target	EN	Target
Turkish	Dense	-	51.11	33.56	74.11	59.11	76.33	42.80	56.45	42.35
	LayerMoE	84	49.91	36.55	75.33	61.78	76.28	49.60	56.05	43.51
	NeuronMoE	50 (-40%)	49.83	34.50	74.44	61.33	76.22	47.83	56.21	43.37
Hungarian	Dense	-	51.11	34.85	74.11	59.67	76.33	44.63	56.45	43.68
	LayerMoE	84	49.91	39.73	75.33	64.22	76.28	51.66	56.05	44.86

Table 3: Stage 1 results for Turkish and Hungarian extension on Llama-3.2-3B. NeuronMoE achieves substantial parameter efficiency while successfully extending to additional low-resource languages. All values in %. Results for LayerMoE baseline are from Zhang et al. (2025) .

Layer-Expert	High-AP	Ratio (%)
L0 base_layer	0	0.00
L0 expert_0	0	0.00
L0 expert_1	27	0.14
L0 expert_2	0	0.00
L0 expert_3	61	0.31
L0 expert_4	19	0.10

Table 4: High-AP neuron counts for Greek across experts in selected layers (total neurons per layer/expert: 19,456). Multi-expert layers show distinct specialization patterns, validating allocation decisions.

Figure 3: Heatmap of high-AP neuron ratios for Greek across all experts. Color intensity indicates Greek specialization strength (red: high, blue: low/none). Multi-expert layers (0, 21, 23-27) show distinct specialization patterns across experts, while single-expert middle layers remain largely unspecialized, validating our neuron-guided allocation strategy.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Layers	Unique Neurons (EN+EL)
0-2	342, 79, 11
3-10	18, 15, 11, 8, 7, 8, 8, 7
11-18	6, 9, 14, 9, 18, 81, 20, 19
19-27	28, 38, 232, 51, 208, 114, 111, 238, 223
Total	1,933

Appendix

Table 5: Cross-lingual neuron diversity across layers (Llama-3.2-3B, Greek extension).

L	EN	EL	TR	HU	L	EN	EL	TR	HU
0	322	21	8	8	14	7	2	1	1
1	73	6	5	3	15	10	8	4	6
2	8	3	3	1	16	11	70	10	8
3	11	7	17	6	17	4	16	15	8
4	13	2	0	1	18	10	9	9	15
5	9	2	2	2	19	9	19	22	23

Appendix

Table 6: Complete layer-wise comparison of language-specific neuron counts across English (base Llama-3.2-3B), Greek, Turkish, and Hungarian

Optimization:
Parameter	Value
Optimizer	AdamW
Learning rate	1e-4
Adam $\beta_{1}$ , $\beta_{2}$	0.9, 0.999
Adam $\epsilon$	1e-8
Weight decay	0.0

Appendix

Table 7: Complete training hyperparameters for all experiments. All models trained with identical settings for fair comparison.

Layers	Greek	Turkish	Hungarian
0-2 (Early)	6,2,1	6,2,1	5,2,1
3-10 (Early-Mid)	1 $\times$ 8	1 $\times$ 8	1 $\times$ 8
11-18 (Mid-Late)	1 $\times$ 7,2	1 $\times$ 8	1 $\times$ 8
19-27 (Late)	1,1,4,1,4, 2,2,4,4	1,1,1,2,2, 4,4,4,4,3	1,1,1,2,1, 3,4,6,3,2
Total	49	50	47

Appendix

Table 8: Layer-wise expert allocation for Turkish and Hungarian on Llama-3.2-3B (28 layers). Both languages show similar allocation patterns to Greek, with concentration in input-proximal and output-proximal layers.

Method	Expert Allocation (Layer 0-23)
LayerMoE	[5, 4, 3, 5, 4, 3, 2, 3, 3, 2, 2, 2, [ 2, 2, 2, 2, 2, 3, 3, 4, 4, 4, 3, 3]
NeuronMoE	[4, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, [ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 6]

Appendix

Table 9: Per-layer expert allocation for LayerMoE and NeuronMoE on Qwen-1.5-1.8B (24 layers). LayerMoE uses attention-layer similarity; NeuronMoE uses cross-lingual neuron diversity analysis.

Expert	Total	High-AP	Ratio (%)
base	19,456	0	0.00
exp_0	19,456	0	0.00
exp_1	19,456	27	0.14
exp_2	19,456	0	0.00
exp_3	19,456	61	0.31
exp_4	19,456	19	0.10

Appendix

Table 10: Complete expert specialization statistics for Greek. High-AP neurons indicate strong Greek specialization developed during Stage 1 training. Total neurons per layer/expert: 19,456.

Model	Language	ARC	Belebele	HellaSwag	MMLU
Dense	English	51.11	74.11	76.33	56.45
	Nepali	24.12	42.44	32.54	33.77
	Bengali	26.26	44.78	33.26	33.48
NeuronMoE-S1 (NE)	English	47.44	71.44	75.29	54.71
	Nepali	24.89	44.67	33.07	34.63
NeuronMoE-S1 (BN)	English	48.72	75.33	76.01	55.28

Appendix

Table 11: Stage 1 results for Bengali and Nepali extensions on Llama-3.2-3B.

Multilingual Language Models

Multilingual language models are trained and adapted to understand and generate text across languages, scripts, and cultural contexts.

See all 265 papers

Mixture of Experts for Low-Resource LLMs
May 17, 2026Ori Bar Joseph, Smadar Arvatz, Noam Kayzer +2Mixture-of-Experts Language Models Expert Routing
Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptation
Apr 4, 2026Kening Zheng, Wei-Chieh Huang, Jiahao Huo +9Mixture-of-Experts Language Models Mixture of Experts
University of Illinois Chicago · University of Maryland · HKUST (Guangzhou)
Leveraging Routing Dynamics in Mixture-of-Experts Models for Efficient Language Adaptation
May 28, 2026Aditi Khandelwal, Marius Mosbach, Verna Dankers +2Multilingual Language Models Continual Learning for LLMs
Mila – Quebec AI Institute & McGill University

May 17, 2026J/K move · Enter open · S save

Mixture of Experts for Low-Resource LLMs

Ori Bar Joseph, Smadar Arvatz, Noam Kayzer, Dan Revital+1

Mixture-of-Experts Language Models Expert Routing LLM Routing Low-Resource Language Processing

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) architectures enable efficient model scaling, yet expert routing behavior across underrepresented languages remains poorly understood. We analyze routing dynamics in two architecturally distinct MoE models -- a pure Transformer (Qwen3-30B-A3B) and a hybrid Mamba-Transformer (Nemotron-3-Nano-30B-A3B) -- using Hebrew as a morphologically rich, low-resource testbed. Both pre-trained models exhibit deep-layer routing collapse: usage entropy drops sharply in final layers and tokens concentrate on a narrow expert subset, a pattern largely absent for English. Continual pre-training (CPT) on balanced bilingual data substantially corrects this imbalance, increasing entropy and shifting routing toward shared, language-agnostic experts; supervised fine-tuning (SFT) alone achieves less complete correction. Extending the analysis to Japanese reveals quantitatively consistent collapse signatures, providing cross-linguistic evidence that the phenomenon is a systematic consequence of pre-training underrepresentation rather than any language-intrinsic property. Routing improvements correlate with consistent downstream benchmark gains, positioning routing entropy and expert specialization as principled diagnostics for multilingual capacity in MoE systems.

L	EN	EL	TR	HU	L	EN	EL	TR	HU
0	322	21	8	8	14	7	2	1	1
1	73	6	5	3	15	10	8	4	6
2	8	3	3	1	16	11	70	10	8
3	11	7	17	6	17	4	16	15	8
4	13	2	0	1	18	10	9	9	15
5	9	2	2	2	19	9	19	22	23

L	EN	EL	TR	HU	L	EN	EL	TR	HU
0	322	21	8	8	14	7	2	1	1
1	73	6	5	3	15	10	8	4	6
2	8	3	3	1	16	11	70	10	8
3	11	7	17	6	17	4	16	15	8
4	13	2	0	1	18	10	9	9	15
5	9	2	2	2	19	9	19	22	23

Abstract

Multilingual Language Models

Explore similar work

Mixture of Experts for Low-Resource LLMs

Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptation

Leveraging Routing Dynamics in Mixture-of-Experts Models for Efficient Language Adaptation

L	EN	EL	TR	HU	L	EN	EL	TR	HU
0	322	21	8	8	14	7	2	1	1
1	73	6	5	3	15	10	8	4	6
2	8	3	3	1	16	11	70	10	8
3	11	7	17	6	17	4	16	15	8
4	13	2	0	1	18	10	9	9	15
5	9	2	2	2	19	9	19	22	23