Knowledge Localization in Mixture-of-Experts LLMs Using Cross-Lingual Inconsistency

Lucas Bandarkar; Alan Ansell; Trevor Cohn

cs.CLMar 17, 2026

Knowledge Localization in Mixture-of-Experts LLMs Using Cross-Lingual Inconsistency

Authors: Lucas Bandarkar, Alan Ansell, Trevor Cohn

Organizations: Google Research · University of California, Los Angeles · University of Melbourne

Abstract

Modern LLMs continue to exhibit significant variance in behavior across languages, such as being able to recall factual information in some languages but not others. While typically studied as a problem to be mitigated, in this work, we propose leveraging this cross-lingual inconsistency as a tool for interpretability in mixture-of-experts (MoE) LLMs. Our knowledge localization framework contrasts routing for sets of languages where the model correctly recalls information from languages where it fails. This allows us to isolate model components that play a functional role in answering about a piece of knowledge. Our method proceeds in two stages: (1) querying the model with difficult factual questions across a diverse set of languages to generate "success" and "failure" activation buckets and then (2) applying a statistical contrastive analysis to the MoE router logits to identify experts important for knowledge. To validate the necessity of this small number of experts for answering a knowledge question, we deactivate them and re-ask the question. We find that despite only deactivating about 20 out of 6000 experts, the model no longer answers correctly in over 40% of cases. Generally, this method provides a realistic and scalable knowledge localization approach to address increasingly complex LLMs.

Figures & tables

Figure 1: Overview of our knowledge localization framework for MoEs that uses Cross-Lingual Inconsistency for Contrastive Identification ( XICI )

GLM-4.5-Air
Dataset	E	M	G
Num. Questions	383	250	150
Num. Languages	12	31	42
English Acc	28.5%	42.0%	54.7%
Avg. Acc	21.5%	24.5%	21.2%
All Correct	9	0	0

Table 1: Prevalence of cross-lingual inconsistency (“Mixed Results”). We abbreviate ECLeKTic to E , MultiLoKo to M , and Global MMLU prehistory to G . Accuracy displayed is averaged over all languages and questions. Statistically, (1) more languages and (2) accuracy closer to 50% increases the chances of inconsistency.

	Model	GLM-4.5-Air				Qwen3-30B-A3B-Instruct-2507
	Details	45 MoE layers, 5760 total routed experts				48 MoE layers, 6144 total experts
	Configuration	max 25, $\tau$ = 0.01, layers 6-40				max 25, $\tau$ = 0.005, layers 6-42
	Dataset	ECLeKTic	MultiLoKo	G-MMLU	Total	ECLeKTic	MultiLoKo	G-MMLU	Total
1	Num. Q’s w/ Inconsistency	202	158	122	482	156	139	124	419
2	Num. Languages	12	31	42	-	12	31	42	-
3	Blacklisted experts	316	290	285	-	298	270	240	-

Table 2: Results from our causal ablation of attributed experts. Row 1 is the number of questions we apply XICI to, while Row 4 is the number of questions where experts were actually identified.

Model	GLM		Qwen
Method	Eng	X-ling	Eng	X-ling
Num Q Variants	11	31	11	31
All Correct	70	0	37	0
All Incorrect	104	92	88	111
Mixed Results	76	158	125	139
Num. Q’s w/	47	125	56	102

Table 3: Table comparing the use of English paraphrases ( Eng ) rather than Cross-lingual translations ( X-ling ).

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Impact of different “max experts” values for our method, when applied to GLM-4.5-Air on MultiLoKo questions.

Figure 3: Location of experts identified for GLM-4.5-Air.

Figure 4: Location of experts identified for Qwen3-30B-A3B.

Concept Localization in Language Models

See all 10 papers

XPERT: Expert Knowledge Transfer for Effective Training of Language Models
May 9, 2026Chang Liu, Boyu Shi, Xu Yang +1Mixture-of-Experts Language Models Knowledge Augmentation for Language Models
School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China.
The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert Level
Apr 2, 2026Jeremy Herbst, Stefan Wermter, Jae Hee LeeMixture-of-Experts Interpretability Mixture-of-Experts Models
Department of Informatics, University of Hamburg, Hamburg, Germany.
Mixture of Experts for Low-Resource LLMs
May 17, 2026Ori Bar Joseph, Smadar Arvatz, Noam Kayzer +2Mixture-of-Experts Language Models Expert Routing

May 9, 2026J/K move · Enter open · S save

XPERT: Expert Knowledge Transfer for Effective Training of Language Models

Chang Liu, Boyu Shi, Xu Yang, Xin Geng

School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China.

Mixture-of-Experts Language Models Knowledge Augmentation for Language Models LLM Training

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) language models organize knowledge into explicitly routed expert modules, making expert-level representations traceable and analyzable. By analyzing expert activation patterns in MoE large language models (LLMs), we find that a subset of experts is consistently activated across diverse knowledge domains. These common experts encode cross-domain, generalizable knowledge that is closely related to model generalization, naturally raising the question of how such identifiable expert knowledge can be practically reused. Motivated by this observation, we propose XPERT, a framework that extracts, consolidates, and reuses expert knowledge from pre-trained MoE LLMs to support more effective training of language models across different model scales. XPERT identifies cross-domain experts via inference-only analysis, refines their representations through tensor decomposition, and adapts the extracted knowledge to reuse in downstream models. Experiments on language understanding and dialogue generation benchmarks show that models benefiting from reused expert knowledge achieve consistently stronger performance and faster convergence compared to strong baselines. These results highlight MoE LLMs as structured and reusable knowledge sources, and demonstrate the value of expert-level knowledge reuse for improving model training.

Abstract

Concept Localization in Language Models

Explore similar work

XPERT: Expert Knowledge Transfer for Effective Training of Language Models

The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert Level

Mixture of Experts for Low-Resource LLMs