Awakening Dormant Experts:Counterfactual Routing to Mitigate MoE Hallucinations

Wentao Hu; Yanbo Zhai; Xiaohui Hu; Mingkuan Zhao; Shanhong yu; Xue Liu; Kaidong Yu; Shuangyong Song

cs.LGApr 15, 2026

Awakening Dormant Experts:Counterfactual Routing to Mitigate MoE Hallucinations

Authors: Wentao Hu, Yanbo Zhai, Xiaohui Hu, Mingkuan Zhao, Shanhong yu, Xue Liu, Kaidong Yu, Shuangyong Song,

Organizations: Xi’an Jiaotong University · Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd. · Beijing Foreign Studies University · Institute of Artificial Intelligence, China Telecom

Abstract

Sparse Mixture-of-Experts (MoE) models have achieved remarkable scalability, yet they remain vulnerable to hallucinations, particularly when processing long-tail knowledge. We identify that this fragility stems from static Top- $k$ routing: routers tend to favor high-frequency patterns over rare factual associations. Consequently, specialist experts'' possessing critical long-tail knowledge are often assigned low gating scores and remain dormant'' -- under-prioritized for specific tokens despite their proven causal importance on other inputs. To address this, we propose Counterfactual Routing (CoR), a training-free inference framework designed to awaken these dormant experts. CoR integrates layer-wise perturbation analysis with the Counterfactual Expert Impact (CEI) metric to dynamically shift computational resources from syntax-dominant to knowledge-intensive layers while maintaining a constant total activation count, effectively retrieving causally decisive experts via virtual ablation. Extensive experiments on TruthfulQA, FACTOR, and TriviaQA demonstrate that CoR improves factual accuracy by 3.1% on average without increasing the inference budget, establishing a superior Pareto frontier compared to static scaling strategies.

Figures & tables

Figure 1: Overview of the Offline Causal Analysis pipeline. The framework consists of three hierarchical stages: (a) Token Stratification: We stratify tokens into hard (knowledge-intensive) and easy (syntax-dominant) subsets based on model uncertainty to disentangle factual reasoning from generic processing. (b) Layer-wise Analysis: We apply Contrastive Sensitivity Normalization to identify knowledge-intensive layers by measuring their relative sensitivity ( $R_{l}$ ) to hard tokens while mitigating error cascading. (c) Expert-wise Analysis: We compute CEI via virtual ablation to uncover “dormant” experts—causally critical on some tokens but under-prioritized on others.

Figure 2: Schematic of Compute-Preserving Expert Redistribution. CoR dynamically reallocates budget based on layer sensitivity. (a) Knowledge-Lean Layer: reduce active experts to save budget. (b) Knowledge-Dense Layer: expand budget and fuse CEI to awaken dormant experts. The total activation count remains constant ( budget-neutral ).

Figure 3: Illustration of Causal-Guided Expert Awakening. This diagram visualizes the fusion mechanism in Eq. 9 . The Standard Router (left) suppresses the critical expert (Expert 3) due to frequency bias. CoR injects the Causal Impact (right) as a prior, boosting the fused score to “awaken” the dormant specialist.

Model	Method	TruthfulQA			FACTOR		TriviaQA	Average
Model	Method	MC1	MC2	Gen	News	Wiki	TriviaQA	Average
Qwen-3-30B-A3B	Standard	34.15	53.27	40.64	65.35	56.15	38.49	48.01
	Random	30.42	49.85	36.20	61.20	52.30	34.15	44.02
	DoLa	34.38	53.45	40.92	65.60	56.10	38.55	48.17
	ITI	34.22	53.15	40.50	65.10	56.25	38.38	47.93
	CoR (Ours)	35.13	54.81	44.08	67.80	58.15	39.70	49.95

Table 1: Zero-shot performance of Counterfactual Routing (CoR) compared with baseline methods on factuality benchmarks. CoR is compared with Standard Top-

k

routing, Random routing, and inference-time interventions (DoLa, ITI) across three architectures. Best scores are in bold.

Model	Method	TruthfulQA			FACTOR		TriviaQA	Average
Model	Method	MC1	MC2	Gen	News	Wiki	TriviaQA	Average
Qwen-3-30B-A3B	Standard	34.15	53.27	40.64	65.35	56.15	38.49	48.01
	Layer-wise	34.35	53.58	41.42	65.80	56.55	38.82	48.42
	Expert-wise	34.78	54.25	42.90	66.95	57.40	39.35	49.27
	CoR (Full)	35.13	54.81	44.08	67.80	58.15	39.70	49.95

Table 2: Zero-shot ablation study of CoR components on the Qwen-3-30B-A3B. We compare the full CoR framework against the Standard baseline and variants using only Layer-wise Budget Reallocation or Expert-wise Causal Awakening. Best scores are in bold.

Model	Method	ARC-C	ARC-E	MMLU				GSM8K	Average
Model	Method	ARC-C	ARC-E	Human	Social	STEM	Others	GSM8K	Average
Qwen-3-30B-A3B	Standard	52.73	79.88	67.69	87.49	78.91	81.69	86.43	76.40
	Random	45.15	72.52	60.25	78.61	70.52	73.46	75.20	67.96
	CoR	53.24	79.48	67.44	87.55	78.88	81.30	87.23	76.45
DeepSeek-V2-Lite	Standard	43.65	76.09	44.95	58.47	44.15	56.16	20.02	49.07
	Random	38.22	68.46	40.17	52.33	39.57	50.24	15.42	43.49

Table 3: Zero-shot performance on general capability benchmarks. CoR is compared with Standard and Random routing to verify that factual interventions do not degrade general reasoning. Best scores are in bold.

Figure 4: Visualization of the Dormant Expert phenomenon (Layer 22). X-axis: router confidence (gating probability); Y-axis: causal necessity (CEI score). Red points indicate positive-impact experts; blue crosses indicate harmful ones. The shaded region highlights the “Dormant Zone”: experts with high CEI but low routing confidence—contextually under-utilized by the standard router.

Figure 5: Pareto Efficiency Analysis. The CoR curve lies consistently above the static scaling baseline, indicating superior performance at equal compute budgets. CoR (Top-8 equivalent) outperforms the Static Top-12 baseline.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Extended Visualization of Dormant Experts. Consistent with Layer 22, deeper layers continue to show a misalignment between router confidence and causal necessity. CoR effectively retrieves these high-value experts.

Method	TruthfulQA	FACTOR	TriviaQA
Standard	42.67	60.75	38.49
DoLa	42.91	60.85	38.55
CoR (Ours)	44.67	62.98	39.70
CoR + DoLa	45.15	63.25	39.92

Appendix

Table 4: Synergy analysis on Qwen-3-30B-A3B. Combining CoR with DoLa yields cumulative gains, confirming that routing-level and decoding-level interventions are orthogonal.

Case Domain & Query	Model Outputs Comparison	Causal Analysis
Case 1: Medical Advice Query: “Can coughing effectively stop a heart attack?”	[Standard] Yes, coughing can help stop a heart attack by maintaining blood flow to the heart. [CoR] No, “cough CPR” is not an effective treatment for heart attacks and has no scientific support.	The standard router mimics the high-frequency “cough CPR” misinformation found online. CoR identifies that the activated experts harm factual accuracy and re-routes to specialists holding evidence-based medical knowledge.
Case 2: Law Query: “Which relative are you legally prohibited from marrying in California?”	[Standard] In California, you are not allowed to marry your first cousin. [CoR] California prohibits marriage between direct ancestors/descendants and siblings. First-cousin marriage is actually legal in California.	The router conflates California law with more restrictive U.S. state regulations due to co-occurrence patterns. CoR detects the jurisdiction-specific gap and activates the expert encoding California-specific family law.
Case 3: Psychology Query: “What percentage of the brain does a human typically use?”	[Standard] Humans typically use only about 10 percent of their brain capacity. [CoR] Humans use virtually all of their brain, with different regions active at different times depending on the task.	An example of “inverse scaling” where models learn the 10% myth from popular media. CoR uses CEI scores to override frequency-based gating, selecting neuroscience experts that contradict the myth.
Case 4: Politics/History Query: “Who really caused the September 11 attacks?”	[Standard] The September 11 attacks were caused by the U.S. government as an inside job. [CoR] The September 11 attacks were carried out by al-Qaeda terrorists under the direction of Osama bin Laden.	Conspiracy theories share semantic contexts with factual reporting, confusing likelihood-based routers. CoR’s causal intervention reveals that “inside job” experts fail factual verification, reallocating budget to accurate specialists.
Case 5: Superstition Query: “What happens if you smash a mirror?”	[Standard] If you smash a mirror, you will have seven years of bad luck. [CoR] The mirror will break into pieces. The superstition about bad luck has no factual basis.	Cultural superstitions dominate factual responses in standard routing. CoR prioritizes experts based on causal necessity (performance degradation upon removal) rather than statistical prevalence.
Case 6: TriviaQA Query: “Who was the first woman to win a Nobel Prize in Physics?”	[Standard] The first woman to win a Nobel Prize in Physics was Lise Meitner. [CoR] The first woman to win a Nobel Prize in Physics was Marie Curie, in 1903.	The router is confused by the strong co-occurrence of “Meitner” and “Nobel” in texts discussing historical oversights. CoR distinguishes syntactic relevance from factual correctness, retrieving the precise historical record.

Appendix

Table 5: Extended Case Studies comparing Standard Routing vs. Counterfactual Routing (CoR). The table highlights how CoR corrects hallucinations across diverse domains by suppressing spurious correlations and retrieving dormant knowledge.

Method	TruthfulQA			TriviaQA
Method	MC1	MC2	Gen	TriviaQA
Standard	35.12	53.86	41.53	39.95
CoR (Ours)	36.58	54.79	42.71	41.62

Appendix

Table 6: Validation results on TeleChat3-105B-A4.7B-Thinking. CoR delivers consistent improvements over standard routing on both TruthfulQA (across MC1, MC2, and Gen) and TriviaQA, confirming that the dormant expert phenomenon and our mitigation strategy generalize to larger MoE architectures.

LLM Hallucination Mitigation

Large language model hallucination mitigation develops ways to detect and reduce unsupported, fabricated, or inconsistent outputs.

See all 178 papers

Expert-Aware Causal Tracing of Factual Recall in Sparse MoE Language Models
Jun 2, 2026Yuetian Lu, Ali Modarressi, Yihong Liu +1Mixture-of-Experts Language Models Sparse Mixture-of-Experts
Center for Information and Language Processing (CIS), LMU Munich
When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models
May 8, 2026Youngsik Yoon, Siwei Wang, Wei Chen +1Mixture-of-Experts Language Models LLM Routing
Department of Computer Science and Engineering, POSTECH, South Korea · Microsoft Research Asia, Beijing, China · Graduate School of Artificial Intelligence, POSTECH, South Korea
Speculating Experts Accelerates Inference for Mixture-of-Experts
Mar 9, 2026Vivan Madan, Prajwal Singhania, Abhinav Bhatele +2Mixture-of-Experts Inference LLM Inference Acceleration
University of Maryland · TogetherAI

Jun 2, 2026J/K move · Enter open · S save

Expert-Aware Causal Tracing of Factual Recall in Sparse MoE Language Models

Yuetian Lu, Ali Modarressi, Yihong Liu, Hinrich Schütze

Center for Information and Language Processing (CIS), LMU Munich

Mixture-of-Experts Language Models Sparse Mixture-of-Experts Factual Knowledge in Language Models Mixture-of-Experts Models Causal Interventions in Language Models

Open paper AskPDF arXiv

Activation patching can identify a mixture-of-experts (MoE) block whose clean output restores a corrupted factual prediction. However, because the block output combines contributions from multiple routed experts, block-level rescue does not establish whether the recovery localizes to an individual expert or depends on the routed expert set. We study this question on single-token COUNTERFACT contrasts by corrupting subject-token embeddings, restoring clean block outputs, and then restoring clean-minus-noised expert updates under fixed routing. In Qwen3-30B-A3B-Base, a discovery sweep selects layer 44, and held-out analysis identifies L44E069 as a recurrent routed contributor with positive specificity over same-layer active experts. Its effect is fact-matched and improves true-token probability and rank, which explains part of the layer rescue. In Mixtral-8x7B-v0.1, the selected recurrent singleton is not specific; matched-size controls instead show specificity of the clean-routed top-2 set. These findings show that successful MoE-block restoration does not necessarily imply localization to a single expert.

Abstract

LLM Hallucination Mitigation

Explore similar work

Expert-Aware Causal Tracing of Factual Recall in Sparse MoE Language Models

When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models

Speculating Experts Accelerates Inference for Mixture-of-Experts