Routing Drift Alone Does Not Diagnose Failure in Merged MoE LLMs

Yuanyi Wang; Yanggan Gu; Su Lu; Guanghao Zhu; Pengkai Wang; Yifan Yang; Congkai Xie; Zhaoyi Yan

cs.AISep 26, 2026

Routing Drift Alone Does Not Diagnose Failure in Merged MoE LLMs

Authors: Yuanyi Wang, Yanggan Gu, Su Lu, Guanghao Zhu, Pengkai Wang, Yifan Yang, Congkai Xie, Zhaoyi Yan,

Organizations: The Hong Kong Polytechnic University, PolyU

Abstract

Model merging efficiently combines specialized large language models (LLMs) without joint retraining, but can substantially alter expert routing in Mixture-of-Experts (MoE) models. Such routing drift is often interpreted as routing failure, raising a fundamental question that remains unclear: does routing drift after MoE merging actually indicate routing failure, and what evidence should justify repair? We investigate these questions across DeepSeekMoE, OLMoE, and Qwen3-MoE proposing a routing analysis toolkit for controlled counterfactual interventions and token-level analysis. By crossing source and merged router inputs and parameters, we attribute most expert reassignments to input shifts rather than parameter changes at the same layer. However, source-relative routing differences poorly predict next-token likelihood gains from source-route restoration, and different expert selections can produce directionally similar mixture outputs. We therefore operationalize routing failure as task loss recoverable under a specified routing intervention, with non-routing parameters fixed. These tests detect recoverable loss under deliberate router corruption, whereas source-route restoration does not establish reliable task benefits in the evaluated merged models. Motivated by these, we propose Selective Router Repair (SRR) as a case study, and find that source-specialist token-likelihood advantages do not reliably identify beneficial local corrections. Together, these findings show that routing drift alone is insufficient evidence of routing failure: source-informed corrections must be judged by their task-level intervention effects. The analysis toolkit and SRR code are released.

Figures & tables

Figure 1: Merge-induced representation drift and SRR updates. Illustrative joint t-SNE of aligned router-input representations in OLMoE. Arrows track the same domain-balanced tokens from Source to Merged in (b) and their corresponding shifts after SRR in (c).

Figure 2: Most post-merge route changes are representation-induced. (a) Expert-set distances for 64 aligned events per setting under component replacements. (b) Full-sample attribution among changed routes. Protocols and intervals appear in Appendices A.4 and A.8 .

Figure 3: Routing drift alone is insufficient evidence of routing failure. (a–c) JS divergence versus source-route NLL gain (log- $x$ , symmetric-log $y$ ). (d) Task-margin changes with 95% bootstrap intervals. Full tests appear in Appendices A.6 – A.8 .

Figure 4: Different routes, similar mixture outputs. (a) Mean cosines. (b) Paired observed-minus-random cosine (95% CIs). (c) Cumulative event-wise cosine gaps.

Figure 5: Task effects of controlled routing interventions. (a) Corrupted vs. clean-route replay accuracy. (b) Paired replay gains with 95% bootstrap CIs across all parent–permutation settings.

Source-route replay: science
Model / merge	$N$	$\Delta\mathrm{Acc}$ (pp)	$\Delta m$ ( $10^{-3}$ )
DeepSeekMoE / Avg.	256	$0.00\,[-1.56,+1.56]$	$-5.2\,[-18.2,+7.9]$
DeepSeekMoE / TA	256	$+0.78\,[-0.78,+2.34]$	$+2.1\,[-11.8,+17.0]$
OLMoE / Avg.	256	$+1.17\,[-1.56,+3.91]$	$+2.5\,[-26.1,+31.2]$
OLMoE / TA	256	$+1.17\,[-1.17,+3.52]$	$-6.2\,[-41.8,+28.5]$
Qwen3-MoE / Avg.	64	$-1.56\,[-4.69,0.00]$	$+8.8\,[-12.0,+28.2]$

Table 1: Natural routing alternatives. Changes from native routing with paired 95% bootstrap CIs.

m

denotes the correct-choice margin.

Figure 6: Local utility and input response of SRR. (a,b) Positive-support NLL gains over native ( $U$ ) and opposite ( $D$ ) routing, with 95% bootstrap CIs. (c) Input-response/direct RMS ratios across eight settings and five updated layers.

DeepSeekMoE-16B-A3B
	Benchmarks (%)								Summary		95% CI (pp)
Method	MMLU	Hella Swag	ARC-C	ARC-E	PIQA	Wino Grande	BoolQ	GSM8K	Avg.	$\Delta$ Avg.	Lower	Upper
Average	57.60	80.02	57.68	83.50	82.48	76.09	81.74	48.67	70.97	—	—	—
+ HARC	57.58	80.05	57.68	84.01	82.32	76.48	82.05	49.51	71.21	+0.236	—	—
+ SRR	57.68	79.99	57.68	83.50	82.48	76.09	81.80	48.67	70.99	+0.015	-0.011	+0.041
Task Arithmetic	58.04	80.03	58.11	83.92	82.32	76.48	80.83	45.26	70.62	—	—	—

Table 2: Task scores across three merged MoEs. Avg. equally weights eight benchmarks;

\Delta

Avg. compares each method with its own Parent (pp). The 95% CI (pp) is the paired item-bootstrap 95% confidence interval for the SRR–Parent difference in percentage points. HARC scores use the same matched requests.

Contrast	Estimate	95% CI
Native ( $U$ )	$-0.341$	$[-3.323,\,2.631]$
Opposite ( $D$ )	$-0.997$	$[-5.653,\,3.868]$

Table 3: Matched selection effects. Positive-minus-control source-direction gains (

10^{-3}

nats/token; 440 matched pairs).

Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A1: Aligned representations in OLMoE. Joint t-SNE at layer 15: router inputs (top) and routed-mixture outputs (bottom) under Source, Merged, and SRR. Red and teal arrows highlight displayed token displacements associated with merging and SRR, respectively.

Figure A2: Aligned representations in DeepSeekMoE. Joint t-SNE of router inputs (top) and MoE-block outputs (bottom) at layer 27 under Source, Merged, and SRR. Display conventions follow Figure A1 .

Figure A3: Aligned representations in Qwen3-MoE. Joint t-SNE of router inputs (top) and MoE-block outputs (bottom) at layer 47 under Source, Merged, and SRR. Display conventions follow Figure A1 .

Analysis	Sampling frame	Record / inference unit
Natural route origin	256 prompts (64/domain); all continuation tokens and sparse layers	Token–layer event / prompt UUID
Structural diagnostic	Same 256 prompts; final five sparse layers; eight hash-selected events per prompt (2,048/cell)	Counterfactual event / prompt UUID
Token-local route patch	80 changed-route events per architecture–parent cell	Counterfactual event / prompt UUID
Local reference and redundancy	One hash-selected event per prompt (256/cell); 112–149 changed-route events/cell	Prompt event / prompt UUID
Task-grounded route patch	256 fixed items/domain/cell; every non-padding question–choice token and sparse layer	Multiple-choice item / item
Matched random routes	32 matched alternatives for each changed-route event	Prompt event / prompt UUID

Appendix

Table A1: Sampling frames for the original DeepSeekMoE/OLMoE analyses. Hash selections do not inspect outcomes. Qwen3-specific counts appear in Appendix A.8 .

Architecture / Parent	Changed route (%)	Representation (%)	Router parameters (%)
DeepSeekMoE / Average	$33.9\,[32.2,35.7]$	$77.7\,[77.1,78.2]$	$1.79\,[1.65,1.94]$
DeepSeekMoE / Task Arithmetic	$46.7\,[44.9,48.7]$	$78.5\,[78.1,79.0]$	$1.14\,[1.04,1.23]$
OLMoE / Average	$42.7\,[41.3,44.1]$	$85.8\,[85.4,86.1]$	$0.62\,[0.57,0.68]$
OLMoE / Task Arithmetic	$54.3\,[52.9,55.7]$	$87.0\,[86.6,87.4]$	$0.36\,[0.33,0.40]$

Appendix

Table A2: Local four-state attribution. Values are percentages with 95% prompt-bootstrap intervals; attribution shares condition on a changed source–merged expert set.

Figure A4: Attribution across sparse layers. Four-state shares among changed routes for DeepSeekMoE/OLMoE under Average and TA. Categories are defined in Appendix A.1 .

Figure A5: Attribution by domain. Representation-induced and router-parameter-induced shares use separate color scales; each cell reports its percentage.

Figure A6: Representation and routing across OLMoE depth. Original-space hidden-state cosine distance and Top-

k

disagreement co-vary across 16 layers. This association is distinct from the crossed intervention.

Architecture / Parent	Route / Top- $k$	Route / JS	State / Top- $k$	State / JS
DeepSeekMoE / Average	$.490\,[.463,.517]$	$.510\,[.472,.548]$	$.502\,[.476,.528]$	$.509\,[.472,.547]$
DeepSeekMoE / Task Arithmetic	$.491\,[.465,.517]$	$.510\,[.478,.542]$	$.503\,[.477,.529]$	$.511\,[.477,.547]$
OLMoE / Average	$.509\,[.483,.533]$	$.519\,[.482,.556]$	$.515\,[.489,.541]$	$.520\,[.481,.559]$
OLMoE / Task Arithmetic	$.525\,[.502,.548]$	$.483\,[.450,.515]$	$.525\,[.502,.549]$	$.487\,[.454,.519]$

Appendix

Table A3: Primary AUROC comparisons with 95% prompt-bootstrap intervals. Route/state denote source-routing/source-hidden-state gains;

0.5

is chance. None of the 16 comparisons survives Holm correction.

Figure A7: Complete changed-set predictor family. Spearman associations with source-policy gains (left) and source-hidden-state gains (right). Primary predictors are set distance and full-distribution JS.

Architecture / Parent	$\Delta\mathrm{NLL}$ [95% CI]	Events / prompts
DeepSeekMoE / Average	$-0.0339\,[-0.0675,-0.0062]$	80 / 71
DeepSeekMoE / Task Arithmetic	$-0.0326\,[-0.0682,-0.0038]$	80 / 74
OLMoE / Average	$-0.1623\,[-0.4078,-0.0026]$	80 / 66
OLMoE / Task Arithmetic	$-0.1076\,[-0.1947,-0.0400]$	80 / 68

Appendix

Table A4: Token-local source-route replay. Positive native-minus-replay NLL favors replay. Intervals resample prompts.

Model / Parent	Domain	Margin $\Delta$ [95% CI]	Accuracy $\Delta$ [95% CI]
DS / Average	Science	$-0.0052\,[-0.0182,+0.0079]$	$+0.0000\,[-0.0156,+0.0156]$
	Mathematics	$-0.0122\,[-0.0259,+0.0020]$	$-0.0234\,[-0.0547,+0.0078]$
	Computer science	$-0.0005\,[-0.0146,+0.0132]$	$+0.0234\,[-0.0117,+0.0586]$
DS / Task Arithmetic	Science	$+0.0021\,[-0.0118,+0.0170]$	$+0.0078\,[-0.0078,+0.0234]$
	Mathematics	$-0.0107\,[-0.0259,+0.0044]$	$-0.0273\,[-0.0664,+0.0117]$
	Computer science	$-0.0005\,[-0.0161,+0.0146]$	$+0.0312\,[-0.0039,+0.0664]$

Appendix

Table A5: Source-route task effects on 256 items per row. Margin and accuracy differences favor replay when positive; accuracy is a fraction. Intervals use paired item bootstrap. DS/OL denote DeepSeekMoE/OLMoE.

Parent	Distance	Spearman $\rho$ [95% CI]	AUROC [95% CI]
Average	Full-routing JS	$-.036\,[-.112,.038]$	$.474\,[.431,.518]$
	Top- $k$ set	$.015\,[-.061,.092]$	$.506\,[.473,.540]$
	Selected-weight JS	$-.022\,[-.099,.058]$	$.482\,[.437,.530]$
Task Arithmetic	Full-routing JS	$.028\,[-.045,.103]$	$.514\,[.475,.553]$
	Top- $k$ set	$-.009\,[-.081,.058]$	$.500\,[.469,.530]$
	Selected-weight JS	$.037\,[-.039,.108]$	$.522\,[.483,.561]$

Appendix

Table A6: Qwen3 structural diagnostics on changed-set events. Intervals are pointwise 95% prompt-bootstrap intervals. Selected-weight JS aligns weights on the union of selected experts.

Quantity	Average	Task Arithmetic
Overall flip rate	$26.46\,[25.01,27.96]$	$33.20\,[31.48,34.94]$
Representation-induced	$96.87\,[96.75,96.98]$	$96.90\,[96.80,97.01]$
Router-parameter-induced	$0.034\,[0.027,0.041]$	$0.040\,[0.033,0.047]$
Interaction-required	$0.929\,[0.873,0.989]$	$0.774\,[0.728,0.823]$
Redundantly sufficient	$2.171\,[2.086,2.259]$	$2.282\,[2.207,2.357]$

Appendix

Table A7: Qwen3 routing attribution (%; 95% prompt-bootstrap intervals). The first row includes all events; remaining rows condition on a source–merged set change.

Parent	Margin $\Delta$ [95% CI]	Accuracy $\Delta$ [95% CI]
Average	$+.00880\,[-.01196,+.02822]$	$-.015625\,[-.046875,0]$
Task Arithmetic	$+.00180\,[-.01685,+.01968]$	$-.015625\,[-.046875,0]$

Appendix

Table A8: Qwen3 source-route replay on 64 ARC-Challenge items. Positive changes favor replay; accuracy changes are fractions. Intervals use paired item bootstrap.

Model / Parent	Source wins [95% CI]	Ties	Merged best [95% CI]	Candidates, mean [range]
DS / Average	$49.6\,[43.4,55.9]\%$	$3.1\%$	$10.9\,[7.0,14.8]\%$	$9.34\,[7,12]$
DS / Task Arithmetic	$48.8\,[43.0,55.1]\%$	$0.4\%$	$12.9\,[9.0,17.2]\%$	$9.69\,[7,12]$
OL / Average	$50.0\,[43.8,56.2]\%$	$0.8\%$	$13.3\,[9.4,17.6]\%$	$9.31\,[7,12]$
OL / Task Arithmetic	$45.7\,[39.8,52.0]\%$	$2.0\%$	$9.8\,[6.2,13.7]\%$	$9.55\,[7,12]$

Appendix

Table A9: NLL reference quality within the fixed local candidate set. Source wins are strict; merged-best counts admit ties within

10^{-6}

. Intervals resample prompts.

Model / Parent	Events	Best expert pair	Source-route mixture	One-swap mixture
DS / Average	115	$0.050\,[0.035,0.067]$	$0.913\,[0.887,0.936]$	$0.951\,[0.936,0.964]$
DS / Task Arithmetic	149	$0.043\,[0.031,0.056]$	$0.896\,[0.867,0.923]$	$0.962\,[0.950,0.972]$
OL / Average	112	$0.074\,[0.050,0.101]$	$0.976\,[0.969,0.982]$	$0.990\,[0.988,0.992]$
OL / Task Arithmetic	142	$0.084\,[0.062,0.108]$	$0.965\,[0.956,0.972]$	$0.987\,[0.984,0.989]$
QW / Average	91	$0.041\,[0.019,0.065]$	$0.953\,[0.936,0.967]$	$0.980\,[0.974,0.984]$
QW / Task Arithmetic	118	$0.096\,[0.066,0.132]$	$0.957\,[0.940,0.970]$	$0.975\,[0.964,0.983]$

Appendix

Table A10: Directional mixture comparisons on matched changed-route events. Values are mean cosines with 95% prompt-bootstrap intervals. DS/OL/QW denote DeepSeekMoE/OLMoE/Qwen3-MoE.

Architecture	Average	Task Arithmetic
DeepSeekMoE	$0.997\,[0.994,0.999]$	$0.998\,[0.997,0.999]$
OLMoE	$0.997\,[0.996,0.998]$	$0.996\,[0.995,0.997]$
Qwen3-MoE	$1.000\,[0.999,1.000]$	$1.000\,[0.999,1.000]$

Appendix

Table A11: Mixture cosine on unchanged-route reference events. These events are complementary to, not matched replicates of, the changed-route sample. Values are means with 95% prompt-bootstrap intervals.

Architecture / Parent	Observed source	Matched random	Difference [95% CI]
DeepSeekMoE / Average	0.902	0.769	$+0.133\,[+0.109,+0.158]$
DeepSeekMoE / Task Arithmetic	0.895	0.748	$+0.147\,[+0.127,+0.168]$
OLMoE / Average	0.976	0.855	$+0.122\,[+0.102,+0.143]$
OLMoE / Task Arithmetic	0.966	0.854	$+0.112\,[+0.095,+0.129]$
Qwen3-MoE / Average	0.953	0.840	$+0.113\,[+0.099,+0.128]$
Qwen3-MoE / Task Arithmetic	0.957	0.841	$+0.116\,[+0.102,+0.131]$

Appendix

Table A12: Observed source routing versus 32 matched random alternatives per event. Intervals apply to the paired observed-minus-random cosine, not separately to the absolute means.

Contrast	DeepSeekMoE	OLMoE
Source–Average	$0.149\,[0.138,0.160]$	$0.114\,[0.106,0.122]$
Source–TA	$0.209\,[0.195,0.224]$	$0.159\,[0.150,0.168]$
Base–source	$0.271\,[0.255,0.287]$	$0.215\,[0.204,0.226]$
Source–source router only	$0.058\,[0.053,0.063]$	$0.022\,[0.019,0.024]$

Appendix

Table A13: Set-distance controls outside merging, with 95% prompt-bootstrap intervals.

Injected corruption $\rightarrow$ clean-native routing replay
Parent	Accuracy change [interval]	Margin change [interval]
Average	$+28.125\;[+20.313,\,+35.547]$	$+0.7677\;[+0.5411,\,+0.9934]$
Task Arithmetic	$+28.125\;[+21.094,\,+35.156]$	$+0.5588\;[+0.3675,\,+0.7549]$
Native merged routing $\rightarrow$ frozen LC routing replay
Average	$-0.391\;[-1.172,\,0.000]$	$-0.0003\;[-0.0037,\,+0.0032]$
Task Arithmetic	$+0.391\;[0.000,\,+1.172]$	$-0.0026\;[-0.0066,\,+0.0008]$

Appendix

Table C14: Original OLMoE replay controls. Accuracy changes are percentage points; margins retain score units. Brackets are 95% paired item-bootstrap intervals. The injected and LC comparisons are distinct experiments.

Parent	Permutation seed	$k$	Corrupted	Replay	Recovery [95% CI]	Holm $p<.05$
Average	17017	1	53.12	50.78	$-2.34\;[-6.25,+1.56]$	No
		4	38.28	50.78	$+12.50\;[+7.03,+18.36]$	Yes
		16	17.97	50.78	$+32.81\;[+25.39,+40.23]$	Yes
	29029	1	49.22	50.39	$+1.17\;[-3.52,+5.86]$	No
		4	37.89	50.78	$+12.89\;[+6.64,+18.76]$	Yes
		16	23.44	50.78	$+27.34\;[+19.92,+34.77]$	Yes

Appendix

Table C15: Graded OLMoE corruption on 256 items per parent. Accuracy is in %; recovery and its paired 95% interval are in pp. Clean accuracy is 50.78%. Holm decisions cover all twelve parent–seed–scope comparisons.

Figure C8: Graded routing interventions. (a) Clean, corrupted, and replay accuracy, averaging two fixed permutations per parent. (b) All twelve paired recovery effects with 95% item-bootstrap intervals. (c) Corrections and regressions, averaged within parent and scope. (d) Independent natural LC effects. Seeds are not model replicates.

Quantity	Value	Quantity	Value
Likelihood temperature $\tau$	$0.5$	Relative ridge $\gamma$	$10^{-3}$
Sign consistency $\kappa$	$0.75$	Update scale $\eta$	$0.0625$
Minimum activity $a_{\min}$	$0.002$	Profile/activity floor	$10^{-8}$
Pairs per domain/layer	$2$	Ridge energy floor	$10^{-12}$
Eligible router layers	Final $5$	PCG iteration limit	$4096$
True-residual tolerance	$10^{-6}$

Appendix

Table D16: Fixed SRR settings. Pair quotas apply to each domain at each eligible layer.

Cell	Weighted positive	Uniform positive	All uniform	Shuffled positive
DS / Average	.01774	.01682	.01624	.01761
DS / TA	.05171	.05065	.04973	.05406
OL / Average	.06645	.05756	.05471	.06211
OL / TA	.11473	.10365	.09864	.10738

Appendix

Table D17: Held-out residual-fitting MSE in a separate weighting ablation; lower is better. These point estimates are not task scores and have no paired uncertainty. DS/OL denote DeepSeekMoE/OLMoE.

Positive support
Setting	$n$	Source $U$	Source $D$	Fitted $U$	Fitted $D$
OL / Average	101	$+1.38$ [-1pt] $[-2.18,+5.14]$	$+2.77$ [-1pt] $[-4.52,+10.60]$	$+2.08$ [-1pt] $[-1.40,+5.87]$	$+4.16$ [-1pt] $[-3.27,+12.07]$
OL / TA	122	$-0.88$ [-1pt] $[-6.01,+4.52]$	$-4.59$ [-1pt] $[-14.68,+7.04]$	$+0.69$ [-1pt] $[-4.29,+6.05]$	$-1.46$ [-1pt] $[-11.29,+10.24]$
QW / Average	117	$-0.56$ [-1pt] $[-2.21,+0.95]$	$-1.44$ [-1pt] $[-3.31,-0.08]$	$-0.56$ [-1pt] $[-2.19,+0.94]$	$-1.44$ [-1pt] $[-3.26,-0.06]$
QW / TA	131	$+0.49$ [-1pt] $[-3.51,+4.58]$	$+1.44$ [-1pt] $[-3.95,+7.34]$	$+0.49$ [-1pt] $[-3.48,+4.71]$	$+1.44$ [-1pt] $[-3.85,+7.45]$
Pooled	471	$+0.11$ [-1pt] $[-1.77,+2.05]$	$-0.45$ [-1pt] $[-3.86,+3.37]$	$+0.67$ [-1pt] $[-1.24,+2.62]$	$+0.67$ [-1pt] $[-2.82,+4.46]$

Appendix

Table E18: Local intervention gains by support group. Entries are estimates and 95% prompt-bootstrap intervals in

10^{-3}

nats/token.

U

compares the proposed direction with native routing;

D

compares it with the opposite direction. Support groups are reported separately, not pooled as independent samples.

Figure E9: Local source-supervision diagnostics. (a) Likelihood advantage across 1,024 events (asinh-spaced axis; labels in nats/token). (b) Directional contrast

D

by support group. (c) Positive-support and matched-control ECDFs. (d) Local and matched effects with prompt-bootstrap intervals. These are fixed-continuation NLL measurements.

Figure F10: Fitted and executed corrections. (a) Native expert-set changes. (b) Direct versus executed pair gaps on 5,120 events. (c) Centered-logit RMS, normalized by each parent before equal-setting averaging. (d) Identity-matched task differences. RMS components are not additive; the task association is descriptive.

Merge	Layer	Direct [95% CI]	Input response [95% CI]	Executed [95% CI]
Avg	11	$0.0135\;[0.0116,0.0153]$	$0\;[0,0]$	$0.0135\;[0.0116,0.0153]$
	12	$0.013\;[0.0113,0.0147]$	$0.00341\;[0.00272,0.00415]$	$0.0135\;[0.0118,0.015]$
	13	$0.0138\;[0.0121,0.0155]$	$0.00587\;[0.00463,0.00744]$	$0.0148\;[0.0132,0.0164]$
	14	$0.0151\;[0.0134,0.0168]$	$0.00835\;[0.00613,0.0109]$	$0.0189\;[0.0159,0.0222]$
	15	$0.0196\;[0.0165,0.0233]$	$0.0077\;[0.00648,0.00896]$	$0.0221\;[0.0187,0.0258]$
TA	11	$0.0151\;[0.0127,0.0176]$	$0\;[0,0]$	$0.0151\;[0.0127,0.0176]$

Appendix

Table F19: OLMoE SRR pair-gap decomposition. Each cell is a pair-logit-gap RMS with a pointwise 95% prompt-bootstrap interval, conditional on its fixed Parent/SRR checkpoints. RMS terms do not add.

Merge	Layer	Direct [95% CI]	Input response [95% CI]	Executed [95% CI]
Avg	43	$0.0323\;[0.0279,0.0365]$	$0\;[0,0]$	$0.0323\;[0.0279,0.0365]$
	44	$0.0357\;[0.0303,0.0408]$	$0.00439\;[0.00319,0.00569]$	$0.0358\;[0.0304,0.0408]$
	45	$0.0272\;[0.0213,0.0334]$	$0.00511\;[0.00415,0.00604]$	$0.0272\;[0.021,0.0339]$
	46	$0.0266\;[0.0233,0.0298]$	$0.0233\;[0.00859,0.037]$	$0.035\;[0.0265,0.0452]$
	47	$0.0332\;[0.0275,0.0388]$	$0.0204\;[0.0134,0.0268]$	$0.0429\;[0.0339,0.0517]$
TA	43	$0.0353\;[0.0303,0.0402]$	$0\;[0,0]$	$0.0353\;[0.0303,0.0402]$

Appendix

Table F20: Qwen3-MoE SRR pair-gap decomposition. Each cell is a pair-logit-gap RMS with a pointwise 95% prompt-bootstrap interval, conditional on its fixed Parent/SRR checkpoints. RMS terms do not add.

Merge	Layer	Cross	Top- $k$ change (%)	Weight $\ell_{1}$ /2	$\\|\Delta h\\|/\\|h\\|$	Native–FP64
Avg	11	0	0.781	0.00042	0	0.0185
	12	-1.8e-07	0.781	0.000836	0.00363	0.0211
	13	-6.54e-08	3.91	0.00198	0.00643	0.0247
	14	3.36e-07	2.34	0.00173	0.00947	0.0208
	15	-1.13e-07	4.69	0.00255	0.0112	0.0291
TA	11	0	1.56	0.000725	0	0.0207

Appendix

Table F21: OLMoE SRR routing and numerical diagnostics. One hash-selected event per prompt and updated layer. Cross is the mean centered-logit direct–response inner product. The native–FP64 difference is the largest absolute BF16-forward versus reconstructed-logit discrepancy; algebra residual is normalized. Route changes and dispatch-weight distance use native routing.

Merge	Layer	Cross	Top- $k$ change (%)	Weight $\ell_{1}$ /2	$\\|\Delta h\\|/\\|h\\|$	Native–FP64	Algebra resid.
Avg	43	0	3.91	0.00558	0	0.0451	0
	44	-7.73e-08	7.03	0.00772	0.00338	0.0526	6.87e-18
	45	-4.01e-07	7.81	0.00846	0.00635	0.0476	0
	46	-2.88e-07	3.12	0.00635	0.00871	0.0543	0
	47	-2.05e-07	7.03	0.00794	0.00917	0.0613	0
TA	43	0	3.91	0.00417	0	0.0434	0

Appendix

Table F22: Qwen3-MoE SRR routing and numerical diagnostics. One hash-selected event per prompt and updated layer. Cross is the mean centered-logit direct–response inner product. The native–FP64 difference is the largest absolute BF16-forward versus reconstructed-logit discrepancy; algebra residual is normalized. Route changes and dispatch-weight distance use native routing.

Benchmark	Items	Scoring field
MMLU	14,042	acc,none
HellaSwag	10,042	acc_norm,none
ARC-Challenge	1,172	acc_norm,none
ARC-Easy	2,376	acc_norm,none
PIQA	1,838	acc_norm,none
WinoGrande	1,267	acc,none

Appendix

Table G23: Evaluation items and scoring fields. Task scores receive equal weight.

Expert Routing

Expert routing in mixture-of-experts models selects which subnetworks process each input, balancing accuracy, computation, and expert workload.

See all 119 papers

Jun 2, 2026

When Model Merging Breaks Routing: Training-Free Calibration for MoE

Model merging has emerged as a cost-effective approach for consolidating the capabilities of multiple LLMs without retraining. However, existing merging techniques, largely based on linear parameter arithmetic or optimization, struggle when applied to Mixture-of-Experts (MoE) architectures. We identify a critical failure mode in MoE merging, termed routing breakdown, in which the merged router fails to dispatch tokens to suitable experts. Routing breakdown stems from the sensitivity of the non-linear softmax and discrete Top-k routing mechanisms to parameter perturbations from merging, a sensitivity further amplified by load-balancing constraints imposed during MoE pretraining. Because fine-tuned experts exhibit distinct specializations, even modest misrouting can cause severe performance degradation. To address this issue, we propose Hessian-Aware Router Calibration (HARC), a training-free framework that leverages second-order curvature information to realign the merged router. This approach admits a closed-form solution that can be efficiently solved using a matrix-free conjugate gradient method. Experiments on mathematical reasoning and code generation tasks show that HARC effectively mitigates routing breakdown across diverse MoE merging baselines and leads to substantial performance improvements. Our code is available at https://github.com/huangcb01/HARC.

Canbin Huang, Tianyuan Shi, Xiaojun Quan +3

School of Computer Science and Engineering, Sun Yat-sen University, China · Shenzhen Loop Area Institute, China · Meituan, Inc., China +1

Mixture of Experts LLM Routing Language Model Merging Mixture-of-Experts Models

Ask PDF

May 8, 2026

When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models

Mixture-of-Experts (MoE) language models route each token to a small subset of experts, but whether the routes selected by a trained top-

k

router are good ones is rarely evaluated directly. Holding the model fixed, we compare each standard route against sampled equal-compute alternatives for the same token and score each by the next-token probability it assigns to the realized token in a verified reasoning trajectory. The result is sharply token-conditional: the standard router is well-aligned with route utility on confident tokens but uninformative on the fragile tokens that drive hard reasoning, where lower-loss equal-compute routes consistently exist inside the frozen model but are not selected. The same pattern holds across Qwen3-30B-A3B, GPT-OSS-20B, DeepSeek-V2-Lite, and OLMoE-1B-7B, and follows structurally from how standard top-

k

training evaluates routing decisions: the language modeling loss scores only the executed route, and load balancing depends only on aggregate routing statistics. A minimal router-only update to the final-layer router, leaving every expert and every other router frozen, is sufficient to shift pass@K on AIME 2024+2025 and HMMT 2025 for both Qwen3-30B-A3B and GPT-OSS-20B, suggesting that at least part of the failure reflects router-reachable misallocation rather than expert capacity alone.

Youngsik Yoon, Siwei Wang, Wei Chen +1

Department of Computer Science and Engineering, POSTECH, South Korea · Microsoft Research Asia, Beijing, China · Graduate School of Artificial Intelligence, POSTECH, South Korea

Mixture-of-Experts Language Models LLM Routing Mixture-of-Experts Models

Ask PDF

May 29, 2026

PR2: Predictive Routing Replay for MoE-Based LLM Reinforcement Learning

Mixture of Experts (MoE) Large Language Models (LLMs) achieve strong performance at scale. However, reinforcement learning (RL) on MoE-based LLMs often suffers from training instability. A root cause is router drift, i.e., expert activations can change drastically across model updates and differ between disaggregated rollout and training phases, causing large rollout--training mismatch and unstable importance sampling weights in PPO-style RL algorithms. Routing replay mitigates this issue by freezing the replay route within each reasoning trajectory, but it ignores how the router evolves under off-policy updates and thus causes router staleness. To address this limitation, we propose Predictive Routing Replay (PR2), which augments each router with a lightweight evolution predictor that learns to anticipate short-horizon router evolution. During the rollout phase, we use the predictive routing distribution to apply top-

k

routing, enabling gradients to reach experts that are likely to become active after updates. During the training phase, we replay the resulting predicted route to retain consistency for stable importance estimation. Theoretical analysis and experiments support that PR2 reduces routing-induced mismatch, improves RL stability, and yields stronger performance across various reasoning benchmarks.

Daize Dong, Junlin Chen, Haolong Jia +9

♣Rutgers University · ♦AMD · ⋆MBZUAI

RL for Language Model Reasoning Expert Routing LLM Routing RL Post-Training Off-Policy RL

Ask PDF

Abstract

Expert Routing

Explore similar work

When Model Merging Breaks Routing: Training-Free Calibration for MoE

When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models

PR2: Predictive Routing Replay for MoE-Based LLM Reinforcement Learning