Routing Divergence Is Not Evidence of Behavioral Influence in Same-Weight MoE Self-Distillation

Cedric Caruzzo; Donggeun Yoo; Tae Soo Kim

cs.LGAug 16, 2026

Routing Divergence Is Not Evidence of Behavioral Influence in Same-Weight MoE Self-Distillation

Authors: Cedric Caruzzo, Donggeun Yoo, Tae Soo Kim

Organizations: Lunit

Abstract

Two Mixture-of-Experts (MoE) forward passes can share every weight yet route the same token through different experts. This creates a possible blind spot in same-weight self-distillation, where a demonstration-conditioned teacher supervises a query-only student. We study this mismatch in its single-step form, with frozen weights rather than as a proxy for a full training trajectory. An exact blockwise decomposition separates a routing term, which changes gates at fixed content, from a dense-like content term. Across seven open-weight checkpoints and two domains, the routing term spans only $1.6\times$ as a fraction of block output, while its residual-stream exposure spans $3.2\times$ . Exposure is ordered by the routed block's share of the residual. Scaling the always-on backbone in two confirmatory models moves exposure monotonically; common-mode controls support a mass-and-coherence mechanism rather than denominator dilution alone. Preregistered PubMedQA patches on three models show that the full routing term moves outputs by less than half the natural context effect and is largely reproduced by matched-norm noise, whereas the content term is strongly direction-specific. Scale and merged-expert probes show that the narrow block-level range is not universal, although exposure remains small at the tested boundaries. Router movement alone is therefore not evidence of behavioral influence: measure exposure first, and use a behavioral intervention when the decision matters.

Figures & tables

Panel of Figure 1 — Figure 1: The decomposition and its causal chain. Top: two forward passes of the same model with the same weights, one with a demonstration in context (teacher) and one without (student), produce different gate vectors and hidden states at each MoE block. The change in the routed output $\Delta_{\mathrm{routed}}=A+B$ splits into a routing-mismatch term $A$ (the same content $h^{S}$ sent through the two passes’ different gates) and a dense-like remainder $B$ (the teacher routing applied to the content shift). This paper measures $A$ and its share of the residual stream (Eq. 4 ). Bottom: the routing factor varies less than exposure in the primary screen, while backbone share maps the block-level perturbation into residual exposure. Neither the observed range nor exposure alone is a behavioral guarantee.

Model	experts/ $k$	shared path	$r_{A/\mathrm{blk}}$	$r_{\mathrm{blk}/\mathrm{res}}$	$r_{A/\mathrm{res}}$
Qwen3-30B-A3B	128/8	–	0.133/0.139	0.174/0.178	0.019/0.021
Mixtral-8x7B	8/2	–	0.094/0.110	0.271/0.283	0.022/0.028
OLMoE-1B-7B	64/8	–	0.124/0.122	0.481/0.515	0.056/0.061
DeepSeek-V2-Lite	64/6	always-on	0.090/0.102	0.356/0.364	0.027/0.033
Gravity-16B-A3B	64/8	always-on	0.098/0.108	0.470/0.511	0.044/0.053
Qwen1.5-MoE-A2.7B	60/4	input-gated	0.085/0.102	0.464/0.500	0.036/0.048

Table 1: The seven primary checkpoints. Values are per-layer-first body aggregates, shown as PubMedQA/GSM8K. Backbone share is computed directly as

\lVert y\rVert/\lVert\mathrm{res}\rVert

before averaging across layers; exposure uses the pre-block residual. Prompt-bootstrap intervals are in Appendix B .

Figure 2: Routing factor

r_{A/\mathrm{blk}}

for the seven primary checkpoints on PubMedQA (circles) and GSM8K (squares). Whiskers are

95\%

confidence intervals from a bootstrap over the

200

prompts. The precise checkpoint estimates occupy the observed

0.085

–

0.139

range despite substantial differences in expert count, top-

k

, shared paths, and gate conventions. The shaded range is a description of these checkpoints, not a universal confidence band.

Figure 3: Exposure

r_{A/\mathrm{res}}

against directly aggregated backbone share

r_{\mathrm{blk}/\mathrm{res}}

on PubMedQA (circles) and GSM8K (squares). Each model’s domain points are connected. The plot uses

\operatorname{mean}_{\ell}(\sum\lVert y\rVert/\sum\lVert\mathrm{res}\rVert)

rather than deriving the horizontal coordinate from exposure, so the association is descriptive rather than an algebraic restatement of Eq. 4 . Qwen3 is least exposed and OLMoE most exposed; the causal test of the backbone lever is Section 5 .

Figure 4: Backbone rescaling on the two confirmatory checkpoints: DeepSeek-V2-Lite (top) and Gravity (bottom). Left: post-block exposure against the frozen-numerator null. Center: routing-coherence response under the mass-removal, matched-norm, and content-replacement controls at matched input drift. Right: top-

k

Jaccard and gate TV across the dose trajectory. Full removal changes routing substantially; the claim rests on the graded response and control ordering.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

	PubMedQA		GSM8K
Model	$r_{A/\mathrm{blk}}$ CI	$r_{A/\mathrm{res}}$ CI	$r_{A/\mathrm{blk}}$ CI	$r_{A/\mathrm{res}}$ CI
Qwen3-30B-A3B	[0.131, 0.135]	[0.019, 0.020]	[0.137, 0.141]	[0.021, 0.021]
Mixtral-8x7B	[0.091, 0.096]	[0.021, 0.023]	[0.108, 0.111]	[0.028, 0.029]
OLMoE-1B-7B	[0.121, 0.127]	[0.055, 0.058]	[0.120, 0.125]	[0.060, 0.062]
DeepSeek-V2-Lite	[0.088, 0.092]	[0.026, 0.027]	[0.100, 0.104]	[0.032, 0.033]
Gravity-16B-A3B	[0.097, 0.099]	[0.044, 0.045]	[0.107, 0.109]	[0.052, 0.053]

Appendix

Table 2: Prompt-level

95\%

bootstrap CIs (resampling the

n{=}200

probe prompts,

2000

resamples) for the two headline ratios, both domains. These intervals quantify prompt-sampling uncertainty for each checkpoint rather than variation across checkpoints.

Checkpoint	$r_{A/\mathrm{blk}}$	$r_{A/\mathrm{res}}$
Qwen3-30B-A3B	0.133/0.139	0.019/0.021
Qwen3-235B-A22B	0.188/0.200	0.030/0.030
DeepSeek-V2-Lite	0.090/0.102	0.027/0.033
DeepSeek-V2 (236B)	0.114/0.139	0.012/0.019
Beyonder-4x7B-v3	0.014/0.011	0.006/0.005

Appendix

Table 3: Boundary checkpoints, shown as PubMedQA/GSM8K. Family pairs are descriptive scale probes; the merged checkpoint tests transfer outside jointly trained expert families. Exposure uses the pre-block screen convention.

Model	$\alpha$	$r_{A/\mathrm{res}}$ post	null	nA	Jaccard
DeepSeek-V2-Lite	1.00	0.0245	0.0245	1.000	1.000
	0.75	0.0278	0.0247	1.095	0.813
	0.50	0.0315	0.0249	1.209	0.690
	0.25	0.0375	0.0250	1.407	0.578
	0.00	0.0395	0.0250	1.567	0.438
Gravity	1.00	0.0403	0.0403	1.000	1.000

Appendix

Table 4: PubMedQA backbone-rescaling dose-response. The null freezes the routing numerator; nA is

\lVert A(\alpha)\rVert/\lVert A(1)\rVert

; Jaccard is routing overlap with

\alpha=1

. DeepSeek-V2-Lite uses

n=200

probe pairs and Gravity

n=100

Model	arm	[0.00,0.15)	[0.15,0.30)	[0.30,0.50)	[0.50,0.90)
DeepSeek-V2-Lite	content replaced	1.018	0.938	0.863	0.958
	matched noise	1.043	1.109	1.124	1.387
	mass removed	1.059	1.129	1.226	1.546
Gravity	content replaced	0.990	1.034	1.099	1.233
	matched noise	0.992	1.060	1.138	1.384
	mass removed	0.994	1.069	1.186	1.480

Appendix

Table 5: PubMedQA matched-drift control ladder. Values are mean routing-coherence response nA by input-drift bin, pooled over intervening doses. At high drift, preserving shared-path mass while replacing content produces a smaller response than matched-norm noise or mass removal.

Model	$\alpha$	$r_{A/\mathrm{res}}$ post	null	nA	Jaccard	ladder [0.50,0.90)
DeepSeek-V2-Lite	1.00	0.0303	0.0303	1.000	1.000
	0.75	0.0364	0.0307	1.153	0.803
	0.50	0.0412	0.0309	1.307	0.678	1.11 / 1.52 / 1.62
	0.25	0.0502	0.0310	1.532	0.566
	0.00	0.0521	0.0310	1.715	0.420
Gravity	1.00	0.0495	0.0495	1.000	1.000

Appendix

Table 6: GSM8K dose-response and the matched-drift ladder’s highest bin. The ladder entries are content replacement / matched noise / mass removal.

Domain	screen factor	$\alpha=1$	0.75	0.50	0.25	0
PubMedQA	0.052	0.0241	0.0297	0.0443	0.0819	0.0520
GSM8K	0.059	0.0280	0.0356	0.0583	0.1156	0.0850

Appendix

Table 7: Moonlight stopped extension. The screen factor is pre-block (

n=200

); the dose trajectory is post-block exposure (

n=100

Mixture-of-Experts Models

Mixture-of-experts models use input-dependent routing to activate specialized subnetworks, increasing model capacity without running every parameter for each input.

See all 454 papers

Routing-Free Mixture-of-Experts
Apr 1, 2026Yilun Liu, Jinru Han, Sikuan Yan +2Mixture-of-Experts Models Expert Load Balancing
Ludwig Maximilian University of Munich · Munich Center for Machine Learning · University of California, Los Angeles
Self-Routing: Parameter-Free Expert Routing from Hidden States
Apr 1, 2026Jama Hussein Mohamud, Drew Wagner, Mirco RavanelliParameter-Free Mixture-of-Experts Routing LLM Routing
Mila – Quebec AI Institute · Universite de Montreal · Concordia University
A Persistent State for Auditable Mixture-of-Experts Routing
Sep 28, 2026Abdurrahman Javat, Allan KazakovLLM Auditing State Tracking
Department of Artificial Intelligence Bahçeşehir University Istanbul, Türkiye

Apr 1, 2026J/K move · Enter open · S save

Routing-Free Mixture-of-Experts

Yilun Liu, Jinru Han, Sikuan Yan, Volker Tresp+1

Ludwig Maximilian University of Munich · Munich Center for Machine Learning · University of California, Los Angeles

Mixture-of-Experts Models Expert Load Balancing

Open paper AskPDF arXiv

Standard Mixture-of-Experts (MoE) models rely on centralized routing mechanisms that introduce rigid inductive biases. We propose Routing-Free MoE which eliminates any hard-coded centralized designs including external routers, Softmax, Top-K and load balancing, instead encapsulating all activation functionalities within individual experts and directly optimized through continuous gradient flow, enabling each expert to determine its activation entirely on its own. We introduce a unified adaptive load-balancing framework to simultaneously optimize both expert-balancing and token-balancing objectives through a configurable interpolation, allowing flexible and customizable resource allocation. Extensive experiments show that Routing-Free MoE can consistently outperform baselines with better scalability and robustness. We analyze its behavior in detail and offer insights that may facilitate future MoE design ad optimization.

Abstract

Mixture-of-Experts Models

Explore similar work

Routing-Free Mixture-of-Experts

Self-Routing: Parameter-Free Expert Routing from Hidden States

A Persistent State for Auditable Mixture-of-Experts Routing