Equifinality in Mixture of Experts: Routing Topology Does Not Determine Language Modeling Quality

Ivan Ternovtsii; Yurii Bilak

cs.AIApr 15, 2026

Equifinality in Mixture of Experts: Routing Topology Does Not Determine Language Modeling Quality

Authors: Ivan Ternovtsii, Yurii Bilak

Organizations: Department of Software Systems, Uzhhorod National University Narodna sq. 3, Uzhhorod, Ukraine, 88000 · HengeBytes

Abstract

Sparse Mixture-of-Experts (MoE) architectures employ increasingly sophisticated routing mechanisms -- learned routers, multi-hop trajectories, token-dependent gating. We ask: does routing topology actually determine language modeling quality? We build a geometric MoE (ST-MoE) using cosine-similarity routing against learned centroids in a low-dimensional space ( $d_{space} = 64$ ), requiring 80% fewer routing parameters than standard linear routers. Through 62 controlled experiments on WikiText-103 at 76--84M parameters trained to convergence (50K steps, 1.64B tokens), we find that routing topology does not determine asymptotic perplexity (PPL): five cosine-routing variants are statistically equivalent within a 1-PPL margin (Two One-Sided Tests [TOST], $p < 0.05$ for all 10 pairwise comparisons; 15 runs across 3 seeds, observed range 33.93--34.72). The finding extends to hash, random-fixed, and top-1 routing (single-seed; graceful 1.1--2.2 PPL degradation) and replicates on OpenWebText (0.03 PPL gap, 6 runs, 3 seeds each). A standard linear router with 5.3 $\times$ more routing parameters reaches PPL 32.76, but iso-parameter cosine routing closes 67% of this gap -- the true mechanism advantage is $\sim$ 1.2%. The mechanistic explanation is convergent redundancy: multi-hop updates are collinear ( $\cos(Δh_0, Δh_1) = 0.805$ ), implementing magnitude amplification rather than compositional reasoning; a single learnable scalar replicates multi-hop performance. As a practical payoff, zero-shot relative-norm halting saves 25% of MoE FLOPs at +0.12% PPL. Expert-level specialization and causal controllability -- which coexist with topology-level equifinality -- are explored in a companion paper.

Figures & tables

Figure 1: ST-MoE Architecture. (A) Full model: token embedding

\to

N

blocks

\to

LM head. (B) Single block: pre-LN attention with RoPE, followed by pre-LN ST-MoE layer, both with residual connections. (C) Multi-hop routing: tokens are projected into a

d_{\text{space}}

-dimensional coordinate space, routed to top-

K

experts via cosine similarity with learned centroids, and accumulate expert updates across

H

hops with semantic position updates between hops.

Scale	$d_{\text{model}}$	Heads	Layers	$M$	Steps	Tokens	Params
Small (36M)	768	12	4	512	5,000	82M	37.6M
Large (76M)	1024	16	6	1024	20,000	328M	76–84M
Marathon (76M)	1024	16	8	1024	50,000	1.64B	76–84M

Table 1: Model configurations used throughout the paper, varying scale, depth, expert count, and training duration.

Small active budget (12 neurons)
Model	Active/Token	PPL	vs Dense
Dense d_ff=12	12	306.01	baseline
Singleton MoE ( $3\times 4\times 1$ )	12	304.50	$-0.5\%$
Medium active budget (48 neurons)
Dense d_ff=48	48	302.24	baseline
Block MoE Wide ( $1\times 6\times 8$ )	48	298.91	$\mathbf{-1.1\%}$

Table 2: MoE vs Dense at iso-FLOP (same active computation per token).

Figure 2: Dense vs MoE Training Comparison. The iso-FLOP dense baseline (

d_{\text{ff}}{=}12

, same active FLOPs) tracks MoE throughout training; the iso-parameter dense baseline (

d_{\text{ff}}{=}1120

{\sim}93\times

more active FLOPs) dominates after step 5K. MoE crosses the iso-FLOP dense at

{\sim}

10K steps, confirming that sparse routing provides value at matched active computation.

Figure 3: Convergence Marathon. Training curves for all configurations at 76M parameters, 50K steps (1.64B tokens). Among cosine-routing variants, all topologies converge to a statistically equivalent band (TOST,

\delta{=}1

PPL, all pairs

p<0.05

; observed range 33.93–34.72 across 15 runs). The standard linear router (dashed cyan) falls outside this band at PPL 32.76, demonstrating that routing capacity matters even when routing topology does not.

Figure 4: Multi-Seed Validation. Training curves for Wide 1

\times

12 and Deep 3

\times

4 with seeds 42 and 137. Same-topology curves track nearly identically throughout training: Wide

\Delta{=}1.2\%

, Deep

\Delta{=}0.1\%

. All four runs converge within a 0.74 PPL band. Full 5-variant

\times

3-seed validation (15 runs total) confirms all topologies converge within a 0.79 PPL band (33.93–34.72), with inter-variant spread (

0.60

PPL) only

5.0\times

the average seed noise (

0.12

PPL avg std).

Model	Architecture	Final PPL	Mean $\pm$ Std (3 seeds)	vs iso-FLOP Dense	Routing Params	Train Time
Baselines
Dense iso-FLOP	d_ff=12	34.49	—	—	—	3.5h
Dense iso-param	d_ff=1120 ( $93\times$ active)	29.36	—	$-14.9\%$	—	3.9h
ST-MoE Wide 1 $\times$ 12	cosine, 1 hop, top-12	33.93	$34.22\pm 0.25$	$-1.6\%$	1.57M	7.4h
ST-MoE Deep 3 $\times$ 4	cosine, 3 hops, top-4	34.62	$34.65\pm 0.03$	$+0.4\%$	1.57M	8.9h
Cosine Routing Interventions (Exps 033–035)

Table 3: Convergence results at 76–84.7M parameters, 1.64B tokens (RTX 3090). Among cosine-routing variants, all topologies are statistically equivalent within 1 PPL (TOST,

p<0.05

for all 10 pairwise comparisons). Mean

\pm

Std computed over 3 seeds (42, 137, 7) for each cosine variant; all 15 runs fall within a 0.79 PPL band. Broader topology conditions (Exp 065) and rank-16 experts (Exp 070) extend the comparison. MoE matches the iso-FLOP dense baseline (

d_{\text{ff}}{=}12

) while using the same active computation but

{\sim}93\times

more total parameters. The iso-parameter dense baseline (

d_{\text{ff}}{=}1120

93\times

more active FLOPs) outperforms all MoE variants, confirming MoE’s value is conditional computation, not per-parameter efficiency.

Checkpoint	Normal PPL	Frozen PPL	$\Delta$	Normal Jaccard
20K steps (underfitting)	268.60	286.76	$+6.8\%$	0.102
50K steps (converged)	34.62	37.26	$+7.6\%$	0.970

Table 4: Frozen routing at two training stages. At convergence, the model already selects 97% identical experts across hops, yet freezing still degrades PPL by 7.6%—re-routing provides a small but genuine correction.

Figure 5: Cross-Seed Expert Alignment. Left: vocabulary projection Jaccard similarity (best-match) is

{\sim}500\times

above random—experts develop partially overlapping functional roles across seeds. Right: raw

W_{\text{up}}

cosine similarity is indistinguishable from random—these overlapping functions are realized through entirely different weight parameterizations. This divergence is the quantitative signature of equifinality.

Figure 6: Pareto Frontier. (a) MLP experts enable a smooth quality-compute tradeoff; static experts show a sharp cliff. (b) Halting response: MLP average hops decrease gradually with

\varepsilon

, while static experts exhibit a phase transition.

$\varepsilon$	Avg Hops	FLOP Savings	$\Delta$ PPL
0.02	2.82	6.0%	$-0.08\%$
0.05	2.65	11.6%	$+0.00\%$
0.10	2.21	26.3%	$+0.14\%$

Table 5: MLP halting Pareto points (Exp 014). Zero-shot on pre-trained model.

$\varepsilon$	Avg Hops	FLOP Savings	$\Delta$ PPL
0.02	2.58	14.2%	$+0.00\%$
0.05	2.44	18.8%	$+0.03\%$
0.10	2.25	25.0%	$+0.12\%$
0.15	2.08	30.7%	$+0.43\%$

Table 6: Halting at convergence (Exp 044c). The converged model is equally amenable to halting.

Regime	$M$	Expert Size	Best Topology	PPL	vs Dense
Micro	512	1	Deep ( $3\times 4$ )	304.50	$-0.5\%$
Macro	64	8	Wide ( $1\times 6$ )	298.91	$\mathbf{-1.1\%}$

Table 7: Architecture Dualism: optimal topology depends on expert granularity.

Architecture	$M$	Expert Size	$\text{top-}k$ /hop	Per-hop Rank	PPL
Singleton Deep (Exp 012)	512	1	4	4	304.50
Granular Deep (Exp 022)	128	4	4	16	337.26

Table 8: Centroid density vs. per-hop rank: rank-4 with 512 centroids beats rank-16 with 128 centroids.

Architecture	$M$	es	Hops	$k$ /hop	Hop Rank	PPL
Multi-hop (trajectory navigation)
Singleton Deep (Exp 012)	512	1	3	4	4	304.50
Macro Deep (Exp 018)	64	8	3	2	16	337.02
Hybrid (Exp 021)	64	8	2	3	24	319.85
Granular Deep (Exp 022)	128	4	3	4	16	337.26
Single-hop (parallel selection)

Table 9: Complete hop-width ablation at iso-FLOP (48 active neurons).

Condition	PPL@3-hop	PPL@2-hop ( $\Delta$ )	PPL@1-hop ( $\Delta$ )	FLOPs at 1-hop
Baseline	302.54	312.92 (+3.4%)	366.48 (+21.1%)	$-67\%$
Curriculum	316.14	316.18 (+0.0%)	318.62 (+0.8%)	$-67\%$

Table 10: Trajectory Curriculum: the model becomes lazy (Exp 024).

Config	$K$	$\tau$	Best PPL	$\Delta$ vs baseline
Baseline	12	30	33.93	—
$K{=}12$ , $\tau{=}15$	12	15	34.06	$+0.13$
$K{=}24$ , $\tau{=}30$	24	30	34.12	$+0.19$
$K{=}24$ , $\tau{=}15$	24	15	33.85	$\mathbf{-0.08}$
$K{=}24$ , $\tau{=}\text{learned}$	24	11–15	34.16	$+0.23$
$K{=}48$ , $\tau{=}30$	48	30	33.92	$-0.01$

Table 11:

K\times\tau

ablation (Exp 051). Best PPL in bold . All runs:

N{=}1024

, Wide 1-hop, seed 42, 50K steps.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Architecture
Parameter	Value
$d_{\text{model}}$	1024
Attention heads	16 (head dim = 64)
Layers	8
Experts per layer ( $M$ )	1024
Expert size (rank)	1

Appendix

Table 12: Complete training configuration for Marathon-scale experiments.

Pair	Mean $\Delta$ Loss	95% CI	Zero in CI?
Wide vs Deep	$-0.023$	[ $-0.031$ , $-0.015$ ]	No
Wide vs Magnitude	$-0.004$	[ $-0.010$ , $+0.003$ ]	Yes
Wide vs Asymmetric	$-0.020$	[ $-0.027$ , $-0.012$ ]	No
Wide vs Decoupled	$-0.007$	[ $-0.016$ , $+0.001$ ]	Yes
Deep vs Magnitude	$+0.020$	[ $+0.012$ , $+0.026$ ]	No
Deep vs Asymmetric	$+0.004$	[ $-0.004$ , $+0.011$ ]	Yes

Appendix

Table 13: Paired bootstrap 95% CIs on per-batch loss differences (all cosine variant pairs).

Margin ( $\delta$ )	Pairs Equivalent (of 10)	Interpretation
0.02 loss ( $\approx 0.6$ PPL)	5/10	Strict: half equivalent
0.03 loss ( $\approx 1.0$ PPL)	10/10	All equivalent
0.05 loss ( $\approx 1.6$ PPL)	10/10	All equivalent (generous)

Appendix

Table 14: TOST equivalence results at three margins.

Configuration	Seed 42 vs 137	95% CI	Significant?
Wide 1 $\times$ 12	$-0.013$	[ $-0.021$ , $-0.005$ ]	Yes
Deep 3 $\times$ 4	$-0.003$	[ $-0.010$ , $+0.005$ ]	No

Appendix

Table 15: Intra-seed variance (paired bootstrap, seeds 42 vs 137).

Pair	Standard	Block ( $b{=}5$ )	Block ( $b{=}10$ )
Wide vs Deep	Equiv.	Equiv.	Equiv.
Wide vs Mag	Equiv.	Equiv.	Equiv.
Wide vs Asym	Equiv.	Equiv.	Equiv.
Wide vs Decoup	Equiv.	Equiv.	Equiv.
Deep vs Mag	Equiv.	Equiv.	Equiv.
Deep vs Asym	Equiv.	Equiv.	Equiv.

Appendix

Table 16: Block bootstrap TOST equivalence (

\delta=0.03

loss). CIs are at most

1.09\times

wider than independent bootstrap; all conclusions are preserved.

Figure 7: Depth advantage during underfitting. At 36M (a) and 76M (b), MLP experts amplify the Deep-vs-Wide gap. However, this advantage vanishes at convergence (Section 4.3 ): at 50K steps, Wide (PPL 33.93) beats Deep (PPL 34.62).

Expert Type	Scale	Deep PPL	Wide PPL	Depth Advantage
Static $V_{\text{sem}}$	36M	321.43	322.65	$+0.4\%$
MLP (rank-1)	36M	304.50	312.61	$+2.7\%$
Static $V_{\text{sem}}$	76M	144.95	149.80	$+3.3\%$
MLP (rank-1)	76M	136.70	149.80	$+8.8\%$

Appendix

Table 17: Depth advantage with static vs MLP experts at two scales.

Figure 8: Scale Emergence. (a) Depth advantage grows with scale. (b) Navigational diversity (HopDiv) increases

7\times

from 36M to 76M. (c) Trajectory displacement grows. (d) MLP improvement compounds:

-4.6\%

at 36M

\to

-5.7\%

at 76M.

Exp	Description	Key Result	Section
001–003	Routing mode ablation	Semantic re-route best	Section 3
004–009	Architecture search	$d_{\text{space}}{=}64$ optimal	Section 3
010	MLP expert introduction	$-4.4\%$ PPL	App. C
012–013	Scale emergence	Depth $+8.8\%$ at 76M	App. C
014, 017	Halting + profiling	25% FLOPs saved	Section 6
018–023	Hop-width ablation	Centroid density key	Section 7.1

Appendix

Table 18: Selected experiment index (62 total). Experiments cited to Ternovtsii and Bilak [ 1 ] are detailed in the companion paper. Full logs in supplementary materials.

Explore similar work

When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models
May 8, 2026Youngsik Yoon, Siwei Wang, Wei Chen +1Mixture-of-Experts Language Models LLM Routing
Department of Computer Science and Engineering, POSTECH, South Korea · Microsoft Research Asia, Beijing, China · Graduate School of Artificial Intelligence, POSTECH, South Korea
Geometric Routing Enables Causal Expert Control in Mixture of Experts
Apr 15, 2026Ivan Ternovtsii, Yurii BilakMixture of Experts Mixture-of-Experts Models
Department of Software Systems, Uzhhorod National University, Narodna sq. 3, Uzhhorod, Ukraine, 88000 · HengeBytes
MoE Routing Testbed: Studying Expert Specialization and Routing Behavior at Small Scale
Apr 8, 2026Tobias Falke, Nicolas Anastassacos, Samson Tan +6Mixture of Experts Sparse Mixture-of-Experts
Amazon AGI

May 8, 2026J/K move · Enter open · S save

When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models

Youngsik Yoon, Siwei Wang, Wei Chen, Jungseul Ok

Department of Computer Science and Engineering, POSTECH, South Korea · Microsoft Research Asia, Beijing, China · Graduate School of Artificial Intelligence, POSTECH, South Korea

Mixture-of-Experts Language Models LLM Routing Mixture-of-Experts Models

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) language models route each token to a small subset of experts, but whether the routes selected by a trained top-

k

router are good ones is rarely evaluated directly. Holding the model fixed, we compare each standard route against sampled equal-compute alternatives for the same token and score each by the next-token probability it assigns to the realized token in a verified reasoning trajectory. The result is sharply token-conditional: the standard router is well-aligned with route utility on confident tokens but uninformative on the fragile tokens that drive hard reasoning, where lower-loss equal-compute routes consistently exist inside the frozen model but are not selected. The same pattern holds across Qwen3-30B-A3B, GPT-OSS-20B, DeepSeek-V2-Lite, and OLMoE-1B-7B, and follows structurally from how standard top-

k

training evaluates routing decisions: the language modeling loss scores only the executed route, and load balancing depends only on aggregate routing statistics. A minimal router-only update to the final-layer router, leaving every expert and every other router frozen, is sufficient to shift pass@K on AIME 2024+2025 and HMMT 2025 for both Qwen3-30B-A3B and GPT-OSS-20B, suggesting that at least part of the failure reflects router-reachable misallocation rather than expert capacity alone.

Abstract

Explore similar work

When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models

Geometric Routing Enables Causal Expert Control in Mixture of Experts

MoE Routing Testbed: Studying Expert Specialization and Routing Behavior at Small Scale