Sparse Mixture-of-Experts (MoE) architectures employ increasingly sophisticated routing mechanisms -- learned routers, multi-hop trajectories, token-dependent gating. We ask: does routing topology actually determine language modeling quality? We build a geometric MoE (ST-MoE) using cosine-similarity routing against learned centroids in a low-dimensional space (dspace=64), requiring 80% fewer routing parameters than standard linear routers. Through 62 controlled experiments on WikiText-103 at 76--84M parameters trained to convergence (50K steps, 1.64B tokens), we find that routing topology does not determine asymptotic perplexity (PPL): five cosine-routing variants are statistically equivalent within a 1-PPL margin (Two One-Sided Tests [TOST], p<0.05 for all 10 pairwise comparisons; 15 runs across 3 seeds, observed range 33.93--34.72). The finding extends to hash, random-fixed, and top-1 routing (single-seed; graceful 1.1--2.2 PPL degradation) and replicates on OpenWebText (0.03 PPL gap, 6 runs, 3 seeds each). A standard linear router with 5.3× more routing parameters reaches PPL 32.76, but iso-parameter cosine routing closes 67% of this gap -- the true mechanism advantage is ∼1.2%. The mechanistic explanation is convergent redundancy: multi-hop updates are collinear (cos(Δh0,Δh1)=0.805), implementing magnitude amplification rather than compositional reasoning; a single learnable scalar replicates multi-hop performance. As a practical payoff, zero-shot relative-norm halting saves 25% of MoE FLOPs at +0.12% PPL. Expert-level specialization and causal controllability -- which coexist with topology-level equifinality -- are explored in a companion paper.
Figures & tables
Figure 1: ST-MoE Architecture. (A) Full model: token embedding →N blocks → LM head. (B) Single block: pre-LN attention with RoPE, followed by pre-LN ST-MoE layer, both with residual connections. (C) Multi-hop routing: tokens are projected into a dspace -dimensional coordinate space, routed to top- K experts via cosine similarity with learned centroids, and accumulate expert updates across H hops with semantic position updates between hops.
Scale
dmodel
Heads
Layers
M
Steps
Tokens
Params
Small (36M)
768
12
4
512
5,000
82M
37.6M
Large (76M)
1024
16
6
1024
20,000
328M
76–84M
Marathon (76M)
1024
16
8
1024
50,000
1.64B
76–84M
Table 1: Model configurations used throughout the paper, varying scale, depth, expert count, and training duration.
Model
Active/Token
PPL
vs Dense
Small active budget (12 neurons)
Dense d_ff=12
12
306.01
baseline
Singleton MoE ( 3×4×1 )
12
304.50
−0.5%
Medium active budget (48 neurons)
Dense d_ff=48
48
302.24
baseline
Block MoE Wide ( 1×6×8 )
48
298.91
−1.1%
Table 2: MoE vs Dense at iso-FLOP (same active computation per token).
Figure 2: Dense vs MoE Training Comparison. The iso-FLOP dense baseline ( dff=12 , same active FLOPs) tracks MoE throughout training; the iso-parameter dense baseline ( dff=1120 , ∼93× more active FLOPs) dominates after step 5K. MoE crosses the iso-FLOP dense at ∼ 10K steps, confirming that sparse routing provides value at matched active computation.
Figure 3: Convergence Marathon. Training curves for all configurations at 76M parameters, 50K steps (1.64B tokens). Among cosine-routing variants, all topologies converge to a statistically equivalent band (TOST, δ=1 PPL, all pairs p<0.05 ; observed range 33.93–34.72 across 15 runs). The standard linear router (dashed cyan) falls outside this band at PPL 32.76, demonstrating that routing capacity matters even when routing topology does not.
Figure 4: Multi-Seed Validation. Training curves for Wide 1 × 12 and Deep 3 × 4 with seeds 42 and 137. Same-topology curves track nearly identically throughout training: Wide Δ=1.2% , Deep Δ=0.1% . All four runs converge within a 0.74 PPL band. Full 5-variant × 3-seed validation (15 runs total) confirms all topologies converge within a 0.79 PPL band (33.93–34.72), with inter-variant spread ( 0.60 PPL) only 5.0× the average seed noise ( 0.12 PPL avg std).
Model
Architecture
Final PPL
Mean ± Std (3 seeds)
vs iso-FLOP Dense
Routing Params
Train Time
Baselines
Dense iso-FLOP
d_ff=12
34.49
—
—
—
3.5h
Dense iso-param
d_ff=1120 ( 93× active)
29.36
—
−14.9%
—
3.9h
ST-MoE Wide 1 × 12
cosine, 1 hop, top-12
33.93
34.22±0.25
−1.6%
1.57M
7.4h
ST-MoE Deep 3 × 4
cosine, 3 hops, top-4
34.62
34.65±0.03
+0.4%
1.57M
8.9h
Cosine Routing Interventions (Exps 033–035)
Table 3: Convergence results at 76–84.7M parameters, 1.64B tokens (RTX 3090). Among cosine-routing variants, all topologies are statistically equivalent within 1 PPL (TOST, p<0.05 for all 10 pairwise comparisons). Mean ± Std computed over 3 seeds (42, 137, 7) for each cosine variant; all 15 runs fall within a 0.79 PPL band. Broader topology conditions (Exp 065) and rank-16 experts (Exp 070) extend the comparison. MoE matches the iso-FLOP dense baseline ( dff=12 ) while using the same active computation but ∼93× more total parameters. The iso-parameter dense baseline ( dff=1120 , 93× more active FLOPs) outperforms all MoE variants, confirming MoE’s value is conditional computation, not per-parameter efficiency.
Checkpoint
Normal PPL
Frozen PPL
Δ
Normal Jaccard
20K steps (underfitting)
268.60
286.76
+6.8%
0.102
50K steps (converged)
34.62
37.26
+7.6%
0.970
Table 4: Frozen routing at two training stages. At convergence, the model already selects 97% identical experts across hops, yet freezing still degrades PPL by 7.6%—re-routing provides a small but genuine correction.
Figure 5: Cross-Seed Expert Alignment. Left: vocabulary projection Jaccard similarity (best-match) is ∼500× above random—experts develop partially overlapping functional roles across seeds. Right: raw Wup cosine similarity is indistinguishable from random—these overlapping functions are realized through entirely different weight parameterizations. This divergence is the quantitative signature of equifinality.
Figure 6: Pareto Frontier. (a) MLP experts enable a smooth quality-compute tradeoff; static experts show a sharp cliff. (b) Halting response: MLP average hops decrease gradually with ε , while static experts exhibit a phase transition.
Table 6: Halting at convergence (Exp 044c). The converged model is equally amenable to halting.
Regime
M
Expert Size
Best Topology
PPL
vs Dense
Micro
512
1
Deep ( 3×4 )
304.50
−0.5%
Macro
64
8
Wide ( 1×6 )
298.91
−1.1%
Table 7: Architecture Dualism: optimal topology depends on expert granularity.
Architecture
M
Expert Size
top-k /hop
Per-hop Rank
PPL
Singleton Deep (Exp 012)
512
1
4
4
304.50
Granular Deep (Exp 022)
128
4
4
16
337.26
Table 8: Centroid density vs. per-hop rank: rank-4 with 512 centroids beats rank-16 with 128 centroids.
Architecture
M
es
Hops
k /hop
Hop Rank
PPL
Multi-hop (trajectory navigation)
Singleton Deep (Exp 012)
512
1
3
4
4
304.50
Macro Deep (Exp 018)
64
8
3
2
16
337.02
Hybrid (Exp 021)
64
8
2
3
24
319.85
Granular Deep (Exp 022)
128
4
3
4
16
337.26
Single-hop (parallel selection)
Table 9: Complete hop-width ablation at iso-FLOP (48 active neurons).
Condition
PPL@3-hop
PPL@2-hop ( Δ )
PPL@1-hop ( Δ )
FLOPs at 1-hop
Baseline
302.54
312.92 (+3.4%)
366.48 (+21.1%)
−67%
Curriculum
316.14
316.18 (+0.0%)
318.62 (+0.8%)
−67%
Table 10: Trajectory Curriculum: the model becomes lazy (Exp 024).
Config
K
τ
Best PPL
Δ vs baseline
Baseline
12
30
33.93
—
K=12 , τ=15
12
15
34.06
+0.13
K=24 , τ=30
24
30
34.12
+0.19
K=24 , τ=15
24
15
33.85
−0.08
K=24 , τ=learned
24
11–15
34.16
+0.23
K=48 , τ=30
48
30
33.92
−0.01
Table 11: K×τ ablation (Exp 051). Best PPL in bold . All runs: N=1024 , Wide 1-hop, seed 42, 50K steps.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Architecture
dmodel
1024
Attention heads
16 (head dim = 64)
Layers
8
Experts per layer ( M )
1024
Expert size (rank)
1
Appendix
Table 12: Complete training configuration for Marathon-scale experiments.
Pair
Mean Δ Loss
95% CI
Zero in CI?
Wide vs Deep
−0.023
[ −0.031 , −0.015 ]
No
Wide vs Magnitude
−0.004
[ −0.010 , +0.003 ]
Yes
Wide vs Asymmetric
−0.020
[ −0.027 , −0.012 ]
No
Wide vs Decoupled
−0.007
[ −0.016 , +0.001 ]
Yes
Deep vs Magnitude
+0.020
[ +0.012 , +0.026 ]
No
Deep vs Asymmetric
+0.004
[ −0.004 , +0.011 ]
Yes
Appendix
Table 13: Paired bootstrap 95% CIs on per-batch loss differences (all cosine variant pairs).
Margin ( δ )
Pairs Equivalent (of 10)
Interpretation
0.02 loss ( ≈0.6 PPL)
5/10
Strict: half equivalent
0.03 loss ( ≈1.0 PPL)
10/10
All equivalent
0.05 loss ( ≈1.6 PPL)
10/10
All equivalent (generous)
Appendix
Table 14: TOST equivalence results at three margins.
Configuration
Seed 42 vs 137
95% CI
Significant?
Wide 1 × 12
−0.013
[ −0.021 , −0.005 ]
Yes
Deep 3 × 4
−0.003
[ −0.010 , +0.005 ]
No
Appendix
Table 15: Intra-seed variance (paired bootstrap, seeds 42 vs 137).
Pair
Standard
Block ( b=5 )
Block ( b=10 )
Wide vs Deep
Equiv.
Equiv.
Equiv.
Wide vs Mag
Equiv.
Equiv.
Equiv.
Wide vs Asym
Equiv.
Equiv.
Equiv.
Wide vs Decoup
Equiv.
Equiv.
Equiv.
Deep vs Mag
Equiv.
Equiv.
Equiv.
Deep vs Asym
Equiv.
Equiv.
Equiv.
Appendix
Table 16: Block bootstrap TOST equivalence ( δ=0.03 loss). CIs are at most 1.09× wider than independent bootstrap; all conclusions are preserved.
Figure 7: Depth advantage during underfitting. At 36M (a) and 76M (b), MLP experts amplify the Deep-vs-Wide gap. However, this advantage vanishes at convergence (Section 4.3 ): at 50K steps, Wide (PPL 33.93) beats Deep (PPL 34.62).
Expert Type
Scale
Deep PPL
Wide PPL
Depth Advantage
Static Vsem
36M
321.43
322.65
+0.4%
MLP (rank-1)
36M
304.50
312.61
+2.7%
Static Vsem
76M
144.95
149.80
+3.3%
MLP (rank-1)
76M
136.70
149.80
+8.8%
Appendix
Table 17: Depth advantage with static vs MLP experts at two scales.
Figure 8: Scale Emergence. (a) Depth advantage grows with scale. (b) Navigational diversity (HopDiv) increases 7× from 36M to 76M. (c) Trajectory displacement grows. (d) MLP improvement compounds: −4.6% at 36M →−5.7% at 76M.
Exp
Description
Key Result
Section
001–003
Routing mode ablation
Semantic re-route best
Section 3
004–009
Architecture search
dspace=64 optimal
Section 3
010
MLP expert introduction
−4.4% PPL
App. C
012–013
Scale emergence
Depth +8.8% at 76M
App. C
014, 017
Halting + profiling
25% FLOPs saved
Section 6
018–023
Hop-width ablation
Centroid density key
Section 7.1
Appendix
Table 18: Selected experiment index (62 total). Experiments cited to Ternovtsii and Bilak [ 1 ] are detailed in the companion paper. Full logs in supplementary materials.
Mixture-of-Experts (MoE) language models route each token to a small subset of experts, but whether the routes selected by a trained top-k router are good ones is rarely evaluated directly. Holding the model fixed, we compare each standard route against sampled equal-compute alternatives for the same token and score each by the next-token probability it assigns to the realized token in a verified reasoning trajectory. The result is sharply token-conditional: the standard router is well-aligned with route utility on confident tokens but uninformative on the fragile tokens that drive hard reasoning, where lower-loss equal-compute routes consistently exist inside the frozen model but are not selected. The same pattern holds across Qwen3-30B-A3B, GPT-OSS-20B, DeepSeek-V2-Lite, and OLMoE-1B-7B, and follows structurally from how standard top-k training evaluates routing decisions: the language modeling loss scores only the executed route, and load balancing depends only on aggregate routing statistics. A minimal router-only update to the final-layer router, leaving every expert and every other router frozen, is sufficient to shift pass@K on AIME 2024+2025 and HMMT 2025 for both Qwen3-30B-A3B and GPT-OSS-20B, suggesting that at least part of the failure reflects router-reachable misallocation rather than expert capacity alone.
Youngsik Yoon, Siwei Wang, Wei Chen +1
Department of Computer Science and Engineering, POSTECH, South Korea · Microsoft Research Asia, Beijing, China · Graduate School of Artificial Intelligence, POSTECH, South Korea
Sparse Mixture-of-Experts (MoE) models scale parameters while fixing active computation per token, but the specialization of individual experts remains opaque. In a companion paper we showed that routing topology is quality-neutral: five structurally different configurations converge to statistically equivalent language modeling quality. Here we show that expert identity is nonetheless causally meaningful: individual rank-1 experts are monosemantic by construction, and cosine-similarity routing in a low-dimensional metric space makes their specialization directly inspectable. We present four lines of evidence. First, projecting expert output vectors through the unembedding matrix yields a Semantic Dictionary: 15% of experts are monosemantic specialists spanning 10 categories (temporal, geographic, cardinal, discourse, emotional, financial, military, scientific). Second, routing exhibits a frequency-to-syntax gradient: early layers separate tokens by word frequency, deeper layers by syntactic class (Zipf-confound controls, all p<0.001). Third, causal interventions confirm these labels: steering toward a temporal expert's centroid increases P(temporal) by +321% (median across 44 prompts); suppressing a geographic expert drops P(geographic) by -23%; rewriting an expert's output vector halves target-category probability, and effects compose additively across layers. Fourth, the interventions are not unique to cosine routing: linear routers support comparable steering, but only cosine routing provides geometric transparency -- expert specialization is readable directly from the centroid matrix. MoE expert-level specialization is a first-class interpretability primitive: architecturally monosemantic, causally validated, and controllable at inference with zero overhead.
Ivan Ternovtsii, Yurii Bilak
Department of Software Systems, Uzhhorod National University, Narodna sq. 3, Uzhhorod, Ukraine, 88000 · HengeBytes
Sparse Mixture-of-Experts (MoE) architectures are increasingly popular for frontier large language models (LLM) but they introduce training challenges due to routing complexity. Fully leveraging parameters of an MoE model requires all experts to be well-trained and to specialize in non-redundant ways. Assessing this, however, is complicated due to lack of established metrics and, importantly, many routing techniques exhibit similar performance at smaller sizes, which is often not reflective of their behavior at large scale. To address this challenge, we propose the MoE Routing Testbed, a setup that gives clearer visibility into routing dynamics at small scale while using realistic data. The testbed pairs a data mix with clearly distinguishable domains with a reference router that prescribes ideal routing based on these domains, providing a well-defined upper bound for comparison. This enables quantifiable measurement of expert specialization. To demonstrate the value of the testbed, we compare various MoE routing approaches and show that balancing scope is the crucial factor that allows specialization while maintaining high expert utilization. We confirm that this observation generalizes to models 35x larger.