Sparse mixture-of-experts (MoE) models expand parameter capacity by activating a small subset of experts per token. Expert selection directly affects model capabilities under a fixed active-expert budget. Dense evaluation of all experts reveals weak agreement between single-router rankings and expert-utility evaluation rankings, especially in middle and deep layers. Richer signals for selection and computation incur additional cost: selection based on internal expert activations requires preliminary computation, while processing multiple full representations separately through the same expert repeats its computation. We propose Multi-View MoE to improve expert selection with multiple routers and reuse their preferences to modulate expert computation under a fixed expert budget. Its Multi-Router submodule aggregates sigmoid scores from independently initialized routers. These routers develop highly differentiated expert preferences during training, allowing experts strongly favored by one router or jointly supported by several routers to be selected. Multi-View further uses each selected expert's relative scores across routers to mix learned latent views into its input, reusing routing preferences for both selection and computation. Each selected expert executes one forward pass, preserving the active-expert count, expert FLOPs, and all-to-all communication volume. On a 24-layer 10B-A1B model trained on 708B tokens, full Multi-View MoE improves the nine-task downstream mean by 1.18 percentage points over the baseline. Multi-Router alone yields a 0.6-percentage-point gain and improves the rank correlation between router scores and evaluated expert utility, with relative increases of 8.1% globally and approximately 58% in the layer with the lowest single-router alignment. Controlled mixing interventions show the model uses input differences that depend on both the expert and the token.
Figures & tables
Figure 1: Router-conditioned expert inputs in Multi-View. Equal total scores can express different preferences across routing heads and yield different mixed inputs. In this illustrative k=2 example, experts i and j have score pairs (0.4,0.1) and (0.2,0.3) , both summing to 0.5 . Normalizing across routing heads for each expert gives view proportions (0.8,0.2) and (0.4,0.6) , respectively. Routing heads and view projections read the same hidden state; each selected expert runs once. The balancing bias is set to zero. Shared experts, output gating, and up-projection are omitted.
Figure 2: Agreement between single-router scores and empirical expert-utility evaluation on the 24-layer 10B-A1B backbone. (a) Spearman correlation across training. The logarithmic training-step axis expands the early stage and compresses later intervals to show early-training changes more clearly. (b) Layer-wise means over the 201 logged measurements used in the router comparison; shaded bands show ±1 standard deviation. Scores exclude the load-balancing bias. Expert utility is evaluated using each expert’s output L2 norm, normalized by its mean over sampled tokens (App. B.5 ).
Figure 3: Comparison of latent-space MoE architectures. Top: (a) LatentMoE uses a router and a shared latent input; (b) Multi-Router sums scores from separately parameterized sigmoid routing heads for top- k selection while retaining the shared input; (c) Multi-View adds distinct down-projections and expert-specific view mixing. Bottom: expansion of the boxed routed path in (c), with matching colors and numbered markers. Balancing bias b is added to summed scores only for selection. Output gates gi follow Eq. 3 . For each selected expert, raw sigmoid scores are normalized across routers to form view proportions am,i , whose sum is one. Each selected expert processes its mixed input once. The aggregated representation y is up-projected and combined with the shared-expert output shown above. K=2 views and selected experts i,j are illustrative; k is the number of active experts out of N . Bars depict example routing scores. Normalization modules and residual paths are omitted for clarity.
Exams
Knowledge
Science
Reasoning
Method
CMMLU
MMLU
Redux
Pro
Avg.
HotpotQA
SimpleQA
PreciseWiki
Avg.
GPQA
BBH
Overall
Baseline (LatentMoE)
53.71
57.48
59.03
27.28
49.38
22.00
3.35
8.64
11.33
30.00
36.10
33.07
Multi-Router ( R=2 )
53.77
57.50
58.90
27.71
49.47
22.13
3.85
9.16
11.71
32.86
37.31
33.69
Multi-View ( K=2 )
55.20
58.67
60.93
27.12
50.48
22.54
3.41
9.02
11.66
35.60
35.76
34.25
Δ (MV − Baseline)
+1.49
+1.19
+1.90
−0.16
+1.10
+0.54
+0.06
+0.38
+0.33
+5.60
−0.34
+1.18
Table 1: Downstream scores (%) at 708B training tokens. Baseline: latent-space MoE; MR: Multi-Router ( R=2 ); MV: score-derived Multi-View ( K=R=2 ; Eq. 5 ). Tasks are grouped into exams, knowledge, science, and reasoning. Avg. is the unweighted within-group mean; Overall is the unweighted mean across all nine tasks. Bold marks the highest displayed score in each column. The final row reports MV minus Baseline in percentage points. Prompt and scoring protocols appear in App. B.4 .
Figure 4: Training loss of the baseline, Multi-Router, and Multi-View on 10B-A1B through 708B tokens. (a) Loss difference from the baseline; the inset shows the full loss trajectories. (b) Magnification of the blue-bordered window in (a), covering 50–230B tokens. (c) Magnification of the purple-bordered window in the full-loss inset, showing absolute loss over 620–708B tokens. All curves use the same causal exponential moving average (EMA), with a 2,000-step span ( α=2/2001 ), initialized at the first observation. The final losses reported in the text average the last 1,000 unsmoothed steps.
Figure 5: Expert-specific inputs and controlled mixing interventions in the score-derived Multi-View checkpoint at 708B tokens. (a) Differences among the eight experts selected for the same token: the mean absolute pairwise difference in view-1 proportions Dr (top) and the mean pairwise input distance normalized by the mean input norm Dv (bottom). Zero denotes identical proportions or inputs, respectively. Curves show the median across tokens at each layer; bands show the 25th–75th percentiles. (b) Final-layer interventions on 256 sequences, changing view proportions while holding S and output gates gi fixed to numerical precision. Whiskers are 95% sequence-bootstrap CIs; crosses mark permutation-seed means. (c) Single-layer interventions on 80 sequences. Bands are pointwise 95% CIs; downstream routing is recomputed. Δ NLL is intervention loss minus the original loss.
Figure 6: Router-score alignment with normalized expert-output norms. (a) Global Spearman correlation over training; solid curves use a 51-record moving mean and pale curves show raw measurements. (b) Layer-wise means and one-standard-deviation bands over 201 matched logged measurements; metric and aggregation definitions appear in App. B.5 . (c) Correlation changes across all 24 layers. Multi-Router improves global correlation by a relative 8.1% , with the largest relative layer-wise gain of 57.7% at layer 20.
Method
Spearman ρ
Single router
0.1245
Multi-Router
0.1346
Table 2: Global alignment between pre-bias router scores and normalized expert-output norms, averaged over 201 matched logged measurements. Multi-Router improves rank correlation by a relative 8.1% . Definitions appear in App. B.5 .
Method
Δ loss vs. baseline ( ↓ )
Single router (baseline; loss 2.3825 )
0.0000
Multi-Router ( R=2 , sigmoid-sum)
−0.0045
Multi-Router ( R=3 )
−0.0051
Fusion position ( R=2 )
Logit-sum
collapses to one router
Probability-normalized sum
−0.0037
Table 3: Multi-Router training ablations on 10B-A1B. Loss differences use the single-router baseline and a ±1% token window around the common ablation checkpoint. All post-sigmoid aggregation operators reduce loss; the final row reports the observed run-pair difference.
Variant ( K=2 )
Δ loss vs. baseline ( ↓ )
Multi-Router ( R=2 )
−0.0045
Learned per-expert view weights
Decoupled gate, init 0.5
−0.0048
Decoupled gate, asym. init
−0.0050
Two independent gates
−0.0044
Coupling between scoring and views
Table 4: Input-mixing training ablations on 10B-A1B. The learned-weight variants use per-expert mixing parameters. In the view-fed coupling pair, both scoring and expert computation read the views; stopping the weight-to-view gradient increases loss by 0.0114 while preserving forward weights at fixed parameters.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Variant
Expert FLOPs
Throughput
Multi-Router ( R=2 )
1.00×
17,375
Geometric mean
1.00×
17,424
Product
1.00×
17,564
Coupled views
1.00×
16,449
Detached views
1.00×
16,908
Appendix
Table 5: Same-cluster throughput in tokens/s/card for the listed 10B-A1B ablation configurations. Coupled and detached variants use view-fed scoring. Expert FLOPs are normalized to the backbone. Main-framework parameter counts are given in Eq. 6 .
Component
Setting
Layers / hidden size D
24 / 1024
Normalization / embeddings
RMSNorm / untied
Attention
MLA, 16 attention heads, gated output
Query / KV latent rank
512 / 512
QK dims per attention head (no RoPE + RoPE)
192 + 64
Value dim per attention head
256
Appendix
Table 6: Architecture of the 10B-A1B model.
Figure 7: Training-loss changes for router-fusion variants with R=2 . All evaluated variants improve over the single-router baseline. The shaded range shows the observed run-pair difference.
Figure 8: Additional analyses of the 708B-token Multi-View checkpoint. (a) Mean view-1 share r1,i by expert, with expert ranks determined independently in each layer. (b) Agreement between the two routing heads on 256 sequences. (c) Single-view interventions on 80 sequences, with downstream routing recomputed; bands are pointwise 95% CIs. (d) Final-layer effects grouped by clean token-NLL quintile on 256 sequences. The x-axis indexes five groups with equal numbers of predictions; error bars show 95% intervals.
Intervention
Δ NLL
95% CI
Identity replay
0.0003
[−0.001,0.002]
Token permutation
2.23
[2.05,2.41]
Uniform mixture
6.62
[6.20,7.09]
Expert permutation
13.40
[12.56,14.35]
View 1 only
18.81
[17.58,20.12]
View 2 only
20.42
[19.32,21.56]
Appendix
Table 7: Final-layer interventions on 256 held-out sequences. All values, including CIs, are in 10−3 nats/token. Expert selection and output gates gi are fixed to numerical precision; only view-mixing proportions change.
Mixture-of-Experts language models use input-dependent routing to activate a small subset of expert networks, increasing capacity without proportional compute.
Sparse Mixture-of-Experts (MoE) architectures route each token through a subset of experts at each layer independently. We propose viewing MoE computation through the lens of expert paths -- the sequence of expert selections a token makes across all layers. This perspective reveals that, despite NL possible paths for N experts across L layers, tokens in practice cluster into a small fraction of paths that align with linguistic function, yet the vast majority of paths remain unexplored, representing a statistical inefficiency. This motivates architectures that constrain the effective path space to amplify this natural concentration. As one instantiation, we introduce \pathmoe{}, which shares router parameters across blocks of consecutive layers. Analysis confirms that \pathmoe{} amplifies the emergent path structure: it produces more concentrated path clusters, better cross-layer consistency, and greater robustness to routing perturbations. Experiments on 0.9B and 16B parameter \pathmoe{} models demonstrate consistent improvements on perplexity and downstream tasks over independent routing, while eliminating the need for auxiliary losses. These results establish expert paths as a useful design axis for MoE architectures, complementary to existing work on independent routing mechanisms.
Mixture-of-experts (MoE) models have become an important approach for scaling model parameters. Modern MoE architectures increasingly combine two design choices: fine-grained expert segmentation and shared experts. The former increases expressiveness by enabling more expert combinations at a fixed activation ratio, and the latter introduces experts that are active for every token. We identify an exploration dilemma between these designs: shared experts can make routing less exploratory by reducing the variability of token representations. This conflicts with the greater need for exploration induced by fine-grained expert segmentation due to the combinatorially enlarged routing space. Consequently, the additional expressiveness offered by fine-grained segmentation may not be fully realized in practice. MirrorMoE is proposed to reconcile this exploration dilemma by shrinking the exploration space through token-adaptive suppression of unnecessary experts. MirrorMoE consistently improves performance across model scales.
Bingcong Li, Hao Ma, Michael Muehlebach +2
ETH Zurich · The University of Hong Kong · Max Planck Institute for Intelligent Systems +1
Mixture of Experts (MoE) models enable parameter-efficient scaling through sparse expert activations, yet optimizing their inference and memory costs remains challenging due to limited understanding of their specialization behavior. We present a systematic analysis of expert specialization in MoEs through two complementary approaches: domain-specific routing patterns and an early decoding framework that tracks expert contributions to output representations. Our analysis of the DeepSeekMoE model reveals that despite having 64 routed experts with 6 active for each layer's computation, the model predominantly relies on a few specialized experts, with the top-weighted expert's output closely approximating the full ensemble prediction. We quantitatively validate these findings through a systematic analysis of the token routing distribution, demonstrating that very few experts handle over 50% of routing decisions across different specialized domains. Hidden state similarity between single and ensemble experts for every layer is extremely high, with some layers having cosine similarity as high as 0.95 and perplexity increasing by only 5% when using a single expert across all three domains. Our results indicate that Mixture of Experts models exhibit concentrated expertise highlighting potential opportunities for inference optimization through targeted expert pruning while maintaining model performance and opening avenues towards studying localization of learned knowledge in these models.
The Pennsylvania State University, University Park, PA 16802 USA · University of Maryland, College Park, MD 20742 USA · Harvard University, Cambridge, MA 02138 USA