Organizations: Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · Baidu Inc.
Mixture-of-Experts (MoE) decouples model capacity from per-token computation, yet their scalability remains limited by the physical dimensions of depth and width. To overcome this, we propose Mixture of Universal Experts (MOUE),a MoE generalization introducing a novel scaling dimension: Virtual Width. In general, MoUE aims to reuse a universal layer-agnostic expert pool across layers, converting depth into virtual width under a fixed per-token activation budget. However, two challenges remain: a routing path explosion from recursive expert reuse, and a mismatch between the exposure induced by reuse and the conventional load-balancing objectives. We address these with three core components: a Staggered Rotational Topology for structured expert sharing, a Universal Expert Load Balance for depth-aware exposure correction, and a Universal Router with lightweight trajectory state for coherent multi-step routing. Empirically, MoUE consistently outperforms matched MoE baselines by up to 1.3% across scaling regimes, enables progressive conversion of existing MoE checkpoints with up to 4.2% gains, and reveals a new scaling dimension for MoE architectures.
Figures & tables
Figure 1: Overview of MoUE . A shared pool of Universal Experts is accessible from multiple layers, enabling recursive reuse under a fixed activation budget.
Figure 2: Expert similarity heatmap across shallow (L2, L3) and deep (L8, L9) layers in a trained MoE. Adjacent layers exhibit strong similarity, and specific experts re-emerge across distant layers, motivating cross-layer expert reuse.
Figure 3: An illustration of MoUE . Each layer routes tokens to a small set of experts, consisting of layer-local experts and a shared pool of Universal Experts (UEs). UEs are connected to multiple layers (under a constrained topology), enabling cross-layer reuse while keeping the per-token activation budget fixed.
Figure 4: LBL only focuses on balancing expert selection within each layer, while ULBL promotes balanced and diverse expert selection both across depth and width.
Model
Act.
TP.
VP.
Commonsense & Reading Comprehension
Continued
LM
Avg.
SciQ
PIQA
WG
ARC-E
ARC-C
Hella.
LogiQA
BoolQ
Lam.
MoE 12A2 L12
0.1B
0.16B
0.16B
52.0
62.9
49.4
41.3
19.9
30.1
19.7
57.3
36.4
41.0
Width scaling
MoUE 18A2
0.1B
0.16B
0.19B
52.6
62.6
50.2
41.1
21.6
30.1
21.4
57.6
36.2
41.5
MoUE 24A2
0.1B
0.16B
0.22B
52.5
63.3
50.9
41.8
21.6
29.3
19.7
59.6
36.1
41.6
MoUE 36A2
0.1B
0.16B
0.28B
52.6
63.3
51.2
41.4
20.4
31.0
23.2
60.1
37.1
42.3
Table 1: Main results of MoUE width/depth expansion on two Qwen-3 style MoE baselines (MoE-0.16B and MoE-0.7B). We expand the virtual width ( ×2,×3,×4 ) and depth ( ×2,×3 ) via cross-layer expert reuse.
Figure 5: Left: LM training loss versus training steps. Middle: Validation loss versus training FLOPs. Right: Max/Mean ratio versus training steps (routing skew / load-balance indicator).
Model
Act.
VP.
ARC-c
ARC-e
GSM8K
HellaS.
HumanE.
NQ
Tri.QA
Wino.
MMLU
AVG
JetMoE-8E, adapted by supervised fine-tuning.
Vanilla MoE
2.2B
8.0B
64.4
79.0
44.7
78.7
42.7
20.4
49.5
56.6
47.7
53.7
MoUE 10A2
2.2B
9.7B
66.8
78.8
42.2
78.4
48.2
20.9
50.7
54.1
46.3
54.1
MoUE 12A2
2.2B
11.3B
65.1
79.4
45.1
80.3
42.7
21.4
50.5
56.8
48.2
54.4
MoUE 16A2
2.2B
14.6B
65.8
79.9
45.0
79.5
45.7
20.5
51.2
57.9
47.9
54.8
OLMoE-64E, adapted by supervised fine-tuning.
Table 2: Comprehensive evaluation of open-source MoE baselines and corresponding MoUE settings. Each baseline is an open-source MoE model; below each are three MoUE variants with their specific metrics.
Figure 6: Left: Average accuracy versus total parameters, with point size representing activated parameters. Middle: Average accuracy versus virtual parameters, with point size representing activated parameters. Right: Average accuracy versus activated parameters.
Figure 8: Token Routing Proportion to Universal Experts per Layer after SFT warm-start training.
Figure 9: Activation differences of universal experts across domains. Experiments are conducted with 4 universal experts per layer.
Figure 10: Effect of the number of universal experts in warm-start experiments.
Hyperparameter
Value
Notes
Sequence length
4096
shared
Global batch size
256
shared
Optimizer
AdamW
β=(0.9,0.95) , weight decay 0.1
LR schedule
cosine
shared
Peak LR
5×10−4
shared
Warmup fraction
5% tokens
shared
Appendix
Table 5: Training and routing hyperparameters used in our experiments. Values are reported for the Qwen-3 style MoE backbones and are kept identical between MoE and MoUE unless explicitly noted.
Mixture of Experts (MoE) models route inputs to specialized subnetworks, aiming to scale capacity and adapt computation without activating every parameter.
Most Mixture-of-Experts (MoE) transformers preassign each expert to one layer for the whole of training. We study whether a brief global-access phase can learn a compatible expert-to-layer allocation that is then executed privately. UniPool-Lock gives every layer its own router over one global pool, balances aggregate pool usage, and scores experts with a scale-stable NormRouter. After the short ownership-learning phase, it assigns each expert to one layer, locks the disjoint allocation, and continues training as a layer-private MoE. With the same expert-FFN budget and routed expert FLOPs, this recipe lowers held-out loss relative to vanilla MoE by 0.024-0.037 across five dense-equivalent scales from 182M to 1.5B, including -0.0247 at 1.5B after 60B tokens. The ownership-learning phase lasts 2K steps, about 3.3% of training, and post-lock step time is within -0.7% to +2.8% of vanilla MoE in our throughput measurements. Controls locate the gain in full-pool training and in the allocation it produces: a random disjoint allocation fixed at initialization, trained with the same router and losses, matches vanilla, whereas locking the allocation learned during the full-pool phase recovers nearly all of the persistent full-pool gain, and substituting a random allocation at the lock forfeits about half of it. Keeping full-pool access throughout training (UniPool-Full) also improves over vanilla MoE at four scales and outperforms it with only 66.7% (182M) to 50% (469M and 650M) of its expert parameters.
Minbin Huang, Han Shi, Chuanyang Zheng +5
The Chinese University of Hong Kong · Huawei Technologies · The University of Hong Kong
Mixture-of-experts (MoE) models have become an important approach for scaling model parameters. Modern MoE architectures increasingly combine two design choices: fine-grained expert segmentation and shared experts. The former increases expressiveness by enabling more expert combinations at a fixed activation ratio, and the latter introduces experts that are active for every token. We identify an exploration dilemma between these designs: shared experts can make routing less exploratory by reducing the variability of token representations. This conflicts with the greater need for exploration induced by fine-grained expert segmentation due to the combinatorially enlarged routing space. Consequently, the additional expressiveness offered by fine-grained segmentation may not be fully realized in practice. MirrorMoE is proposed to reconcile this exploration dilemma by shrinking the exploration space through token-adaptive suppression of unnecessary experts. MirrorMoE consistently improves performance across model scales.
Bingcong Li, Hao Ma, Michael Muehlebach +2
ETH Zurich · The University of Hong Kong · Max Planck Institute for Intelligent Systems +1
As model parameters scale up, sparsely activated Mixture-of-Experts (MoE) approaches have become essential for model scaling. However, mainstream models currently adopt hierarchical architectures, where experts in different layers, although topologically independent, exhibit substantial similarity and redundancy in their stored content. Compared to the hierarchically arranged Local Expert in existing mainstream models, we hypothesize that inter-layer shared Meta Expert hold greater potential. To this end, we conduct a multi-dimensional empirical comparison between Local and Meta Experts, and find that models incorporating Meta Experts exhibit lower parameter redundancy, together with lower cross-entropy loss and higher downstream task accuracy. With the activated quantity and the expert granularity held fixed, our model matches the performance of Vanilla MoE using only 70--80% of the experts, substantially improving the model's scaling potential. Beyond removing inter-layer redundancy among experts and improving information storage efficiency, our theoretical analysis further shows that, compared with Local Experts, Meta Experts provide hidden states with more flexible residual pathways, thereby improving information retrieval efficiency.