Mixture of Universal Experts: Scaling Virtual Width via Depth-Width Transformation

Yilong Chen; Naibin Gu; Junyuan Shang; Zhenyu Zhang; Yuchen Feng; Jiawei Sheng; Tingwen Liu; Shuohuan Wang

cs.LGMar 5, 2026

Mixture of Universal Experts: Scaling Virtual Width via Depth-Width Transformation

Authors: Yilong Chen, Naibin Gu, Junyuan Shang, Zhenyu Zhang, Yuchen Feng, Jiawei Sheng, Tingwen Liu, Shuohuan Wang,

Organizations: Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · Baidu Inc.

Abstract

Mixture-of-Experts (MoE) decouples model capacity from per-token computation, yet their scalability remains limited by the physical dimensions of depth and width. To overcome this, we propose Mixture of Universal Experts (MOUE),a MoE generalization introducing a novel scaling dimension: Virtual Width. In general, MoUE aims to reuse a universal layer-agnostic expert pool across layers, converting depth into virtual width under a fixed per-token activation budget. However, two challenges remain: a routing path explosion from recursive expert reuse, and a mismatch between the exposure induced by reuse and the conventional load-balancing objectives. We address these with three core components: a Staggered Rotational Topology for structured expert sharing, a Universal Expert Load Balance for depth-aware exposure correction, and a Universal Router with lightweight trajectory state for coherent multi-step routing. Empirically, MoUE consistently outperforms matched MoE baselines by up to 1.3% across scaling regimes, enables progressive conversion of existing MoE checkpoints with up to 4.2% gains, and reveals a new scaling dimension for MoE architectures.

Figures & tables

Figure 1: Overview of MoUE . A shared pool of Universal Experts is accessible from multiple layers, enabling recursive reuse under a fixed activation budget.

Figure 2: Expert similarity heatmap across shallow (L2, L3) and deep (L8, L9) layers in a trained MoE. Adjacent layers exhibit strong similarity, and specific experts re-emerge across distant layers, motivating cross-layer expert reuse.

Figure 3: An illustration of MoUE . Each layer routes tokens to a small set of experts, consisting of layer-local experts and a shared pool of Universal Experts (UEs). UEs are connected to multiple layers (under a constrained topology), enabling cross-layer reuse while keeping the per-token activation budget fixed.

Figure 4: LBL only focuses on balancing expert selection within each layer, while ULBL promotes balanced and diverse expert selection both across depth and width.

Model	Act.	TP.	VP.	Commonsense & Reading Comprehension						Continued		LM	Avg.
Model	Act.	TP.	VP.	SciQ	PIQA	WG	ARC-E	ARC-C	Hella.	LogiQA	BoolQ	Lam.	Avg.
MoE 12A2 L12	0.1B	0.16B	0.16B	52.0	62.9	49.4	41.3	19.9	30.1	19.7	57.3	36.4	41.0
Width scaling
MoUE 18A2	0.1B	0.16B	0.19B	52.6	62.6	50.2	41.1	21.6	30.1	21.4	57.6	36.2	41.5
MoUE 24A2	0.1B	0.16B	0.22B	52.5	63.3	50.9	41.8	21.6	29.3	19.7	59.6	36.1	41.6
MoUE 36A2	0.1B	0.16B	0.28B	52.6	63.3	51.2	41.4	20.4	31.0	23.2	60.1	37.1	42.3

Table 1: Main results of MoUE width/depth expansion on two Qwen-3 style MoE baselines (MoE-0.16B and MoE-0.7B). We expand the virtual width (

\times 2,\times 3,\times 4

) and depth (

\times 2,\times 3

) via cross-layer expert reuse.

Figure 5: Left: LM training loss versus training steps. Middle: Validation loss versus training FLOPs. Right: Max/Mean ratio versus training steps (routing skew / load-balance indicator).

Model	Act.	VP.	ARC-c	ARC-e	GSM8K	HellaS.	HumanE.	NQ	Tri.QA	Wino.	MMLU	AVG
JetMoE-8E, adapted by supervised fine-tuning.
Vanilla MoE	2.2B	8.0B	64.4	79.0	44.7	78.7	42.7	20.4	49.5	56.6	47.7	53.7
MoUE 10A2	2.2B	9.7B	66.8	78.8	42.2	78.4	48.2	20.9	50.7	54.1	46.3	54.1
MoUE 12A2	2.2B	11.3B	65.1	79.4	45.1	80.3	42.7	21.4	50.5	56.8	48.2	54.4
MoUE 16A2	2.2B	14.6B	65.8	79.9	45.0	79.5	45.7	20.5	51.2	57.9	47.9	54.8
OLMoE-64E, adapted by supervised fine-tuning.

Table 2: Comprehensive evaluation of open-source MoE baselines and corresponding MoUE settings. Each baseline is an open-source MoE model; below each are three MoUE variants with their specific metrics.

Figure 6: Left: Average accuracy versus total parameters, with point size representing activated parameters. Middle: Average accuracy versus virtual parameters, with point size representing activated parameters. Right: Average accuracy versus activated parameters.

Figure 7: Expert Utilization Heatmap. LE: Local Expert, UE: Universal Expert

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Token Routing Proportion to Universal Experts per Layer after SFT warm-start training.

Figure 9: Activation differences of universal experts across domains. Experiments are conducted with 4 universal experts per layer.

Figure 10: Effect of the number of universal experts in warm-start experiments.

Hyperparameter	Value	Notes
Sequence length	4096	shared
Global batch size	256	shared
Optimizer	AdamW	$\beta=(0.9,0.95)$ , weight decay $0.1$
LR schedule	cosine	shared
Peak LR	$5\times 10^{-4}$	shared
Warmup fraction	$5\%$ tokens	shared

Appendix

Table 5: Training and routing hyperparameters used in our experiments. Values are reported for the Qwen-3 style MoE backbones and are kept identical between MoE and MoUE unless explicitly noted.

Mixture of Experts

Mixture of Experts (MoE) models route inputs to specialized subnetworks, aiming to scale capacity and adapt computation without activating every parameter.

See all 224 papers

UniPool: Learning Expert-to-Layer Ownership from Brief Global Access
May 7, 2026Minbin Huang, Han Shi, Chuanyang Zheng +5Mixture-of-Experts Language Models Mixture of Experts
The Chinese University of Hong Kong · Huawei Technologies · The University of Hong Kong
Rethinking Routing Exploration in Modern Mixture-of-Experts
Oct 15, 2026Bingcong Li, Hao Ma, Michael Muehlebach +2Mixture-of-Experts Models Expert Routing
ETH Zurich · The University of Hong Kong · Max Planck Institute for Intelligent Systems +1
Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE
Oct 20, 2026Ermo Hua, Xiangyu Hong, Baiting Wu +11Sparse Mixture-of-Experts Mixture-of-Experts Models
Tsinghua University · Shanghai AI Laboratory

May 7, 2026J/K move · Enter open · S save

UniPool: Learning Expert-to-Layer Ownership from Brief Global Access

Minbin Huang, Han Shi, Chuanyang Zheng, Yimeng Wu+4

The Chinese University of Hong Kong · Huawei Technologies · The University of Hong Kong

Mixture-of-Experts Language Models Mixture of Experts Sparse Mixture-of-Experts Mixture-of-Experts Expert Specialization

Open paper AskPDF arXiv

Most Mixture-of-Experts (MoE) transformers preassign each expert to one layer for the whole of training. We study whether a brief global-access phase can learn a compatible expert-to-layer allocation that is then executed privately. UniPool-Lock gives every layer its own router over one global pool, balances aggregate pool usage, and scores experts with a scale-stable NormRouter. After the short ownership-learning phase, it assigns each expert to one layer, locks the disjoint allocation, and continues training as a layer-private MoE. With the same expert-FFN budget and routed expert FLOPs, this recipe lowers held-out loss relative to vanilla MoE by 0.024-0.037 across five dense-equivalent scales from 182M to 1.5B, including -0.0247 at 1.5B after 60B tokens. The ownership-learning phase lasts 2K steps, about 3.3% of training, and post-lock step time is within -0.7% to +2.8% of vanilla MoE in our throughput measurements. Controls locate the gain in full-pool training and in the allocation it produces: a random disjoint allocation fixed at initialization, trained with the same router and losses, matches vanilla, whereas locking the allocation learned during the full-pool phase recovers nearly all of the persistent full-pool gain, and substituting a random allocation at the lock forfeits about half of it. Keeping full-pool access throughout training (UniPool-Full) also improves over vanilla MoE at four scales and outperforms it with only 66.7% (182M) to 50% (469M and 650M) of its expert parameters.

Abstract

Mixture of Experts

Explore similar work

UniPool: Learning Expert-to-Layer Ownership from Brief Global Access

Rethinking Routing Exploration in Modern Mixture-of-Experts

Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE