As model parameters scale up, sparsely activated Mixture-of-Experts (MoE) approaches have become essential for model scaling. However, mainstream models currently adopt hierarchical architectures, where experts in different layers, although topologically independent, exhibit substantial similarity and redundancy in their stored content. Compared to the hierarchically arranged Local Expert in existing mainstream models, we hypothesize that inter-layer shared Meta Expert hold greater potential. To this end, we conduct a multi-dimensional empirical comparison between Local and Meta Experts, and find that models incorporating Meta Experts exhibit lower parameter redundancy, together with lower cross-entropy loss and higher downstream task accuracy. With the activated quantity and the expert granularity held fixed, our model matches the performance of Vanilla MoE using only 70--80% of the experts, substantially improving the model's scaling potential. Beyond removing inter-layer redundancy among experts and improving information storage efficiency, our theoretical analysis further shows that, compared with Local Experts, Meta Experts provide hidden states with more flexible residual pathways, thereby improving information retrieval efficiency.
Figures & tables
Figure 1: Comparison between Vanilla and Meta-Expert models. All models are trained under the optimal setting of the Vanilla model, at varying depths and scales. Models with Meta Experts achieve comparable loss to the Vanilla baselines with only 70–80% as many experts, or 75–90% as many parameters, and the savings in both experts and parameters grow as the models get deeper and larger, indicating the favorable scalability of Meta Experts.
Figure 2: The Implementation of Local and Meta Experts. In contrast to the layer-private Local Experts, Meta Experts are shared across all layers.
Figure 3: The Quantitative and Topological Comparison between Meta and Local Experts.
Figure 4: Pilot experiments on the Meta/Local Expert ratio. We vary the total-number proportion of Meta Experts ( 0% – 100% ) and their per-layer activation proportion ( 25% – 75% ), while keeping the total expert count and the total activation count fixed so that all models share similar FLOPs. Higher Meta-Expert proportions consistently reduce both the cross-entropy loss and the expert redundancy ρ ; increasing the activation share of Local Experts reduces redundancy but degrades the loss.
Figure 5: Scaling-ladder on the Meta/Local Expert ratio. With the per-layer activation fixed to 4 Meta Experts and 4 Local Experts, we scale the total-number ratio of Meta Experts ( 25% / 50% / 75% ) across training compute (FLOPs). Although slightly worse than Pure-Meta-Expert Setting, the 75% ratio achieves the best cross-entropy loss over all Partial-Meta Experts setting at every compute budget, and is adopted as the base configuration in all subsequent experiments.
Figure 6: Pilot experiments on the Meta/Local Expert topology. We vary where Local Experts are used exclusively (the first and/or the last 1/2/4/8 layers), and how the Meta Experts are hierarchically organized ( 2 / 4 / 8 sub-pools, accessed in a looped manner, either chunked or interleaved). Both interventions generally degrade the loss, corroborating the benefit of Meta Experts, while a lighter degree of intervention—only the first and the last two layers restricted to Local Experts, or the Meta Experts looped only twice—slightly improves it.
Figure 7: Scaling-ladder on the Meta/Local Expert topology. We scale the two best pilot configurations—Local Experts exclusively at both shallow and deep ends, and interleaved sub-pool partitioning of the Meta Experts—across training compute (FLOPs). Restricting the first two and the last two layers to Local Experts, together with 2 sub-pools, achieves the best performance at every compute budget; as the model becomes deeper, more Local-Experts-only layers and a larger hierarchy of Meta Experts further help, suggesting that deeper models call for a larger hierarchical expert topology.
Figure 8: Scaling-ladder on cumulative Meta-Expert designs. Starting from a 75% Meta-Expert total-number ratio (MR75), we incrementally add topology refinements—Local Experts only in the first and last two layers (F2L2) and two looped Meta-Expert sub-pools (Loop2)—and finally combine both (F2L2 + Loop2). All models are compared against the Vanilla MoE (dashed line at 1.0 ) in loss ratio L/LVanilla (left) and expert redundancy ratio R/RVanilla (right) across training compute (FLOPs). Every refinement improves over Vanilla at every compute budget, with the largest gains under the combined design; the advantage in both loss and redundancy becomes more pronounced as training compute grows.
Dataset
Vanilla
MR75
MR75
MR75
MR75
+ Loop2
+ F2L2
+ F2L2 + Loop2
MMLU
0.4087
0.4120
0.4175
0.3928
0.4052
C-Eval
0.4354
0.4443
0.4487
0.4160
0.4302
CMMLU
0.4203
0.4161
0.4344
0.4198
0.4249
TriviaQA
0.1924
0.1964
0.1932
0.2014
0.2042
BBH
0.2972
0.2912
0.3030
0.3061
0.2984
Table 1: Downstream results for the 6B-A1.6B models after training on 200B tokens. MMLU / C-Eval / CMMLU are 5-shot accuracy, OpenBookQA is normalized accuracy, and BBH / TriviaQA are generation exact-match. The best result per dataset is in bold . Vanilla is the Local-Experts-only baseline (Vanilla MoE); MR75 activates Meta Experts at a 75% meta ratio; + Loop2 partitions the Meta Experts into two interleaved sub-pools; and + F2L2 restricts the first two and last two layers to Local Experts only.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Depth L
Hyperparameter
12
16
20
24
Hidden size dmodel
768
1024
1280
1536
Attention heads
6
8
10
12
GQA query groups
1
1
1
2
Full-attention layers ( L/4 )
3
4
5
6
GDN key/value heads
6/12
8/16
10/20
12/24
Appendix
Table 2: Per-depth architecture of the scaling ladder. Every rung uses 128 routed experts plus one gated shared expert, top- 8 routing, a period-4 GDN/full-attention pattern (one quarter full attention), and a head dimension of 256 . Widths scale jointly with depth. The table specifies the shared vanilla backbone at each rung; the Meta Experts variants replace only the routed-expert layers and keep the same 128×L routed-expert budget, so every variant at a given rung shares these dimensions. GDN key/value entries are head counts (each head has dimension 128 ).
Depth L
Setting
12
16
20
24
Global batch size (seqs)
96
160
256
384
Micro-batch size
2
2
2
2
Training steps
6747
8436
9763
11101
Warmup steps
43
26
16
11
WSD-decay steps
675
844
976
1110
Appendix
Table 3: Per-depth training schedule and compute of the scaling ladder. Batch size, learning rate, and token budget depend only on depth. GBS is the global batch size in sequences of length 8192 ; the token budget is GBS×8192×iters . FLOPs are the training-compute values used for the compute-versus-loss analysis. Warmup and WSD-decay are given in optimizer steps.
Family
Variant
Meta/layer
Local
Topk (m + l)
Local-only
Meta
(share)
layers
loop
Vanilla
—
— (0%)
128
0 + 8
—
1
Ratio
MR25
32 (25%)
96
4 + 4
—
1
MR50
64 (50%)
64
4 + 4
—
1
MR75
96 (75%)
32
4 + 4
—
1
First/last
F1L1
96 (75%)
32
4 + 4
first/last 1
1
Appendix
Table 4: The Meta Experts variants. Each row is one variant, run at all four depths L∈{12,16,20,24} ; only the routed-expert layers differ from the vanilla backbone of Table 2 , and every variant keeps the 128×L routed-expert budget (iso-param) and the per-token top- 8 activation (iso-FLOP). “Meta/layer” denotes the per-layer-equivalent allocation used to construct the shared cross-layer Meta-expert pool, while “Local” denotes the number of layer-private Local experts per layer; topk lists meta + local experts activated per token. “Local-only layers” names the boundary layers restricted to 128 Local experts (no meta); “Meta loop” is the number of interleaved sub-pools the shared Meta pool is looped through. The last column is the label used in the main text.
Variant
Meta/layer
Local
Topk (m + l)
Meta loop
Meta pool ( L=16 )
mpl90
90
32
4 + 4
2
1080
mpl84
84
32
4 + 4
2
1008
mpl72
72
32
4 + 4
2
864
mpl60
60
32
4 + 4
2
720
mpl48
48
32
4 + 4
2
576
Appendix
Table 5: Shared-Meta param-reduction ablation, run at all four depths off the F2L2 + Loop2 point. Only the per-layer Meta width is shrunk from the r75 base of 96 ; Local ( 32 ) and the per-token top- 8 ( 4 meta +4 local) are unchanged, so the activated FLOPs are fixed while the total routed-expert count (stored parameters) falls below the 128L budget. The shared Meta pool is Meta/layer×(L−4) (the four boundary layers are Local-only, so they carry no meta). The L=16 pool size is shown as an example.
Mixture-of-Experts (MoE) models route each token to a subset of expert networks, increasing capacity while keeping per-token computation sparse. In many deployed MoEs, the number of active experts is fixed across layers and tasks, although layer roles and expert redundancy vary with depth and demand varies with difficulty. Existing approaches address only part of this setting: layer-wise allocations are usually determined offline and reused for all tasks, while token-level methods vary expert activation using local routing signals without task-level context. We propose MetaNet, a support-set controller that predicts, for each layer, an expert-retention threshold and a bounded routing bias. The backbone, experts, and router remain frozen. On DeepSeek-MoE-16B-Chat, MetaNet provides a tunable accuracy-expert-activation trade-off. Relative to fixed k=6, a conservative setting activates 3.61 experts on average (40% fewer) and achieves comparable MMLU accuracy (0.489 vs. 0.474), whereas an aggressive setting activates 2.28 experts on average (62% fewer) with accuracy approximately 3.7 percentage points lower. The MMLU-trained controller also transfers to C-Eval without retraining, activating 2.90 experts on average (52% fewer than fixed k=6) at 0.386 accuracy.
Rongfeng Wang, Shichao Weng, Zhiqiang Wang +4
Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · Nanjing College, University of Chinese Academy of Sciences, Nanjing, China · Nanjing Institute of Information Superbahn, Nanjing, China +3
Mixture-of-Experts (MoE) architectures decouple model capacity from computational cost, yet incur high memory footprints as parameters grow linearly with the number of experts. Recurrent Transformers achieve parameter efficiency by reusing layer weights, but typically lack the capacity for competitive language modeling. We propose Mixture of Reused Experts (MoRE), a hybrid that shares expert pools across groups of adjacent layers. Each layer retains its own router but selects from a larger shared pool, expanding the diversity of routing combinations without additional parameters. To enable shared experts to distinguish between layers, we introduce lightweight learnable depth embeddings that condition each layer's input before routing. Experiments across three model scales (114M-1.15B parameters) show that MoRE consistently achieves lower perplexity and stronger downstream performance than standard MoEs and state-of-the-art weight-sharing architectures at matched compute and parameter budgets, with only minimal modifications to existing MoE implementations.
Eric S. Qiu, Utku Umur Acikalin, Justin Lovelace +4
Department of Computer Science, Cornell University
Mixture of Experts (MoE) architecture has become the standard for state-of-the-art large language models, owing to its computational efficiency through sparse expert activation. However, sparsity through finer expert granularity is becoming increasingly difficult to achieve due to fundamental training challenges such as expert collapse and load imbalance. In this work, we explore and leverage intra-expert activation sparsity as a complementary and underexplored dimension of sparsity in MoE models. Surprisingly, substantial intra-expert sparsity is readily available in existing pre-trained MoE models, without any modification to the activation function or model parameters, providing up to 90% sparsity within each expert without significant accuracy loss. We explore intra-expert activation sparsity across eight off-the-shelf MoE models ranging from 1B to 400B parameters, and extend the MoE execution pipeline of vLLM to leverage intra-expert activation sparsity by skipping the computations of inactive neurons, on top of its existing optimizations, achieving up to 2.5 times speedup in MoE layer execution and 1.2 times end-to-end speedup compared to the original dense vLLM baseline.
Jongseok Park, Sunga Kim, Zhenyu Gu +2
University of California, Berkeley · Work done during an internship at Advanced Micro Devices, Inc. · Advanced Micro Devices, Inc.