Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE

Ermo Hua; Xiangyu Hong; Baiting Wu; Che Jiang; Haozheng Hou; Yicheng Gu; Hanjing Wang; Qirui Zeng

cs.AIOct 20, 2026

Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE

Authors: Ermo Hua, Xiangyu Hong, Baiting Wu, Che Jiang, Haozheng Hou, Yicheng Gu, Hanjing Wang, Qirui Zeng,

Organizations: Tsinghua University · Shanghai AI Laboratory

Abstract

As model parameters scale up, sparsely activated Mixture-of-Experts (MoE) approaches have become essential for model scaling. However, mainstream models currently adopt hierarchical architectures, where experts in different layers, although topologically independent, exhibit substantial similarity and redundancy in their stored content. Compared to the hierarchically arranged Local Expert in existing mainstream models, we hypothesize that inter-layer shared Meta Expert hold greater potential. To this end, we conduct a multi-dimensional empirical comparison between Local and Meta Experts, and find that models incorporating Meta Experts exhibit lower parameter redundancy, together with lower cross-entropy loss and higher downstream task accuracy. With the activated quantity and the expert granularity held fixed, our model matches the performance of Vanilla MoE using only 70--80% of the experts, substantially improving the model's scaling potential. Beyond removing inter-layer redundancy among experts and improving information storage efficiency, our theoretical analysis further shows that, compared with Local Experts, Meta Experts provide hidden states with more flexible residual pathways, thereby improving information retrieval efficiency.

Figures & tables

Figure 1: Comparison between Vanilla and Meta-Expert models. All models are trained under the optimal setting of the Vanilla model, at varying depths and scales. Models with Meta Experts achieve comparable loss to the Vanilla baselines with only 70–80% as many experts, or 75–90% as many parameters, and the savings in both experts and parameters grow as the models get deeper and larger, indicating the favorable scalability of Meta Experts.

Figure 2: The Implementation of Local and Meta Experts. In contrast to the layer-private Local Experts, Meta Experts are shared across all layers.

Figure 3: The Quantitative and Topological Comparison between Meta and Local Experts.

Figure 4: Pilot experiments on the Meta/Local Expert ratio. We vary the total-number proportion of Meta Experts (

0\%

–

100\%

) and their per-layer activation proportion (

25\%

–

75\%

), while keeping the total expert count and the total activation count fixed so that all models share similar FLOPs. Higher Meta-Expert proportions consistently reduce both the cross-entropy loss and the expert redundancy

\rho

; increasing the activation share of Local Experts reduces redundancy but degrades the loss.

Figure 5: Scaling-ladder on the Meta/Local Expert ratio. With the per-layer activation fixed to 4 Meta Experts and 4 Local Experts, we scale the total-number ratio of Meta Experts (

25\%

50\%

75\%

) across training compute (FLOPs). Although slightly worse than Pure-Meta-Expert Setting, the

75\%

ratio achieves the best cross-entropy loss over all Partial-Meta Experts setting at every compute budget, and is adopted as the base configuration in all subsequent experiments.

Figure 6: Pilot experiments on the Meta/Local Expert topology. We vary where Local Experts are used exclusively (the first and/or the last 1/2/4/8 layers), and how the Meta Experts are hierarchically organized (

2

4

8

sub-pools, accessed in a looped manner, either chunked or interleaved). Both interventions generally degrade the loss, corroborating the benefit of Meta Experts, while a lighter degree of intervention—only the first and the last two layers restricted to Local Experts, or the Meta Experts looped only twice—slightly improves it.

Figure 7: Scaling-ladder on the Meta/Local Expert topology. We scale the two best pilot configurations—Local Experts exclusively at both shallow and deep ends, and interleaved sub-pool partitioning of the Meta Experts—across training compute (FLOPs). Restricting the first two and the last two layers to Local Experts, together with

2

sub-pools, achieves the best performance at every compute budget; as the model becomes deeper, more Local-Experts-only layers and a larger hierarchy of Meta Experts further help, suggesting that deeper models call for a larger hierarchical expert topology.

Figure 8: Scaling-ladder on cumulative Meta-Expert designs. Starting from a

75\%

Meta-Expert total-number ratio (MR75), we incrementally add topology refinements—Local Experts only in the first and last two layers (F2L2) and two looped Meta-Expert sub-pools (Loop2)—and finally combine both (F2L2 + Loop2). All models are compared against the Vanilla MoE (dashed line at

1.0

) in loss ratio

L/L_{\mathrm{Vanilla}}

(left) and expert redundancy ratio

R/R_{\mathrm{Vanilla}}

(right) across training compute (FLOPs). Every refinement improves over Vanilla at every compute budget, with the largest gains under the combined design; the advantage in both loss and redundancy becomes more pronounced as training compute grows.

Dataset	Vanilla	MR75	MR75	MR75	MR75
			+ Loop2	+ F2L2	+ F2L2 + Loop2
MMLU	0.4087	0.4120	0.4175	0.3928	0.4052
C-Eval	0.4354	0.4443	0.4487	0.4160	0.4302
CMMLU	0.4203	0.4161	0.4344	0.4198	0.4249
TriviaQA	0.1924	0.1964	0.1932	0.2014	0.2042
BBH	0.2972	0.2912	0.3030	0.3061	0.2984

Table 1: Downstream results for the 6B-A1.6B models after training on 200B tokens. MMLU / C-Eval / CMMLU are 5-shot accuracy, OpenBookQA is normalized accuracy, and BBH / TriviaQA are generation exact-match. The best result per dataset is in bold . Vanilla is the Local-Experts-only baseline (Vanilla MoE); MR75 activates Meta Experts at a 75% meta ratio; + Loop2 partitions the Meta Experts into two interleaved sub-pools; and + F2L2 restricts the first two and last two layers to Local Experts only.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

	Depth $L$
Hyperparameter	12	16	20	24
Hidden size $d_{\text{model}}$	768	1024	1280	1536
Attention heads	6	8	10	12
GQA query groups	1	1	1	2
Full-attention layers ( $L/4$ )	3	4	5	6
GDN key/value heads	6/12	8/16	10/20	12/24

Appendix

Table 2: Per-depth architecture of the scaling ladder. Every rung uses

128

routed experts plus one gated shared expert, top-

8

routing, a period-4 GDN/full-attention pattern (one quarter full attention), and a head dimension of

256

. Widths scale jointly with depth. The table specifies the shared vanilla backbone at each rung; the Meta Experts variants replace only the routed-expert layers and keep the same

128\times L

routed-expert budget, so every variant at a given rung shares these dimensions. GDN key/value entries are head counts (each head has dimension

128

	Depth $L$
Setting	12	16	20	24
Global batch size (seqs)	96	160	256	384
Micro-batch size	2	2	2	2
Training steps	6747	8436	9763	11101
Warmup steps	43	26	16	11
WSD-decay steps	675	844	976	1110

Appendix

Table 3: Per-depth training schedule and compute of the scaling ladder. Batch size, learning rate, and token budget depend only on depth. GBS is the global batch size in sequences of length

8192

; the token budget is

\mathrm{GBS}\times 8192\times\text{iters}

. FLOPs are the training-compute values used for the compute-versus-loss analysis. Warmup and WSD-decay are given in optimizer steps.

Family	Variant	Meta/layer	Local	Topk (m $+$ l)	Local-only	Meta
		(share)			layers	loop
Vanilla	—	— (0%)	128	0 $+$ 8	—	1
Ratio	MR25	32 (25%)	96	4 $+$ 4	—	1
	MR50	64 (50%)	64	4 $+$ 4	—	1
	MR75	96 (75%)	32	4 $+$ 4	—	1
First/last	F1L1	96 (75%)	32	4 $+$ 4	first/last $1$	1

Appendix

Table 4: The Meta Experts variants. Each row is one variant, run at all four depths

L\in\{12,16,20,24\}

; only the routed-expert layers differ from the vanilla backbone of Table 2 , and every variant keeps the

128\times L

routed-expert budget (iso-param) and the per-token top-

8

activation (iso-FLOP). “Meta/layer” denotes the per-layer-equivalent allocation used to construct the shared cross-layer Meta-expert pool, while “Local” denotes the number of layer-private Local experts per layer; topk lists meta

+

local experts activated per token. “Local-only layers” names the boundary layers restricted to

128

Local experts (no meta); “Meta loop” is the number of interleaved sub-pools the shared Meta pool is looped through. The last column is the label used in the main text.

Variant	Meta/layer	Local	Topk (m $+$ l)	Meta loop	Meta pool ( $L{=}16$ )
mpl90	90	32	4 $+$ 4	2	$1080$
mpl84	84	32	4 $+$ 4	2	$1008$
mpl72	72	32	4 $+$ 4	2	$864$
mpl60	60	32	4 $+$ 4	2	$720$
mpl48	48	32	4 $+$ 4	2	$576$

Appendix

Table 5: Shared-Meta param-reduction ablation, run at all four depths off the F2L2

+

Loop2 point. Only the per-layer Meta width is shrunk from the r75 base of

96

; Local (

32

) and the per-token top-

8

(

4

Sparse Mixture-of-Experts

Sparse mixture-of-experts models route each input to a small subset of specialized networks to increase capacity while limiting computation.

See all 121 papers

Meta-Learning Where to Allocate Experts: Task-Conditioned Layer-Wise Compression for MoEs
Aug 27, 2026Rongfeng Wang, Shichao Weng, Zhiqiang Wang +4Mixture-of-Experts Inference Mixture-of-Experts Models
Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · Nanjing College, University of Chinese Academy of Sciences, Nanjing, China · Nanjing Institute of Information Superbahn, Nanjing, China +3
MoRE: Mixture of Reused Experts
Sep 16, 2026Eric S. Qiu, Utku Umur Acikalin, Justin Lovelace +4Mixture-of-Experts Language Models Sparse Mixture-of-Experts
Department of Computer Science, Cornell University
Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Model Execution
May 9, 2026Jongseok Park, Sunga Kim, Zhenyu Gu +2Activation Sparsity Mixture-of-Experts Inference
University of California, Berkeley · Work done during an internship at Advanced Micro Devices, Inc. · Advanced Micro Devices, Inc.

Aug 27, 2026J/K move · Enter open · S save

Meta-Learning Where to Allocate Experts: Task-Conditioned Layer-Wise Compression for MoEs

Rongfeng Wang, Shichao Weng, Zhiqiang Wang, Xinyu Liu+3

Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · Nanjing College, University of Chinese Academy of Sciences, Nanjing, China · Nanjing Institute of Information Superbahn, Nanjing, China+3

Mixture-of-Experts Inference Mixture-of-Experts Models LLM Inference Acceleration Meta-Learning

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) models route each token to a subset of expert networks, increasing capacity while keeping per-token computation sparse. In many deployed MoEs, the number of active experts is fixed across layers and tasks, although layer roles and expert redundancy vary with depth and demand varies with difficulty. Existing approaches address only part of this setting: layer-wise allocations are usually determined offline and reused for all tasks, while token-level methods vary expert activation using local routing signals without task-level context. We propose MetaNet, a support-set controller that predicts, for each layer, an expert-retention threshold and a bounded routing bias. The backbone, experts, and router remain frozen. On DeepSeek-MoE-16B-Chat, MetaNet provides a tunable accuracy-expert-activation trade-off. Relative to fixed k=6, a conservative setting activates 3.61 experts on average (40% fewer) and achieves comparable MMLU accuracy (0.489 vs. 0.474), whereas an aggressive setting activates 2.28 experts on average (62% fewer) with accuracy approximately 3.7 percentage points lower. The MMLU-trained controller also transfers to C-Eval without retraining, activating 2.90 experts on average (52% fewer than fixed k=6) at 0.386 accuracy.

Abstract

Sparse Mixture-of-Experts

Explore similar work

Meta-Learning Where to Allocate Experts: Task-Conditioned Layer-Wise Compression for MoEs

MoRE: Mixture of Reused Experts

Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Model Execution