Meta-Learning Where to Allocate Experts: Task-Conditioned Layer-Wise Compression for MoEs

Rongfeng Wang; Shichao Weng; Zhiqiang Wang; Xinyu Liu; Yang Yi; Peilong Zhou; Hongwei Tang

cs.CLAug 27, 2026

Meta-Learning Where to Allocate Experts: Task-Conditioned Layer-Wise Compression for MoEs

Authors: Rongfeng Wang, Shichao Weng, Zhiqiang Wang, Xinyu Liu, Yang Yi, Peilong Zhou, Hongwei Tang

Organizations: Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · Nanjing College, University of Chinese Academy of Sciences, Nanjing, China · Nanjing Institute of Information Superbahn, Nanjing, China · Dobot Robotics, Shenzhen, China ·

Abstract

Mixture-of-Experts (MoE) models route each token to a subset of expert networks, increasing capacity while keeping per-token computation sparse. In many deployed MoEs, the number of active experts is fixed across layers and tasks, although layer roles and expert redundancy vary with depth and demand varies with difficulty. Existing approaches address only part of this setting: layer-wise allocations are usually determined offline and reused for all tasks, while token-level methods vary expert activation using local routing signals without task-level context. We propose MetaNet, a support-set controller that predicts, for each layer, an expert-retention threshold and a bounded routing bias. The backbone, experts, and router remain frozen. On DeepSeek-MoE-16B-Chat, MetaNet provides a tunable accuracy-expert-activation trade-off. Relative to fixed k=6, a conservative setting activates 3.61 experts on average (40% fewer) and achieves comparable MMLU accuracy (0.489 vs. 0.474), whereas an aggressive setting activates 2.28 experts on average (62% fewer) with accuracy approximately 3.7 percentage points lower. The MMLU-trained controller also transfers to C-Eval without retraining, activating 2.90 experts on average (52% fewer than fixed k=6) at 0.386 accuracy.

Figures & tables

Panel of Figure 1 — Figure 1: An MoE layer: the router activates 2 of 6 experts per token; inactive experts grayed out.

Panel of Figure 2 — Figure 2: Overview of MetaNet. Left: support examples $S_{\tau}$ pass through the frozen backbone; routing statistics are fed to MetaNet $g_{\phi}$ , producing a per-layer retention threshold $\rho_{\tau,l}$ and prior score $\mathbf{p}_{\tau,l}$ . Right: at query time, the corresponding bounded bias is added to gate logits (2) ; cumulative-mass thresholding with $\rho_{\tau,l}$ determines $k_{\tau,l}$ (3) ; $\mathrm{TopK}_{k_{\tau,l}}$ selects experts (4) . Orange: activated; gray: skipped. Backbone parameters are unchanged.

MMLU (meta-test, 12 tasks $\times$ 16 queries)
Method	Accuracy	Act. ratio	Mean $k$
Fixed $k{=}6$	0.474	0.094	6.00
Fixed $k{=}12$	0.495	0.188	12.00
Fixed $k{=}4$	0.432	0.063	4.00
Fixed $k{=}3$	0.417	0.047	3.00
HyperRouter-style	0.406	0.094	6.00

Table 1: Main results on MMLU and C-Eval-Full. Bold accuracy marks the stronger MetaNet setting at or below the native

k{=}6

budget; bold active-ratio and mean-

k

values mark the minimum routed-expert workload in each panel. In the MMLU panel, fixed

k{=}12

is an upper-budget reference and is excluded from the bold accuracy comparison. The C-Eval fixed-

k{=}12

result was measured only on C-Eval-10 and is reported separately in Table 7 . The conservative setting uses a lower budget penalty weight

\lambda_{\mathrm{budget}}

Backbone	Method	MMLU	C-Eval	Mean $k$
DeepSeek-MoE 16B-Chat ( $K_{\mathrm{nat}}{=}6$ , 64 exp.)	Fixed $k{=}6$	0.474	0.444	6.00
	Fixed $k{=}4$	0.432	0.443	4.00
	Fixed $k{=}2$	0.365	0.377	2.00
	MetaNet	0.438	0.386	2.28
OLMoE-1B-7B ( $K_{\mathrm{nat}}{=}8$ , 64 exp.)	Fixed $k{=}8$	0.521	0.363	8.00
	Fixed $k{=}4$	0.479	0.350	4.00

Table 2: Cross-backbone generalization (Exp. II). MetaNet trained on DeepSeek and evaluated zero-shot on OLMoE-1B-7B [ 43 ] . Fixed-

k

rows report each backbone’s native and lower budgets; the final column reports mean

k

on MMLU.

Dataset	Method	Accuracy	Act. ratio
syn-pattern	Original	1.00	1.00
	Static	1.00	0.75
	MetaNet	0.98	0.75
syn-pattern-shift	Original	0.99	1.00
	Static	0.58	0.75
	MetaNet	0.66	0.75

Table 3: Cross-architecture proof-of-concept results (Experiment III, GoogLeNet backbone).

Variant	Acc.	Act. ratio	Mean $k$	Gate agr.	Low-use
Fixed $k{=}6$	0.474	0.094	6.00	–	–
Budget-only	0.375	0.045	2.89	0.038	0.734
Prior-only	0.484	0.188	12.00	0.622	0.318
Full method	0.438	0.036	2.28	0.943	0.004

Table 4: Method-component ablation on MMLU. Dashes denote controller diagnostics omitted for fixed routing.

Variant	Acc.	Act. ratio	Mean $k$
Fixed $k{=}6$ (all layers)	0.474	0.094	6.00
All layers	0.438	0.036	2.28
Shallow-9 (0–8)	0.396	0.087	5.59
Middle-9	0.427	0.074	4.75
Deep-9 (18–26)	0.469	0.069	4.40
FrontBack-10	0.427	0.081	5.16

Table 5: Layer-mask ablation on MMLU.

Figure 3: Support-derived layer–expert routing profile for the selected MMLU checkpoint (

27

routed layers

\times

64

experts; color denotes mean normalized routing mass).

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Backbone & modules
Model	DeepSeek-MoE-16B-Chat
Frozen	backbone, experts, original gate
Trainable	MetaNet only
Routed experts / layer; native top- $k$ ( $K_{\mathrm{nat}}$ )	64; 6
Retention profile

Appendix

Table 6: Hyperparameter configuration (

R=12

) used in the main experiments.

MMLU: 12 subjects, 192 queries
Method	Acc.	Act. ratio	Mean $k$	Lat. (ms)
Fixed $k{=}6$	0.474	0.094	6.00	542.94
Fixed $k{=}12$	0.495	0.188	12.00	557.65
HyperRouter-style ∗	0.406	0.094	6.00	539.32
Probe Pruning adapted	0.188	0.094	6.00	180.45
AdaMoE adapted ∗	0.479	0.090	5.76	577.90

Appendix

Table 7: Expanded comparison under the common inference harness. “Adapted” denotes an inference-compatible decision rule rather than an exact reproduction. C-Eval-10 is separate from C-Eval-Full in Table 1 . ∗ The original method requires training or replacement of routing components.

Scope	Profile	Acc.	Act. ratio	Mean $k$	Lat. (ms)
MMLU	bt0p3	0.460	0.027	1.75	527.07
	bt0p4	0.472	0.031	1.98	527.96
	bt0p5	0.477	0.040	2.55	516.02
	bt0p8	0.494	0.045	2.88	502.15
C-Eval-10	bt0p3	0.375	0.036	2.30	457.05
	bt0p4	0.487	0.102	6.50	457.41

Appendix

Table 8: Additional operating-point sweep. Active ratio is the realized mean top-

k

divided by

64

. Latency is measured in the same three-GPU harness.

Method	Seed	Acc.	Act. ratio	Mean $k$	Lat. (ms)
Fixed $k{=}6$	0	0.484	0.094	6.00	537.83
Fixed $k{=}6$	1	0.526	0.094	6.00	538.22
Fixed $k{=}12$	0	0.503	0.188	12.00	538.69
Fixed $k{=}12$	1	0.531	0.188	12.00	540.36
MetaNet (conservative)	0	0.490	0.080	5.13	514.58
MetaNet (conservative)	1	0.521	0.074	4.75	511.83

Appendix

Table 9: Repeated MMLU evaluation. Active ratio is mean top-

k

divided by

64

Mixture-of-Experts Inference

Mixture-of-experts inference studies how to run sparsely activated expert models efficiently while preserving their quality.

See all 177 papers

Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE
Oct 20, 2026Ermo Hua, Xiangyu Hong, Baiting Wu +11Sparse Mixture-of-Experts Mixture-of-Experts Models
Tsinghua University · Shanghai AI Laboratory
Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inference
Apr 9, 2026Baihui Liu, Kaiyuan Tian, Wei Wang +3Mixture-of-Experts Inference Mixture of Experts
National Key Laboratory of Parallel and Distributed Computing, College of Computer Science and Technology, National University of Defense Technology
SPARE: Sparsity Allocation via Routing Entropy for Efficient MoEs
Oct 21, 2026Yifan Yang, Hanling Yi, Dongquan Yang +4Mixture-of-Experts Pruning Sparse Mixture-of-Experts
Intellifusion Inc.

Oct 20, 2026J/K move · Enter open · S save

Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE

Ermo Hua, Xiangyu Hong, Baiting Wu, Che Jiang+10

Tsinghua University · Shanghai AI Laboratory

Sparse Mixture-of-Experts Mixture-of-Experts Models

Open paper AskPDF arXiv

As model parameters scale up, sparsely activated Mixture-of-Experts (MoE) approaches have become essential for model scaling. However, mainstream models currently adopt hierarchical architectures, where experts in different layers, although topologically independent, exhibit substantial similarity and redundancy in their stored content. Compared to the hierarchically arranged Local Expert in existing mainstream models, we hypothesize that inter-layer shared Meta Expert hold greater potential. To this end, we conduct a multi-dimensional empirical comparison between Local and Meta Experts, and find that models incorporating Meta Experts exhibit lower parameter redundancy, together with lower cross-entropy loss and higher downstream task accuracy. With the activated quantity and the expert granularity held fixed, our model matches the performance of Vanilla MoE using only 70--80% of the experts, substantially improving the model's scaling potential. Beyond removing inter-layer redundancy among experts and improving information storage efficiency, our theoretical analysis further shows that, compared with Local Experts, Meta Experts provide hidden states with more flexible residual pathways, thereby improving information retrieval efficiency.

Abstract

Mixture-of-Experts Inference

Explore similar work

Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE

Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inference

SPARE: Sparsity Allocation via Routing Entropy for Efficient MoEs