Organizations: Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · Nanjing College, University of Chinese Academy of Sciences, Nanjing, China · Nanjing Institute of Information Superbahn, Nanjing, China · Dobot Robotics, Shenzhen, China · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
Mixture-of-Experts (MoE) models route each token to a subset of expert networks, increasing capacity while keeping per-token computation sparse. In many deployed MoEs, the number of active experts is fixed across layers and tasks, although layer roles and expert redundancy vary with depth and demand varies with difficulty. Existing approaches address only part of this setting: layer-wise allocations are usually determined offline and reused for all tasks, while token-level methods vary expert activation using local routing signals without task-level context. We propose MetaNet, a support-set controller that predicts, for each layer, an expert-retention threshold and a bounded routing bias. The backbone, experts, and router remain frozen. On DeepSeek-MoE-16B-Chat, MetaNet provides a tunable accuracy-expert-activation trade-off. Relative to fixed k=6, a conservative setting activates 3.61 experts on average (40% fewer) and achieves comparable MMLU accuracy (0.489 vs. 0.474), whereas an aggressive setting activates 2.28 experts on average (62% fewer) with accuracy approximately 3.7 percentage points lower. The MMLU-trained controller also transfers to C-Eval without retraining, activating 2.90 experts on average (52% fewer than fixed k=6) at 0.386 accuracy.
Figures & tables
Figure 1: An MoE layer: the router activates 2 of 6 experts per token; inactive experts grayed out.
Figure 2: Overview of MetaNet. Left: support examples Sτ pass through the frozen backbone; routing statistics are fed to MetaNet gϕ , producing a per-layer retention threshold ρτ,l and prior score pτ,l . Right: at query time, the corresponding bounded bias is added to gate logits (2) ; cumulative-mass thresholding with ρτ,l determines kτ,l (3) ; TopKkτ,l selects experts (4) . Orange: activated; gray: skipped. Backbone parameters are unchanged.
Method
Accuracy
Act. ratio
Mean k
MMLU (meta-test, 12 tasks × 16 queries)
Fixed k=6
0.474
0.094
6.00
Fixed k=12
0.495
0.188
12.00
Fixed k=4
0.432
0.063
4.00
Fixed k=3
0.417
0.047
3.00
HyperRouter-style
0.406
0.094
6.00
Table 1: Main results on MMLU and C-Eval-Full. Bold accuracy marks the stronger MetaNet setting at or below the native k=6 budget; bold active-ratio and mean- k values mark the minimum routed-expert workload in each panel. In the MMLU panel, fixed k=12 is an upper-budget reference and is excluded from the bold accuracy comparison. The C-Eval fixed- k=12 result was measured only on C-Eval-10 and is reported separately in Table 7 . The conservative setting uses a lower budget penalty weight λbudget .
Backbone
Method
MMLU
C-Eval
Mean k
DeepSeek-MoE 16B-Chat ( Knat=6 , 64 exp.)
Fixed k=6
0.474
0.444
6.00
Fixed k=4
0.432
0.443
4.00
Fixed k=2
0.365
0.377
2.00
MetaNet
0.438
0.386
2.28
OLMoE-1B-7B ( Knat=8 , 64 exp.)
Fixed k=8
0.521
0.363
8.00
Fixed k=4
0.479
0.350
4.00
Table 2: Cross-backbone generalization (Exp. II). MetaNet trained on DeepSeek and evaluated zero-shot on OLMoE-1B-7B [ 43 ] . Fixed- k rows report each backbone’s native and lower budgets; the final column reports mean k on MMLU.
Dataset
Method
Accuracy
Act. ratio
syn-pattern
Original
1.00
1.00
Static
1.00
0.75
MetaNet
0.98
0.75
syn-pattern-shift
Original
0.99
1.00
Static
0.58
0.75
MetaNet
0.66
0.75
Table 3: Cross-architecture proof-of-concept results (Experiment III, GoogLeNet backbone).
Variant
Acc.
Act. ratio
Mean k
Gate agr.
Low-use
Fixed k=6
0.474
0.094
6.00
–
–
Budget-only
0.375
0.045
2.89
0.038
0.734
Prior-only
0.484
0.188
12.00
0.622
0.318
Full method
0.438
0.036
2.28
0.943
0.004
Table 4: Method-component ablation on MMLU. Dashes denote controller diagnostics omitted for fixed routing.
Variant
Acc.
Act. ratio
Mean k
Fixed k=6 (all layers)
0.474
0.094
6.00
All layers
0.438
0.036
2.28
Shallow-9 (0–8)
0.396
0.087
5.59
Middle-9
0.427
0.074
4.75
Deep-9 (18–26)
0.469
0.069
4.40
FrontBack-10
0.427
0.081
5.16
Table 5: Layer-mask ablation on MMLU.
Figure 3: Support-derived layer–expert routing profile for the selected MMLU checkpoint ( 27 routed layers ×64 experts; color denotes mean normalized routing mass).
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Backbone & modules
Model
DeepSeek-MoE-16B-Chat
Frozen
backbone, experts, original gate
Trainable
MetaNet only
Routed experts / layer; native top- k ( Knat )
64; 6
Retention profile
Appendix
Table 6: Hyperparameter configuration ( R=12 ) used in the main experiments.
Method
Acc.
Act. ratio
Mean k
Lat. (ms)
MMLU: 12 subjects, 192 queries
Fixed k=6
0.474
0.094
6.00
542.94
Fixed k=12
0.495
0.188
12.00
557.65
HyperRouter-style ∗
0.406
0.094
6.00
539.32
Probe Pruning adapted
0.188
0.094
6.00
180.45
AdaMoE adapted ∗
0.479
0.090
5.76
577.90
Appendix
Table 7: Expanded comparison under the common inference harness. “Adapted” denotes an inference-compatible decision rule rather than an exact reproduction. C-Eval-10 is separate from C-Eval-Full in Table 1 . ∗ The original method requires training or replacement of routing components.
Scope
Profile
Acc.
Act. ratio
Mean k
Lat. (ms)
MMLU
bt0p3
0.460
0.027
1.75
527.07
bt0p4
0.472
0.031
1.98
527.96
bt0p5
0.477
0.040
2.55
516.02
bt0p8
0.494
0.045
2.88
502.15
C-Eval-10
bt0p3
0.375
0.036
2.30
457.05
bt0p4
0.487
0.102
6.50
457.41
Appendix
Table 8: Additional operating-point sweep. Active ratio is the realized mean top- k divided by 64 . Latency is measured in the same three-GPU harness.
Method
Seed
Acc.
Act. ratio
Mean k
Lat. (ms)
Fixed k=6
0
0.484
0.094
6.00
537.83
Fixed k=6
1
0.526
0.094
6.00
538.22
Fixed k=12
0
0.503
0.188
12.00
538.69
Fixed k=12
1
0.531
0.188
12.00
540.36
MetaNet (conservative)
0
0.490
0.080
5.13
514.58
MetaNet (conservative)
1
0.521
0.074
4.75
511.83
Appendix
Table 9: Repeated MMLU evaluation. Active ratio is mean top- k divided by 64 .
As model parameters scale up, sparsely activated Mixture-of-Experts (MoE) approaches have become essential for model scaling. However, mainstream models currently adopt hierarchical architectures, where experts in different layers, although topologically independent, exhibit substantial similarity and redundancy in their stored content. Compared to the hierarchically arranged Local Expert in existing mainstream models, we hypothesize that inter-layer shared Meta Expert hold greater potential. To this end, we conduct a multi-dimensional empirical comparison between Local and Meta Experts, and find that models incorporating Meta Experts exhibit lower parameter redundancy, together with lower cross-entropy loss and higher downstream task accuracy. With the activated quantity and the expert granularity held fixed, our model matches the performance of Vanilla MoE using only 70--80% of the experts, substantially improving the model's scaling potential. Beyond removing inter-layer redundancy among experts and improving information storage efficiency, our theoretical analysis further shows that, compared with Local Experts, Meta Experts provide hidden states with more flexible residual pathways, thereby improving information retrieval efficiency.
Mixture-of-Experts (MoE) has become a dominant architecture for scaling large language models due to their sparse activation mechanism. However, the substantial number of expert activations creates a critical latency bottleneck during inference, especially in resource-constrained deployment scenarios. Existing approaches that reduce expert activations potentially lead to severe model performance degradation. In this work, we introduce the concept of activation budget as a constraint on the number of expert activations and propose Alloc-MoE, a unified framework that optimizes budget allocation coordinately at both the layer and token levels to minimize performance degradation. At the layer level, we introduce Alloc-L, which leverages sensitivity profiling and dynamic programming to determine the optimal allocation of expert activations across layers. At the token level, we propose Alloc-T, which dynamically redistributes activations based on routing scores, optimizing budget allocation without increasing latency. Extensive experiments across multiple MoE models demonstrate that Alloc-MoE maintains model performance under a constrained activation budget. Especially, Alloc-MoE achieves 1.15× prefill and 1.34× decode speedups on DeepSeek-V2-Lite at half of the original budget.
Baihui Liu, Kaiyuan Tian, Wei Wang +3
National Key Laboratory of Parallel and Distributed Computing, College of Computer Science and Technology, National University of Defense Technology
Mixture-of-Experts (MoE) architectures scale model capacity efficiently, yet their substantial memory footprint poses a major barrier to deployment. Expert pruning reduces this footprint, but most methods focus on which experts to remove and prune every layer equally. We propose SPARE (Sparsity Allocation via Routing Entropy), a training-free, selector-agnostic method that decides how many experts to remove from each layer using routing entropy, the entropy of tokens' normalized Top-K gate weights on calibration data. When a token's gate weight is spread evenly, removing one active expert discards little of it, so SPARE prunes layers with more diffuse routing more aggressively. A perturbation bound under random pruning formalizes this intuition, and controlled interventions with a fixed selector link concentrated routing to larger output changes. With the same selector and budget, SPARE improves average accuracy over uniform allocation on Mixtral, Qwen3, and DeepSeek models, especially at high pruning ratios. At 50% expert pruning, it raises the eight-task average of Qwen3-30B-A3B from 57.2% to 63.8%, and the math, code, and science average of DeepSeek-R1 from 80.1 to 82.8.