Organizations: Zhejiang University · Shanghai Innovation Institute · Shanghai AI Laboratory · Bytedance Seed · Fudan University · Chinese University of Hong Kong.
Mixture-of-Experts (MoE) effectively scales model capacity while preserving computational efficiency through sparse expert activation. However, training high-quality MoEs from scratch is prohibitively expensive. A promising alternative is to convert pretrained dense models into sparse MoEs. Existing dense-to-MoE methods fall into two categories: dynamic structural pruning that converts dense models into MoE architectures with moderate sparsity to balance performance and inference efficiency, and downcycling approaches that use pretrained dense models to initialize highly sparse MoE architectures. However, existing methods break the intrinsic activation patterns within dense models, leading to suboptimal expert construction. In this work, we argue that the Gated Linear Unit (GLU) mechanism provides a natural blueprint for dense-to-MoE conversion. We show that the fine-grained neural-wise activation patterns of GLU reveal a coarse-grained structure, uncovering an inherent MoE architecture composed of consistently activated universal neurons and dynamically activated specialized neurons. Leveraging this discovery, we introduce ExpertWeaver, a training-free framework that partitions neurons according to their activation patterns and constructs shared experts and specialized routed experts with layer-adaptive configurations. Our experiments demonstrate that ExpertWeaver significantly outperforms existing methods, both as a training-free dynamic structural pruning technique and as a downcycling strategy for superior MoE initialization.
Figures & tables
Figure 1: Neuron activation patterns across diverse tasks. We visualize the middle layer’s activation patterns from Qwen2.5-7B on a subset of Flan-v2. a) Activation distribution of neurons across different tasks. b) Activation distribution of neurons within individual task clusters, where tasks belonging to the same cluster are enclosed in boxes.
Method
MMLU(5)
HellaSwag(10)
ARC-e
ARC-C(25)
PiQA
Avg.
LLaMA3-8B, 50% sparsity
Dense
65.3
82.1
77.9
57.9
80.8
72.8
Wanda
55.8
75.0
72.0
51.3
77.3
66.3
SparseGPT
57.4
75.5
72.0
50.3
78.1
66.7
Magnitude
48.3
41.9
53.3
38.7
65.8
49.6
AbsTopk-GLU
58.8
79.4
72.4
51.6
78.6
68.2
Table 1: Performance of AbsTopk-GLU vs. unstructured pruning at 50% sparsity. (Numbers in parentheses denote the number of shots for evaluation; no annotation means zero-shot.)
Figure 2: Neuron Coefficient of Variation Across Layers .
Figure 3: The ExpertWeaver Framework. a) The GLU in the MLP layer contains three weight matrices, where the same color denotes corresponding neuron slices. b) Neuron activation patterns are captured using a multi-task calibration dataset. c) The CVs are computed to determine the budget for shared vs. routed experts. (d) Neurons are clustered according to their activation patterns to form one shared expert and multiple routed experts.
Method
MMLU(5)
HellaSwag(10)
ARC-e
ARC-c
PiQA
Avg.
LLaMA3-8B, 25% sparsity
Dense
65.3
82.1
77.9
57.9
80.8
72.8
LLM-Pruner
24.2
51.3
58.9
32.4
74.4
48.2
FLAP
33.4
48.0
50.0
29.3
68.3
45.8
CMoE
41.6
65.9
63.1
41.5
73.9
57.2
ExpertWeaver
47.0
69.8
64.4
44.3
76.3
60.4
Table 2: Comparison with structured pruning methods under 25% sparsity.
Figure 4: Comparison of Downcycling, Upcycling, and From-Scratch Training. Comparison of training loss, evaluation loss, and downstream task performance using the same OLMoE model configuration under three different MoE initialization paradigms.
Model
MMLU(5)
HellaSwag(10)
ARC-e
ARC-c(25)
PIQA
WinoGrande
LogiQA
SciQ
Average
Dense Models
Qwen2.5-7B
74.1
80.2
77.5
63.7
79.7
73.2
36.4
95.2
72.5
Qwen2.5-3B
65.6
74.6
73.9
56.5
78.8
68.1
33.5
95.2
68.3
Qwen2.5-1.5B
60.9
68.0
72.5
54.9
75.9
63.8
31.9
93.4
65.2
Llama-3.2-3B
56.1
76.4
71.6
50.5
77.4
69.9
30.6
92.7
65.7
OLMo-7B
30.7
77.1
68.7
45.1
79.6
66.5
27.5
88.6
60.8
Table 3: Downcycling Performance Comparison. Best results are in bold, second-best are underlined. Models in gray (Qwen2.5-{7B, 3B, 1.5B}, Llama-3.2-3B, and OLMoE) are for context, as they were trained on significantly more data. Our ExpertWeaver models (OLMo-based and Qwen2.5-based) were trained on 200B tokens. The OLMo-based version has 1B active parameters (7B-A1B), and the Qwen2.5-based version has 3.5B active parameters (7B-A3.5B). We compare against the 500B token checkpoint (*) of OLMoE. Blank entries for OpenMoE indicate unavailable results.
Figure 5: Training Loss Comparison with Different MoE Initialization Strategies.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Ablation studies on key hyperparameters of ExpertWeaver. a) MMLU performance heatmap for different shared expert ratios, controlled by the hyperparameters αmin and αmax in Eq. 5 . The diagonal where αmin=αmax represents a static configuration with a uniform ratio across all layers. b) MMLU performance varying specialization threshold τ from Eq. 4 . c) MMLU performance varying expert granularity.
Method
MMLU(5)
ARC-C(25)
GSM8K
HumanEval
IFEval
Avg.
LLaMA-MoE-3B-7B
28.24
44.03
4.62
12.02
28.10
23.40
OLMoE-1B-7B
53.79
55.63
40.94
40.48
35.49
45.27
LLaMA-MoE-v2
40.90
40.20
55.00
51.20
36.00
44.66
ExpertWeaver-Instruct
50.60
69.80
57.10
50.20
33.10
52.16
Appendix
Table 4: Instruct Model Performance Comparison
GSM8k
HumanEval
GPQADiamond
LiveCodeBench
Base Model
Qwen2.5-7B
DeepSeek-R1-Distill-Qwen-7B
Sparsity
25%
12.5%
25%
12.5%
12.5%
LLM-Pruner
2.0
-
14.6
10.4
33.3
6.1
FLAP
14.8
-
57.6
9.8
36.3
2.3
ExpertWeaver
34.9
18.9
64.6
32.9
37.3
16.8
Appendix
Table 5: Training-Free Pruning on Code Benchmarks. We compare ExpertWeaver against static pruning methods on several code-related benchmarks. The base models used for each benchmark are specified in the table.
GPU
Method
RPS ↑
OTPS ↑
TTPS ↑
TTFT ↓
TPOT ↓
ITL ↓
(ms)
(ms)
(ms)
A100
ExpertWeaver
22.88
2928.86
14644.31
440.6
40.3
40.3
Qwen2.5-7B
18.85
2413.22
12066.10
675.4
47.8
48.2
H100
ExpertWeaver
44.04
5637.42
28187.10
542.2
9.8
9.8
Qwen2.5-7B
41.41
5299.96
26499.78
1404.5
18.5
18.5
Appendix
Table 6: Inference throughput comparison across different GPU architectures. RPS: Requests Per Second. OTPS: Output Tokens Per Second. TTPS: Total Tokens Per Second. TTFT: Time To First Token. TPOT: Time Per Output Token. ITL: Inter-Token Latency.
Method
MMLU
HellaSwag(10)
ARC-e
ARC-c(25)
PiQA
Avg.
Qwen2.5-7B, 25% sparsity
Dense
74.2
80.3
77.8
63.8
80.0
75.2
Ablation on Data Source
ExpertWeaver Flan (default)
61.6
72.3
71.5
53.5
76.3
67.0
ExpertWeaver C4
61.2
71.6
72.1
49.1
74.8
65.8
Ablation on Data Quantity & Diversity
Appendix
Table 7: Ablation Studies on the Calibration Set. We analyze the impact of calibration set size, diversity, and source.
Determines logical relationship between sentences.
mnli, qnli, rte, wnli, anli
Coreference Resolution
Identifies expressions referring to the same entity.
wsc, dpr, winogender
Appendix
Table 8: Tasks from Flan-v2 used in the calibration set.
OLMo 575M
OLMo 1.3B
OLMoE 1.3B-A676M
Model Dimension
2048
2048
2048
FFN Dimension
1024
8192
1024
Attention Heads
16
16
16
Key/Value Heads
16
16
16
Layers
16
16
16
Vocabulary Size
50280
50280
50280
Appendix
Table 9: Model Configurations of OLMo Downcycling Experiment
Hyperparameter
Value
Optimizer
AdamW
Learning Rate
4e-4
Min Learning Rate
4e-5
Global Batch Size
1024
Sequence Length
4096
Training Iterations
50,000
Appendix
Table 10: Hyperparameters for Continued Pre-training.
Figure 7: Shared vs. Routed Expert Ratio Across Layers. The figure shows the layer-wise configuration of shared and routed expert ratios for Qwen2.5-7B, as determined by ExpertWeaver.
Figure 8: Expert Specialization in the ExpertWeaver Model.
Mixture-of-Experts language models use input-dependent routing to activate a small subset of expert networks, increasing capacity without proportional compute.