Organizations: Zhejiang University · Shanghai Innovation Institute · Shanghai AI Laboratory · Bytedance Seed · Fudan University · Chinese University of Hong Kong.
Mixture-of-Experts (MoE) effectively scales model capacity while preserving computational efficiency through sparse expert activation. However, training high-quality MoEs from scratch is prohibitively expensive. A promising alternative is to convert pretrained dense models into sparse MoEs. Existing dense-to-MoE methods fall into two categories: dynamic structural pruning that converts dense models into MoE architectures with moderate sparsity to balance performance and inference efficiency, and downcycling approaches that use pretrained dense models to initialize highly sparse MoE architectures. However, existing methods break the intrinsic activation patterns within dense models, leading to suboptimal expert construction. In this work, we argue that the Gated Linear Unit (GLU) mechanism provides a natural blueprint for dense-to-MoE conversion. We show that the fine-grained neural-wise activation patterns of GLU reveal a coarse-grained structure, uncovering an inherent MoE architecture composed of consistently activated universal neurons and dynamically activated specialized neurons. Leveraging this discovery, we introduce ExpertWeaver, a training-free framework that partitions neurons according to their activation patterns and constructs shared experts and specialized routed experts with layer-adaptive configurations. Our experiments demonstrate that ExpertWeaver significantly outperforms existing methods, both as a training-free dynamic structural pruning technique and as a downcycling strategy for superior MoE initialization.
Figures & tables
Figure 1: Neuron activation patterns across diverse tasks. We visualize the middle layer’s activation patterns from Qwen2.5-7B on a subset of Flan-v2. a) Activation distribution of neurons across different tasks. b) Activation distribution of neurons within individual task clusters, where tasks belonging to the same cluster are enclosed in boxes.
Method
MMLU(5)
HellaSwag(10)
ARC-e
ARC-C(25)
PiQA
Avg.
LLaMA3-8B, 50% sparsity
Dense
65.3
82.1
77.9
57.9
80.8
72.8
Wanda
55.8
75.0
72.0
51.3
77.3
66.3
SparseGPT
57.4
75.5
72.0
50.3
78.1
66.7
Magnitude
48.3
41.9
53.3
38.7
65.8
49.6
AbsTopk-GLU
58.8
79.4
72.4
51.6
78.6
68.2
Table 1: Performance of AbsTopk-GLU vs. unstructured pruning at 50% sparsity. (Numbers in parentheses denote the number of shots for evaluation; no annotation means zero-shot.)
Figure 2: Neuron Coefficient of Variation Across Layers .
Figure 3: The ExpertWeaver Framework. a) The GLU in the MLP layer contains three weight matrices, where the same color denotes corresponding neuron slices. b) Neuron activation patterns are captured using a multi-task calibration dataset. c) The CVs are computed to determine the budget for shared vs. routed experts. (d) Neurons are clustered according to their activation patterns to form one shared expert and multiple routed experts.
Method
MMLU(5)
HellaSwag(10)
ARC-e
ARC-c
PiQA
Avg.
LLaMA3-8B, 25% sparsity
Dense
65.3
82.1
77.9
57.9
80.8
72.8
LLM-Pruner
24.2
51.3
58.9
32.4
74.4
48.2
FLAP
33.4
48.0
50.0
29.3
68.3
45.8
CMoE
41.6
65.9
63.1
41.5
73.9
57.2
ExpertWeaver
47.0
69.8
64.4
44.3
76.3
60.4
Table 2: Comparison with structured pruning methods under 25% sparsity.
Figure 4: Comparison of Downcycling, Upcycling, and From-Scratch Training. Comparison of training loss, evaluation loss, and downstream task performance using the same OLMoE model configuration under three different MoE initialization paradigms.
Model
MMLU(5)
HellaSwag(10)
ARC-e
ARC-c(25)
PIQA
WinoGrande
LogiQA
SciQ
Average
Dense Models
Qwen2.5-7B
74.1
80.2
77.5
63.7
79.7
73.2
36.4
95.2
72.5
Qwen2.5-3B
65.6
74.6
73.9
56.5
78.8
68.1
33.5
95.2
68.3
Qwen2.5-1.5B
60.9
68.0
72.5
54.9
75.9
63.8
31.9
93.4
65.2
Llama-3.2-3B
56.1
76.4
71.6
50.5
77.4
69.9
30.6
92.7
65.7
OLMo-7B
30.7
77.1
68.7
45.1
79.6
66.5
27.5
88.6
60.8
Table 3: Downcycling Performance Comparison. Best results are in bold, second-best are underlined. Models in gray (Qwen2.5-{7B, 3B, 1.5B}, Llama-3.2-3B, and OLMoE) are for context, as they were trained on significantly more data. Our ExpertWeaver models (OLMo-based and Qwen2.5-based) were trained on 200B tokens. The OLMo-based version has 1B active parameters (7B-A1B), and the Qwen2.5-based version has 3.5B active parameters (7B-A3.5B). We compare against the 500B token checkpoint (*) of OLMoE. Blank entries for OpenMoE indicate unavailable results.
Figure 5: Training Loss Comparison with Different MoE Initialization Strategies.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Ablation studies on key hyperparameters of ExpertWeaver. a) MMLU performance heatmap for different shared expert ratios, controlled by the hyperparameters αmin and αmax in Eq. 5 . The diagonal where αmin=αmax represents a static configuration with a uniform ratio across all layers. b) MMLU performance varying specialization threshold τ from Eq. 4 . c) MMLU performance varying expert granularity.
Method
MMLU(5)
ARC-C(25)
GSM8K
HumanEval
IFEval
Avg.
LLaMA-MoE-3B-7B
28.24
44.03
4.62
12.02
28.10
23.40
OLMoE-1B-7B
53.79
55.63
40.94
40.48
35.49
45.27
LLaMA-MoE-v2
40.90
40.20
55.00
51.20
36.00
44.66
ExpertWeaver-Instruct
50.60
69.80
57.10
50.20
33.10
52.16
Appendix
Table 4: Instruct Model Performance Comparison
GSM8k
HumanEval
GPQADiamond
LiveCodeBench
Base Model
Qwen2.5-7B
DeepSeek-R1-Distill-Qwen-7B
Sparsity
25%
12.5%
25%
12.5%
12.5%
LLM-Pruner
2.0
-
14.6
10.4
33.3
6.1
FLAP
14.8
-
57.6
9.8
36.3
2.3
ExpertWeaver
34.9
18.9
64.6
32.9
37.3
16.8
Appendix
Table 5: Training-Free Pruning on Code Benchmarks. We compare ExpertWeaver against static pruning methods on several code-related benchmarks. The base models used for each benchmark are specified in the table.
GPU
Method
RPS ↑
OTPS ↑
TTPS ↑
TTFT ↓
TPOT ↓
ITL ↓
(ms)
(ms)
(ms)
A100
ExpertWeaver
22.88
2928.86
14644.31
440.6
40.3
40.3
Qwen2.5-7B
18.85
2413.22
12066.10
675.4
47.8
48.2
H100
ExpertWeaver
44.04
5637.42
28187.10
542.2
9.8
9.8
Qwen2.5-7B
41.41
5299.96
26499.78
1404.5
18.5
18.5
Appendix
Table 6: Inference throughput comparison across different GPU architectures. RPS: Requests Per Second. OTPS: Output Tokens Per Second. TTPS: Total Tokens Per Second. TTFT: Time To First Token. TPOT: Time Per Output Token. ITL: Inter-Token Latency.
Method
MMLU
HellaSwag(10)
ARC-e
ARC-c(25)
PiQA
Avg.
Qwen2.5-7B, 25% sparsity
Dense
74.2
80.3
77.8
63.8
80.0
75.2
Ablation on Data Source
ExpertWeaver Flan (default)
61.6
72.3
71.5
53.5
76.3
67.0
ExpertWeaver C4
61.2
71.6
72.1
49.1
74.8
65.8
Ablation on Data Quantity & Diversity
Appendix
Table 7: Ablation Studies on the Calibration Set. We analyze the impact of calibration set size, diversity, and source.
Determines logical relationship between sentences.
mnli, qnli, rte, wnli, anli
Coreference Resolution
Identifies expressions referring to the same entity.
wsc, dpr, winogender
Appendix
Table 8: Tasks from Flan-v2 used in the calibration set.
OLMo 575M
OLMo 1.3B
OLMoE 1.3B-A676M
Model Dimension
2048
2048
2048
FFN Dimension
1024
8192
1024
Attention Heads
16
16
16
Key/Value Heads
16
16
16
Layers
16
16
16
Vocabulary Size
50280
50280
50280
Appendix
Table 9: Model Configurations of OLMo Downcycling Experiment
Hyperparameter
Value
Optimizer
AdamW
Learning Rate
4e-4
Min Learning Rate
4e-5
Global Batch Size
1024
Sequence Length
4096
Training Iterations
50,000
Appendix
Table 10: Hyperparameters for Continued Pre-training.
Figure 7: Shared vs. Routed Expert Ratio Across Layers. The figure shows the layer-wise configuration of shared and routed expert ratios for Qwen2.5-7B, as determined by ExpertWeaver.
Figure 8: Expert Specialization in the ExpertWeaver Model.
Mixture-of-Experts language models use input-dependent routing to activate a small subset of expert networks, increasing capacity without proportional compute.
The scaling of Large Language Models (LLMs) has driven significant performance gains but created substantial challenges in inference efficiency. While Mixture of Experts (MoEs) architectures address this by decoupling model size from inference cost, training MoEs from scratch is often unstable and compute intensive. Conversion of pre-trained dense models into sparse MoEs has emerged as an alternative solution; however, existing methods typically rely on heuristic neuron clustering or random splitting to partition the Feed-Forward Network (FFN) into experts. In this work, we propose DOT-MoE, a novel framework that formulates the decomposition of dense layers as a Differentiable Optimal Transport (DOT) problem. Instead of static heuristics, we model neuron assignment as a balanced transport problem, utilizing differentiable Sinkhorn-Knopp iterations to enforce strict expert capacity constraints. Furthermore, we utilize Straight-Through Estimators (STE) to jointly learn the discrete neuron-to-expert assignment and the token-to-expert routing policy end-to-end. Extensive experiments across multiple architectures and benchmarks demonstrate that DOT-MoE significantly outperforms structured pruning, heuristic clustering, and random-split baselines, retaining 90% of the original dense model's performance while reducing active parameters by 50%.
Mixture of Experts (MoE) architecture has become the standard for state-of-the-art large language models, owing to its computational efficiency through sparse expert activation. However, sparsity through finer expert granularity is becoming increasingly difficult to achieve due to fundamental training challenges such as expert collapse and load imbalance. In this work, we explore and leverage intra-expert activation sparsity as a complementary and underexplored dimension of sparsity in MoE models. Surprisingly, substantial intra-expert sparsity is readily available in existing pre-trained MoE models, without any modification to the activation function or model parameters, providing up to 90% sparsity within each expert without significant accuracy loss. We explore intra-expert activation sparsity across eight off-the-shelf MoE models ranging from 1B to 400B parameters, and extend the MoE execution pipeline of vLLM to leverage intra-expert activation sparsity by skipping the computations of inactive neurons, on top of its existing optimizations, achieving up to 2.5 times speedup in MoE layer execution and 1.2 times end-to-end speedup compared to the original dense vLLM baseline.
Jongseok Park, Sunga Kim, Zhenyu Gu +2
University of California, Berkeley · Work done during an internship at Advanced Micro Devices, Inc. · Advanced Micro Devices, Inc.
Large language models impose substantial memory demands and arithmetic costs during inference. Dense-to-mixture-of-experts (MoE) conversion is a model optimization technique that aims to reduce these arithmetic costs. Dense-to-MoE conversion partitions pretrained feed-forward network (FFN) neurons into experts and selects a subset for each token. Leading methods fall into two categories: analytic methods produce a converted model quickly using heuristics, while learning methods produce a model much more slowly by optimizing via gradients. Both approaches, however, fix the same active expert count, or "utilization," at every layer. We introduce Spectral-MoE, which learns to allocate a utilization budget heterogeneously across layers. Its name draws a metaphor between separating a dense FFN into experts and separating a light beam into its spectrum. Spectral-MoE consists of two distinct approaches: a static approach that uses a single per-layer scalar which is shared across tokens, and a dynamic approach which emits a per-token schedule. We demonstrate that both allocators improve performance over uniform on both the analytic-based and learning-based SOTA dense-to-MoE methods. At 25% FFN utilization, dynamic allocation reduces WikiText-2 perplexity relative to uniform by 1.04, 2.96, and 2.71 points on the learned base for Llama-2-7B, Llama-3-8B, and Qwen2.5-7B, respectively. The corresponding reductions on the analytic base are 2.45, 6.26, and 2.17 points. On the learned base, dynamic allocation improves six-task mean accuracy over uniform by an average of 1.5, 4.2, and 3.6 percentage points across the three models at FFN utilizations of 12.5%, 25%, and 50%, respectively. These results establish per-layer expert utilization as a novel optimization axis in dense-to-MoE conversion.