Mixture-of-Experts (MoE) models reduce computation through sparse activation, but storing their expert parameters remains a major memory bottleneck. We introduce GS-MoE, a training-free compression framework that addresses redundancy across and within experts through two compression axes: shared-subspace rank and expert intermediate width. Groupwise low-rank representations capture shared structure across experts while retaining expert-specific projections, and Nyström-based neuron compression reduces intermediate width within each expert. For fixed shared subspaces, we derive per-expert reconstruction-error reductions from the reconstruction objective and use them to guide expert grouping. Budgets are allocated hierarchically using a gradient-free layer-importance score to guide the allocation of rank and width budgets across layers, which eigenvalues and leverage then distribute as non-uniform ranks across groups and intermediate widths across experts. Experiments demonstrate the complementary benefits of compression along both axes, particularly at high compression rates on fine-grained MoE models. On DeepSeek-MoE-16B-Base, at a 60% reduction in total model parameters, GS-MoE retains approximately 90% of the uncompressed model's average reasoning performance, compared with 73% for D2-MoE, while achieving 20% lower out-of-domain perplexity. We provide a code implementation at: https://github.com/cbacary/GS-MoE
Figures & tables
Figure 1: Overview of GS-MoE. Compared with an uncompressed expert (left), GS-MoE partitions experts into equally sized groups with shared input and output projections Pg↑ and Pg↓ , while retaining expert-specific projections Ag,j↑ and Ag,j↓ , where j indexes experts within group g , so Ag,j⋆ corresponds to Ae⋆ for the associated expert e∈Eg . Group ranks and expert intermediate widths are allocated non-uniformly under a fixed parameter budget. Colored cells indicate retained parameters, and hatched regions indicate removed dimensions. Two experts per group are shown for illustration; routing weights and gated projections are omitted.
Table 1: Comparison of GS-MoE to other methods. Best results are in bold .
Grouped assignment
Group rank allocation
Wiki.
C4
PTB
✗
✗
29.49
33.64
36.49
✗
✓
29.42
33.89
36.56
✓
✗
25.23
28.83
30.35
✓
✓
25.02
28.30
29.37
Table 2: Ablation of grouped assignment and group rank allocation for GS-MoE factorization only on DeepSeek-MoE-16B-Base at 60% compression with G=8 . Lower perplexity is better.
Figure 2: (a–c) Expert-to-expert dissimilarity via Obj. 7 on Qwen2-57B-A14B and DeepSeek-MoE-16B-Base. White boxes indicate assigned groups; darker colors indicate lower dissimilarity. (d) Leverage-based retained width ke relative to a uniform split within each layer on DeepSeek-MoE-16B-Base at 60% compression ( ν=0.55 ), with experts sorted by width; the black line marks the 70-neuron floor. (e) Under the same setting, ke/dff versus tokens routed to each expert (mean within-layer Spearman ρ=−0.04 ). (f) Sensitivity of expert-subspace grouping to the number of restarts and iterations.
Figure 3: Active subspaces per token on DeepSeek-MoE-16B-Base ( G=8 ), counted as distinct groups among the top-6 experts. (a) Guided versus random grouping over all tokens and layers. (b) Per-layer distributions for guided grouping, with means shown in the white line.
Mixtral-8x7B
Method
Tokens/sec
D 2 -MoE ( Gu et al., 2025 )
112.84
TD-MoE ( Xu et al., 2026 )
102.53
MoE-SVD ( Li et al., 2025b )
351.44
GS-MoE ( G=1 )
374.77
GS-MoE ( G=2 )
363.78
Table 3: Throughput (tokens/sec, ↑ ) at 60% compression.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Compression Rates
η
ν
G
DeepSeek
20%
0.1
0.75
1
40%
0.1
0.65
2
60%
0.1
0.55
8
Mixtral
20%
0.01
0.85
1
40%
0.01
0.85
1
60%
0.01
0.85
1
Appendix
Table 4: Default compression hyperparameters for the main comparisons. Ablation settings are specified separately.
Figure 4: Perplexity of GS-MoE on DeepSeek-16B-Base at 55% compression. (a) Varying the number of groups G . (b) Varying ν .
(a) Per-expert width allocation
Uniform
Non-uniform
Ratio
ν
Wiki
C4
PTB
Wiki
C4
PTB
20%
0.75
6.17
9.08
9.33
6.07
8.98
9.26
40%
0.65
6.83
10.16
10.05
6.59
10.02
9.81
60%
0.55
8.10
12.15
11.65
7.87
11.93
11.29
(b) Layer-wise allocation
Appendix
Table 5: Perplexity ( ↓ ) of GS-MoE on Qwen2-57B-A14B under uniform and non-uniform allocation. (a) Uniform vs. leverage-based expert widths; ν controls the split between width and rank compression. (b) Uniform vs. non-uniform layer-wise allocation, calibrated on Alpaca. Lower perplexity for each dataset and setting is in bold .
Table 6: Subsets of GS-MoE on three different models. Factorization is GS-MoE’s group shared subspace method with full intermediate width. Nyström is GS-MoE with only expert intermediate width reduction.
Mixture-of-experts models use input-dependent routing to activate specialized subnetworks, increasing model capacity without running every parameter for each input.
Mixture-of-Experts (MoE) models have shown strong potential in scaling language models efficiently by activating only a small subset of experts per input. However, their widespread deployment remains limited due to the high memory overhead associated with storing all expert parameters, particularly as the number of experts increases. To address this challenge, prior works have explored expert dropping and merging strategies, yet they often suffer from performance drop at high compression ratios. In this paper, we introduce PuzzleMoE, a training-free MoE compression method that achieves both high accuracy and efficient inference through two key innovations: First, PuzzleMoE performs sparse expert merging by identifying element-wise weight redundancy and specialization. It uses a dual-mask to capture both shared and expert-specific parameters. Second, to avoid the overhead of storing binary masks and signs, PuzzleMoE introduces a bit-packed encoding scheme that reuses underutilized exponent bits, enabling efficient MoE inference on GPUs. Extensive experiments demonstrate that PuzzleMoE can compress MoE models by up to 50% while maintaining accuracy across various tasks. Specifically, it outperforms prior MoE compression methods by up to 16.7% on MMLU at 50% compression ratio, and achieves up to 1.28\times inference speedup.
Yushu Zhao, Zheng Wang, Minjia Zhang
1Tsinghua University · †Work done while intern at UIUC. · University of Illinois Urbana-Champaign.
Mixture-of-Experts (MoE) models enable efficient scaling of large language models but face critical deployment challenges due to massive memory requirements. Existing pruning methods either incur prohibitive search costs or neglect the dynamic interdependencies between experts. To address these challenges, we present OMP-MoE, a novel training-free compression framework for reducing expert redundancy in MoE-based LLMs. Based on observations of expert contribution patterns, we reformulate the pruning problem as a sparse signal reconstruction task solved through Orthogonal Matching Pursuit. Specifically, our method first treats individual expert contributions as dictionary atoms and selects experts that greedily minimize reconstruction error with linear computational complexity. Then, we optimize cross-layer expert allocation through a water-filling strategy that accounts for both reconstruction quality and routing stability. Finally, we introduce OMP-MoE†, an adaptive inference mechanism that dynamically adjusts expert activation based on energy prediction. Comprehensive experiments on Qwen, DeepSeek-V2, GPT-OSS, and Mixtral MoE demonstrate consistent improvements over existing methods at 25-50% pruning ratios. For Qwen3-30B-A3B at 50% compression, we retain 93.3% of original performance, achieving 33× faster search and 1.55× inference speedup. Codes will be available after acceptance.
Dezhi Li, Lujun Li, Qiyuan Zhu +4
The Hong Kong University of Science and Technology
Mixture-of-Experts (MoE) models achieve strong performance through conditional computation, but their large parameter footprint poses deployment challenges. Prior MoE compression approaches catastrophically fail when evaluated on general-purpose benchmarks beyond commonsense reasoning. We trace this failure to the granularity of compression: important capabilities are distributed across experts but concentrated in FFN sparse intermediate dimensions. To identify these dimensions, we use Fisher importance which outperforms activation-, router-score-, and magnitude-based alternatives, and identifies tiny sets of task-critical dimensions: in Qwen1.5-MoE, removing as few as 12 of 1.35M routed-FFN intermediate dimensions collapses GSM8K accuracy while largely preserving factual-knowledge performance. Building on this, we propose Fisher-MoE, which operates within FFN to remove intermediate dimensions ranked by Fisher importance. At the same 50% MoE compression ratio, Fisher-MoE preserves model capability, while reducing weight memory by ~45% and improving inference throughput by 21%. These findings suggest intermediate dimension granularity is an effective unit for both compression and ranking where capability concentrates in MoE models.