GS-MoE: Training-Free MoE Compression via Groupwise Subspace Sharing and Pruning

Carson Cary; Yukai Jin; Yihao Huang; Zhankai Ye; Fangyi Li; Ting Hua; Xin Liu; Shangqian Gao

cs.LGOct 23, 2026

GS-MoE: Training-Free MoE Compression via Groupwise Subspace Sharing and Pruning

Authors: Carson Cary, Yukai Jin, Yihao Huang, Zhankai Ye, Fangyi Li, Ting Hua, Xin Liu, Shangqian Gao

Organizations: Florida State University · University of Pennsylvania · University of Notre Dame

Abstract

Mixture-of-Experts (MoE) models reduce computation through sparse activation, but storing their expert parameters remains a major memory bottleneck. We introduce GS-MoE, a training-free compression framework that addresses redundancy across and within experts through two compression axes: shared-subspace rank and expert intermediate width. Groupwise low-rank representations capture shared structure across experts while retaining expert-specific projections, and Nyström-based neuron compression reduces intermediate width within each expert. For fixed shared subspaces, we derive per-expert reconstruction-error reductions from the reconstruction objective and use them to guide expert grouping. Budgets are allocated hierarchically using a gradient-free layer-importance score to guide the allocation of rank and width budgets across layers, which eigenvalues and leverage then distribute as non-uniform ranks across groups and intermediate widths across experts. Experiments demonstrate the complementary benefits of compression along both axes, particularly at high compression rates on fine-grained MoE models. On DeepSeek-MoE-16B-Base, at a 60% reduction in total model parameters, GS-MoE retains approximately 90% of the uncompressed model's average reasoning performance, compared with 73% for D $^2$ -MoE, while achieving 20% lower out-of-domain perplexity. We provide a code implementation at: https://github.com/cbacary/GS-MoE

Figures & tables

Figure 1: Overview of GS-MoE. Compared with an uncompressed expert (left), GS-MoE partitions experts into equally sized groups with shared input and output projections $P_{g}^{\uparrow}$ and $P_{g}^{\downarrow}$ , while retaining expert-specific projections $A_{g,j}^{\uparrow}$ and $A_{g,j}^{\downarrow}$ , where $j$ indexes experts within group $g$ , so $A_{g,j}^{\star}$ corresponds to $A_{e}^{\star}$ for the associated expert $e\in\mathcal{E}_{g}$ . Group ranks and expert intermediate widths are allocated non-uniformly under a fixed parameter budget. Colored cells indicate retained parameters, and hatched regions indicate removed dimensions. Two experts per group are shown for illustration; routing weights and gated projections are omitted.

DeepSeekMoE-16B-Base (64 experts, top-6 activated)
Ratio	Method	Wiki. $\downarrow$	PTB $\downarrow$	OpenB.	ARC-e	WinoG.	HellaS.	ARC-c	PIQA	MathQA	Avg. $\uparrow$
0%	Original	6.38	9.47	0.32	0.75	0.70	0.58	0.44	0.79	0.31	0.56
20%	NAEE ( 2024 )	9.44	15.02	0.32	0.71	0.66	0.55	0.40	0.77	0.29	0.53
20%	MoE-I 2 ( 2024 )	7.69	11.59	0.26	0.71	0.68	0.49	0.38	0.73	0.29	0.51
	D 2 -MoE ( 2025 )	6.84	11.10	0.30	0.74	0.69	0.55	0.41	0.76	0.31	0.54
	GS-MoE (Ours)	7.35	10.64	0.33	0.75	0.70	0.57	0.43	0.79	0.30	0.55

Table 1: Comparison of GS-MoE to other methods. Best results are in bold .

Grouped assignment	Group rank allocation	Wiki.	C4	PTB
✗	✗	29.49	33.64	36.49
✗	✓	29.42	33.89	36.56
✓	✗	25.23	28.83	30.35
✓	✓	25.02	28.30	29.37

Table 2: Ablation of grouped assignment and group rank allocation for GS-MoE factorization only on DeepSeek-MoE-16B-Base at 60% compression with

G=8

. Lower perplexity is better.

Panel of Figure 2 — Figure 2: (a–c) Expert-to-expert dissimilarity via Obj. 7 on Qwen2-57B-A14B and DeepSeek-MoE-16B-Base. White boxes indicate assigned groups; darker colors indicate lower dissimilarity. (d) Leverage-based retained width $k_{e}$ relative to a uniform split within each layer on DeepSeek-MoE-16B-Base at 60% compression ( $\nu=0.55$ ), with experts sorted by width; the black line marks the 70-neuron floor. (e) Under the same setting, $k_{e}/d_{\mathrm{ff}}$ versus tokens routed to each expert (mean within-layer Spearman $\rho=-0.04$ ). (f) Sensitivity of expert-subspace grouping to the number of restarts and iterations.

Panel of Figure 3 — Figure 3: Active subspaces per token on DeepSeek-MoE-16B-Base ( $G=8$ ), counted as distinct groups among the top-6 experts. (a) Guided versus random grouping over all tokens and layers. (b) Per-layer distributions for guided grouping, with means shown in the white line.

Mixtral-8x7B
Method	Tokens/sec
D 2 -MoE ( Gu et al., 2025 )	112.84
TD-MoE ( Xu et al., 2026 )	102.53
MoE-SVD ( Li et al., 2025b )	351.44
GS-MoE ( $G=1$ )	374.77
GS-MoE ( $G=2$ )	363.78

Table 3: Throughput (tokens/sec,

\uparrow

) at 60% compression.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Compression Rates	$\eta$	$\nu$	$G$
DeepSeek	20%	0.1	0.75	1
	40%	0.1	0.65	2
	60%	0.1	0.55	8
Mixtral	20%	0.01	0.85	1
	40%	0.01	0.85	1
	60%	0.01	0.85	1

Appendix

Table 4: Default compression hyperparameters for the main comparisons. Ablation settings are specified separately.

Panel of Figure 4 — Figure 4: Perplexity of GS-MoE on DeepSeek-16B-Base at 55% compression. (a) Varying the number of groups $G$ . (b) Varying $\nu$ .

(a) Per-expert width allocation
		Uniform			Non-uniform
Ratio	$\nu$	Wiki	C4	PTB	Wiki	C4	PTB
20%	0.75	6.17	9.08	9.33	6.07	8.98	9.26
40%	0.65	6.83	10.16	10.05	6.59	10.02	9.81
60%	0.55	8.10	12.15	11.65	7.87	11.93	11.29
(b) Layer-wise allocation

Appendix

Table 5: Perplexity (

\downarrow

) of GS-MoE on Qwen2-57B-A14B under uniform and non-uniform allocation. (a) Uniform vs. leverage-based expert widths;

\nu

controls the split between width and rank compression. (b) Uniform vs. non-uniform layer-wise allocation, calibrated on Alpaca. Lower perplexity for each dataset and setting is in bold .

DeepSeekMoE-16B-Base (64 experts, top-6 activated)
Ratio	Method	Wiki. $\downarrow$	PTB $\downarrow$	OpenB.	ARC-e	WinoG.	HellaS.	ARC-c	PIQA	MathQA	Avg. $\uparrow$
0%	Original	6.38	9.47	0.32	0.75	0.70	0.58	0.44	0.79	0.31	0.56
20%	Factorization	8.11	11.47	0.30	0.74	0.70	0.54	0.43	0.77	0.28	0.538
	Nyström	7.30	10.53	0.33	0.76	0.70	0.57	0.44	0.79	0.32	0.558
	GS-MoE (Ours)	7.35	10.64	0.33	0.75	0.70	0.57	0.43	0.79	0.30	0.552
40%	Factorization	12.97	16.13	0.27	0.71	0.68	0.48	0.39	0.74	0.26	0.506

Appendix

Table 6: Subsets of GS-MoE on three different models. Factorization is GS-MoE’s group shared subspace method with full intermediate width. Nyström is GS-MoE with only expert intermediate width reduction.

Mixture-of-Experts Models

Mixture-of-experts models use input-dependent routing to activate specialized subnetworks, increasing model capacity without running every parameter for each input.

See all 456 papers

PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference
Nov 6, 2025Yushu Zhao, Zheng Wang, Minjia ZhangLLM Compression Mixture-of-Experts Models
1Tsinghua University · †Work done while intern at UIUC. · University of Illinois Urbana-Champaign.
OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit
Sep 9, 2026Dezhi Li, Lujun Li, Qiyuan Zhu +4Mixture-of-Experts Pruning Mixture-of-Experts Language Models
The Hong Kong University of Science and Technology
Less is MoE: Trimming Experts in Domain-Specialist Language Models
Jun 4, 2026Haoze He, Xinkai Zou, Xuan Jiang +4Mixture-of-Experts Pruning LLM Pruning
Carnegie Mellon University · UCSD · MIT

Nov 6, 2025J/K move · Enter open · S save

PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference

Yushu Zhao, Zheng Wang, Minjia Zhang

1Tsinghua University · †Work done while intern at UIUC. · University of Illinois Urbana-Champaign.

LLM Compression Mixture-of-Experts Models

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) models have shown strong potential in scaling language models efficiently by activating only a small subset of experts per input. However, their widespread deployment remains limited due to the high memory overhead associated with storing all expert parameters, particularly as the number of experts increases. To address this challenge, prior works have explored expert dropping and merging strategies, yet they often suffer from performance drop at high compression ratios. In this paper, we introduce PuzzleMoE, a training-free MoE compression method that achieves both high accuracy and efficient inference through two key innovations: First, PuzzleMoE performs sparse expert merging by identifying element-wise weight redundancy and specialization. It uses a dual-mask to capture both shared and expert-specific parameters. Second, to avoid the overhead of storing binary masks and signs, PuzzleMoE introduces a bit-packed encoding scheme that reuses underutilized exponent bits, enabling efficient MoE inference on GPUs. Extensive experiments demonstrate that PuzzleMoE can compress MoE models by up to 50% while maintaining accuracy across various tasks. Specifically, it outperforms prior MoE compression methods by up to 16.7% on MMLU at 50% compression ratio, and achieves up to 1.28\times inference speedup.

Abstract

Mixture-of-Experts Models

Explore similar work

PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference

OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit

Less is MoE: Trimming Experts in Domain-Specialist Language Models