LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing

Jiawei Hao; Zhiwei Hao; Jianyuan Guo; Li Shen; Yong Luo; Han Hu; Dan Zeng

cs.LGMar 13, 2026

LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing

Authors: Jiawei Hao, Zhiwei Hao, Jianyuan Guo, Li Shen, Yong Luo, Han Hu, Dan Zeng

Organizations: School of Information and Electronics, Beijing Institute of Technology, Beijing, China · Department of Computer Science, City University of Hong Kong, Hong Kong, China · Shenzhen Campus of Sun Yat-sen University, Shenzhen, China · School of Computer Science, Wuhan University, Wuhan, China ·

Abstract

Mixture-of-Experts (MoE) based Large Language Models (LLMs) have demonstrated impressive performance and computational efficiency. However, their deployment is often constrained by substantial memory demands, primarily due to the need to load numerous expert modules. While existing expert compression techniques like pruning or merging attempt to mitigate this, they often suffer from irreversible knowledge loss or high training overhead. In this paper, we propose a novel expert compression paradigm termed expert replacing, which replaces redundant experts with parameter-efficient modules and recovers their capabilities with low training costs. We find that even a straightforward baseline of this paradigm yields promising performance. Building on this foundation, we introduce LightMoE, a framework that enhances the paradigm by introducing adaptive expert selection, hierarchical expert construction, and an annealed recovery strategy. Experimental results show that LightMoE matches the performance of LoRA fine-tuning at a 30% compression ratio. Even under a more aggressive 50% compression rate, it outperforms existing methods and achieves average performance improvements of 5.6% across five diverse tasks. These findings demonstrate that LightMoE strikes a superior balance among memory efficiency, training efficiency, and model performance.

Figures & tables

Figure 1: Performance comparison across different compression ratios on the Math task. While the directly replacing strategy performs comparably to MC-SMoE, both suffer from significant performance degradation, particularly at higher compression ratios

Figure 2: Overview of the proposed LightMoE framework. ( Left ) A standard MoE layer. ( Right ) The LightMoE workflow, comprising three key steps: (1) scoring experts and selecting those with lower scores as compression candidates, (2) grouping the selected candidates, and (3) replacing each group with a shared base augmented with lightweight, expert-specific adaptation parameters.

Panel of Figure 3 — Figure 3: Analysis of expert importance in OLMoE-1B-7B-SFT.

			Math	Code	Commonsense	Specialized Tasks
Ratio	Methods	# Params	GSM8K	HumanEval	8 Sub-Tasks	Intent	Translation	Average
0%	Original	-	49.3	55.0	60.6	9.8	13.6	37.7
0%	Full FT	6.92B	61.2	49.7	54.2	79.2	37.1	56.3
0%	LoRA	0.45B	59.8	54.2	61.4	73.0	29.2	55.5
	MoBE	0.45B	52.9	41.3	58.3	69.4	30.3	50.4
	HC-SMoE	0.45B	40.6	41.7	55.5	64.0	32.4	46.8

Table 1: Main performance comparison across methods and tasks at different compression ratios. ‘*’ indicates the full method as described in the original paper. “# Params” is the number of trainable parameters. Best results are shown in bold and second-best results are underlined . Our method LightMoE consistently achieves good performance among all tasks under different compression settings.

Ratio	Selection Methods	Math	Code
Ratio	Selection Methods	GSM8K	HumanEval
30%	Uniform	58.1	55.1
	Average	59.4	55.5
	Adaptive	59.7	55.7
50%	Uniform	57.1	47.9
	Average	54.6	43.0

Table 2: Comparison of different experts selection methods for LightMoE on the Math and Code tasks. Our adaptive thresholding method can select the most appropriate experts for replacing, adapting to different compression ratios.

Ratio	K-means	Dominant (Ours)
30%	60.0	59.7
40%	56.1	57.9
50%	54.9	56.1

Table 3: Comparison of expert grouping strategies for OLMoE on the Math task at different compression ratios.

Figure 4: Comparison of different end ratios for LightMoE on the Math task. Directly replacing (blue points) is consistently sub-optimal, which shows the effectiveness of our annealed expert replacement strategy (red points).

Figure 5: Comparison of different ranks at different compression ratios on the Math task.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

			Math	Code	Common.	Specialized Tasks
Ratio	Methods	# Params	GSM8K	HumanEval	8 Sub-Tasks	Intent	Translation	Average
0%	Original	-	3.0	53.9	61.8	0.0	12.2	26.2
0%	LoRA	0.96B	38.9	58.0	62.3	81.0	19.9	52.0
	MoBE	0.96B	54.3	47.7	59.7	80.2	27.0	53.8
	HC-SMoE	0.96B	36.8	51.6	57.8	75.8	26.9	49.8
	MC-SMoE	0.96B	48.4	53.7	57.0	60.6	26.8	49.3

Appendix

Table 4: Performance comparison on DeepSeek model across methods and tasks at different compression ratios.

Ratio	Number of Tokens
Ratio	$2^{13}$	$2^{15}$	$2^{17}$	$2^{19}$
30%	57.4	59.1	59.7	59.6
40%	55.1	56.6	57.9	58.4
50%	53.9	54.5	56.1	56.3

Appendix

Table 5: Ablation study on the impact of sample size on OLMoE performance.

Ratio	Clipping Range $\max\Delta$
Ratio	0	0.1	0.15	0.2	0.25	0.3	$\infty$
30%	58.1	59.4	59.7	59.7	59.3	58.6	58.6
50%	57.1	55.6	55.8	56.1	54.1	54.3	54.3

Appendix

Table 6: Sensitivity analysis of the clipping range parameter

\text{max}\Delta

for OLMoE on the Math task.

Figure 6: Comparison of different group sizes at different compression ratios for OLMoE on the Math task.

Figure 7: Comparison of different annealing schedules. The dashed line represents the linear decay strategy, while solid lines depict exponential decay strategies with varying

\gamma

. All schedules converge to zero at the end ratio

\epsilon

Ratio	Linear	Exponential Decay
Ratio	Linear	$\gamma=1.0$	$\gamma=3.0$	$\gamma=5.0$
30%	59.7	59.8	59.6	58.7
40%	57.9	56.1	58.2	58.4
50%	56.1	54.5	54.4	56.0

Appendix

Table 7: Comparison of linear and exponential annealing schedules for OLMoE on the Math task.

Ratio	Total	Memory	MoE Active	Latency
Ratio	Params	Usage	Params	(ms/token)
0%	6.92 B	12.89 GB	0.81 B	8.09
30%	4.82 B	9.09 GB	0.77 B	8.71
40%	4.14 B	7.85 GB	0.73 B	8.32
50%	3.47 B	6.63 GB	0.70 B	8.01

Appendix

Table 8: Inference efficiency comparison on the Math task under different compression ratios.

Figure 8: Training loss trajectories of LightMoE versus the Directly Replacing baseline on the Math task with OLMoE at a 50% compression ratio.

Ratio	Avg #	Base Threshold $\hat{p}$
(%)	Exp.	Math	Code	Common.	Intent	Trans.
Model: OLMoE-1B-7B-SFT
30	$\approx$ 24	0.110	0.095	0.185	0.055	0.036
40	$\approx$ 31	0.180	0.159	0.275	0.098	0.059
50	$\approx$ 38	0.268	0.240	0.382	0.168	0.092
Model: DeepSeek-V2-Lite

Appendix

Table 9: Base threshold

\hat{p}

settings calibrated for OLMoE and DeepSeek models. The average number of selected experts per layer (Avg # Exp.) corresponds to the target compression ratio.

Mixture-of-Experts Language Models

Mixture-of-Experts language models use input-dependent routing to activate a small subset of expert networks, increasing capacity without proportional compute.

See all 216 papers

OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit
Sep 9, 2026Dezhi Li, Lujun Li, Qiyuan Zhu +4Mixture-of-Experts Pruning Mixture-of-Experts Language Models
The Hong Kong University of Science and Technology
PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference
Nov 6, 2025Yushu Zhao, Zheng Wang, Minjia ZhangLLM Compression Mixture-of-Experts Models
1Tsinghua University · †Work done while intern at UIUC. · University of Illinois Urbana-Champaign.
REAM: Merging Improves Pruning of Experts in LLMs
Apr 6, 2026Saurav Jha, Maryam Hashemzadeh, Ali Saheb Pasand +3LLM Compression Mixture-of-Experts Pruning
Mila – Quebec AI Institute · Polytechnique Montréal · Université de Montréal +3

Sep 9, 2026J/K move · Enter open · S save

OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit

Dezhi Li, Lujun Li, Qiyuan Zhu, Hao Gu+3

The Hong Kong University of Science and Technology

Mixture-of-Experts Pruning Mixture-of-Experts Language Models LLM Compression LLM Inference Acceleration

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) models enable efficient scaling of large language models but face critical deployment challenges due to massive memory requirements. Existing pruning methods either incur prohibitive search costs or neglect the dynamic interdependencies between experts. To address these challenges, we present OMP-MoE, a novel training-free compression framework for reducing expert redundancy in MoE-based LLMs. Based on observations of expert contribution patterns, we reformulate the pruning problem as a sparse signal reconstruction task solved through Orthogonal Matching Pursuit. Specifically, our method first treats individual expert contributions as dictionary atoms and selects experts that greedily minimize reconstruction error with linear computational complexity. Then, we optimize cross-layer expert allocation through a water-filling strategy that accounts for both reconstruction quality and routing stability. Finally, we introduce OMP-MoE†, an adaptive inference mechanism that dynamically adjusts expert activation based on energy prediction. Comprehensive experiments on Qwen, DeepSeek-V2, GPT-OSS, and Mixtral MoE demonstrate consistent improvements over existing methods at 25-50% pruning ratios. For Qwen3-30B-A3B at 50% compression, we retain 93.3% of original performance, achieving 33

\times

faster search and 1.55

\times

inference speedup. Codes will be available after acceptance.

Abstract

Mixture-of-Experts Language Models

Explore similar work

OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit

PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference

REAM: Merging Improves Pruning of Experts in LLMs