REAM: Merging Improves Pruning of Experts in LLMs

Saurav Jha; Maryam Hashemzadeh; Ali Saheb Pasand; Ali Parviz; Min-Joong Lee; Boris Knyazev

cs.AIApr 6, 2026

REAM: Merging Improves Pruning of Experts in LLMs

Authors: Saurav Jha, Maryam Hashemzadeh, Ali Saheb Pasand, Ali Parviz, Min-Joong Lee, Boris Knyazev

Organizations: Mila – Quebec AI Institute · Polytechnique Montréal · Université de Montréal · McGill University ·

Abstract

Mixture-of-Experts (MoE) large language models (LLMs) are among the top-performing architectures. The largest models, often with hundreds of billions of parameters, pose significant memory challenges for deployment. Traditional approaches to reduce memory requirements include weight pruning and quantization. Motivated by the Router-weighted Expert Activation Pruning (REAP) that prunes experts, we propose a novel method, Router-weighted Expert Activation Merging (REAM). Instead of removing experts, REAM groups them and merges their weights, better preserving original performance. We evaluate REAM against REAP and other baselines across multiple MoE LLMs on diverse multiple-choice (MC) question answering and generative (GEN) benchmarks. Our results reveal a trade-off between MC and GEN performance that depends on the mix of calibration data. By controlling the mix of general, math and coding data, we examine the Pareto frontier of this trade-off and show that REAM often outperforms the baselines and in many cases is comparable to the original uncompressed models.

Figures & tables

Panel of Figure 1 — Figure 1: Illustration of REAM components: a) Comparison of expert compression strategies reducing $N{=}9$ experts to $N^{\prime}{=}4$ . HC-SMoE merging ( Chen et al., 2025 ) clusters all experts by output similarity regardless of saliency (e.g., E1 and E7 grouped together). Pruning retains the top-4 salient experts unchanged and discards the rest. Our REAM’s pseudo-pruning selects the top-4 experts as protected centroids and absorbs remaining experts into their nearest centroid via saliency-weighted merging, leaving other groups as singletons. b) Compared to baseline pruning and merging methods \raisebox{-.9pt} {1}⃝ that collect the activations from the original uncompressed model for all layers at once, REAM \raisebox{-.9pt} {2}⃝ recomputes the per-layer activations after merging each MoE layer before processing the next layer.

Figure 2: Discriminative (MC) vs. Generative (GEN) trade-off depending on the calibration data mixture: benchmark scores with 64 ( left ) and 96 ( right ) experts for REAP, HC-SMoE, and REAM across ten mixing ratios of the calibration data with Qwen3-30B-A3B-Instruct-2507. The marker sizes are proportional to the The-Stack-Smol share of the mixture.

Method	$N$	C4:Math:Code	IFEval	AIME25	GSM8K	GPQA	HumanEval	LCB	GEN
Original	128	–	90.4	56.7	89.3	47.0	93.3	48.6	70.9
Freq	96	0:0.3:0.7	87.8	$\mathbf{60.0}$	82.9	36.9	93.9	44.0	67.6
HC-SMoE	96	0.5:0:0.5	88.2	$\mathbf{60.0}$	84.7	34.3	91.5	45.9	67.4
REAP	96	0.2:0.25:0.55	89.6	50.0	87.9	39.4	94.5	50.3	68.6
REAM	96	0:0.5:0.5	89.9	$\mathbf{60.0}$	86.3	38.4	93.3	$\mathbf{51.0}$	69.8

Table 1: Results at 96 experts on Qwen3-30B-A3B-Instruct-2507. Each method uses the calibration mixture achieving its best GEN score; bold is the best among compressed models.

Panel of Figure 3 — Figure 3: Additional analyses for 96 experts: a) Pearson correlation $r$ between calibration datasets (C4, Math, Code) and MC/GEN scores, and between MC and GEN scores themselves, for each merging method. b) Pareto frontiers where each point is one of 10 calibration mixtures. Filled markers denote Pareto-optimal configurations not simultaneously dominated on MC and GEN by any other mixture of the same method, and hollow markers denote dominated ones. The hypervolume (HV) measures the area of the MC $\times$ GEN plane dominated by each method’s frontier relative to a shared reference point, quantifying its overall performance ceiling. Per-method offsets are applied for better visibility.

Model	Method	$N$	IFEval	AIME25	GSM8K	GPQA	HumanEval	LCB	GEN
Qwen3-80B-A3B	Original	512	93.4	80.0	78.6	47.0	95.1	43.2	72.9
	REAP	384	92.8	66.7	77.7	42.4	94.5	43.6	69.6
	REAM	384	93.4	73.3	78.1	46.5	93.9	43.7	71.5
Qwen3-Coder	Original	512	89.6	80.0	85.4	42.4	92.7	47.5	72.9
	REAP	384	87.5	70.0	86.4	37.9	94.5	47.7	70.7
	REAM	384	89.3	80.0	85.3	40.4	94.5	48.0	72.9

Table 2: GEN benchmark results on additional models with a 25% expert reduction: 512

\to

384 experts for Qwen3-Next-80B-A3B-Instruct and Qwen3-Coder-Next, and 128

\to

96 for GLM-4.5-Air. The calibration mixture is fixed at C4 : Math : Code = 0 : 0.3 : 0.7 to favor GEN tasks; bold is the best among compressed models.

Panel of Figure 4 — Figure 4: Ablation of REAM components with 96 experts: (a) MC and GEN scores for each ablation variant; (b) Per-task score drop ( $\Delta$ ) relative to the full REAM performance.

Figure 5: Correlation between avg. pre-logit ranks and AVG benchmark scores across 10 calibration ratios for 96 experts.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

C4	Math	Code	Description
0.3	0.3	0.3	Balanced
0.5	0.5	0.0	C4 + math only
0.5	0.0	0.5	C4 + code only
0.0	0.5	0.5	Math + code only
0.2	0.5	0.3	Math-leaning
0.1	0.8	0.1	Math-heavy

Appendix

Table 3: Calibration dataset mixing ratios used in experiments. Each row defines the proportion of C4 (general text), math (NuminaMath), and code (The-Stack-Smol).

Figure 6: Effect of calibration data mixture on MC–GEN trade-off. Each panel shows discriminative (MC) vs. generative (GEN) benchmark scores for Freq, REAP, HC-SMoE, and REAM across ten mixing ratios of C4, Math, and Code datasets, with marker size proportional to each dataset’s share of the mixture. Results are shown at two expert-count targets: 64 (50% reduction) and 96 (25% reduction). The star denotes the performance of the original Qwen3-30B-A3B-Instruct with 128 experts.

Method	Ratio	IFEval	AIME25	GSM8K	GPQA	HumanEval	LCB	GEN
Original	—	89.6	80.0	85.4	42.4	92.7	47.5	72.9
REAP	0.0/0.3/0.7	87.5	70.0	86.4	37.9	94.5	47.7	70.7
	0.1/0.1/0.8	87.5	53.3	89.7	35.9	95.1	47.6	68.2
	0.2/0.25/0.55	86.6	60.0	87.6	37.9	93.3	47.0	68.7
	0.2/0.5/0.3	88.1	60.0	86.1	34.3	89.6	42.7	66.8
REAM full	0.0/0.3/0.7	89.3	80.0	85.3	40.4	94.5	48.0	72.9

Appendix

Table 4: Further GEN benchmark results for Qwen3-Coder-Next ( Cao et al., 2026 ) compressed from

512

to 384 experts (25% reduction in

N

) , with group size

C=32

. The ratio column denotes the calibration mixture (C4 : Math : Stack-Smol). Bold marks the best score in each column across all rows.

Figure 7: Pareto frontiers of expert-merging methods at 64 retained experts. Each point is one of 10 calibration mixtures; filled markers denote Pareto-optimal configurations (not simultaneously dominated on both MC and GEN by any other mixture of the same method) and hollow markers denote dominated ones. The hypervolume (HV) measures the area of the MC

\times

GEN plane dominated by each method’s frontier relative to a shared reference point, quantifying its overall performance ceiling. HV and

n/10

counts are computed on the original scores. Per-method offsets are then applied for better visibility.

LLM Compression

Large language model compression reduces the memory, compute, and context costs of LLMs while aiming to preserve their capabilities.

See all 273 papers

OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit
Sep 9, 2026Dezhi Li, Lujun Li, Qiyuan Zhu +4Mixture-of-Experts Pruning Mixture-of-Experts Language Models
The Hong Kong University of Science and Technology
Shape Mutating Expert Compression:LorExperts and BTExperts
Aug 7, 2026Inesh Chakrabarti, Sourjya Roy, Bowen Bao +3Mixture-of-Experts Language Models Low-Rank Matrix Decomposition
Advanced Micro Devices
LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing
Mar 13, 2026Jiawei Hao, Zhiwei Hao, Jianyuan Guo +4Mixture-of-Experts Language Models LLM Compression
School of Information and Electronics, Beijing Institute of Technology, Beijing, China · Department of Computer Science, City University of Hong Kong, Hong Kong, China · Shenzhen Campus of Sun Yat-sen University, Shenzhen, China +2

Sep 9, 2026J/K move · Enter open · S save

OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit

Dezhi Li, Lujun Li, Qiyuan Zhu, Hao Gu+3

The Hong Kong University of Science and Technology

Mixture-of-Experts Pruning Mixture-of-Experts Language Models LLM Compression LLM Inference Acceleration

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) models enable efficient scaling of large language models but face critical deployment challenges due to massive memory requirements. Existing pruning methods either incur prohibitive search costs or neglect the dynamic interdependencies between experts. To address these challenges, we present OMP-MoE, a novel training-free compression framework for reducing expert redundancy in MoE-based LLMs. Based on observations of expert contribution patterns, we reformulate the pruning problem as a sparse signal reconstruction task solved through Orthogonal Matching Pursuit. Specifically, our method first treats individual expert contributions as dictionary atoms and selects experts that greedily minimize reconstruction error with linear computational complexity. Then, we optimize cross-layer expert allocation through a water-filling strategy that accounts for both reconstruction quality and routing stability. Finally, we introduce OMP-MoE†, an adaptive inference mechanism that dynamically adjusts expert activation based on energy prediction. Comprehensive experiments on Qwen, DeepSeek-V2, GPT-OSS, and Mixtral MoE demonstrate consistent improvements over existing methods at 25-50% pruning ratios. For Qwen3-30B-A3B at 50% compression, we retain 93.3% of original performance, achieving 33

\times

faster search and 1.55

\times

inference speedup. Codes will be available after acceptance.

Abstract

LLM Compression

Explore similar work

OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit

Shape Mutating Expert Compression:LorExperts and BTExperts

LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing