GEAR2: Pruned Experts Live On through Gated Reuse of Activated Experts and a Missing-Mass Residual
Authors: Linliang Chen, Yan Zhong, Wang Kang, Sai Li, Ji Wu, Ji Liu
Organizations: 1Hangzhou International Innovation Institute, Beihang University, Hangzhou, China · School of Mathematical Sciences, Peking University, Beijing, China · School of Integrated Circuit Science and Engineering, Beihang University, Beijing, China · Department of Electronic Engineering, Tsinghua University, Beijing, China
Mixture-of-Experts (MoE) Large Language Models (LLMs) activate only a few experts per token while generally keeping all the experts in GPU memory for high efficiency. Expert pruning reduces this GPU memory footprint cost, yet discards the contribution of pruned experts. Existing remedies either rely on static substitutes, merge experts at the cost of altering their functions, or require costly recovery training. In this paper, we find that the functionalities of pruned experts can be efficiently reconstructed from the activated remaining experts. We propose GEAR (Gated Expert Activation Reuse), which reuses the scaled outputs of reliable activated experts without any training or additional expert computation. In addition, we propose GEAR2 to further recover what reuse cannot express with an ultra-lightweight residual. On five MoE LLMs from 16B to 106B parameters with 25% and 50% of experts pruned, GEAR improves over its training-free counterparts in nearly every setting (up to 4 pp.) in terms of accuracy, and GEAR2 outperforms baseline approaches (up to 2.75 pp.) in every setting on nine zero-shot benchmarks, with larger advantages (up to 2.72 pp. higher) under more aggressive pruning (up to 50%).
Figures & tables
Figure 1: Existing routes for mitigating the loss from expert pruning. (a) An MoE Transformer block. (b) Pruning discards the contribution of pruned experts. (c) Merging alters the functions of merged experts. (d) Recovery fine-tuning is costly.
Figure 2: Overview of GEAR and GEAR2 . (a) The pruned model keeps its router and activated retained experts; (b) each pruned expert reuses the scaled output of a reliable proxy and falls back to the mean otherwise; (c) GEAR2 adds a missing-mass-weighted low-rank residual, trained by light distillation with all original parameters frozen.
Moonlight
Qwen3
DeepSeek
Gemma-4
GLM
Method
25%
50%
25%
50%
25%
50%
25%
50%
25%
50%
Original
69.99
74.48
65.93
75.10
75.76
HC-SMoE
56.87
44.54
64.98
46.90
62.66
53.84
–
–
–
–
MoRA
67.48
55.36
72.50
65.83
63.68
54.40
–
–
–
–
REAP
66.68
55.37
73.56
67.41
63.19
54.36
69.61
59.20
74.33
68.81
MoNE
67.37
56.68
73.87
70.25
64.47
53.71
73.11
62.39
74.31
68.67
Table 1: Nine-task average accuracy (%) after pruning 25% or 50% of routed experts.
Method
RTE
BoolQ
OBQA
PIQA
ARC-C
ARC-E
Wino.
Hella.
MMLU
Avg.
Original
66.06
80.24
45.20
80.79
57.94
82.41
71.43
78.36
67.48
69.99
HC-SMoE
53.43
61.31
27.80
62.73
26.02
50.63
54.06
34.96
29.93
44.54
MoRA
61.73
70.92
35.40
77.31
36.43
65.32
61.01
61.47
28.66
55.36
REAP
46.21
56.51
40.80
77.42
42.83
66.25
70.80
72.76
24.76
55.37
MoNE
59.57
72.26
39.20
78.29
36.43
59.89
69.53
72.01
22.97
56.68
GEAR
60.65
72.11
39.60
78.51
36.26
60.06
70.40
72.22
22.97
56.98
Table 2: Accuracy (%) on nine zero-shot tasks at 50% pruning.
Figure 3: (a) Per-task gain of GEAR2 over MoNE at 50% pruning (pp). (b) Accuracy lost to pruning versus the gain of GEAR2 over MoNE: the gain grows with the loss. Dashed line: least-squares fit.
Method
Mask
Avg.
Δ
Mean
–
62.39
−0.59
No gate
No
61.76
−1.22
Random #1
Yes
62.02
−0.96
Random #2
Yes
61.41
−1.57
Reverse ε
Yes
61.98
−1.00
GEAR
Yes
62.98
ref.
Table 3: Reliability-gate ablation on Gemma-4 at 50% pruning. Mean: every pruned expert falls back to μi ; masked variants mask as many pairs as GEAR. Δ : relative to GEAR.
Base
Res.
m(x)
Avg.
Δ
Mean
–
–
70.25
−1.99
GEAR
–
–
71.41
−0.83
Zero
✓
✓
71.62
−0.62
Mean
✓
✓
71.61
−0.63
GEAR
✓
✗
68.62
−3.62
GEAR
✓
✓
72.24
ref.
Table 4: Residual ablation on Qwen3 at 50% pruning. Res.: low-rank residual; ✗: m(x) replaced by 1. Last row: GEAR2 . Δ : relative to GEAR2 .
Figure 4: Layer-wise hidden states at 50% pruning, projected onto two dimensions. Qualitative examples, selected by the error reduction of GEAR2 . Err: normalized L2 error of the hidden state relative to the original model at the shown layer.
Metric
MoNE
GEAR
GEAR 2
Training
–
–
64 steps, 32K tokens
Stats.
μi
μi,S,ε
μi,S,ε
Storage (GiB)
0.047
0.056
0.079
Extra experts
0
0
0
Latency (s)
2.54
3.15
3.47
Table 5: Recovery cost and inference overhead on Qwen3 at 50% pruning; the pruned model occupies 29.9 GiB in bf16.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Method
RTE
BoolQ
OBQA
PIQA
ARC-C
ARC-E
Wino.
Hella.
MMLU
Avg.
Original
66.06
80.24
45.20
80.79
57.94
82.41
71.43
78.36
67.48
69.99
HC-SMoE
59.57
70.34
36.40
68.99
41.72
70.45
56.59
53.93
53.80
56.87
MoRA
70.76
79.57
43.00
80.79
53.15
79.12
69.38
74.76
56.79
67.48
REAP
55.60
79.11
45.60
81.12
54.27
78.28
72.61
78.11
55.39
66.68
MoNE
62.09
79.24
45.60
81.39
55.46
79.46
71.82
78.02
53.22
67.37
GEAR
63.18
79.14
46.00
81.28
55.46
79.55
71.67
77.68
54.99
67.66
Appendix
Table 6: Accuracy (%) on nine zero-shot tasks at 25% pruning.
Model
vs. MoNE
vs. REAP
vs. GEAR
Moonlight
+2.41 [1.76, 3.03]
+3.72 [2.82, 4.62]
+2.11 [1.48, 2.75]
Qwen3
+1.99 [1.46, 2.56]
+4.83 [4.02, 5.67]
+0.83 [0.34, 1.36]
DeepSeek
+3.44 [2.65, 4.27]
+2.79 [1.84, 3.75]
+1.95 [1.22, 2.69]
Gemma-4
+2.14 [1.29, 2.96]
+5.33 [4.32, 6.35]
+1.55 [0.79, 2.32]
GLM
+0.96 [0.31, 1.64]
+0.82 [0.06, 1.56]
+1.34 [0.68, 2.02]
Appendix
Table 7: Improvement of GEAR2 at 50% pruning (points), with 95% paired bootstrap confidence intervals.
Model
Mean
GEAR
GEAR 2
Qwen3
64.79
65.95
66.62
DeepSeek-V2-Lite
53.55
55.37
57.16
Gemma-4
52.12
54.56
55.28
Appendix
Table 8: Nine-task average accuracy (%) with REAP masks at 50% pruning. Compensation consistently improves over the mean estimate.
Qwen3
DeepSeek
Gemma-4
Task
Mean
GEAR
GEAR 2
Mean
GEAR
GEAR 2
Mean
GEAR
GEAR 2
RTE
78.34
75.45
75.09
53.43
55.60
57.04
52.35
55.96
53.43
BoolQ
86.42
85.66
84.07
69.45
68.96
72.35
69.20
68.35
69.76
OBQA
35.60
38.20
40.40
36.80
36.80
39.40
36.00
40.60
40.20
PIQA
76.88
77.15
77.58
75.19
77.04
78.29
75.84
76.82
78.24
ARC-C
47.61
49.83
49.49
34.22
37.97
40.10
32.00
36.86
37.63
Appendix
Table 9: Accuracy (%) with REAP masks at 50% pruning.
GEAR2
Moonlight
DeepSeek
Seed 1, rank 32
58.51
57.15
Seed 2, rank 32
59.09
56.51
Seed 3, rank 32
58.82
56.58
Seed 1, rank 16
58.44
56.27
MoNE
56.68
53.71
Appendix
Table 10: Nine-task average accuracy (%) of GEAR2 at 50% pruning across training seeds and ranks.
τ
1
0.99
0.98
0.95
0.90
GEAR
51.68
56.98
56.55
56.54
56.67
Appendix
Table 11: Nine-task average accuracy (%) of GEAR on Moonlight at 50% pruning under different thresholds; τ=1 disables the gate.
25%
50%
Batch
REAP
GEAR 2
REAP
GEAR 2
1
74.33
74.39
68.81
69.63
2
74.16
74.36
67.74
69.23
3
74.09
74.16
68.57
69.85
Appendix
Table 12: Nine-task average accuracy (%) of GLM-4.5-Air with three calibration batches. GEAR2 outperforms REAP on every batch.
Model
Windows
Mean
GEAR 2
Closer
Moonlight
150
0.265
0.232
111/150
DeepSeek
150
0.222
0.168
142/150
GLM
150
0.249
0.239
99/150
Appendix
Table 13: Average hidden-state error at 50% pruning over token windows, and the number of windows in which GEAR2 is closer to the original model than the mean estimate.
Mixture-of-Experts (MoE) large language models (LLMs) are among the top-performing architectures. The largest models, often with hundreds of billions of parameters, pose significant memory challenges for deployment. Traditional approaches to reduce memory requirements include weight pruning and quantization. Motivated by the Router-weighted Expert Activation Pruning (REAP) that prunes experts, we propose a novel method, Router-weighted Expert Activation Merging (REAM). Instead of removing experts, REAM groups them and merges their weights, better preserving original performance. We evaluate REAM against REAP and other baselines across multiple MoE LLMs on diverse multiple-choice (MC) question answering and generative (GEN) benchmarks. Our results reveal a trade-off between MC and GEN performance that depends on the mix of calibration data. By controlling the mix of general, math and coding data, we examine the Pareto frontier of this trade-off and show that REAM often outperforms the baselines and in many cases is comparable to the original uncompressed models.
Saurav Jha, Maryam Hashemzadeh, Ali Saheb Pasand +3
Mila – Quebec AI Institute · Polytechnique Montréal · Université de Montréal +3
Mixture-of-Experts (MoE) models reduce computation through sparse expert activation, but deployment still requires storing a large number of expert parameters. We study post-training expert pruning using task-domain calibration data as a subset-selection problem: given a pretrained MoE layer, select a fixed number of experts that best reconstruct its original output. Exact subset search is combinatorial and becomes intractable for modern MoE models with many experts per layer. We develop OptEP, a scalable continuous relaxation that directly optimizes differentiable expert-selection variables, followed by a lightweight reconstruction-based update of the retained expert weights. Across multiple MoE architectures and tasks spanning mathematical reasoning and code generation, our approach improves over the strongest existing expert-pruning baselines, with absolute accuracy gains of up to 15.05% on Qwen3-30B-A3B, 14.55% on Qwen3-235B-A22B, 11.06% on GLM-4.5-Air, and 16.20% on Mixtral-8x7B. The improvements are particularly pronounced under aggressive pruning, demonstrating that reconstruction-driven expert selection and weight adaptation can preserve substantially more of the original model's performance while reducing the number of retained experts.
Jelena Markovic-Voronov, Petros Prastakos, Kayhan Behdin +7
Mixture-of-Experts (MoE) models enable efficient scaling of large language models but face critical deployment challenges due to massive memory requirements. Existing pruning methods either incur prohibitive search costs or neglect the dynamic interdependencies between experts. To address these challenges, we present OMP-MoE, a novel training-free compression framework for reducing expert redundancy in MoE-based LLMs. Based on observations of expert contribution patterns, we reformulate the pruning problem as a sparse signal reconstruction task solved through Orthogonal Matching Pursuit. Specifically, our method first treats individual expert contributions as dictionary atoms and selects experts that greedily minimize reconstruction error with linear computational complexity. Then, we optimize cross-layer expert allocation through a water-filling strategy that accounts for both reconstruction quality and routing stability. Finally, we introduce OMP-MoE†, an adaptive inference mechanism that dynamically adjusts expert activation based on energy prediction. Comprehensive experiments on Qwen, DeepSeek-V2, GPT-OSS, and Mixtral MoE demonstrate consistent improvements over existing methods at 25-50% pruning ratios. For Qwen3-30B-A3B at 50% compression, we retain 93.3% of original performance, achieving 33× faster search and 1.55× inference speedup. Codes will be available after acceptance.
Dezhi Li, Lujun Li, Qiyuan Zhu +4
The Hong Kong University of Science and Technology