GEAR2: Pruned Experts Live On through Gated Reuse of Activated Experts and a Missing-Mass Residual
Authors: Linliang Chen, Yan Zhong, Wang Kang, Sai Li, Ji Wu, Ji Liu
Organizations: 1Hangzhou International Innovation Institute, Beihang University, Hangzhou, China · School of Mathematical Sciences, Peking University, Beijing, China · School of Integrated Circuit Science and Engineering, Beihang University, Beijing, China · Department of Electronic Engineering, Tsinghua University, Beijing, China
Mixture-of-Experts (MoE) Large Language Models (LLMs) activate only a few experts per token while generally keeping all the experts in GPU memory for high efficiency. Expert pruning reduces this GPU memory footprint cost, yet discards the contribution of pruned experts. Existing remedies either rely on static substitutes, merge experts at the cost of altering their functions, or require costly recovery training. In this paper, we find that the functionalities of pruned experts can be efficiently reconstructed from the activated remaining experts. We propose GEAR (Gated Expert Activation Reuse), which reuses the scaled outputs of reliable activated experts without any training or additional expert computation. In addition, we propose GEAR2 to further recover what reuse cannot express with an ultra-lightweight residual. On five MoE LLMs from 16B to 106B parameters with 25% and 50% of experts pruned, GEAR improves over its training-free counterparts in nearly every setting (up to 4 pp.) in terms of accuracy, and GEAR2 outperforms baseline approaches (up to 2.75 pp.) in every setting on nine zero-shot benchmarks, with larger advantages (up to 2.72 pp. higher) under more aggressive pruning (up to 50%).
Figures & tables
Figure 1: Existing routes for mitigating the loss from expert pruning. (a) An MoE Transformer block. (b) Pruning discards the contribution of pruned experts. (c) Merging alters the functions of merged experts. (d) Recovery fine-tuning is costly.
Figure 2: Overview of GEAR and GEAR2 . (a) The pruned model keeps its router and activated retained experts; (b) each pruned expert reuses the scaled output of a reliable proxy and falls back to the mean otherwise; (c) GEAR2 adds a missing-mass-weighted low-rank residual, trained by light distillation with all original parameters frozen.
Moonlight
Qwen3
DeepSeek
Gemma-4
GLM
Method
25%
50%
25%
50%
25%
50%
25%
50%
25%
50%
Original
69.99
74.48
65.93
75.10
75.76
HC-SMoE
56.87
44.54
64.98
46.90
62.66
53.84
–
–
–
–
MoRA
67.48
55.36
72.50
65.83
63.68
54.40
–
–
–
–
REAP
66.68
55.37
73.56
67.41
63.19
54.36
69.61
59.20
74.33
68.81
MoNE
67.37
56.68
73.87
70.25
64.47
53.71
73.11
62.39
74.31
68.67
Table 1: Nine-task average accuracy (%) after pruning 25% or 50% of routed experts.
Method
RTE
BoolQ
OBQA
PIQA
ARC-C
ARC-E
Wino.
Hella.
MMLU
Avg.
Original
66.06
80.24
45.20
80.79
57.94
82.41
71.43
78.36
67.48
69.99
HC-SMoE
53.43
61.31
27.80
62.73
26.02
50.63
54.06
34.96
29.93
44.54
MoRA
61.73
70.92
35.40
77.31
36.43
65.32
61.01
61.47
28.66
55.36
REAP
46.21
56.51
40.80
77.42
42.83
66.25
70.80
72.76
24.76
55.37
MoNE
59.57
72.26
39.20
78.29
36.43
59.89
69.53
72.01
22.97
56.68
GEAR
60.65
72.11
39.60
78.51
36.26
60.06
70.40
72.22
22.97
56.98
Table 2: Accuracy (%) on nine zero-shot tasks at 50% pruning.
Figure 3: (a) Per-task gain of GEAR2 over MoNE at 50% pruning (pp). (b) Accuracy lost to pruning versus the gain of GEAR2 over MoNE: the gain grows with the loss. Dashed line: least-squares fit.
Method
Mask
Avg.
Δ
Mean
–
62.39
−0.59
No gate
No
61.76
−1.22
Random #1
Yes
62.02
−0.96
Random #2
Yes
61.41
−1.57
Reverse ε
Yes
61.98
−1.00
GEAR
Yes
62.98
ref.
Table 3: Reliability-gate ablation on Gemma-4 at 50% pruning. Mean: every pruned expert falls back to μi ; masked variants mask as many pairs as GEAR. Δ : relative to GEAR.
Base
Res.
m(x)
Avg.
Δ
Mean
–
–
70.25
−1.99
GEAR
–
–
71.41
−0.83
Zero
✓
✓
71.62
−0.62
Mean
✓
✓
71.61
−0.63
GEAR
✓
✗
68.62
−3.62
GEAR
✓
✓
72.24
ref.
Table 4: Residual ablation on Qwen3 at 50% pruning. Res.: low-rank residual; ✗: m(x) replaced by 1. Last row: GEAR2 . Δ : relative to GEAR2 .
Figure 4: Layer-wise hidden states at 50% pruning, projected onto two dimensions. Qualitative examples, selected by the error reduction of GEAR2 . Err: normalized L2 error of the hidden state relative to the original model at the shown layer.
Metric
MoNE
GEAR
GEAR 2
Training
–
–
64 steps, 32K tokens
Stats.
μi
μi,S,ε
μi,S,ε
Storage (GiB)
0.047
0.056
0.079
Extra experts
0
0
0
Latency (s)
2.54
3.15
3.47
Table 5: Recovery cost and inference overhead on Qwen3 at 50% pruning; the pruned model occupies 29.9 GiB in bf16.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Method
RTE
BoolQ
OBQA
PIQA
ARC-C
ARC-E
Wino.
Hella.
MMLU
Avg.
Original
66.06
80.24
45.20
80.79
57.94
82.41
71.43
78.36
67.48
69.99
HC-SMoE
59.57
70.34
36.40
68.99
41.72
70.45
56.59
53.93
53.80
56.87
MoRA
70.76
79.57
43.00
80.79
53.15
79.12
69.38
74.76
56.79
67.48
REAP
55.60
79.11
45.60
81.12
54.27
78.28
72.61
78.11
55.39
66.68
MoNE
62.09
79.24
45.60
81.39
55.46
79.46
71.82
78.02
53.22
67.37
GEAR
63.18
79.14
46.00
81.28
55.46
79.55
71.67
77.68
54.99
67.66
Appendix
Table 6: Accuracy (%) on nine zero-shot tasks at 25% pruning.
Model
vs. MoNE
vs. REAP
vs. GEAR
Moonlight
+2.41 [1.76, 3.03]
+3.72 [2.82, 4.62]
+2.11 [1.48, 2.75]
Qwen3
+1.99 [1.46, 2.56]
+4.83 [4.02, 5.67]
+0.83 [0.34, 1.36]
DeepSeek
+3.44 [2.65, 4.27]
+2.79 [1.84, 3.75]
+1.95 [1.22, 2.69]
Gemma-4
+2.14 [1.29, 2.96]
+5.33 [4.32, 6.35]
+1.55 [0.79, 2.32]
GLM
+0.96 [0.31, 1.64]
+0.82 [0.06, 1.56]
+1.34 [0.68, 2.02]
Appendix
Table 7: Improvement of GEAR2 at 50% pruning (points), with 95% paired bootstrap confidence intervals.
Model
Mean
GEAR
GEAR 2
Qwen3
64.79
65.95
66.62
DeepSeek-V2-Lite
53.55
55.37
57.16
Gemma-4
52.12
54.56
55.28
Appendix
Table 8: Nine-task average accuracy (%) with REAP masks at 50% pruning. Compensation consistently improves over the mean estimate.
Qwen3
DeepSeek
Gemma-4
Task
Mean
GEAR
GEAR 2
Mean
GEAR
GEAR 2
Mean
GEAR
GEAR 2
RTE
78.34
75.45
75.09
53.43
55.60
57.04
52.35
55.96
53.43
BoolQ
86.42
85.66
84.07
69.45
68.96
72.35
69.20
68.35
69.76
OBQA
35.60
38.20
40.40
36.80
36.80
39.40
36.00
40.60
40.20
PIQA
76.88
77.15
77.58
75.19
77.04
78.29
75.84
76.82
78.24
ARC-C
47.61
49.83
49.49
34.22
37.97
40.10
32.00
36.86
37.63
Appendix
Table 9: Accuracy (%) with REAP masks at 50% pruning.
GEAR2
Moonlight
DeepSeek
Seed 1, rank 32
58.51
57.15
Seed 2, rank 32
59.09
56.51
Seed 3, rank 32
58.82
56.58
Seed 1, rank 16
58.44
56.27
MoNE
56.68
53.71
Appendix
Table 10: Nine-task average accuracy (%) of GEAR2 at 50% pruning across training seeds and ranks.
τ
1
0.99
0.98
0.95
0.90
GEAR
51.68
56.98
56.55
56.54
56.67
Appendix
Table 11: Nine-task average accuracy (%) of GEAR on Moonlight at 50% pruning under different thresholds; τ=1 disables the gate.
25%
50%
Batch
REAP
GEAR 2
REAP
GEAR 2
1
74.33
74.39
68.81
69.63
2
74.16
74.36
67.74
69.23
3
74.09
74.16
68.57
69.85
Appendix
Table 12: Nine-task average accuracy (%) of GLM-4.5-Air with three calibration batches. GEAR2 outperforms REAP on every batch.
Model
Windows
Mean
GEAR 2
Closer
Moonlight
150
0.265
0.232
111/150
DeepSeek
150
0.222
0.168
142/150
GLM
150
0.249
0.239
99/150
Appendix
Table 13: Average hidden-state error at 50% pruning over token windows, and the number of windows in which GEAR2 is closer to the original model than the mean estimate.