What We Keep of What We Forget: Replacement Operators and Routing Semantics in Training-Free MoE Expert Pruning
Organizations: School of Electrical Engineering and Computer Science (SEECS), National University of Sciences and Technology (NUST) · Deep Learning Laboratory, National Center of Artificial Intelligence (NCAI), NUST
Abstract
Training-free expert pruning makes two decisions: which experts to remove, and what the layer computes when its router still selects a removed expert. Published methods change both at once. Holding the mask fixed, we compare replacement operators from three families on OLMoE, DeepSeek-V2-Lite and Moonlight, with Granite as a fourth router, using paired, multiplicity-corrected tests. The largest effect is a choice usually left implicit: the survivors' gate scale. On routers trained without top-k renormalization, renormalizing the survivors' gates, a common implementation choice, costs 0.8-8.5 points and reverses whether rerouting beats MoNE's mean-output replacement; on two routers trained with renormalization, the trained scale wins at 50% pruning. We introduce the tilted novice, a closed-form low-rank map of the pruned expert's input whose rank-0 case is MoNE's constant. At 50% pruning it adds 1.2-1.8% parameters and gains 3.45 and 1.37 points over the constant on OLMoE and Moonlight, stable across calibration draws and largely unchanged by 4- and 3-bit quantization. No operator dominates: trained-scale rerouting is best on DeepSeek-V2-Lite, and a partial copy of each pruned expert is statistically indistinguishable from the tilted novice on two models. Evaluation can change conclusions: at 25% pruning MoNE's constant keeps 92-97% of dense multiple-choice accuracy but only 5-65% of GSM8K and 0-17% of code scores, and the calibration domain moves the mask 4-9 times more than resampling. We propose a five-item reporting checklist.
Figures & tables
| Reroute, gate scale | |||||
|---|---|---|---|---|---|
| Model | Pruned | Constant | Tilt r32 | Trained | Other |
| OLMoE | 25% | 61.19 | 63.02 | 62.65 | 58.15 |
| 50% | 49.09 | 52.54 | 51.48 | 48.08 | |
| 75% | 41.04 | 43.84 | 42.94 | 39.85 | |
| DeepSeek | 25% | 65.40 | 65.56 | 65.66 | 64.87 |
| 50% | 59.84 | 60.32 | 61.19 | 56.52 | |
| Tilt constant | Reroute constant | Tilt reroute | |||
|---|---|---|---|---|---|
| Model | 0-shot | 5-shot | 0-shot | 5-shot | 5-shot |
| OLMoE | +3.45 [+2.62, +4.27]* | +3.01 [+2.25, +3.85]* | +2.38 [+1.42, +3.33]* | +2.03 [+1.16, +2.90]* | +0.98 [+0.12, +1.87]* |
| DeepSeek | +0.48 [−0.03, +0.99] | −0.29 [−0.88, +0.29] | +1.35 [+0.65, +2.08]* | +2.46 [+1.76, +3.17]* | −2.75 [−3.53, −2.00]* |
| Moonlight | +1.37 [+0.73, +2.01]* | +1.54 [+0.70, +2.38]* | −0.18 [−1.04, +0.68] | +1.28 [+0.42, +2.16]* | +0.25 [−0.57, +1.08] |
| Model | Operator | WikiText | C4 | Zyda-2 C | Acc. |
|---|---|---|---|---|---|
| OLMoE | Tilt r32 | 18.3 | 23.7 / 23.8 | 22.3 / 22.5 | 19.5 |
| Reroute | 11.1 | 12.4 | 9.3 | 13.5 | |
| Neuron copy | — | 28.0 | 26.6 | — | |
| W-SVD copy | — | 16.1 | 14.2 | — | |
| DeepSeek | Tilt r32 | 19.0 | 13.9 / 14.3 | 14.1 / 14.1 | 6.6 |
| Reroute | 27.4 | 29.8 | 27.2 | 18.5 |
| Setting | Nine-task | Big-5 | MMLU |
|---|---|---|---|
| OLMoE 25% | −4.50 [−5.48, −3.57]* | −5.59* | −2.08* |
| OLMoE 50% | −3.39 [−4.66, −2.12]* | −4.31* | −2.04* |
| OLMoE 75% | −3.09 [−4.52, −1.68]* | −4.83* | +0.73* |
| DeepSeek 25% | −0.79 [−1.51, −0.07]* | −0.67* | −1.40* |
| DeepSeek 50% | −4.67 [−5.71, −3.59]* | −6.28* | −7.35* |
| DeepSeek 75% | −8.47 [−9.98, −7.01]* | −10.84* | +0.73* |
| MoNE mask | REAP mask | ||||
|---|---|---|---|---|---|
| Setting | Constant | Renorm | Keep | Renorm | Keep |
| OLMoE 25% | 61.19 | 58.15 | 62.65 | — | — |
| OLMoE 50% | 49.09 | 48.08 | 51.48 | 47.12 | 49.00 |
| DeepSeek 25% | 65.40 | 64.87 | 65.66 | — | — |
| DeepSeek 50% | 59.84 | 56.52 | 61.19 | 56.58 | 60.46 |
| Pruned | 25% | 50% | 75% |
|---|---|---|---|
| Constant | 57.95 | 45.27 | 41.30 |
| Tilt r32 | 59.24 | 48.64 | 42.88 |
| Keep (other scale) | 57.11 | 44.65 | 40.87 |
| Renorm (trained scale) | 57.39 | 46.74 | 41.14 |
| Remap nearest | — | 43.23 | — |
| Renorm keep | +0.28 | +2.08* ‡ | +0.27 |
| Model | Dense | Constant | Tilt r32 | Trained-scale reroute | Neuron copy |
|---|---|---|---|---|---|
| OLMoE | 66.76 | 49.09 | 52.54 | 51.48 | 52.63 |
| DeepSeek-V2-Lite | 67.15 | 59.84 | 60.32 | 61.19 | 60.18 |
| Moonlight | 71.48 | 60.24 | 61.61 | 60.06 | 60.94 |
| Setting | Constant | Tilt | Reroute | Tilt constant | Reroute constant |
|---|---|---|---|---|---|
| OLMoE 25% | 2.43 | 1.82 | 2.73 | −0.61 [−1.59, +0.38] | +0.30 [−0.76, +1.29] |
| OLMoE 50% | 0.00 | 0.00 | 0.68 | +0.00 [+0.00, +0.00] | +0.68 [+0.30, +1.14]* ‡ |
| DeepSeek 25% | 24.56 | 23.12 | 23.81 | −1.44 [−3.11, +0.30] | −0.76 [−2.58, +1.06] |
| DeepSeek 50% | 3.79 | 4.62 | 6.37 | +0.83 [−0.53, +2.20] | +2.58 [+1.21, +4.02]* ‡ |
| Moonlight 25% | 32.60 | 32.75 | 42.76 | +0.15 [−1.97, +2.35] | +10.16 [+7.66, +12.66]* ‡ |
| Moonlight 50% | 0.38 | 1.36 | 1.52 | +0.99 [+0.30, +1.67]* † | +1.14 [+0.38, +1.90]* † |
| GSM8K vs general | General A vs B | ||
|---|---|---|---|
| Model | 50% | 25% | 50% |
| OLMoE | 65.2% | 44.1% | 93.6% |
| DeepSeek | 70.3% | 50.0% | 95.8% |
| Moonlight | 66.1% | 60.8% | 95.8% |
| Operator | Weights (GB) | Prefill | Kept | Decode |
|---|---|---|---|---|
| OLMoE | ||||
| Dense | 13.84 | 1.000 | — | 1.000 |
| Constant (MoNE) | 7.40 | 0.752 | 100% | 0.885 |
| Tilt r32 | 7.53 | 0.837 | 66% | 1.076 |
| Keep reroute | 7.40 | 0.982 | 7% | 0.946 |
| DeepSeek-V2-Lite | ||||
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | 25% | 50% | 75% |
|---|---|---|---|
| OLMoE | 3.4 / 32.9 | 18.3 / 19.5 | 41.2 / 10.9 |
| DeepSeek | −4.9 / 9.0 | 19.0 / 6.6 | 26.3 / 6.3 |
| Moonlight | 17.7 / 4.6 | 18.7 / 12.2 | 37.7 / 11.8 |
| Setting | Nine-task | MMLU | WinoGrande |
|---|---|---|---|
| OLMoE 25% | +0.06 | +3.81* | −3.39 |
| OLMoE 50% | −0.16 | +9.61* | −6.95 |
| DeepSeek 25% | −1.90* | −5.55* | −2.53 |
| DeepSeek 50% | −3.89* | −6.53* | −6.08 |
| Moonlight 25% | +1.06* | +5.70* | −3.00 |
| Moonlight 50% | −3.86* | +17.38* | −8.21 |
| Family | Comparison | [95% CI] | table | all | Verdict | |
|---|---|---|---|---|---|---|
| Fig. 3 | OLMoE 25% tilt constant | +1.83 [+1.21, +2.47]* | 2.0e 4 | 0.005 | 0.021 | both |
| Fig. 3 | OLMoE 25% keep constant | +1.46 [+0.85, +2.11]* | 2.0e 4 | 0.005 | 0.021 | both |
| Fig. 3 | OLMoE 25% tilt keep | +0.37 [ 0.28, +1.00] | 0.262 | 1 | 1 | n.s. |
| Fig. 3 | OLMoE 50% tilt constant | +3.45 [+2.62, +4.27]* | 2.0e 4 | 0.005 | 0.021 | both |
| Fig. 3 | OLMoE 50% keep constant | +2.38 [+1.42, +3.33]* | 2.0e 4 | 0.005 | 0.021 | both |
| Fig. 3 | OLMoE 50% tilt keep | +1.07 [+0.07, +2.08]* | 0.035 | 0.381 | 1 | lost |
| OLMoE | DeepSeek | Moonlight | ||||
| Arm | Added | Acc. | Added | Acc. | Added | Acc. |
| Constant (MoNE) | 0 | 49.09 | 0 | 59.84 | 0 | 60.24 |
| Rerouting, trained scale | 0 | 51.48 | 0 | 61.19 | 0 | 60.06 |
| Tilt r8 | 16.8 M | 51.92 | 27.3 M | 59.42 | 27.3 M | 60.87 |
| Tilt r32 | 67.1 M | 52.54 | 109.1 M | 60.32 | 109.1 M | 61.61 |
| Tilt r64 | 134.3 M | 53.52 | — | — | — | — |
Mixture-of-Experts Pruning
Mixture-of-Experts pruning removes or trims experts to reduce model memory and computation while preserving useful capabilities.
See all 68 papers