Organizations: School of Electrical Engineering and Computer Science (SEECS), National University of Sciences and Technology (NUST) · Deep Learning Laboratory, National Center of Artificial Intelligence (NCAI), NUST
Training-free expert pruning makes two decisions: which experts to remove, and what the layer computes when its router still selects a removed expert. Published methods change both at once. Holding the mask fixed, we compare replacement operators from three families on OLMoE, DeepSeek-V2-Lite and Moonlight, with Granite as a fourth router, using paired, multiplicity-corrected tests. The largest effect is a choice usually left implicit: the survivors' gate scale. On routers trained without top-k renormalization, renormalizing the survivors' gates, a common implementation choice, costs 0.8-8.5 points and reverses whether rerouting beats MoNE's mean-output replacement; on two routers trained with renormalization, the trained scale wins at 50% pruning. We introduce the tilted novice, a closed-form low-rank map of the pruned expert's input whose rank-0 case is MoNE's constant. At 50% pruning it adds 1.2-1.8% parameters and gains 3.45 and 1.37 points over the constant on OLMoE and Moonlight, stable across calibration draws and largely unchanged by 4- and 3-bit quantization. No operator dominates: trained-scale rerouting is best on DeepSeek-V2-Lite, and a partial copy of each pruned expert is statistically indistinguishable from the tilted novice on two models. Evaluation can change conclusions: at 25% pruning MoNE's constant keeps 92-97% of dense multiple-choice accuracy but only 5-65% of GSM8K and 0-17% of code scores, and the calibration domain moves the mask 4-9 times more than resampling. We propose a five-item reporting checklist.
Figures & tables
Figure 1: Protocol. Selection is done once and held fixed; only the replacement operator varies. Calibration noise is measured by refitting operators on disjoint draws with the mask unchanged; test-item noise by the paired bootstrap; multiplicity by Holm.
Figure 2: Seven operators in three families act on the same mask; they differ only in what the layer computes when the router selects a pruned expert ( Ei , dashed). Remapping, renormalization and COMPEL move the slot’s work onto survivors ( Section 5 ). x~=x−xˉi ; P is the pruned experts’ softmax mass for the token. The tilt stores μi (already in MoNE’s model), Ui , Vi and two r -vectors; xˉi is folded into the feature offset.
Reroute, gate scale
Model
Pruned
Constant
Tilt r32
Trained
Other
OLMoE
25%
61.19
63.02
62.65
58.15
50%
49.09
52.54
51.48
48.08
75%
41.04
43.84
42.94
39.85
DeepSeek
25%
65.40
65.56
65.66
64.87
50%
59.84
60.32
61.19
56.52
Table 4: Nine-task zero-shot accuracy (%), MoNE masks, calibration draw A. Dense: OLMoE 66.76, DeepSeek 67.15, Moonlight 71.48. Rerouting at the trained scale is keep-scale on OLMoE and DeepSeek and renormalized on Moonlight; the other scale is the opposite arm (Moonlight’s from Table 11 ). Bold: highest in the row, significant or not; intervals are in Figure 3 .
Figure 3: Operator differences in nine-task accuracy with 95% item-bootstrap intervals, MoNE masks, calibration draw A. Filled marker: the interval excludes zero. Small dots: the tilt on calibration draws B and C. † / ‡ : survives Holm correction within the figure / over all 111 comparisons ( Section H.1 ). “Keep” is rerouting at the router’s trained gate scale on every model (renormalized on Moonlight).
Figure 4: The tilt’s gain over the constant against the damage the constant leaves (dense − constant), nine-task, MoNE masks, draw A, 95% intervals; filled: the interval excludes zero. Diamonds: Granite ( Section 5.3 ), whose contrasts are outside the Holm families. Where the constant loses more than 10 points every gain is significant; below 3 points none is.
Figure 5: Nine-task accuracy at 50% pruning against parameters added to MoNE’s constant model (MoNE masks, draw A); contrasts with intervals are in Table H.6 . Tilt: ranks 8, 32, 64 and 128 on OLMoE, rank 32 elsewhere. Extra experts: MoNE’s mask keeping 33 and 34 of 64 experts per layer, evaluated in the same process as every other arm. Copies at the tilt’s rank-32 budget, and at four times it on OLMoE.
Tilt − constant
Reroute − constant
Tilt − reroute
Model
0-shot
5-shot
0-shot
5-shot
5-shot
OLMoE
+3.45 [+2.62, +4.27]*
+3.01 [+2.25, +3.85]*
+2.38 [+1.42, +3.33]*
+2.03 [+1.16, +2.90]*
+0.98 [+0.12, +1.87]*
DeepSeek
+0.48 [−0.03, +0.99]
−0.29 [−0.88, +0.29]
+1.35 [+0.65, +2.08]*
+2.46 [+1.76, +3.17]*
−2.75 [−3.53, −2.00]*
Moonlight
+1.37 [+0.73, +2.01]*
+1.54 [+0.70, +2.38]*
−0.18 [−1.04, +0.68]
+1.28 [+0.42, +2.16]*
+0.25 [−0.57, +1.08]
Table 6: Gain over the constant, zero-shot and 5-shot, nine-task, 50%, MoNE masks, draw A, 95% intervals. Reroute: rerouting at the router’s trained gate scale. 5-shot accuracy (constant / tilt / reroute): OLMoE 52.23 / 55.24 / 54.26, DeepSeek 62.88 / 62.59 / 65.34, Moonlight 62.38 / 63.92 / 63.66.
Model
Operator
WikiText
C4
Zyda-2 C
Acc.
OLMoE
Tilt r32
18.3
23.7 / 23.8
22.3 / 22.5
19.5
Reroute
11.1
12.4
9.3
13.5
Neuron copy
—
28.0
26.6
—
W-SVD copy
—
16.1
14.2
—
DeepSeek
Tilt r32
19.0
13.9 / 14.3
14.1 / 14.1
6.6
Reroute
27.4
29.8
27.2
18.5
Table 7: Share of the constant’s damage recovered at 50% pruning (%), MoNE masks. Tilt cells on C4 and Zyda-2 give calibration draws A / B; all other cells use draw A. Reroute: rerouting at the trained gate scale. Copies are the equal-budget copies of Section 4.3 .
Setting
Nine-task
Big-5
MMLU
OLMoE 25%
−4.50 [−5.48, −3.57]*
−5.59*
−2.08*
OLMoE 50%
−3.39 [−4.66, −2.12]*
−4.31*
−2.04*
OLMoE 75%
−3.09 [−4.52, −1.68]*
−4.83*
+0.73*
DeepSeek 25%
−0.79 [−1.51, −0.07]*
−0.67*
−1.40*
DeepSeek 50%
−4.67 [−5.71, −3.59]*
−6.28*
−7.35*
DeepSeek 75%
−8.47 [−9.98, −7.01]*
−10.84*
+0.73*
Table 8: Renormalized minus keep-scale rerouting, same mask (MoNE), 95% intervals for the nine-task mean. Every nine-task row survives Holm within the table; all but DeepSeek-V2-Lite 25% also survive over all comparisons. At 75% both operators score below chance on MMLU (22.9–23.7%, chance 25%), so the positive MMLU difference there does not indicate retained knowledge.
MoNE mask
REAP mask
Setting
Constant
Renorm
Keep
Renorm
Keep
OLMoE 25%
61.19
58.15
62.65
—
—
OLMoE 50%
49.09
48.08
51.48
47.12
49.00
DeepSeek 25%
65.40
64.87
65.66
—
—
DeepSeek 50%
59.84
56.52
61.19
56.58
60.46
Table 9: Nine-task accuracy of rerouting under each gate scale, against MoNE’s constant on its own mask. REAP’s mask was evaluated at 50% only; “renorm” on the REAP mask is REAP with its own routing semantics (REAP did not evaluate these models).
Figure 6: Each operator that changes the survivors’ gate scale or moves a pruned slot’s work onto survivors, minus rerouting at the scale the router was trained with: keep-scale on OLMoE and DeepSeek-V2-Lite (softmax, trained without top- k renormalization) and on Moonlight (sigmoid, trained with top- k renormalization: rerouting at its trained, renormalized scale; Section 3 ), renormalized on Granite (softmax, trained with renormalization; Section 5.3 ). Nine-task, MoNE masks, draw A, 95% intervals; filled marker: the interval excludes zero; ‡ / † : survives Holm over all comparisons / within its table only; Granite’s remap row is outside the Holm families. Rescaling changes the survivors’ gates (renorm on the first two models, keep on Granite); remapping hands the slot to a survivor at the unchanged gate value; COMPEL moves pruned weights into the survivors.
Pruned
25%
50%
75%
Constant
57.95
45.27
41.30
Tilt r32
59.24
48.64
42.88
Keep (other scale)
57.11
44.65
40.87
Renorm (trained scale)
57.39
46.74
41.14
Remap nearest
—
43.23
—
Renorm − keep
+0.28
+2.08* ‡
+0.27
Table 10: Granite-3.0-3B-A800M (softmax router trained with top- k renormalization), MoNE masks, draw A. Top: nine-task accuracy (dense 61.82; guessing about 40.5); bold: highest in the column. Bottom: contrasts; * the 95% interval excludes zero; ‡ : survives Holm over all comparisons (the other contrasts are outside the Holm families). Every interval, with big-5 and MMLU, is in Table H.11 .
Model
Dense
Constant
Tilt r32
Trained-scale reroute
Neuron copy
OLMoE
66.76
49.09
52.54
51.48
52.63
DeepSeek-V2-Lite
67.15
59.84
60.32
61.19
60.18
Moonlight
71.48
60.24
61.61
60.06
60.94
Table 12: Representative replacement operators at 50% expert pruning, on MoNE masks and the same Zyda-2 calibration. Values are mean accuracy over nine zero-shot tasks (%). Trained-scale reroute: keep-scale on OLMoE and DeepSeek-V2-Lite, renormalized on Moonlight. The tilt and the neuron copy add similar parameter budgets.
Figure 7: Share of the dense model’s score kept at 25% pruning with MoNE’s constant and general calibration: nine-task multiple choice, GSM8K (5-shot, strict match), HumanEval (pass@1) and MBPP (pass@1, 3-shot). Multiple-choice tasks have chance floors and generation tasks near-zero floors, so the bars are not comparable fractions of ability; above chance, multiple choice keeps 79%, 93% and 91% ( Table A.9 ).
Setting
Constant
Tilt
Reroute
Tilt − constant
Reroute − constant
OLMoE 25%
2.43
1.82
2.73
−0.61 [−1.59, +0.38]
+0.30 [−0.76, +1.29]
OLMoE 50%
0.00
0.00
0.68
+0.00 [+0.00, +0.00]
+0.68 [+0.30, +1.14]* ‡
DeepSeek 25%
24.56
23.12
23.81
−1.44 [−3.11, +0.30]
−0.76 [−2.58, +1.06]
DeepSeek 50%
3.79
4.62
6.37
+0.83 [−0.53, +2.20]
+2.58 [+1.21, +4.02]* ‡
Moonlight 25%
32.60
32.75
42.76
+0.15 [−1.97, +2.35]
+10.16 [+7.66, +12.66]* ‡
Moonlight 50%
0.38
1.36
1.52
+0.99 [+0.30, +1.67]* †
+1.14 [+0.38, +1.90]* †
Table 13: GSM8K (5-shot, strict match) absolute scores and gain over the constant on the same mask, general calibration, 95% item-bootstrap intervals. Dense: OLMoE 45.49, DeepSeek 37.53, Moonlight 73.84. At 50% every model keeps at most 17% of its dense score under every operator, so differences there cannot rank operators; OLMoE 50% scores 0 under both the constant and the tilt. Reroute: rerouting at the trained gate scale. † / ‡ : survives Holm within the table / over all comparisons.
GSM8K vs general
General A vs B
Model
50%
25%
50%
OLMoE
65.2%
44.1%
93.6%
DeepSeek
70.3%
50.0%
95.8%
Moonlight
66.1%
60.8%
95.8%
Table 14: Share of pruned experts that the GSM8K-calibrated and general (Zyda-2) masks have in common (chance: 50% and 25%). Last column: two disjoint general draws, both masks from MoNE’s released code.
Figure 8: Nine-task gain over the constant on the same mask, with general (Zyda-2) and GSM8K calibration, for the tilt and rerouting at the trained gate scale (renormalized on Moonlight; Section 3 ). One GSM8K calibration draw.
Operator
Weights (GB)
Prefill
Kept
Decode
OLMoE
Dense
13.84
1.000
—
1.000
Constant (MoNE)
7.40
0.752
100%
0.885
Tilt r32
7.53
0.837
66%
1.076
Keep reroute
7.40
0.982
7%
0.946
DeepSeek-V2-Lite
Table 16: Weights and HuggingFace eager timings relative to the dense model at 50% pruning (prompt 512 tokens, 64 decoded). Prefill at batch 32; decode per token at batch 1. “Kept”: share of MoNE’s prefill saving kept. Timings compare operators within a model; they are not serving-stack times. Weights are measured sizes; on DeepSeek-V2-Lite they exceed two bytes per parameter ( Table B.2 ; 31.41 GB dense, as in Table F.5 ) by a constant 1.14 GB in every row, which does not affect comparisons between operators. Batch 1 and 8: Table F.1 .
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Model
25%
50%
75%
OLMoE
3.4 / 32.9
18.3 / 19.5
41.2 / 10.9
DeepSeek
−4.9 / 9.0
19.0 / 6.6
26.3 / 6.3
Moonlight
17.7 / 4.6
18.7 / 12.2
37.7 / 11.8
Appendix
Table A.6 : The tilt’s share of the constant’s damage recovered by pruning ratio (%): WikiText log-perplexity / nine-task accuracy, MoNE masks, draw A.
Setting
Nine-task
MMLU
WinoGrande
OLMoE 25%
+0.06
+3.81*
−3.39
OLMoE 50%
−0.16
+9.61*
−6.95
DeepSeek 25%
−1.90*
−5.55*
−2.53
DeepSeek 50%
−3.89*
−6.53*
−6.08
Moonlight 25%
+1.06*
+5.70*
−3.00
Moonlight 50%
−3.86*
+17.38*
−8.21
Appendix
Table A.11 : Change in multiple-choice accuracy from GSM8K calibration (GSM8K minus general calibration), constant novice. * = 95% item-bootstrap interval excludes zero; WinoGrande is a point estimate.
Family
Comparison
Δ [95% CI]
p
pHolm table
pHolm all
Verdict
Fig. 3
OLMoE 25% tilt − constant
+1.83 [+1.21, +2.47]*
2.0e − 4
0.005
0.021
both
Fig. 3
OLMoE 25% keep − constant
+1.46 [+0.85, +2.11]*
2.0e − 4
0.005
0.021
both
Fig. 3
OLMoE 25% tilt − keep
+0.37 [ − 0.28, +1.00]
0.262
1
1
n.s.
Fig. 3
OLMoE 50% tilt − constant
+3.45 [+2.62, +4.27]*
2.0e − 4
0.005
0.021
both
Fig. 3
OLMoE 50% keep − constant
+2.38 [+1.42, +3.33]*
2.0e − 4
0.005
0.021
both
Fig. 3
OLMoE 50% tilt − keep
+1.07 [+0.07, +2.08]*
0.035
0.381
1
lost
Appendix
Table H.1 : Every paired comparison with Holm-adjusted p-values (adjusted values capped at 1). Keep: rerouting at the trained gate scale (renormalized on Moonlight).
OLMoE
DeepSeek
Moonlight
Arm
Added
Acc.
Added
Acc.
Added
Acc.
Constant (MoNE)
0
49.09
0
59.84
0
60.24
Rerouting, trained scale
0
51.48
0
61.19
0
60.06
Tilt r8
16.8 M
51.92
27.3 M
59.42
27.3 M
60.87
Tilt r32
67.1 M
52.54
109.1 M
60.32
109.1 M
61.61
Tilt r64
134.3 M
53.52
—
—
—
—
Appendix
Table H.6 : Nine-task accuracy at 50% pruning against the parameters each replacement adds to MoNE’s constant model (MoNE masks, draw A). Extra experts: MoNE’s mask keeping 33 or 34 of 64 experts per layer, with constants for the rest; one extra expert per layer is 100.7 M on OLMoE and 224.9 M on the other two. Copies store a compressed version of each pruned expert at the tilt’s rank-32 budget (1×) or four times it (4×, OLMoE only); their bias replaces the constant’s stored mean, so they add d values per pruned expert less than they store ( Table C.2 lists what they store). Every arm is evaluated in the same process with the same constants; MoNE’s saved keep-33/34 checkpoints, evaluated separately, give 49.95 and 51.22 on OLMoE ( Section H.5 ). † / ‡ : survives Holm within the table / over all comparisons.