When Load-Balancing Goes Too Far: Expert Pruning in Over-Dispersed Mixture-of-Experts Models

Berkcan Kapusuzoglu; Connor Pryor; Sangwoo Cho; Supriyo Chakraborty; Shi-Xiong Zhang; Sambit Sahu; Milind Naphade

cs.LGSep 3, 2026

When Load-Balancing Goes Too Far: Expert Pruning in Over-Dispersed Mixture-of-Experts Models

Authors: Berkcan Kapusuzoglu, Connor Pryor, Sangwoo Cho, Supriyo Chakraborty, Shi-Xiong Zhang, Sambit Sahu, Milind Naphade

Organizations: Capital One, AI Foundations, McLean, VA 22102, USA

Abstract

Expert pruning reduces the memory and serving cost of Mixture-of-Experts (MoE) models by removing low-importance experts identified by the router, assuming router probabilities provide a reliable importance signal. We observe that this assumption breaks down under over-dispersed routing, a regime associated with aggressive load-balancing during training, in which tokens are distributed nearly uniformly across experts and importance signals collapse. In this regime, perplexity does not predict downstream task accuracy: on gpt-oss-20B, the lowest-perplexity pruning configuration yields the worst mathematical reasoning, while the highest-perplexity configuration preserves it. This does not occur under standard routing (e.g., Mixtral-8x7B-Instruct), where perplexity and accuracy degrade together. Pruning under over-dispersed routing also exposes a capability trade-off in which no single scoring metric dominates: activation-aware scoring preserves mathematical reasoning but severely degrades knowledge-intensive science (an 18-point gap on GPQA), whereas frequency-based scoring exhibits the reverse. We propose Minimax Expert Score Allocation (MESA), a domain-aware method that iteratively boosts importance scores for experts serving whichever domain is currently worst-affected, minimizing worst-case domain degradation rather than average accuracy. At 25% expert pruning MESA achieves the smallest worst-case degradation across domains, outperforming activation-aware baselines on 7 of 11 benchmarks at a correspondingly reduced memory footprint, and it generalizes to gpt-oss-120B, Gemma-4-26B-A4B, and OLMoE-1B-7B. Our results indicate that over-dispersed routing is a qualitatively distinct pruning regime in which standard assumptions fail, and that recognizing it is a prerequisite for principled expert pruning of load-balanced MoE models.

Figures & tables

Figure 1: Per-layer routing entropy (normalized to % of maximum) for gpt-oss-20b and Mixtral-8x7B-Instruct. gpt-oss maintains

>

82% across all layers (over-dispersed), while Mixtral drops to 64–68% in later layers as specialization emerges. This gap explains why global importance scores are informative for Mixtral but noisy for gpt-oss.

Method	WikiText-2 PPL ( $\downarrow$ )	GSM8K (%) ( $\uparrow$ )
Baseline (unpruned)	126.02	88.86
MESA (ours)	107.14	92.87
REAP	145.04	89.31
RM-Uniform	83.29	83.24
Random	40.95	78.54

Table 1: WikiText-2 PPL vs. GSM8K accuracy across pruning configurations at

r=0.25

. PPL and task accuracy are uncorrelated in over-dispersed models: random pruning achieves the lowest PPL (40.95) but near-worst task accuracy, while MESA improves GSM8K above the unpruned baseline despite moderate PPL.

Figure 2: PPL vs task accuracy across three benchmarks at

r=0.25

. Each point is one pruning method. The expected negative correlation (lower PPL

=

better accuracy) does not hold:

\rho

is positive on GSM8K and MMLU and near zero on GPQA, confirming PPL is uninformative or misleading for method comparison in over-dispersed models.

	Math Reasoning			Knowledge & Commonsense				Multi-domain & Instruction
Method	AIME’25	GSM8K	GPQA	ARC-C	ARC-E	OBQA	PIQA	MMLU	MMLU-Pro	WinoGr.	IFEval
Baseline	76.67 $\pm$ 3.27	88.86 $\pm$ 0.38	64.47 $\pm$ 1.46	95.30 $\pm$ 0.27	98.32 $\pm$ 0.15	92.99 $\pm$ 0.45	83.34 $\pm$ 0.36	84.01 $\pm$ 0.10	72.51 $\pm$ 0.22	76.78 $\pm$ 1.04	91.67 $\pm$ 0.67
MESA (ours)	63.33 $\pm$ 6.06	92.87 $\pm$ 0.45	65.48 $\pm$ 4.2	92.66 $\pm$ 0.8	97.60 $\pm$ 0.3 ∗	90.38 $\pm$ 1.2	80.73 $\pm$ 1.5	79.51 $\pm$ 0.6	67.25 $\pm$ 0.9	69.59 $\pm$ 2.1	89.92 $\pm$ 1.1
REAP	73.33 $\pm$ 9.7	89.31 $\pm$ 1.8	47.72 $\pm$ 5.6	91.63 $\pm$ 1.0	95.03 $\pm$ 0.7	92.18 $\pm$ 1.0	74.90 $\pm$ 2.0	77.36 $\pm$ 0.8	64.27 $\pm$ 1.2	78.44 $\pm$ 2.4	90.88 $\pm$ 0.9
RM-Uni.	46.67 $\pm$ 8.5	83.24 $\pm$ 1.5	56.85 $\pm$ 5.1	94.36 $\pm$ 0.6	97.56 $\pm$ 0.4	92.18 $\pm$ 0.9	74.85 $\pm$ 1.8	80.31 $\pm$ 0.5	66.32 $\pm$ 0.8	69.27 $\pm$ 2.3	86.07 $\pm$ 1.4
Random	33.33 $\pm$ 7.2	78.54 $\pm$ 2.3	49.75 $\pm$ 4.8	90.01 $\pm$ 1.2	96.46 $\pm$ 0.5	86.97 $\pm$ 1.5	78.61 $\pm$ 1.6	74.76 $\pm$ 0.9	56.06 $\pm$ 1.3	72.91 $\pm$ 2.8	89.68 $\pm$ 1.2

Table 2: Task accuracy (%) at

r=0.25

(25% expert pruning,

32\to 24

experts per layer), 5-seed evaluation with 95% CIs. Best pruned result per task in bold . MESA achieves the most balanced profile, with the best or near-best accuracy on the majority of benchmarks and the smallest worst-case degradation.

Benchmark	MESA (ours)	REAP	EvoESAP	RM-Uniform	Random
AIME’25	$-13.34$	$-3.34$	$-20.00$	$-30.00$	$-43.34$
GSM8K	$+4.01$	$+0.45$	$-10.54$	$-5.62$	$-10.32$
ARC-C	$-2.64$	$-3.67$	$-0.34$	$-0.94$	$-5.29$
ARC-E	$-0.72$	$-3.29$	$-0.68$	$-0.76$	$-1.86$
GPQA Diamond	$+1.01$	$-16.75$	$-20.31$	$-7.62$	$-14.72$
OpenBookQA	$-2.61$	$-0.81$	$-3.81$	$-0.81$	$-6.02$

Table 3: Per-benchmark accuracy change (

\Delta

, percentage points) relative to the unpruned baseline, gpt-oss-20B at

r{=}0.25

, single-seed evaluation across all 11 benchmarks. Best (least-negative or most-positive)

\Delta

per row in bold ; ties bolded.

Method	WikiText-2 PPL
Baseline (8E)	4.78
REAP (6E)	6.00
Random (6E)	8.31
RM-Uniform (6E)	14.35
MESA (6E)	16.43

Table 4: Cross-architecture validation on Mixtral-8x7B-Instruct (

\lambda_{\text{aux}}\approx 0.001

r=0.25

8\to 6

experts). Unlike gpt-oss, Mixtral shows PPL increases with pruning (standard behavior). REAP achieves the best PPL (6.00), while MESA, designed for over-dispersed routing, produces the worst (16.43).

Method	ARC-C	ARC-E	GPQA	GSM8K	MMLU	IFEval	PIQA
Baseline	85.29 $\pm$ 0.24	94.51 $\pm$ 0.15	31.07 $\pm$ 1.21	31.42 $\pm$ 1.25	68.54 $\pm$ 0.16	66.58 $\pm$ 1.34	85.60 $\pm$ 0.34
REAP	82.19 $\pm$ 0.30	92.29 $\pm$ 0.15	28.93 $\pm$ 2.61	35.00 $\pm$ 0.97	64.29 $\pm$ 0.19	65.23 $\pm$ 0.73	83.58 $\pm$ 0.26
EvoESAP	73.63 $\pm$ 0.32	86.14 $\pm$ 0.30	31.68 $\pm$ 1.59	40.56 $\pm$ 0.66	56.65 $\pm$ 0.27	65.62 $\pm$ 0.23	77.21 $\pm$ 0.38
Random	70.13 $\pm$ 0.50	80.61 $\pm$ 0.29	19.70 $\pm$ 1.36	20.71 $\pm$ 0.84	52.56 $\pm$ 0.11	57.98 $\pm$ 1.25	74.95 $\pm$ 0.46
MESA	51.97 $\pm$ 1.20	67.15 $\pm$ 1.13	22.34 $\pm$ 5.74	3.49 $\pm$ 0.48	38.49 $\pm$ 0.34	49.44 $\pm$ 1.51	58.42 $\pm$ 0.71
RM-Uniform	49.77 $\pm$ 1.05	60.23 $\pm$ 0.60	37.66 $\pm$ 0.22	2.96 $\pm$ 0.38	24.60 $\pm$ 2.36	45.95 $\pm$ 1.68	57.04 $\pm$ 1.14

Table 5: Cross-architecture task accuracy (%) on Mixtral-8x7B-Instruct at

r=0.25

(

8\to 6

experts), 5-seed means

\pm

std. Best pruned result per task in bold . REAP retains 98% of baseline average accuracy while MESA retains 63%.

Task (cloze acc_norm)	Unpruned	MESA (prefill)	MESA (gen)	REAP (gen)	RM (gen)
OpenBookQA	45.2	39.8	35.8	38.4	35.8
PIQA	80.6	77.6	74.2	71.9	72.2
WinoGrande	72.0	67.7	59.0	67.6	59.0
ARC-Challenge	55.2	50.0	44.0	40.6	45.8
ARC-Easy	82.3	77.4	72.2	64.0	73.2
HellaSwag	76.0	69.2	58.5	64.9	57.7

Table 6: Generality to a third, independently-trained over-dispersed MoE family: OLMoE-1B-7B-0924 (a fully-open base model; 16 layers, 64 experts, top-8; native routing entropy

\bar{H}=0.925\cdot\log_{2}E

). Loglikelihood/cloze accuracy (%) at

r=0.25

(25% expert pruning,

64\to 48

experts per layer), single seed. Best pruned result per task in bold . On these six commonsense tasks (cloze/acc_norm), MESA calibrated in the matching prefill regime leads every pruned method and beats REAP on all six, retaining 92.8% of unpruned commonsense accuracy.

Task	Unpruned	MESA $r{=}0.25$	MESA $r{=}0.375$	MESA $r{=}0.50$
OpenBookQA	45.2	39.8	35.6	31.2
PIQA	80.6	77.6	72.4	68.1
WinoGrande	72.0	67.7	58.6	53.8
ARC-Challenge	55.2	50.0	43.5	34.5
ARC-Easy	82.3	77.4	73.4	53.7
HellaSwag	76.0	69.2	59.6	46.2

Table 7: OLMoE-1B-7B prefill-MESA ratio sweep (single seed): commonsense accuracy degrades gracefully with no cliff. Eval-matched (prefill) calibration buys

\sim

one extra ratio step of headroom: prefill MESA at

r=0.375

(57.2) matches every gen-route method at

r=0.25

(

57.3

–

57.9

; Table 6 ).

	Math Reasoning			Knowledge & Commonsense				Multi-domain & Instruction
Method	AIME’25	GSM8K	GPQA	ARC-C	ARC-E	OBQA	PIQA	MMLU	MMLU-Pro	WinoGr.	IFEval
Unpruned	75.56	96.41	71.21	96.62	98.78	94.47	90.82	86.58	82.05	80.27	89.81
MESA (ours)	72.22	96.21	67.85	95.53	98.30	92.13	88.03	83.73	79.40	77.56	88.98
REAP	35.56	95.60	53.54	95.82	98.22	90.67	86.85	78.94	67.90	76.95	88.70
RM-Uni.	50.00	95.12	46.63	91.52	96.55	88.13	81.27	78.26	69.56	73.24	86.67

Table 8: Task accuracy (%) on Gemma-4-26B-A4B-it at

r=0.25

(128 routed experts, top-8;

128\to 96

experts per layer), temperature

0.6

, single measurement per cell. Best pruned result per task in bold . MESA matches or exceeds both REAP and RM-Uniform on every one of the 11 benchmarks, with the largest gap on GPQA-Diamond, the hardest task in the suite (

+14.3

pp over REAP,

+21.2

pp over RM-Uniform ).

Method	$\Delta_{\text{R}}$ (%)	$\Delta_{\text{K}}$ (%)	Character
MESA (ours)	$-4.67$	$-1.68$	Balanced
REAP	$-1.45$	$-3.48$	Reasoning-preserving
EvoESAP †	$-15.27$	$-0.51$	Knowledge-preserving
RM-Uniform	$-17.81$	$-0.85$	Knowledge-preserving
Random	$-26.83$	$-3.58$	Severe collapse

Table 9: Reasoning/Knowledge degradation at

r=0.25

(single-seed results).

\Delta_{\text{R}}

: mean accuracy change on AIME’25 and GSM8K;

\Delta_{\text{K}}

: mean change on ARC-C and ARC-E. Each method occupies a distinct position on the capability trade-off; MESA achieves the most balanced trade-off with a net reasoning improvement on GSM8K offsetting its AIME’25 drop.

Figure 3: Capability trade-off: reasoning vs knowledge preservation at

r=0.25

. Each method occupies a distinct position; MESA is closest to the origin (balanced), while REAP favors reasoning preservation.

Method	Scoring compute	Allocation compute	Extra fwd. passes beyond harvest
MESA (ours)	Static per-domain $L\times E$ router-mass matrix, already produced by the calibration harvest	CPU-only iterative minimax reweighting over the $L\times E$ matrix ( $\sim$ 15–20 steps)	0
REAP [ 12 ]	Per-expert activation-norm scoring requires a dedicated forward pass over its calibration set (24,576 samples at 16,384 tokens)	Global budget allocation over per-expert activation-norm scores	Full calibration-set forward pass (activation-norm scoring)
RM	Router-mass matrix only, identical byproduct of the same harvest MESA consumes	Uniform per-layer threshold on aggregate router mass	0

Table 10: Qualitative compute characterization of expert-scoring and allocation for MESA (ours) vs. REAP and Router-Mass (RM). MESA’s allocation step (Algorithm 1 ) operates entirely on a small, pre-computed

L\times E

router-mass matrix (e.g.,

24\times 32

for gpt-oss-20B) rather than requiring additional model forward passes, making it structurally cheaper than activation-based scoring methods such as REAP. No wall-clock timings are claimed here; see § 5.2 for the verified

\sim

\times

calibration-sample-count reduction relative to REAP.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

(a) Samples per domain (fixed $\ell=2048$ )
Samples/domain	Jaccard $J$	$\Delta$ experts	Character
5	0.655	40	Knowledge-biased
10	0.829	18	Moderate
20	1.000	0	✓Reference
50	0.979	2	✓Near-reference
100	0.901	10	Domain-fair-biased

Appendix

Table 11: Calibration sensitivity of MESA expert selection. Jaccard similarity measures agreement with the reference configuration (

s{=}20

\ell{=}2048

) on which experts to prune (

24\times 8=192

binary decisions at

r{=}0.25

). Different calibration budgets explore different regions of the capability Pareto frontier: lower calibration favors knowledge preservation while higher calibration emphasizes domain fairness.

(a) Iterations $T$ (fixed $\eta_{0}{=}0.15$ )
$T$	Worst-domain loss	Jaccard vs. $T{=}80$	Best step
1	0.0524	76.1%	0
3	0.0509	81.1%	2
5	0.0503	86.4%	4
10	0.0492	92.0%	8
15	0.0491	95.9%	13

Appendix

Table 12: MESA hyperparameter sensitivity at

r{=}0.25

. (a) Increasing iterations

T

monotonically improves worst-domain loss and selection agreement (Jaccard vs.

T{=}80

reference). Convergence is effectively reached by

T{=}15

. (b) The boost rate

\eta_{0}

controls convergence speed vs. stability; values in

[0.05,0.25]

achieve near-identical solutions with low oscillation.

Panel of Figure 4 — Figure 4: MESA convergence for different pruning ratios $r$ . The algorithm converges within 6–20 iterations for $r\leq 0.375$ . Higher ratios produce higher loss plateaus but similar convergence dynamics, suggesting the difficulty is in the objective landscape, not the optimizer.

Figure 5: Per-domain routing mass across all 24 layers and 32 experts in gpt-oss-20b. Each cell shows the fraction of routing probability mass that domain $d$ assigns to expert $e$ at layer $l$ . The narrow colorbar range (0.028–0.038 vs. the uniform expectation of $1/32\approx 0.031$ ) confirms over-dispersion: no single expert dominates for any domain. However, subtle per-domain structure is visible; certain experts receive slightly elevated mass for specific domains (e.g., experts 8–12 in layers 16–20 for Code). These weak but consistent signals are exactly what MESA’s domain-conditional scoring exploits: individually negligible, they become collectively meaningful when the minimax objective accumulates evidence across all calibration tokens.

Figure 6: Jaccard overlap between domains’ top-8 most-routed experts per layer, averaged across all 24 layers. High off-diagonal values (0.41–0.47) confirm that domains share nearly half their preferred experts due to over-dispersed routing. The slight asymmetries (e.g., Math–Science 0.47 vs. Code–Instruction 0.41) reflect residual domain structure that domain-fair scoring can exploit.

Figure 7: Per-task accuracy change from baseline at $r=0.25$ for all evaluated pruning methods. The heatmap reveals structured degradation patterns: reasoning tasks (left columns) show large variance across methods (AIME25 spans $-3.3$ to $-43.3$ %), while knowledge tasks (right columns) show uniformly mild degradation ( $<5$ % for most methods). MESA achieves the lightest overall shading (smallest absolute degradation across all tasks), while REAP shows a distinctive pattern of severe GPQA loss ( $-16.8$ %) alongside strong reasoning preservation.

Figure 8: Routing entropy determines importance signal quality. As routing entropy increases toward uniformity, Kendall-

\tau

between global importance ranking and worst-domain ranking decreases sharply. The 85% band (dashed) is a practical diagnostic, not a sharp physical boundary, and separates reliable (Mixtral, 74.3%) from degraded (gpt-oss, 90.8%) regimes. Simulated on 24-layer, 32-expert matrices with 6 calibration domains; real data points marked with stars.

Panel of Figure 9 — Figure 9: Knowledge degradation on GPQA-Diamond. Same model (gpt-oss-20b), same graduate-level electrostatics problem. Left: the unpruned model correctly identifies that system energy is half the interaction energy for a grounded conductor. Right: REAP-pruned model sets up the method of images identically but omits the factor of $1/2$ , a specific physical concept lost during pruning. This pattern, correct technique but missing conceptual knowledge, explains REAP’s 17.8% GPQA gap relative to MESA: the pruned model retains procedural skills but loses domain-specific physical reasoning stored in the removed experts.

Task	Unpruned	MESA (ours)	REAP	RM-Uni.
AIME’25	86.67 $\pm$ 3.33	90.00 $\pm$ 3.33	76.67 $\pm$ 4.71	70.00 $\pm$ 5.44
GSM8K	96.51 $\pm$ 0.38	95.98 $\pm$ 0.45	93.18 $\pm$ 0.62	89.54 $\pm$ 0.88
GPQA-Diamond	74.24 $\pm$ 1.21	65.66 $\pm$ 1.82	55.56 $\pm$ 1.95	48.99 $\pm$ 2.15
ARC-Challenge	96.84 $\pm$ 0.42	95.56 $\pm$ 0.51	92.40 $\pm$ 0.68	88.15 $\pm$ 0.82
ARC-Easy	98.82 $\pm$ 0.25	98.23 $\pm$ 0.31	96.50 $\pm$ 0.45	93.80 $\pm$ 0.60

Appendix

Table 13: Gemma-4-26B-A4B-it

r=0.25

head-to-head at temperature

0

(deterministic decoding), mean

\pm

std over 3 seeds, on the 5 tasks with temperature-

0

replication data. Best pruned result per task in bold ; non-overlapping intervals on GPQA-Diamond. This is an independent replication of the main-text temperature-

0.6

result (Table 8 ) under deterministic decoding, disclosed alongside it rather than in place of it. Additional benchmarks (MMLU, MMLU-Pro, IFEval, PIQA, OpenBookQA, WinoGrande) are pending temperature-

0

replication.

	Math Reasoning			Knowledge & Commonsense				Multi-domain & Instruction
Method	AIME’25	GSM8K	GPQA	ARC-C	ARC-E	OBQA	PIQA	MMLU	MMLU-Pro	WinoGr.	IFEval
Unpruned	74.00	95.63	70.36	96.38	98.67	93.91	84.93	87.76	77.78	81.03	86.50
MESA (ours)	72.22	95.58	58.46	92.38	95.88	93.56	77.95	83.01	64.20	77.27	87.15
REAP	48.89	96.26	51.85	95.14	98.18	93.33	77.93	82.05	68.96	81.16	84.50
RM-Uni.	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00

Appendix

Table 14: Task accuracy (%) on gpt-oss-120B at

r=0.5

(aggressive expert pruning,

128\to 64

experts per layer), single measurement per cell. Best pruned result per task in bold . RM-Uniform collapses to

0.00

on every task at this ratio, a genuine failure of the method at

r=0.5

rather than a missing measurement. On the worst domain (GPQA-Diamond, the hardest task in the suite), MESA leads REAP by

6.6

pp; MESA also leads on AIME, IFEval, MMLU, OpenBookQA, and PIQA. REAP edges ahead on GSM8K, ARC-Challenge, ARC-Easy, MMLU-Pro, and WinoGrande. We report this honestly as a mixed outcome at this aggressive ratio: MESA’s worst-domain protection holds, but it does not dominate every benchmark under such heavy pruning.

Mixture-of-Experts Pruning

Mixture-of-Experts pruning removes or trims experts to reduce model memory and computation while preserving useful capabilities.

See all 59 papers

Fine-Grained Units, Flat Decisions: Hierarchical Micro-Expert Pruning for Mixture-of-Experts Models
Oct 19, 2026Zukang Xu, Xinpei Gao, Xing Hu +2Channel Pruning Mixture-of-Experts Pruning
Houmo AI · University of Science and Technology of China · Southeast University
How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principle
Jun 14, 2026Zongfang Liu, Jinghui Zhang, Zijian Ma +2Mixture-of-Experts Pruning LLM Pruning
1Zhejiang University · 2Westlake University · 3Mohamed bin Zayed University of Artificial Intelligence +1
SPARE: Sparsity Allocation via Routing Entropy for Efficient MoEs
Oct 21, 2026Yifan Yang, Hanling Yi, Dongquan Yang +4Mixture-of-Experts Pruning Sparse Mixture-of-Experts
Intellifusion Inc.

Oct 19, 2026J/K move · Enter open · S save

Fine-Grained Units, Flat Decisions: Hierarchical Micro-Expert Pruning for Mixture-of-Experts Models

Zukang Xu, Xinpei Gao, Xing Hu, Zhe Jiang+1

Houmo AI · University of Science and Technology of China · Southeast University

Channel Pruning Mixture-of-Experts Pruning Mixture-of-Experts Models Structured Pruning Efficient Language Model Inference

Open paper AskPDF arXiv

Micro-expert pruning reduces the capacity of mixture-of-experts (MoE) models by removing coupled channels within routed experts. However, finer pruning granularity alone does not guarantee the preservation of downstream capabilities, as similar language-modeling perplexity can accompany substantially different task performance. We revisit micro-expert mask construction as two distinct decisions: allocating retained capacity across experts and selecting channels within that capacity. We propose UC-RAMP, a two-stage post-training pruning framework that explicitly separates these decisions. CAPE uses cross-domain calibration evidence to redistribute expert widths under an exact layer-wise budget, while RAMP refines channel composition at fixed widths through one-out, one-in exchanges guided by a channel-interaction surrogate. Across three MoE models at 25% and 50% routed-expert channel pruning, UC-RAMP achieves the highest ten-task mean accuracy among evaluated pruning methods in five of six settings. At 50% pruning, it outperforms the strongest evaluated baseline by 8.19, 3.96, and 6.14 percentage points across the three models, respectively. In a single-A100 deployment at 25% pruning, UC-RAMP reduces measured time to first token by 19.7--23.9% and time per output token by 20.2--22.3% relative to the unpruned model. The code is available at https://github.com/xzktx003/UC_RAMP.

Abstract

Mixture-of-Experts Pruning

Explore similar work

Fine-Grained Units, Flat Decisions: Hierarchical Micro-Expert Pruning for Mixture-of-Experts Models

How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principle

SPARE: Sparsity Allocation via Routing Entropy for Efficient MoEs