Prune, Don't Merge: Expert Compression of Mixture-of-Experts LLMs on Multilingual Production Tasks
Organizations: Hostinger Research
Abstract
Mixture-of-Experts language models are cheap to run per token but expensive to hold in GPU memory, which drives the cost of serving them in production. One-shot expert compression shrinks them after a single calibration pass. Its two current state-of-the-art methods are REAP for expert pruning, which deletes the least salient experts, and REAM for expert merging, which merges them into the most salient ones. Both have been evaluated only on academic benchmarks. We compare the two on five multilingual production tasks of a web-hosting company, from classification to the generation of marketing HTML email templates, with two 30B-parameter models (Qwen3-30B-A3B-Instruct-2507 and the hybrid Mamba-attention Nemotron-3.5-Lightning) at 25% and 50% of experts removed and the same calibration data. Pruning wins: on the same rows, REAP is significantly better in 16 of 20 comparisons and REAM in 2. At 25%, REAP keeps 93.6% of Qwen3-30B's generative quality and REAM 62.5%, although on benchmarks the two are within a point for the same model. The damage concentrates in the output contract rather than the content: 89.8-95.2% of each significant loss on email templates comes from hard-gate failures such as invalid JSON, broken mobile layouts and runaway generation, and most failed non-English domain descriptions copy the prompt's English opening phrase. Cross-perplexity and logit divergence, two diagnostics from the REAP paper, also track production quality, as does the rate of repetition loops; n-gram diversity does not. We close with practical guidance for compressing MoE models for production.
Figures & tables
| Task | Type | Output | Decoding | Lang. | Primary metric | Tokens |
|---|---|---|---|---|---|---|
| Website cues | classification | JSON, 2 booleans | greedy | 19 | exact match | 23 |
| Brand names | span extraction | JSON (schema) | greedy | 24 | exact match | 24 |
| Categorisation | open-label categorisation | JSON (schema) | greedy | 33 | judge pass rate | 11 |
| Domain descriptions | short generation | plain text | sampled | 17 | judge pass rate | 20 |
| Email templates | long structured generation | JSON with HTML | sampled | 35 | rubric score (0–100) | 2,724 |
| Base | REAP 25% | REAP 50% | REAM 25% | REAM 50% | |
| Qwen3-30B | |||||
| Website cues | 98.8 | 0.2 | 1.0 | 0.5 | 1.7 |
| Brand names | 64.8 | +3.9 | +4.2 | 0.3 | +5.9 |
| Categorisation | 81.7 | 1.0 | 6.7 | 5.0 | 10.4 |
| Domain descriptions | 93.5 | 13.0 | 22.2 | 31.1 | 30.1 |
| Email templates | 70.2 | +0.9 | 25.0 | 29.3 | 45.1 |
| Score (0–100) | Passes all | Mean score if | Loss from | Runaway | ||
|---|---|---|---|---|---|---|
| Model | production | relaxed | gates (%) | all gates pass | gates (%) | answers |
| Qwen3-30B | 26.1 | 70.2 | 88.7 | 79.2 | 6 | |
| REAP 25% | 51.3 | 71.1 | 89.9 | 79.1 | 14 | |
| REAP 50% | 37.7 | 45.3 | 59.2 | 76.5 | 92.1 | 102 |
| REAM 25% | 0.4 | 40.9 | 54.5 | 75.0 | 89.8 | 61 |
| REAM 50% | 1.9 | 25.1 | 34.4 | 73.1 | 91.7 | 215 |
| Pass (%) | Non-English rows (%) | Pass non-En. | ||||||
| non- | opener | wrong | of which | at base | ||||
| Model | En. | En. | only | lang. | En. | undet. | neither | drift |
| Qwen3-30B | 98.2 | 92.6 | 3.3 | 0.2 | 0 | 1.7 | 95.8 | – |
| REAP 25% | 99.4 | 76.6 | 11.5 | 0.2 | 0 | 1.8 | 86.3 | 82.6 |
| REAP 50% | 96.8 | 66.0 | 17.0 | 6.7 | 97 | 2.2 | 86.7 | 82.5 |
| REAM 25% | 97.7 | 55.1 | 39.6 | 2.4 | 54 | 1.8 | 94.4 | 89.7 |
| Claim (source) | Production evidence (ours) | Verdict |
|---|---|---|
| “Expert pruning is a superior strategy for generative tasks” (REAP, abstract); at 25% on Qwen3-30B-A3B pruning and M-SMoE are within about a point on generation (§5) | REAP better in 16 of 20 comparisons; Qwen3-30B generative retention at 25% 93.6% vs 62.5% | Confirmed and larger, also against REAM |
| REAM beats REAP on generation at 25%: GEN 69.8 vs 68.6, best of ten mixtures (REAM §5.1) | REAP ahead on 4 of 5 tasks at 25% on Qwen3-30B, by up to +30.2 points | Not seen in production |
| At 50%, REAP leads on GEN (62.8 vs 60.7), REAM on multiple-choice (REAM App. A.3) | REAP ahead on three tasks; REAM ahead only on brand names (detection) | Partly confirmed |
| Merging collapses expert functions (“functional subspace collapse”, REAP §3.3) | Not measured on the experts; REAM’s outputs diverge further from the base model’s (next row) | Consistent; not tested directly |
| Merged outputs have higher cross-perplexity and diverge faster with position (REAP §5) | Cross-perplexity higher for REAM in all 20 cells; JSD in 20; position trend not testable on our short and HTML answers | Level confirmed |
| Merged outputs are “more repetitive” (REAP §5) | REAM has 1.3–5.2 times as many runaway email-template answers; distinct-2 without loops does not differ | Confirmed via loops |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Task, model | Base | Variant | Score | Change [95% CI] | Retention (%) [95% CI] | (Holm) |
|---|---|---|---|---|---|---|
| Website cues, Qwen3-30B | 98.8 | REAP 25% | 98.5 | 0.2 [ 0.6, 0.0] | 99.7 [99.4, 100.1] | 1.000 |
| REAP 50% | 97.8 | 1.0 [ 1.5, 0.4] | 99.0 [98.5, 99.6] | 0.009 | ||
| REAM 25% | 98.2 | 0.5 [ 1.0, 0.0] | 99.5 [99.0, 100.0] | 0.529 | ||
| REAM 50% | 97.1 | 1.7 [ 2.4, 1.0] | 98.3 [97.6, 99.0] | 0.001 | ||
| Website cues, Nemotron | 96.0 | REAP 25% | 96.5 | +0.5 [0.0, 1.1] | 100.6 [100.1, 101.1] | 0.491 |
| REAP 50% | 94.7 | 1.3 [ 2.1, 0.5] | 98.6 [97.8, 99.5] | 0.020 |
| Qwen3-30B | Nemotron | |||
|---|---|---|---|---|
| Task | 25% | 50% | 25% | 50% |
| Website cues | +0.2 [ 0.2, 0.8] | +0.7 [0.0, 1.4] | +1.0 [0.3, 1.7] | +3.1 [2.2, 4.1] |
| Brand names | +4.2 [3.0, 5.5] | 1.7 [ 2.9, 0.4] † | +6.0 [4.3, 7.6] | 5.5 [ 7.2, 3.6] † |
| Categorisation | +4.0 [2.5, 5.5] | +3.7 [1.9, 5.5] | +5.3 [3.4, 7.2] | +7.0 [5.0, 9.0] |
| Domain descr. | +18.1 [15.9, 20.3] | +7.8 [5.7, 9.9] | +12.9 [10.3, 15.6] | +5.1 [2.7, 7.3] |
| Email templates | +30.2 [28.4, 32.0] | +20.1 [18.0, 22.2] | +8.5 [7.2, 9.9] | +2.6 [2.0, 3.2] |
| Exact | Detec- | Precision (%) | Change in exact (points) | ||||
|---|---|---|---|---|---|---|---|
| Model | (%) | tion (%) | 95% CI | total | detect. | precis. | |
| Qwen3-30B | 64.8 | 87.2 | 74.3 | [72.3, 76.3] | – | – | – |
| REAP 25% | 68.7 | 91.0 | 75.5 | [73.5, 77.4] | +3.9 | +2.8 | +1.1 |
| REAP 50% | 69.0 | 91.5 | 75.5 | [73.5, 77.4] | +4.2 | +3.2 | +1.0 |
| REAM 25% | 64.5 | 88.2 | 73.1 | [71.1, 75.2] | 0.3 | +0.7 | 1.0 |
| REAM 50% | 70.7 | 95.2 | 74.3 | [72.2, 76.2] | +5.9 | +5.9 | 0.1 |
| Relaxed rules (as scored) | Production rules | ||||||
| Model | parseable | HTML | complete | unsubscribe | mobile | complete | unsubscribe |
| Qwen3-30B | 1.1 | 0.0 | 0.1 | 0.1 | 10.0 | 61.5 | 0.2 |
| REAP 25% | 1.1 | 0.0 | 0.1 | 0.1 | 8.8 | 28.0 | 0.1 |
| REAP 50% | 5.4 | 0.1 | 0.4 | 0.1 | 35.1 | 14.3 | 0.2 |
| REAM 25% | 5.1 | 0.0 | 0.1 | 0.4 | 40.1 | 94.3 | 0.8 |
| REAM 50% | 12.2 | 0.0 | 2.6 | 0.4 | 51.7 | 82.5 | 0.5 |
| Production gates | Relaxed | |||||
|---|---|---|---|---|---|---|
| Model | REAP | REAM | diff. | 95% CI | Holm | diff. |
| Qwen3-30B 25% | 51.3 | 0.4 | +51.0 | [+49.3, +52.6] | 0.001 | +30.2 |
| Qwen3-30B 50% | 37.7 | 1.9 | +35.8 | [+34.0, +37.5] | 0.001 | +20.1 |
| Nemotron 25% | 12.1 | 2.9 | +9.2 | [+7.9, +10.4] | 0.001 | +8.5 |
| Nemotron 50% | 2.6 | 0.1 | +2.5 | [+1.9, +3.1] | 0.001 | +2.6 |
| Change from base | Gap: non-En. En. | |||
| Task, model | English | non-English | [95% CI] | Holm |
| Website cues | ||||
| Qwen3-30B REAP 25% | 0.1 | 0.4 | 0.3 [ 1.0, +0.3] | 1 |
| Qwen3-30B REAP 50% | 0.5 | 1.5 | 1.0 [ 2.1, +0.1] | 1 |
| Qwen3-30B REAM 25% | 0.0 | 1.1 | 1.1 [ 2.1, 0.1] | 0.78 |
| Qwen3-30B REAM 50% | 0.7 | 2.7 | 2.1 [ 3.5, 0.7] | 0.068 |
| Kendall | REAP vs REAM | ||||||
| Better | REAP | quality | same | ||||
| Diagnostic | when | Tasks | mean | better | agrees | order | |
| Cross-perplexity | lower | all | +0.70 | 9/10 | 20/20 | 18/20 | 18/20 |
| Completion JSD a | lower | all | +0.50 | 7/10 | 20/20 | 18/20 | 18/20 |
| Effective rank | higher | all | +0.44 | 6/10 | 15/20 | 13/15 | 13/20 |
| Distinct-2, no loops | higher | gen. | +0.15 | 3/4 | 1/8 | 1/1 | 1/8 |
| Cross-perplexity | Completion-logit JSD (bits) | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Model | Website | Brand | Categorisation | Domain | Website | Brand | Categorisation | Domain | ||
| Qwen3-30B | 1.000 | 1.042 | 1.038 | 1.19 | 1.048 | – | – | – | – | – |
| REAP 25% | 1.001 | 1.073 | 1.11 | 1.50 | 1.065 | 0.000 | 0.008 | 0.013 | 0.056 | 0.005 |
| REAP 50% | 1.005 | 1.15 | 1.32 | 4.00 | 1.12 | 0.000 | 0.024 | 0.034 | 0.168 | 0.018 |
| REAM 25% | 1.003 | 1.086 | 1.30 | 2.13 | 1.13 | 0.001 | 0.010 | 0.030 | 0.101 | 0.018 |
| REAM 50% | 1.009 | 1.16 | 1.57 | 5.90 | 1.20 | 0.001 | 0.029 | 0.049 | 0.215 | 0.029 |
| distinct-2 | Effective rank of pre-logit embeddings | ||||||||
| Domain descr. | Email templates | ||||||||
| Model | within | across | within | across | Website | Brand | Categorisation | Domain | |
| Qwen3-30B | 1.000 | 0.378 | 0.950 | 0.690 | 1,184 | 1,242 | 1,413 | 892 | 1,272 |
| REAP 25% | 1.000 | 0.359 | 0.950 | 0.698 | 1,165 | 1,230 | 1,396 | 876 | 1,257 |
| REAP 50% | 0.998 | 0.332 | 0.928 | 0.696 | 1,104 | 1,180 | 1,332 | 839 | 1,226 |
| REAM 25% | 1.000 | 0.372 | 0.934 | 0.702 | 1,154 | 1,215 | 1,379 | 873 | 1,241 |
| Tokens | Share of tokens (%) | English (%) | Sys. pr. | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Task | Rows | (k) | system | user | answer | rows | tokens | text | En. |
| Website cues | 1,103 | 592 | 0.2 | 94.8 | 5.0 | 77.9 | 74.6 | 74.6 | 1/1 |
| Brand names | 272 | 593 | 59.4 | 39.7 | 0.9 | 18.8 | 18.5 | 66.7 | 2/2 |
| Categorisation | 1,067 | 592 | 20.9 | 76.6 | 2.5 | 20.2 | 15.1 | 31.9 | 4/4 |
| Domain descr. | 3,174 | 592 | 64.3 | 24.3 | 11.4 | 71.0 | 70.0 | 88.8 | 2/2 |
| Email templates | 50 | 592 | 74.6 | 1.4 | 24.0 | 18.0 | 17.6 | 79.2 | 50/50 |
LLM Compression
Large language model compression reduces the memory, compute, and context costs of LLMs while aiming to preserve their capabilities.
See all 301 papers