Benchmarking Composable Compression Techniques in Mixture-of-Experts LLMs

Afsara Benazir; Chen Chen; Rongxiao Qu; Jiabo Huang; Jingtao Li; Lingjuan Lyu

cs.LGAug 22, 2026

Benchmarking Composable Compression Techniques in Mixture-of-Experts LLMs

Authors: Afsara Benazir, Chen Chen, Rongxiao Qu, Jiabo Huang, Jingtao Li, Lingjuan Lyu

Organizations: University of Virginia · Sony AI

Abstract

Mixture-of-Experts (MoE) LLMs scale model capacity efficiently through sparse activation, but their large expert parameter footprint, routing imbalance, and long-context KV-cache growth make deployment difficult on commodity hardware. Practical deployment often requires stacking multiple compression techniques: expert pruning removes redundant experts, weight quantization lowers model memory footprint, and KV-cache compression reduces long-context memory pressure. However, these techniques are typically evaluated in isolation, leaving open how they interact when applied together in realistic deployment pipelines. In this work, we present MoEXBench, a systematic benchmark for evaluating composable MoE compression as an end-to-end deployment workflow. MoEXBench studies 10 MoE models ranging from 30B to 235B total parameters across standard-attention, hybrid linear-attention, and sliding window attention architectures. It evaluates 20%-50% expert pruning rates, 1 to 16 bit weight-quantization schemes, and multiple KV-cache precision settings, applied both individually and in combination. MoEXBench introduces an eight-module evaluation suite that jointly measures composable-compression quality, workload and architecture robustness, pruning/quantization/KV cache sensitivity, and deployment efficiency on commodity hardware. Our results reveal non-trivial interactions among compression methods: composable compression cannot be predicted from standalone techniques, compression rate alone does not reliably predict quality loss or runtime gain, expert pruning is the dominant degradation source, and average quality can hide workload and architecture-specific failures. By releasing normalized module scores, compressed artifacts, and reproducible scripts, MoEXBench enables practical accuracy-memory-latency comparison across MoE families and hardware backends.

Figures & tables

Figure 1: Overview of MoE-XBench. Each radar reports normalized module scores, where higher is better. Performance, robustness, and reliability measure fixed-configuration quality, while pruning, quantization, and KV tolerance summarize controlled sensitivity sweeps around that configuration.

Table 1: Research questions and their corresponding MoE-XBench evaluation modules.

Configuration	Baseline FP16	Prune only	Quant only	Quant + KV	Prune + Quant	Prune + Quant + KV
Expert Pruning	✗	✓	✗	✗	✓	✓
Weight Quantization	✗	✗	✓	✓	✓	✓
KV Cache Compression	✗	✗	✗	✓	✗	✓

Table 2: Compression configurations in MoE-XBench; the last three are composable.

Method	Model	Model Size (Compression Rate)	Workload								Score ${}_{\text{perf}}$	Score ${}_{\text{robust}}$
Method	Model	Model Size (Compression Rate)	GSM8K	MMLU	MuSR	IFEval	HumanEval	BFCLv3	RULER	Avg.	Score ${}_{\text{perf}}$	Score ${}_{\text{robust}}$
Baseline (BF16)	Q-30B-A3B	61.10 GB (0.0%)	0.8360	0.8470	0.4290	0.8170	0.9390	0.8880	0.9441	0.8143	100.00	100.00
Baseline (BF16)	Q-35B-A3B	69.38 GB (0.0%)	0.8741	0.8657	0.4405	0.8207	0.9695	0.8662	0.9470	0.8262	100.00	100.00
Prune only	Q-30B-A3B	44.00 GB (28.0%)	0.8400	0.7420	0.4380	0.7650	0.8110	0.8560	0.9452	0.7710	96.96	88.87
Prune only	Q-35B-A3B	52.04 GB (25.0%)	0.8779	0.8467	0.4444	0.8022	0.9634	0.8554	0.9473	0.8196	96.96	88.87
Quant. only	Q-30B-A3B	18.63 GB (69.5%)	0.8330	0.8450	0.4250	0.8260	0.9270	0.8910	0.9422	0.8127	99.91	97.00

Table 3: Performance comparison across different compression configurations for Qwen3-30B-A3B-Instruct-2507 (Q-30B-A3B) and Qwen3.6-35B-A3B (Q-35B-A3B) on diverse workloads.

Standard MoE
Model	Baseline	Pruning	Quant only	Quant+KV	Prune+Quant	Prune+Quant+KV
Qwen3-30B-A3B-Instruct	7.36	9.46	7.41	7.49	9.55	9.54
Qwen3-Coder-30B-A3B-Instruct	9.54	12.88	9.74	9.71	13.11	13.11
Qwen3-235B-A22B-Instruct-2507	4.31	6.50	4.41	4.41	6.62	6.62
GLM-4.7-Flash	9.86	17.64	10.27	9.85	18.25	18.25
MiniMax-M2	6.79	13.16	7.09	7.13	10.41	10.41

Table 4: Cross-architecture reliability across compression config. PPL,

\downarrow

is better; score,

\uparrow

is better.

Configuration	Fp16/fp16	Q8/Q8	Q4/Q4	Q8/Turbo4	Q8/Turbo3	Turbo4/Turbo3	score kv
Baseline FP16	6.4935	6.4949	6.6130	6.5127	6.5311	6.7846	94.38
Quant + KV	6.5509	6.5492	6.6741	6.5614	6.5795	6.8459	94.37
Prune + KV	8.1727	8.1730	8.3441	8.2070	8.2417	8.6205	93.18
Prune + Quant + KV	8.2479	8.2469	8.4371	8.2856	8.3184	8.7095	93.00

Table 5: KV-cache sensitivity. Higher

\mathrm{score}_{kv}

indicates greater tolerance to KV-cache compression.

Figure 4: Inference efficiency. Peak memory, prefill throughput, and decode throughput on NVIDIA 1xH100 for Qwen3-30B-A3B-Instruct at varying context lengths, across the six compression configurations for Q4_K_M (top) and IQ1_M (bottom) with Q8 KV-cache under bs=1. Both higher compression rate and higher throughput are desirable although their relationship is non-monotonic.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Calibration	GSM8K	MMLU-R	MuSR	HumanEval	MBPP	Math-Hard	IFEval
Official mixture	0.883	0.811	0.431	0.713	0.148	0.539	0.797
C4 general	0.898	0.827	0.418	0.281	0.524	0.297	0.771
Code-only	0.891	0.796	0.427	0.665	0.516	0.557	0.782
Math-only	0.896	0.804	0.438	0.720	0.264	0.560	0.815

Appendix

Table 6: Calibration-dataset ablation for REAP on Qwen3.6-35B-A3B. All variants use the same pruning ratio

r=0.30

, seed, router renormalization, and GGUF Q4_K_M deployment path. Higher is better for all metrics.

Scoring setup	GSM8K	MMLU-R	MuSR
Unpruned Q4_K_M	0.8916	0.8625	0.4365
BF16 REAP	0.8825	0.8111	0.4312
Module quant.-aware REAP	0.8741	0.8107	0.4272
Packed-MoE quant.-aware REAP	0.8779	0.8131	0.4299

Appendix

Table 7: Quantization-aware REAP ablation on Qwen3.6-35B-A3B. All pruned variants use REAP with

r=0.30

and are evaluated after GGUF Q4_K_M quantization through the same deployment path. “Module” quantizes only standard linear modules during scoring, while “Packed MoE” also covers packed routed experts and router weights.

Expert comp. method	Config	GSM8K	MMLU	MuSR	IFEval	HumanEval	BFCLv3	RULER	Avg.	Score_perf	Score_robust
—	Baseline BF16	0.8360	0.8470	0.4290	0.8170	0.9390	0.8880	0.9441	0.8143	100.00	100.00
REAP	Expert only	0.8400	0.7420	0.4380	0.7650	0.8110	0.8560	0.9452	0.7710	94.69	80.41
—	Q4_K_M only	0.8330	0.8450	0.4250	0.8260	0.9270	0.8910	0.9422	0.8127	99.81	97.43
—	Q4_K_M + Q8 KV	0.8378	0.8422	0.4246	0.8336	0.9268	0.8694	0.9435	0.8111	99.61	95.56
REAP	+ Q4_K_M	0.8230	0.7330	0.4310	0.7430	0.8170	0.8410	0.9424	0.7615	93.51	80.82
REAP	+ Q4_K_M + Q8 KV	0.8180	0.7320	0.4270	0.7600	0.8050	0.8420	0.9435	0.7611	93.46	80.41

Appendix

Table 8: Performance comparison of expert compression methods across configurations.

Pruning	Weights	KV cache	PPL $\pm$ stderr	$\Delta$ vs 0% bf16	$\Delta$ vs same-prune bf16
0%	bf16	f16	$6.8200$	–	–
0%	Q4_K_M	f16	$6.8198\pm 0.0444$	$-0.0002$	$-0.0002$
0%	Q4_K_M	q8_0	$6.8180\pm 0.0444$	$-0.0020$	$-0.0020$
20%	bf16	f16	$8.6695\pm 0.0592$	$+1.8495$	–
20%	Q4_K_M	f16	$8.7952\pm 0.0602$	$+1.9752$	$+0.1257$
20%	Q4_K_M	q8_0	$8.7998\pm 0.0602$	$+1.9798$	$+0.1303$

Appendix

Table 9: Pruning sensitivity under fixed quantization and KV-cache settings. For each pruning ratio, the bf16/f16 row serves as the same-prune baseline.

Method	Score (with IQ1_M)	Score (without IQ1_M)
Quant-only	56.12	87.45
Quant+KV (q8)	56.10	87.42
Prune+Quant	55.67	84.35
Prune+Quant+KV (q8)	55.62	84.32
Prune+Quant+KV (q4)	53.21	83.14

Appendix

Table 10: Quantization sensitivity score with and without IQ1_M.

Figure 7: Inference efficiency on Apple-M1-Max. Peak memory, prefill throughput, and decode throughput for Qwen3-30B-A3B-Instruct at context lengths $\{2\mathrm{K},8\mathrm{K},32\mathrm{K}\}$ , across the six compression configurations of Table 2 at two weight-quantization formats (Q4_K_M, IQ1_M) with Q8 KV-cache. Q8_0 is used as the baseline because BF16 caused memory thrashing for this model.

Model	Original HF ID	REAP-pruned HF ID
Qwen3-30B-A3B	Qwen/Qwen3-30B-A3B-Instruct-2507	SamsungSAILMontreal/Qwen3-30B-A3B-Instruct-2507-REAP
Qwen3-Coder-30B-A3B	Qwen/Qwen3-Coder-30B-A3B-Instruct	cerebras/Qwen3-Coder-REAP-25B-A3B
Qwen3-235B-A22B	Qwen/Qwen3-235B-A22B-Instruct-2507	SamsungSAILMontreal/Qwen3-235B-A22B-Instruct-2507-REAP
GLM-4.7-Flash	zai-org/GLM-4.7-Flash	cerebras/GLM-4.7-Flash-REAP-23B-A3B
MiniMax-M2	MiniMaxAI/MiniMax-M2	cerebras/MiniMax-M2-REAP-162B-A10B
Kimi-Linear-48B-A3B	moonshotai/Kimi-Linear-48B-A3B-Instruct	cerebras/Kimi-Linear-REAP-35B-A3B-Instruct

Appendix

Table 11: Original and REAP-pruned Hugging Face checkpoints.

Mixture-of-Experts Inference

Mixture-of-experts inference studies how to run sparsely activated expert models efficiently while preserving their quality.

See all 179 papers

OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit
Sep 9, 2026Dezhi Li, Lujun Li, Qiyuan Zhu +4Mixture-of-Experts Pruning Mixture-of-Experts Language Models
The Hong Kong University of Science and Technology
FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models
Jun 26, 2026Fan Mo, Yuxuan Han, Geng Zhang +2Mixture-of-Experts Pruning LLM Pruning
National University of Singapore
MoEZip: Routing-Aware KV Cache Compression for Sparse Mixture-of-Experts LLMs
Oct 21, 2026Minsang Kim, Seungjun BaekKV-Cache Compression Long-Context Language Model Inference
Korea University · SK Telecom

Sep 9, 2026J/K move · Enter open · S save

OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit

Dezhi Li, Lujun Li, Qiyuan Zhu, Hao Gu+3

The Hong Kong University of Science and Technology

Mixture-of-Experts Pruning Mixture-of-Experts Language Models LLM Compression LLM Inference Acceleration

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) models enable efficient scaling of large language models but face critical deployment challenges due to massive memory requirements. Existing pruning methods either incur prohibitive search costs or neglect the dynamic interdependencies between experts. To address these challenges, we present OMP-MoE, a novel training-free compression framework for reducing expert redundancy in MoE-based LLMs. Based on observations of expert contribution patterns, we reformulate the pruning problem as a sparse signal reconstruction task solved through Orthogonal Matching Pursuit. Specifically, our method first treats individual expert contributions as dictionary atoms and selects experts that greedily minimize reconstruction error with linear computational complexity. Then, we optimize cross-layer expert allocation through a water-filling strategy that accounts for both reconstruction quality and routing stability. Finally, we introduce OMP-MoE†, an adaptive inference mechanism that dynamically adjusts expert activation based on energy prediction. Comprehensive experiments on Qwen, DeepSeek-V2, GPT-OSS, and Mixtral MoE demonstrate consistent improvements over existing methods at 25-50% pruning ratios. For Qwen3-30B-A3B at 50% compression, we retain 93.3% of original performance, achieving 33

\times

faster search and 1.55

\times

inference speedup. Codes will be available after acceptance.

Abstract

Mixture-of-Experts Inference

Explore similar work

OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit

FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models

MoEZip: Routing-Aware KV Cache Compression for Sparse Mixture-of-Experts LLMs