Benchmarking Composable Compression Techniques in Mixture-of-Experts LLMs

Afsara Benazir; Chen Chen; Rongxiao Qu; Jiabo Huang; Jingtao Li; Lingjuan Lyu

cs.LGAug 22, 2026

Benchmarking Composable Compression Techniques in Mixture-of-Experts LLMs

Authors: Afsara Benazir, Chen Chen, Rongxiao Qu, Jiabo Huang, Jingtao Li, Lingjuan Lyu

Organizations: University of Virginia · Sony AI

Abstract

Mixture-of-Experts (MoE) LLMs scale model capacity efficiently through sparse activation, but their large expert parameter footprint, routing imbalance, and long-context KV-cache growth make deployment difficult on commodity hardware. Practical deployment often requires stacking multiple compression techniques: expert pruning removes redundant experts, weight quantization lowers model memory footprint, and KV-cache compression reduces long-context memory pressure. However, these techniques are typically evaluated in isolation, leaving open how they interact when applied together in realistic deployment pipelines. In this work, we present MoEXBench, a systematic benchmark for evaluating composable MoE compression as an end-to-end deployment workflow. MoEXBench studies 10 MoE models ranging from 30B to 235B total parameters across standard-attention, hybrid linear-attention, and sliding window attention architectures. It evaluates 20%-50% expert pruning rates, 1 to 16 bit weight-quantization schemes, and multiple KV-cache precision settings, applied both individually and in combination. MoEXBench introduces an eight-module evaluation suite that jointly measures composable-compression quality, workload and architecture robustness, pruning/quantization/KV cache sensitivity, and deployment efficiency on commodity hardware. Our results reveal non-trivial interactions among compression methods: composable compression cannot be predicted from standalone techniques, compression rate alone does not reliably predict quality loss or runtime gain, expert pruning is the dominant degradation source, and average quality can hide workload and architecture-specific failures. By releasing normalized module scores, compressed artifacts, and reproducible scripts, MoEXBench enables practical accuracy-memory-latency comparison across MoE families and hardware backends.

Figures & tables

Figure 1: Overview of MoE-XBench. Each radar reports normalized module scores, where higher is better. Performance, robustness, and reliability measure fixed-configuration quality, while pruning, quantization, and KV tolerance summarize controlled sensitivity sweeps around that configuration.

Table 1: Research questions and their corresponding MoE-XBench evaluation modules.

Configuration	Baseline FP16	Prune only	Quant only	Quant + KV	Prune + Quant	Prune + Quant + KV
Expert Pruning	✗	✓	✗	✗	✓	✓
Weight Quantization	✗	✗	✓	✓	✓	✓
KV Cache Compression	✗	✗	✗	✓	✗	✓

Table 2: Compression configurations in MoE-XBench; the last three are composable.

Method	Model	Model Size (Compression Rate)	Workload								Score ${}_{\text{perf}}$	Score ${}_{\text{robust}}$
Method	Model	Model Size (Compression Rate)	GSM8K	MMLU	MuSR	IFEval	HumanEval	BFCLv3	RULER	Avg.	Score ${}_{\text{perf}}$	Score ${}_{\text{robust}}$
Baseline (BF16)	Q-30B-A3B	61.10 GB (0.0%)	0.8360	0.8470	0.4290	0.8170	0.9390	0.8880	0.9441	0.8143	100.00	100.00
Baseline (BF16)	Q-35B-A3B	69.38 GB (0.0%)	0.8741	0.8657	0.4405	0.8207	0.9695	0.8662	0.9470	0.8262	100.00	100.00
Prune only	Q-30B-A3B	44.00 GB (28.0%)	0.8400	0.7420	0.4380	0.7650	0.8110	0.8560	0.9452	0.7710	96.96	88.87
Prune only	Q-35B-A3B	52.04 GB (25.0%)	0.8779	0.8467	0.4444	0.8022	0.9634	0.8554	0.9473	0.8196	96.96	88.87
Quant. only	Q-30B-A3B	18.63 GB (69.5%)	0.8330	0.8450	0.4250	0.8260	0.9270	0.8910	0.9422	0.8127	99.91	97.00

Table 3: Performance comparison across different compression configurations for Qwen3-30B-A3B-Instruct-2507 (Q-30B-A3B) and Qwen3.6-35B-A3B (Q-35B-A3B) on diverse workloads.

Standard MoE
Model	Baseline	Pruning	Quant only	Quant+KV	Prune+Quant	Prune+Quant+KV
Qwen3-30B-A3B-Instruct	7.36	9.46	7.41	7.49	9.55	9.54
Qwen3-Coder-30B-A3B-Instruct	9.54	12.88	9.74	9.71	13.11	13.11
Qwen3-235B-A22B-Instruct-2507	4.31	6.50	4.41	4.41	6.62	6.62
GLM-4.7-Flash	9.86	17.64	10.27	9.85	18.25	18.25
MiniMax-M2	6.79	13.16	7.09	7.13	10.41	10.41

Table 4: Cross-architecture reliability across compression config. PPL,

\downarrow

is better; score,

\uparrow

is better.

Configuration	Fp16/fp16	Q8/Q8	Q4/Q4	Q8/Turbo4	Q8/Turbo3	Turbo4/Turbo3	score kv
Baseline FP16	6.4935	6.4949	6.6130	6.5127	6.5311	6.7846	94.38
Quant + KV	6.5509	6.5492	6.6741	6.5614	6.5795	6.8459	94.37
Prune + KV	8.1727	8.1730	8.3441	8.2070	8.2417	8.6205	93.18
Prune + Quant + KV	8.2479	8.2469	8.4371	8.2856	8.3184	8.7095	93.00

Table 5: KV-cache sensitivity. Higher

\mathrm{score}_{kv}

indicates greater tolerance to KV-cache compression.

Figure 4: Inference efficiency. Peak memory, prefill throughput, and decode throughput on NVIDIA 1xH100 for Qwen3-30B-A3B-Instruct at varying context lengths, across the six compression configurations for Q4_K_M (top) and IQ1_M (bottom) with Q8 KV-cache under bs=1. Both higher compression rate and higher throughput are desirable although their relationship is non-monotonic.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Calibration	GSM8K	MMLU-R	MuSR	HumanEval	MBPP	Math-Hard	IFEval
Official mixture	0.883	0.811	0.431	0.713	0.148	0.539	0.797
C4 general	0.898	0.827	0.418	0.281	0.524	0.297	0.771
Code-only	0.891	0.796	0.427	0.665	0.516	0.557	0.782
Math-only	0.896	0.804	0.438	0.720	0.264	0.560	0.815

Appendix

Table 6: Calibration-dataset ablation for REAP on Qwen3.6-35B-A3B. All variants use the same pruning ratio

r=0.30

, seed, router renormalization, and GGUF Q4_K_M deployment path. Higher is better for all metrics.

Scoring setup	GSM8K	MMLU-R	MuSR
Unpruned Q4_K_M	0.8916	0.8625	0.4365
BF16 REAP	0.8825	0.8111	0.4312
Module quant.-aware REAP	0.8741	0.8107	0.4272
Packed-MoE quant.-aware REAP	0.8779	0.8131	0.4299

Appendix

Table 7: Quantization-aware REAP ablation on Qwen3.6-35B-A3B. All pruned variants use REAP with

r=0.30

and are evaluated after GGUF Q4_K_M quantization through the same deployment path. “Module” quantizes only standard linear modules during scoring, while “Packed MoE” also covers packed routed experts and router weights.

Expert comp. method	Config	GSM8K	MMLU	MuSR	IFEval	HumanEval	BFCLv3	RULER	Avg.	Score_perf	Score_robust
—	Baseline BF16	0.8360	0.8470	0.4290	0.8170	0.9390	0.8880	0.9441	0.8143	100.00	100.00
REAP	Expert only	0.8400	0.7420	0.4380	0.7650	0.8110	0.8560	0.9452	0.7710	94.69	80.41
—	Q4_K_M only	0.8330	0.8450	0.4250	0.8260	0.9270	0.8910	0.9422	0.8127	99.81	97.43
—	Q4_K_M + Q8 KV	0.8378	0.8422	0.4246	0.8336	0.9268	0.8694	0.9435	0.8111	99.61	95.56
REAP	+ Q4_K_M	0.8230	0.7330	0.4310	0.7430	0.8170	0.8410	0.9424	0.7615	93.51	80.82
REAP	+ Q4_K_M + Q8 KV	0.8180	0.7320	0.4270	0.7600	0.8050	0.8420	0.9435	0.7611	93.46	80.41

Appendix

Table 8: Performance comparison of expert compression methods across configurations.

Pruning	Weights	KV cache	PPL $\pm$ stderr	$\Delta$ vs 0% bf16	$\Delta$ vs same-prune bf16
0%	bf16	f16	$6.8200$	–	–
0%	Q4_K_M	f16	$6.8198\pm 0.0444$	$-0.0002$	$-0.0002$
0%	Q4_K_M	q8_0	$6.8180\pm 0.0444$	$-0.0020$	$-0.0020$
20%	bf16	f16	$8.6695\pm 0.0592$	$+1.8495$	–
20%	Q4_K_M	f16	$8.7952\pm 0.0602$	$+1.9752$	$+0.1257$
20%	Q4_K_M	q8_0	$8.7998\pm 0.0602$	$+1.9798$	$+0.1303$

Appendix

Table 9: Pruning sensitivity under fixed quantization and KV-cache settings. For each pruning ratio, the bf16/f16 row serves as the same-prune baseline.

Method	Score (with IQ1_M)	Score (without IQ1_M)
Quant-only	56.12	87.45
Quant+KV (q8)	56.10	87.42
Prune+Quant	55.67	84.35
Prune+Quant+KV (q8)	55.62	84.32
Prune+Quant+KV (q4)	53.21	83.14

Appendix

Table 10: Quantization sensitivity score with and without IQ1_M.

Figure 7: Inference efficiency on Apple-M1-Max. Peak memory, prefill throughput, and decode throughput for Qwen3-30B-A3B-Instruct at context lengths $\{2\mathrm{K},8\mathrm{K},32\mathrm{K}\}$ , across the six compression configurations of Table 2 at two weight-quantization formats (Q4_K_M, IQ1_M) with Q8 KV-cache. Q8_0 is used as the baseline because BF16 caused memory thrashing for this model.

Model	Original HF ID	REAP-pruned HF ID
Qwen3-30B-A3B	Qwen/Qwen3-30B-A3B-Instruct-2507	SamsungSAILMontreal/Qwen3-30B-A3B-Instruct-2507-REAP
Qwen3-Coder-30B-A3B	Qwen/Qwen3-Coder-30B-A3B-Instruct	cerebras/Qwen3-Coder-REAP-25B-A3B
Qwen3-235B-A22B	Qwen/Qwen3-235B-A22B-Instruct-2507	SamsungSAILMontreal/Qwen3-235B-A22B-Instruct-2507-REAP
GLM-4.7-Flash	zai-org/GLM-4.7-Flash	cerebras/GLM-4.7-Flash-REAP-23B-A3B
MiniMax-M2	MiniMaxAI/MiniMax-M2	cerebras/MiniMax-M2-REAP-162B-A10B
Kimi-Linear-48B-A3B	moonshotai/Kimi-Linear-48B-A3B-Instruct	cerebras/Kimi-Linear-REAP-35B-A3B-Instruct

Appendix

Table 11: Original and REAP-pruned Hugging Face checkpoints.

Mixture-of-Experts Inference

Mixture-of-experts inference studies how to run sparsely activated expert models efficiently while preserving their quality.

See all 179 papers

Sep 9, 2026

OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit

Mixture-of-Experts (MoE) models enable efficient scaling of large language models but face critical deployment challenges due to massive memory requirements. Existing pruning methods either incur prohibitive search costs or neglect the dynamic interdependencies between experts. To address these challenges, we present OMP-MoE, a novel training-free compression framework for reducing expert redundancy in MoE-based LLMs. Based on observations of expert contribution patterns, we reformulate the pruning problem as a sparse signal reconstruction task solved through Orthogonal Matching Pursuit. Specifically, our method first treats individual expert contributions as dictionary atoms and selects experts that greedily minimize reconstruction error with linear computational complexity. Then, we optimize cross-layer expert allocation through a water-filling strategy that accounts for both reconstruction quality and routing stability. Finally, we introduce OMP-MoE†, an adaptive inference mechanism that dynamically adjusts expert activation based on energy prediction. Comprehensive experiments on Qwen, DeepSeek-V2, GPT-OSS, and Mixtral MoE demonstrate consistent improvements over existing methods at 25-50% pruning ratios. For Qwen3-30B-A3B at 50% compression, we retain 93.3% of original performance, achieving 33

\times

faster search and 1.55

\times

inference speedup. Codes will be available after acceptance.

Dezhi Li, Lujun Li, Qiyuan Zhu +4

The Hong Kong University of Science and Technology

Mixture-of-Experts Pruning Mixture-of-Experts Language Models LLM Compression LLM Inference Acceleration

Ask PDF

Jun 26, 2026

FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models

Mixture-of-Experts (MoE) language models scale model ability with sparsely activated experts, making this architecture a standard recipe for modern large models. However, sparse activation does not remove the deployment burden of storing and serving all experts, and the available deployment budget can vary substantially across devices, users, and workloads. Existing MoE compression methods are still largely fixed-budget, typically optimizing one compressed endpoint at each chosen target budget. We study a different setting: converting a large pretrained MoE LLM into a nested family of deployable subnetworks across budgets. Our method first ranks expert FFN channels by their importance, then lets each expert learn a discrete action to prune its channels. By gradually increasing cost pressure, a single action-training run exports a series of action masks from high to low budgets, each of which identifies a reliable smaller subnetwork nested in the ranked base model. Moreover, we use a single recovery fine-tune at a mid pruning budget (40%) to recover degraded model quality and transfer the recovered model to other unseen budgets. Overall, our framework surpasses recent MoE compression baselines. Specifically, on Qwen2-57B-A14B, our method retains ~99.8% of base performance while pruning 50% of routed expert parameters even without fine-tuning. For deployment, our pruned subnetworks deliver real memory reduction and throughput gains, and further support realtime online budget switching with kernel-level co-design.

Fan Mo, Yuxuan Han, Geng Zhang +2

National University of Singapore

Mixture-of-Experts Pruning LLM Pruning Mixture-of-Experts Language Models LLM Compression LLM Inference Acceleration Structured Pruning+1

Ask PDF

Oct 21, 2026

MoEZip: Routing-Aware KV Cache Compression for Sparse Mixture-of-Experts LLMs

Mixture-of-Experts (MoE) language models scale model capacity efficiently by activating a small subset of parameters per token. However, in long-context inference, memory remains a major bottleneck because the KV cache grows linearly with sequence length. Existing KV cache compression methods reduce this cost for dense transformers, but overlook a problem specific to MoE: KV eviction can change which experts are selected during answer generation, degrading the generation quality. We propose MoEZip, a routing-aware KV cache compression for sparse MoE language models. To estimate the sensitivity of expert routing to perturbation, we propose a novel metric based on the Fisher information matrix of routing distributions. MoEZip combines this metric with attention weights for routing-aware scoring of contexts. Finally, we compose a complementary scoring system that combines eviction methods for dense and sparse-MoE architectures. Experiments show that MoEZip achieves robust performance under aggressive cache compression on various benchmarks. At a KV retention ratio of 10%, MoEZip improves over the SoTA baseline by +24 and +20 points on average on LongBench and RULER, corresponding to 2.3

\times

and 4.9

\times

higher scores, respectively.

Minsang Kim, Seungjun Baek

Korea University · SK Telecom

KV-Cache Compression Long-Context Language Model Inference LLM Inference Acceleration Mixture-of-Experts Language Models Expert Routing

Ask PDF

Abstract

Mixture-of-Experts Inference

Explore similar work

OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit

FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models

MoEZip: Routing-Aware KV Cache Compression for Sparse Mixture-of-Experts LLMs