cs.LGSep 5, 2026
SaveBeyond Retraining-Free MoE Compression: A Cost-Normalized Study of Post-Compression Adjustment
Organizations: Department of Electrical and Computer Engineering, Seoul National University · Interdisciplinary Program in Artificial Intelligence, Seoul National University
Abstract
Retraining-free MoE compression reduces deployment memory by pruning or merging experts, but often treats the compressed checkpoint as the final artifact. We argue that this view is incomplete: compressed MoE checkpoints are better understood as compressed initializations that benefit from a tiny post-compression adjustment stage. Across two MoE LLM backbones, four pruning/merging methods, three expert-retention ratios, and 28 benchmarks, we compare LM fine-tuning and teacher-based KD under matched small-data budgets and measured GPU costs. Using only 3,000 C4 examples and a single epoch of adjustment, Full FT recovers 37.3% of the original-to-compressed performance gap on average. Moreover, LM fine-tuning is more cost-effective than standard token-level KD, and full-parameter adjustment gives the strongest cost--recovery trade-off among the tested scopes. These results suggest that retraining-free compression should be paired with small post-compression adjustment to recover a substantial portion of the performance lost during compression.
Figures & tables
| Model | Ratio | Best(%) | Most(%) | Worst(%) |
|---|---|---|---|---|
| Gemma4 | 50% | 8.63 | 90.76 | 0.61 |
| Gemma4 | 62.5% | 11.38 | 88.25 | 0.37 |
| Gemma4 | 75% | 15.52 | 84.20 | 0.28 |
| Qwen3 | 50% | 8.05 | 91.73 | 0.22 |
| Qwen3 | 62.5% | 15.81 | 84.14 | 0.05 |
| Qwen3 | 75% | 32.15 | 67.84 | 0.01 |
Table 1: Scenario-frequency summary by backbone and expert retention ratio. Results are based on 100 samples from the ELI5 dataset. Best denotes cases where the original selected expert set is preserved, Most denotes partial overlap between the original and compressed selected expert sets, and Worst denotes disjoint selected expert sets. For each model–ratio pair, results are averaged across the corresponding compression methods.
Figure 1: Overall recovery gain of post-compression adjustment strategies. Bars show the average-score recovery gain, measured in percentage points (pp), over the unadjusted compressed baseline. Causal LM fine-tuning consistently outperforms KD under matched trainable scopes, and Full FT achieves the largest recovery.
Figure 2: GPU cost–performance trade-off of post-compression adjustment strategies. The y-axis shows mean average-score recovery gain, while the x-axis shows effective GPU-hours (top), GPU-memory occupancy in GiB-hours (middle), and GPU energy in kWh (bottom). Each point is an unweighted arithmetic mean over 12 method–ratio settings (four compression methods three retention ratios). Both expert-selection cost and training-loop cost are included when applicable. Upper-left is better.
Figure 3: Benchmark-family recovery of Full FT . Bars show the post-adjustment performance, and dashed lines denote the original model and the unadjusted compressed baseline. Full FT yields the largest recovery on chain-of-thought and MCQA benchmarks, while math and coding show only marginal changes.
Figure 4: Recovery gain as the trainable parameter scope expands. For both LM fine-tuning and KD, recovery increases from router-only to router+top- experts, router+all-experts, and full-parameter adjustment. The large gain from router+all-experts to full adjustment suggests that compression-induced noise propagates beyond MoE modules into dense/shared components.
Figure 5: Recovery with 1,024 C4 calibration examples. Bars show the mean score change from the unadjusted compressed model across 28 benchmarks and 12 backbone–compressor–retention settings. Full FT remains the strongest strategy, followed by Router All-Expert FT and Full KD, showing that reducing the main study’s 3,000-example budget preserves the central strategy-level conclusion.
Figure 6: Calibration-domain robustness with 1,024 examples each from C4 and OpenR1-Math-220k, averaged over six matched settings and 28 benchmarks. Each point is an adjustment strategy; upper-right indicates stronger recovery in both domains. Full FT has the largest two-domain mean, and aggregate rankings are strongly associated ( , ).
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
| Setting | Value |
|---|---|
| Calibration samples | 3000 |
| Epochs | 1 |
| Per-device batch size | 2 |
| Gradient accumulation | 4 steps |
| Effective batch size | 8 examples |
| Maximum sequence length | 512 tokens |
Table 2: Common Post Compression Adjustment hyperparameters.
| Category | Tasks | Few-shot / metric settings |
|---|---|---|
| Reasoning and math | gsm8k , gsm8k_platinum , minerva_math , bbh_fewshot , bbh_zeroshot , coqa | GSM8K and GSM8K-Platinum use 5-shot; Minerva Math uses 4-shot; BBH few-shot uses 3-shot; BBH zero-shot and CoQA use 0-shot. Primary metrics are exact-match style harness metrics. |
| Multiple choice | medqa_4options , medmcqa , openbookqa , arc_easy , winogrande , arc_challenge , piqa , mmlu , hellaswag | All are evaluated 0-shot. Metrics are the harness-provided accuracy and normalized-accuracy variants where available. |
| Chain-of-thought | gsm8k_cot , gsm8k_platinum_cot , gsm8k_cot_zeroshot , gsm8k_platinum_cot_zeroshot , bbh_cot_fewshot , bbh_cot_zeroshot | GSM8K-CoT and GSM8K-Platinum-CoT use 8-shot; their zero-shot variants use 0-shot. BBH-CoT few-shot uses 3-shot and BBH-CoT zero-shot uses 0-shot. Metrics are exact-match style harness metrics. |
| Code | humaneval , mbpp , mbpp_plus , humaneval_plus | HumanEval uses 0-shot. MBPP and MBPP-Plus use 3-shot. Metrics are pass@1 style harness metrics. |
| AIME | aime , aime24 , aime25 | All are 0-shot. Generation uses max_gen_toks=2780 , do_sample=false , and temperature 0.0. The metric is exact match. |
Table 3: Downstream evaluation categories and task-level settings.
| Backbone | Paradigm | Method | Target | Compression settings |
|---|---|---|---|---|
| Qwen3 | Pruning | REAP | 128 experts/layer 64, 80, 96 retained experts/layer | Prune fractions 0.50, 0.375, and 0.25. Observation data used the allenai/c4 train split with shuffling, observer batch size 1, 1024 batches per category, maximum sequence length 2048, cosine distance, and seed 42. Router weights were renormalized after pruning. The saved configuration records prune_method=reap , record_pruning_metrics_only=true , renormalize_router_weights=true , expert_sim=ttm , frequency_penalty=true , agglomerative clustering metadata, and average linkage. |
| Qwen3 | Merging | HC-SMoE | 128 logical experts/layer 64, 80, 96 physical experts/layer | Hierarchical expert merging was run on the RTE calibration task with n_sentences=8 , train batch size 2, data_limit=1000000 , partition=8 , start_layer=0 , and group_limit=4 . Experts were grouped by expert-output activation similarity using hierarchical clustering with average linkage and silhouette stopping; ZipIt was used as the merge operator with ingredient=act and dominant=no . |
| Qwen3 | Editing | MoBE | Ratio labels 50, 62.5, 75 | MoBE factor fitting was run over all 48 layers and 128 experts with expert matrix shape , SiLU activation, 8000 fitting epochs, batch size 32, 4 batches, and learning rate 0.07. The fitted matrices were gate_proj and up_proj . The basis count num_B was 8, 8, and 32 for the 50%, 62.5%, and 75% settings; truncation was 438, 768, and 768, respectively. Native MoBE checkpoints were reconstructed to Hugging Face checkpoints for vLLM evaluation. |
| Gemma4 | Pruning | AIMER | 128 experts/layer 64, 80, 96 retained experts/layer | AIMER pruning was calibration-free. The score was computed over each routed expert’s gate, up, and down weights. Runs used metric=aimer , metric_device=auto , prune_highest_score=true , seed 42, bfloat16 loading, device_map=auto , trust_remote_code=true , and model_class=causal-lm . The dense Gemma4 layer.mlp shared component was preserved and not treated as part of AIMER pruning. |
| Gemma4 | Merging | M-SMoE | 128 logical experts/layer 64, 80, 96 physical experts/layer | M-SMoE used the shared C4 calibration shard c4-train.00000-of-01024.json . Router statistics were collected with similarity_base=router-logits , subset_ratio=0.01 , block size 512, and batch size 1. The implementation confirms 128 logical Gemma4 experts and rewrites the 30 routed layers to the requested physical expert count. |
| Gemma4 | Editing | TD-MoE | Target total parameters 14.387B, 17.242B, 20.096B | TD-MoE was applied to all 30 Gemma4 routed layers using the same C4 calibration shard, 256 whitening samples, model sequence length 2048, seed 3, input whitening, SVD decomposition, bfloat16 inference, and device_map=auto . Native TD-MoE checkpoints were materialized back to Hugging Face/vLLM-compatible checkpoints with bfloat16 weights and 10GB maximum shard size. |
Table 4: Compression settings.
Figure 7: Expert Pruning: Overall recovery gain of post-compression adjustment strategies. Bars show mean average-score recovery gain, measured in percentage points (pp), over the unadjusted pruning baseline. Results are averaged over pruning checkpoints, retention ratios, and benchmarks. Full FT achieves the largest recovery, and causal LM fine-tuning outperforms KD under matched trainable scopes.
Figure 8: Expert Pruning: GPU cost–performance trade-off of post-compression adjustment strategies. The y-axis shows mean average-score recovery gain, while the x-axis shows effective GPU-hours (top), GPU-memory occupancy in GiB-hours (middle), and GPU energy in kWh (bottom). Each point is an unweighted arithmetic mean over six pruning method–ratio settings (two pruning methods three retention ratios). Both expert-selection cost and training-loop cost are included when applicable. Upper-left is better.
Figure 9: Expert Pruning: Benchmark-family recovery of Full FT . Bars show post-adjustment performance, and dashed lines denote the original model and the unadjusted pruning baseline. Full FT yields gains on CoT, MCQA, and general benchmarks, while math and coding decrease slightly.
Figure 10: Expert Pruning: Recovery gain as the trainable parameter scope expands. For both LM fine-tuning and KD, recovery increases from router-only to router+top- experts, router+all-experts, and full-parameter adjustment. LM fine-tuning remains stronger than KD at each matched scope.
Figure 11: Expert Merging: Overall recovery gain of post-compression adjustment strategies. Bars show mean average-score recovery gain, measured in percentage points (pp), over the unadjusted merging baseline. Results are averaged over merging checkpoints, retention ratios, and benchmarks. Full FT achieves the largest recovery, and causal LM fine-tuning consistently outperforms KD under matched trainable scopes.
Figure 12: Expert Merging: GPU cost–performance trade-off of post-compression adjustment strategies. The y-axis shows mean average-score recovery gain, while the x-axis shows effective GPU-hours (top), GPU-memory occupancy in GiB-hours (middle), and GPU energy in kWh (bottom). Each point is an unweighted arithmetic mean over six merging method–ratio settings (two merging methods three retention ratios). Both expert-selection cost and training-loop cost are included when applicable. Upper-left is better.
Figure 13: Expert Merging: Benchmark-family recovery of Full FT . Bars show post-adjustment performance, and dashed lines denote the original model and the unadjusted merging baseline. Full FT recovers strongly on CoT, MCQA, general, and math benchmarks, while coding is essentially unchanged.
Figure 14: Expert Merging: Recovery gain as the trainable parameter scope expands. For both LM fine-tuning and KD, recovery increases as the trainable scope expands from router-only to router+top- experts, router+all-experts, and full-parameter adjustment. The large final jump to full-parameter adjustment mirrors the main result and suggests that merging-induced residuals are not confined to MoE expert parameters alone.
Figure 15: Compatibility-mode Expert Editing: Overall recovery gain of post-compression adjustment strategies. Bars show the mean average-score recovery gain, measured in percentage points (pp), over the unadjusted reconstructed Expert Editing baseline. Full FT achieves the largest recovery. LM fine-tuning outperforms KD at every expert-inclusive and full-parameter matched scope; the router-only pair is the sole exception, where Router-only KD yields pp versus pp for Router-only FT . Router-only adjustment provides only limited gains. These results are reported as supplementary compatibility-mode evidence because adjustment is performed on reconstructed standard-MoE weights rather than native factorized Expert Editing parameters.
Figure 16: Compatibility-mode Expert Editing: Benchmark-family recovery of Full FT . Bars show post-adjustment performance, and dashed lines denote the original model and the unadjusted reconstructed Expert Editing baseline. Full FT yields large gains on CoT and MCQA benchmarks, while math shows the weakest recovery in this setting.
Figure 17: Compatibility-mode Expert Editing: Recovery gain as the trainable parameter scope expands. For both LM fine-tuning and KD, recovery increases from router-only adjustment to router+top- experts, router+all-experts, and full-parameter adjustment. The same monotonic scope-expansion trend observed in the main pruning/merging experiments also appears for reconstructed Expert Editing checkpoints.
| Strategy | Score | (pp) | Energy (kWh) | Eff. GPU-h | Mem. (GiB-h) |
|---|---|---|---|---|---|
| Original | 0.484 | +9.31 | – | – | – |
| Compressed Only | 0.390 | +0.00 | – | – | – |
| Full FT | 0.418 | +2.78 | 0.200 | 0.213 | 93.7 |
| Router All-Expert FT | 0.409 | +1.86 | 0.201 | 0.209 | 85.7 |
| Router Top-50 FT | 0.406 | +1.58 | 0.255 | 0.248 | 75.0 |
| Router Top-16 FT | 0.401 | +1.03 | 0.241 | 0.242 | 55.1 |
Table 5: Adjustment with a 1,024-example C4 calibration budget. Values are macro-averaged over 28 evaluation benchmarks and 12 backbone–compressor–retention settings (four backbone–compressor pairs at three retention ratios). is the score change from the unadjusted compressed model in percentage points. Costs are end-to-end means; Mem. is GPU-memory occupancy in GiB-hours. Original and Compressed Only require no adjustment and therefore have no adjustment cost. The Original and Compressed Only scores for this check were re-evaluated; their aggregate gap differs from Table 8 by evaluation-level noise of about 0.05 pp.
| Mean score | Recovery (pp) | Math-calibration cost | |||||
|---|---|---|---|---|---|---|---|
| Strategy | C4 | Math | C4 | Math | kWh | Eff. GPU-h | GiB-h |
| Original | 0.484 | 0.484 | +11.66 | +11.66 | – | – | – |
| Compressed Only | 0.367 | 0.367 | +0.00 | +0.00 | – | – | – |
| Full FT | 0.402 | 0.413 | +3.50 | +4.58 | 0.126 | 0.098 | 70.1 |
| Router All-Expert FT | 0.386 | 0.407 | +1.89 | +4.01 | 0.124 | 0.093 | 64.3 |
| Router Top-50 FT | 0.385 | 0.404 | +1.81 | +3.70 | 0.167 | 0.124 | 52.7 |
Table 6: Calibration-domain comparison on six matched settings (Qwen3/HC-SMoE and Gemma4/AIMER, each at three retention ratios). Both C4 and OpenR1-Math-220k Lozhkov et al. (2025) use 1,024 calibration examples, and scores average the same 28 evaluation benchmarks. Recovery is measured from the unadjusted compressed model. The last three columns report end-to-end adjustment costs for OpenR1-Math calibration; GiB-h is GPU-memory occupancy in GiB-hours.
| Strategy | IFEval | TruthfulQA | ToxiGen | WMDP | CrowS | WinoGender |
|---|---|---|---|---|---|---|
| Original | 0.853 | 0.421 | 0.812 | 0.508 | 0.576 | 0.581 |
| Compressed Only | 0.761 | 0.365 | 0.708 | 0.437 | 0.556 | 0.567 |
| Full FT | 0.720 | 0.361 | 0.693 | 0.477 | 0.577 | 0.610 |
| Router All-Expert FT | 0.752 | 0.360 | 0.702 | 0.461 | 0.572 | 0.591 |
| Router Top-50 FT | 0.755 | 0.360 | 0.711 | 0.460 | 0.570 | 0.588 |
| Router Top-16 FT | 0.760 | 0.360 | 0.712 | 0.455 | 0.569 | 0.582 |
Table 7: Behavioral-proxy results averaged over 12 backbone–compressor–retention settings. Adjusted strategies are the 1,024-example C4 runs of Table 5 . The table jointly reports the two baselines, all LM fine-tuning and KD scopes, and Direct Router Logit Match. Metric directions differ: higher is better for IFEval, TruthfulQA MC1, and WinoGender accuracy; ToxiGen is classification accuracy rather than a monotone safety score; higher WMDP means greater hazardous-knowledge accuracy; and CrowS is more neutral near 0.5. We therefore neither average across columns nor designate a single “safest” condition.
| Strategy | Score | (pp) | Gap (%) | Energy (kWh) | Eff. GPU-h | Memory (GiB-h) |
|---|---|---|---|---|---|---|
| Original | 0.483650 | – | – | – | – | – |
| Compressed Only | 0.390050 | +0.00 | – | – | – | – |
| Full FT | 0.424975 | +3.49 | 37.3 | 0.576679 | 0.622545 | 273.4334 |
| Full KD | 0.411908 | +2.19 | 23.4 | 0.786771 | 0.764422 | 516.9228 |
| Router-only FT | 0.392342 | +0.23 | 2.4 | 0.337770 | 0.263998 | 61.3633 |
| Router-only KD | 0.391792 | +0.17 | 1.9 | 0.531668 | 0.406578 | 207.5186 |
Table 8: Overall performance and measured cost of the 13 Post-Compression Adjustment strategies. Score is the unweighted mean over 28 benchmarks and 12 compressor–retention settings (four compressors three retention ratios). is the score change from Compressed Only in percentage points, and Gap is the recovered fraction of the aggregate Original-to-Compressed gap. Cost columns are unweighted means over the same 12 settings; GPU-memory values are time-integrated occupancy. Expert-selection and online-teacher passes are included when applicable.
| Strategy | 50% retention | 62.5% retention | 75% retention |
|---|---|---|---|
| Each cell: Score; then GPU energy / effective GPU-hours / GPU-memory occupancy | |||
| Original | 0.483650 [1pt]– / – / – | 0.483650 [1pt]– / – / – | 0.483650 [1pt]– / – / – |
| Compressed Only | 0.346475 [1pt]– / – / – | 0.388450 [1pt]– / – / – | 0.435225 [1pt]– / – / – |
| Full FT | 0.359000 [1pt]0.449945 / 0.471098 / 178.8026 | 0.423075 [1pt]0.575316 / 0.620035 / 267.0727 | 0.492850 [1pt]0.704776 / 0.776501 / 374.4249 |
| Full KD | 0.358125 [1pt]0.660507 / 0.614883 / 386.0558 | 0.410475 [1pt]0.786565 / 0.760321 / 511.9236 | 0.467125 [1pt]0.913239 / 0.918064 / 652.7890 |
| Router-only FT | 0.332550 [1pt]0.289110 / 0.226473 / 44.4989 | 0.389950 [1pt]0.335920 / 0.264699 / 60.0503 | 0.454525 [1pt]0.388279 / 0.300822 / 79.5408 |
Table 9: Retention-ratio sensitivity with all three measured GPU-cost metrics. Each cell reports Score on the first line and on the second, where is GPU energy (kWh), is effective GPU-hours, and is GPU-memory occupancy (GiB-hours). Score is an unweighted mean over four compressor–backbone settings and 28 benchmarks at the indicated retention ratio; each cost is the unweighted mean of the corresponding four strategy runs. Expert-selection and online-teacher passes are included when applicable.
| Compressor | Backbone | Paradigm | Compressed | Full FT | Full KD |
|---|---|---|---|---|---|
| Full FT/KD cells: Score ( pp); then E/H/M | |||||
| AIMER | Gemma4 | Pruning | 0.280333 | 0.314933 (+3.46) [1pt]0.088906 / 0.119960 / 24.6553 | 0.297067 (+1.67) [1pt]0.118888 / 0.150347 / 44.9570 |
| HC-SMoE | Qwen3 | Merging | 0.454000 | 0.510667 (+5.67) [1pt]0.557599 / 0.394955 / 339.4779 | 0.493333 (+3.93) [1pt]0.939317 / 0.652387 / 740.1165 |
| M-SMoE | Gemma4 | Merging | 0.297167 | 0.322833 (+2.57) [1pt]0.970750 / 1.490283 / 299.4836 | 0.304800 (+0.76) [1pt]1.010224 / 1.510365 / 411.3137 |
| REAP | Qwen3 | Pruning | 0.528700 | 0.551467 (+2.28) [1pt]0.689461 / 0.484980 / 430.1168 | 0.552433 (+2.37) [1pt]1.078653 / 0.744590 / 871.3040 |
Table 10: Integrated compressor-level performance and cost, averaged over the three retention ratios. Full FT and Full KD cells report Score and the change from that compressor’s unadjusted checkpoint on the first line, followed by GPU energy (kWh), effective GPU-hours, and GPU-memory occupancy (GiB-hours) as . Performance additionally averages 28 benchmarks; each cost averages three runs. Each compressor is paired with one backbone, so compressor and backbone effects are not separately identifiable.
| Strategy | Expert Pruning | Expert Merging | Expert Editing |
|---|---|---|---|
| Each cell: Score ( pp); then E/H/M | |||
| Original | 0.483650 [1pt]– / – / – | 0.483650 [1pt]– / – / – | 0.483650 [1pt]– / – / – |
| Compressed Only | 0.404517 (+0.00) [1pt]– / – / – | 0.375583 (+0.00) [1pt]– / – / – | 0.337283 (+0.00) [1pt]– / – / – |
| Full FT | 0.433200 (+2.87) [1pt]0.389184 / 0.302470 / 227.3860 | 0.416750 (+4.12) [1pt]0.764175 / 0.942619 / 319.4808 | 0.393000 (+5.57) [1pt]– / – / – |
| Full KD | 0.424750 (+2.02) [1pt]0.598771 / 0.447469 / 458.1305 | 0.399067 (+2.35) [1pt]0.974771 / 1.081376 / 575.7151 | 0.377533 (+4.03) [1pt]– / – / – |
| Router-only FT | 0.405483 (+0.10) [1pt]0.292612 / 0.228956 / 52.8989 | 0.379200 (+0.36) [1pt]0.382928 / 0.299040 / 69.8278 | 0.339817 (+0.25) [1pt]– / – / – |
Table 11: Performance–cost summary by compression paradigm with the complete measured cost tuple. For Expert Pruning and Expert Merging, is the 28-benchmark score change from the unadjusted compressed checkpoint, averaged over two compressors and three retention ratios; denotes GPU energy (kWh), effective GPU-hours, and GPU-memory occupancy (GiB-hours), each averaged over the corresponding six runs. Expert Editing reports the same performance change over six compatibility-mode MoBE/TD-MoE settings; its cost tuple is shown as unavailable because those reconstructed checkpoints use a different parameterization and measurement boundary.
| (a) Causal-LM fine-tuning scopes | ||||||||
|---|---|---|---|---|---|---|---|---|
| Family | Orig. | Comp. | Full FT | Router-only FT | Router Top-8 FT | Router Top-16 FT | Router Top-50 FT | Router All-Expert FT |
| General (4) | 0.450962 | 0.435135 | 0.450323 | 0.440379 | 0.447379 | 0.448342 | 0.452844 | 0.454240 |
| Math (6) | 0.412292 | 0.289432 | 0.297187 | 0.289442 | 0.296483 | 0.297693 | 0.299303 | 0.303750 |
| Coding (4) | 0.373688 | 0.242994 | 0.238104 | 0.240187 | 0.247377 | 0.245998 | 0.243981 | 0.241181 |
| CoT (6) | 0.588908 | 0.449946 | 0.540596 | 0.454926 | 0.465842 | 0.468947 | 0.486062 | 0.494618 |
| MCQA (8) | 0.529556 | 0.471557 | 0.514856 | 0.474628 | 0.488238 | 0.490059 | 0.496747 | 0.499443 |
Table 12: Benchmark-family performance averaged over all 12 compressor–retention settings. General contains BBH-Fewshot, BBH-Zeroshot, CoQA, and HellaSwag; the remaining families contain 6 Math, 4 Coding, 6 chain-of-thought (CoT), and 8 multiple-choice QA (MCQA) benchmarks. Cost is not allocated by benchmark because every strategy run evaluates all benchmark families after a single shared adjustment procedure; aggregate run-level costs are reported in Table 8 . Family scores are aggregated from individually rounded benchmark scores.
More in this topic
Mixture-of-Experts Pruning
Mixture-of-Experts pruning removes or trims experts to reduce model memory and computation while preserving useful capabilities.
See all 59 papers