Modality-Aware Expert Pruning for MoE-Based Multimodal Large Language Models
Organizations: Nota Inc.
Abstract
Mixture-of-Experts (MoE) enables multimodal large language models (MLLMs) to scale capacity efficiently, but expert parameters dominate memory even though only a few experts are activated per token: they account for over 90% of model size in modern MoE-MLLMs (e.g., 54 GB out of about 58 GB in BF16 for InternVL3.5-30B-A3B and Qwen3-VL-30B-A3B). Expert pruning has emerged as an effective compression strategy, but existing methods aggregate importance uniformly across tokens, ignoring that the same expert may serve different roles for vision versus language processing. To address this, we propose MAEP (Modality-Aware Expert Pruning), a training-free framework that decomposes expert importance by modality in a single forward pass. MAEP introduces (1) Delta-H Output (), a redistribution-aware metric measuring the MoE output change when an expert is removed, and (2) low-attention vision tokens as a vision-side pruning signal that is dataset-stable: its expert-importance ranking remains stable across different datasets. Text-side importance serves as a preservation signal while vision-side importance serves as a pruning signal, and experts are ranked globally across layers. Experiments on three MoE-based MLLMs across multimodal and text benchmarks show that MAEP outperforms modality-agnostic baselines on the majority of multimodal benchmarks while remaining competitive on text-only tasks, reducing the BF16 expert weight footprint by about 27 GB (about 47% of total memory) so that the full weight footprint of an approximately 58 GB model fits within a single 40 GB GPU. Code is available at https://github.com/nota-github/MAEP.
Figures & tables
| Multimodal | Text-only | Avg. (%) | ||||||||||||
| Model | Ratio | Method | MME | MMBench | MMStar | ChartQA | TextVQA | RealWorldQA | CV-Bench | POPE | MMLU-Redux | IFEval | ARC-C | |
| InternVL3.5 30B-A3B | — | Original † | 2442.2 | 85.2 | 71.4 | 87.3 | 80.2 | 72.2 | 84.6 | 89.4 | 80.9 | 78.9 | 95.2 | 100.0 |
| 25% | HC-SMoE | 2344.8 | 79.8 | 67.1 | 87.0 | 76.2 | 71.6 | 84.5 | 89.8 | 67.4 | 45.6 | 87.4 | 91.9 | |
| REAP | 2326.6 | 84.0 | 68.9 | 83.2 | 78.8 | 72.5 | 83.6 | 89.3 | 69.3 | 69.7 | 92.7 | 95.9 | ||
| MAEP | 2364.9 | 83.2 | 69.3 | 87.0 | 78.9 | 71.6 | 84.3 | 89.7 | 72.3 | 77.1 | 94.1 | 97.7 | ||
| 50% | HC-SMoE | 1907.5 | 66.1 | 56.2 | 75.3 | 66.5 | 68.0 | 79.8 | 88.3 | 33.9 | 20.9 | 53.2 | 74.1 | |
| Model | Used | Ratio | sweep (% retention) | ||||
|---|---|---|---|---|---|---|---|
| 0.1 | 0.3 | 0.5 | 0.7 | 0.9 | |||
| InternVL3.5 | 0.8 | 25% | 97.5 | 98.3 | 98.1 | 98.0 | 98.1 |
| 50% | 84.3 | 85.9 | 86.9 | 89.1 | 65.1 | ||
| Qwen3-VL | 0.3 | 25% | 96.4 | 97.1 | 97.1 | 29.0 | 26.0 |
| 50% | 84.4 | 87.6 | 25.4 | 29.6 | 23.0 | ||
| Gemma4 | 0.1 | 25% | 95.9 | 91.1 | 72.5 | 67.6 | 66.7 |
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Configuration | Memory (GB) | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Model | Total | Expert | Layers | E/L | Top- | Router | BF16 | 25% | 50% |
| InternVL3.5 | 30.8B | 29.0B (94%) | 48 | 128 | 8 | softmax | 57.5 | 44.0 (-23%) | 30.5 (-47%) |
| Qwen3-VL | 31.1B | 29.0B (93%) | 48 | 128 | 8 | softmax | 57.9 | 44.4 (-23%) | 30.9 (-47%) |
| Gemma4-26B-A4B | 25.8B | 22.8B (88%) | 30 | 128 | 8 | softmax | 48.1 | 37.4 (-22%) | 26.8 (-44%) |
| Metric | Measured |
|---|---|
| Resident weight memory at 0 / 25% / 50% pruning | 57.5 / 44.0 / 30.5 GB ( at 50%) |
| decode-throughput ratio (50%-pruned / unpruned) | 0.96–1.02 (parity within run-to-run noise) |
| Decode under a 40 GB memory cap | unpruned ( CPU-offloaded): 1.52 tok/s; 50%-pruned (fully resident): 5.88 tok/s ( ) |
| Large-batch decode-throughput ratio (50%-pruned / unpruned) | 1.13 ( ) / 1.29 ( , ) / 1.37 ( ) |
| Unique activated experts per layer per decode step at | (unpruned) vs. (50%-pruned) |
| Model | Ratio | Method | MM | Text | All |
|---|---|---|---|---|---|
| InternVL3.5-30B-A3B | 25% | HC-SMoE | 97.2 | 77.6 | 91.9 |
| REAP | 97.9 | 90.5 | 95.9 | ||
| MAEP | 98.6 | 95.3 | 97.7 | ||
| 50% | HC-SMoE | 86.4 | 41.4 | 74.1 | |
| REAP | 91.6 | 80.3 | 88.5 | ||
| MAEP | 94.0 | 83.8 | 91.2 |
| Benchmark | DiEP | MAEP (ours) |
|---|---|---|
| MME | 2364.3 / 2195.6 | 2364.9 / 2160.3 |
| MMBench | 79.2 / 66.7 | 83.2 / 80.7 |
| MMStar | 61.8 / 49.7 | 69.3 / 63.3 |
| ChartQA | 54.0 / 40.4 | 87.0 / 76.7 |
| TextVQA | 74.6 / 67.1 | 78.9 / 75.6 |
| RealWorldQA | 70.6 / 64.7 | 71.6 / 71.6 |
| Model | closed-form | |||
|---|---|---|---|---|
| InternVL3.5-30B-A3B | ||||
| Qwen3-VL-30B-A3B-Instruct | ||||
| Gemma4-26B-A4B-Instruct |
| Selector | (stability) | (vs. text; low = distinct) |
|---|---|---|
| Low-attention (ours) | 0.51 | |
| High-attention | 0.30 | 0.47 |
| Full vision (no subset) | 0.30 | |
| Drift-low | 0.19 | 0.01 |
| Drift-high | 0.29 | 0.34 |
| Diversity (farthest-point) | 0.36 | 0.36 |
| Statistic | Low-attention tokens | Base rate |
|---|---|---|
| Fraction outside the top-50% saliency mass | 92.0% | 76.5% |
| Fraction in the saliency-map background (Otsu threshold) | 89.5% | 70.1% |
| Calibration swapped to the ChartNet mixture | Jaccard vs. original | Decisions flipped |
|---|---|---|
| Vision (low-attention) side only | 0.922 | 248 |
| Text side only | 0.776 | 776 |
| Both (full mixture; Table 2 (b)) | 0.780 | 760 |
| Modality | Spearman | Jaccard @ |
|---|---|---|
| Text | 0.999 | 0.97 |
| Low-attn vision | 0.987 | 0.92 |
| High-attn vision | 0.994 | 0.93 |
| Full vision | 0.981 | 0.89 |
| Removal order | KL at text positions |
|---|---|
| Highest- experts first (our sign) | 0.30 |
| Randomly selected experts | 0.48 |
| Lowest- experts first (inverted) | 1.13 |
| Uniform per-layer budget | Global non-uniform budget | |
|---|---|---|
| REAP scoring | 2109.6 / 63.3 / 65.3 / 74.4 (Table 1 ) | 2085.7 / 63.0 / 65.1 / 74.6 |
| MAEP scoring | 2103.9 / 62.2 / 72.1 / 76.5 ( ; Table 2 (a)) | 2160.3 / 63.3 / 76.7 / 75.6 (Table 1 ) |
| Score (uniform per-layer budget) | MME | MMStar | ChartQA | TextVQA |
|---|---|---|---|---|
| MAEP, | 2144.1 | 60.1 | 56.2 | 76.5 |
| MAEP, | 2155.9 | 59.8 | 56.8 | 76.8 |
| MAEP, | 2138.0 | 61.4 | 60.9 | 76.9 |
| MAEP, | 2124.6 | 60.9 | 61.2 | 76.6 |
| MAEP, (Table 2 (a)) | 2103.9 | 62.2 | 72.1 | 76.5 |
| MAEP, | 1675.4 | 49.3 | 44.8 | 58.2 |
Mixture-of-Experts Pruning
Mixture-of-Experts pruning removes or trims experts to reduce model memory and computation while preserving useful capabilities.
See all 68 papers