ProSE: Probing Reconstruction Error for Semantic-guided Expert Skipping in MoE-MLLMs
Organizations: KU Leuven · University of Derby
Abstract
Mixture-of-Experts Multimodal Large Language Models (MoE-MLLMs) offer scalable inference through sparse expert activation, yet substantial computational redundancy remains among the activated experts. Existing expert skipping methods typically calibrate thresholds based on target skipping ratios or final-output divergence, overlooking the accumulation of skipping-induced error across layers and the varying semantic importance of individual tokens. To address that, we propose ProSE, a training-free, semantic-guided expert skipping framework that probes layer-wise reconstruction error to promote representation consistency compared with default activation. Specifically, we construct query-aware semantic masks to identify salient visual tokens and progressively calibrate modality-specific skipping thresholds for each MoE layer. By minimizing masked reconstruction error under a given budget, ProSE maintains semantic consistency after skipping and allows adaptive expert allocation under limited activation budgets. Extensive experiments on 9 benchmarks demonstrate consistent improvements on average over existing expert skipping methods, particularly under aggressive skipping settings. ProSE maintains over 96% performance on KimiVL-A3B-Inst with 83% expert skipping and accelerates prefilling by .
Figures & tables
| Method | MMMU | SQA | TextVQA | ChartQA | OCRBench | MMStar | RealWorldQA | VizWiz | InfoVQA | Avg | Rel (%) |
| 53.4 | 94.9 | 83.4 | 85.1 | 82.9 | 58.7 | 65.1 | 71.9 | 81.3 | 75.2 | 100.0 | |
| skip 63% | |||||||||||
| 51.7 | 93.2 | 80.8 | 78.0 | 84.3 | 61.2 | 46.4 | 69.7 | 76.1 | 71.3 | 94.8 | |
| 26.2 | 14.0 | 23.6 | 19.9 | 26.2 | 14.8 | 20.0 | 16.6 | 20.4 | 20.2 | 26.9 | |
| NAEE [ 24 ] | 50.7 | 93.7 | 81.2 | 79.6 | 83.0 | 57.9 | 50.3 | 70.4 | 76.9 | 71.5 | 95.1 |
| DiEP [ 4 ] | 50.1 | 93.9 | 81.8 | 80.1 | 82.7 | 57.5 | 55.2 | 71.1 | 77.2 | 72.2 | 96.0 |
| Method | MMMU | SQA | TextVQA | ChartQA | OCRBench | MMStar | RealWorldQA | VizWiz | InfoVQA | Avg | Rel (%) |
| 51.7 | 96.2 | 88.7 | 89.5 | 84.3 | 49.4 | 65.8 | 70.4 | 84.0 | 75.6 | 100.0 | |
| skip 67% | |||||||||||
| 46.7 | 92.8 | 85.9 | 87.4 | 80.7 | 54.2 | 63.8 | 68.7 | 76.8 | 73.0 | 96.6 | |
| 26.1 | 50.3 | 55.1 | 52.6 | 43.0 | 32.5 | 42.1 | 49.6 | 43.8 | 43.9 | 58.1 | |
| NAEE [ 24 ] | 46.1 | 92.9 | 82.8 | 84.9 | 81.4 | 53.8 | 60.5 | 69.1 | 76.2 | 72.0 | 95.2 |
| DiEP [ 4 ] | 47.8 | 93.4 | 83.2 | 85.4 | 82.6 | 55.1 | 63.7 | 69.4 | 77.3 | 73.1 | 96.7 |
| Method | TextVQA | ChartQA | OCRBench | MMStar | InfoVQA | Rel (%) | Method | TextVQA | ChartQA | OCRBench | MMStar | InfoVQA | Rel (%) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3VL-MoE-30B-A3B | KimiVL-A3B-Inst | ||||||||||||
| 83.4 | 85.1 | 82.9 | 58.7 | 81.3 | 100.0 | 88.7 | 89.5 | 84.3 | 49.4 | 84.0 | 100.0 | ||
| skip 75%, prune 20% | skip 67%, prune 20% | ||||||||||||
| MoDES +MI | 81.6 | 80.1 | 77.8 | 56.9 | 74.9 | 94.9 | MoDES +MI | 85.2 | 86.3 | 73.6 | 48.7 | 78.1 | 93.9 |
| ProSE +MI | 82.8 | 83.8 | 79.8 | 66.2 | 77.3 | 99.6 | ProSE +MI | 87.1 | 87.3 | 75.8 | 50.9 | 78.1 | 95.7 |
| MoDES +Div | 81.5 | 80.0 | 79.4 | 56.5 | 74.8 | 95.1 | MoDES +Div | 83.5 | 84.9 | 74.1 | 46.9 | 74.3 | 91.9 |
| Method | Calib. (min) | Prefill (ms) | TTFT (ms) | Perform |
|---|---|---|---|---|
| w/o | - | 261.8 | 293.5 | 89.5 0% |
| MoDES | 2.5+4.9 | 155.6 | 219.3 | 84.4 84.2% |
| ProSE | 5.1 | 149.7 ( ) | 208.3 ( ) | 87.0 85.6% |
| Method | MMMU | SQA | TextVQA | ChartQA | MMStar | Rel (%) |
|---|---|---|---|---|---|---|
| 53.4 | 94.9 | 83.4 | 85.1 | 58.7 | 100.0 | |
| COCO 128 | 51.0 | 90.4 | 80.0 | 76.6 | 57.5 | 94.7 |
| GQA 64 | 51.5 | 91.5 | 80.4 | 78.8 | 57.8 | 95.9 |
| GQA 512 | 52.9 | 92.8 | 80.9 | 79.0 | 58.6 | 97.0 |
| Uniform | 49.6 | 91.5 | 79.5 | 76.9 | 57.7 | 94.6 |
| Random | 51.3 | 92.1 | 80.2 | 78.1 | 58.2 | 95.8 |
| Method | MMMU | SQA | TextVQA | ChartQA | MMStar | Rel (%) |
|---|---|---|---|---|---|---|
| skip 67% | ||||||
| w/o early | 49.9 | 95.1 | 86.6 | 88.0 | 51.1 | 99.4 |
| w/o layer | 43.6 | 87.5 | 82.0 | 82.7 | 47.3 | 93.8 |
| Ours | 50.3 | 95.0 | 87.3 | 88.8 | 51.7 | 100.0 |
| skip 83% | ||||||
| w/o early | 48.7 | 92.6 | 84.9 | 86.6 | 49.7 | 99.2 |
Efficient Multimodal Inference
Efficient multimodal inference reduces the compute, memory, latency, and energy needed to process image, video, audio, text, and sensor inputs.
See all 243 papers