The rise of the agentic web has been accompanied by growing demand for on-device Vision-Language Models (VLMs) that perform diverse cross-modal web tasks with enhanced privacy and reduced latency. Mixture-of-Experts (MoE) models are widely adopted in modern VLMs for compute-efficient scalability, yet their multi-expert architecture poses deployment challenges due to substantial memory demands. Mixed-precision quantization is an effective compression technique that reduces bit-widths while adapting to the heterogeneous hardware budgets of edge devices, particularly well-suited for MoE due to its inherent structural sparsity, which causes imbalanced importance among experts. While prior work has applied quantization for efficient MoE inference at the edge, we point out that in MoE-based VLMs, the modality-specific characteristics of visual and text tokens further impact expert quantization. To address this issue, we present MoMoVL, a modality-aware adaptive compression framework for MoE-based VLMs that brings vision-language inference within reach of on-device resource budgets. Motivated by our empirical findings on modality-induced disparities across experts, MoMoVL jointly exploits the per-expert activation profile and per-modality quantization sensitivity to guide bit-width assignment. Experimental results show that MoMoVL improves the accuracy-compression trade-off relative to state-of-the-art mixed-precision baselines and reduces memory footprint by up to 15.44% with a modest drop in accuracy over its uniform-precision baseline. Our code and data are publicly available at https://github.com/sco-edge/MoMoVL.
Figures & tables
Figure 1. Architectural overview of an MoE-based VLM. Visual tokens and text tokens are processed with shared and routed experts using top-k routing. Overview of MoE-based VLM Input embeddings comprising visual and text tokens are processed with their routed experts based on gating weights, along with the shared experts that are always activated.
Figure 2. Modality-specific expert activation in MolmoE-1B on the COCO caption dataset. (a-b) Each cell shows the percentage of visual/text tokens routed to a given expert (summing to 800 % per layer due to top-8 routing); darker colors indicate more frequent activations. (c) Difference in percentage points; red and blue mark visual- and text-centric experts, respectively. All three panels share one color scale (0-61.5 %, ± 61.5 %p in (c)); the first 16 of 64 experts in native index order are shown. Three heatmaps of modality-specific expert activation. Expert activation pattern of MolmoE-1B is shown per-modality. Leftmost and center pane each show visual- and text-centric expert's activation frequency, which are distinct due to their modality-specific characteristics. The rightmost figure shows the subtraction of the two.
Figure 3. The trend of benchmark scores when increasing the proportion of experts quantized to INT4 in MolmoE-1B. Visual- and text-centric experts are selected according to their importance when processing visual tokens and text tokens, respectively. Baseline: MolmoE-1B in BF16 (59.48). Line plot with orange triangles (Text-Centric), blue circles (Visual-Centric), and a horizontal red dotted line (BF16 baseline) Quantizing text-centric experts in INT4 shows better accuracy overall than quantizing visual-centric experts in INT4. Setting INT4 proportion to 30% shows the largest gap of 2.65.
Figure 4. MoMoVL architecture: A workflow overview of an adaptive, modality-aware mixed-precision quantization framework for MoE-based VLMs. Workflow overview of MoMoVL At the first phase, per-expert activation statistics and per-modality quantization sensitivity are measured. At the second phase, the allocator assigns bit-width to each expert based on the collected offline statistics and the provided hardware budgets. At the third phase, mixed-precision quantization is performed on the target model based on the quantization configuration generated at the second phase.
Figure 5. Per-layer ratio of summed absolute gradients of text tokens over visual tokens in MolmoE-1B on the COCO caption dataset. Visual gradients vanish at the last layer due to loss masking and the absence of subsequent attention. Trend of gradient ratio, text over visual, across layers A line plot with blue circles is shown. Ranging from a minimum of 3.96x at Layer 4, then increasing up to Layer 14. The gradient ratio reaches infinity at Layer 15 as visual gradients become 0.
Metric
MolmoE-1B
DeepSeek-VL2-Tiny
Total Parameters
7.2B
3.4B
Expert Parameters
6.4B (89%)
2.5B (74%)
Experts per Layer (Routed / Shared)
64 / 0
64 / 2
Top-K Routing
8
6
Number of MoE Layers
16
11
Table 1. Configuration of evaluated MoE-based VLMs.
Benchmark
No. of Queries
No. of Tokens ( μ )
Token Ratio (Visual/Text)
Visual
Text
MMMU (val)
900
831.50
129.77
6.41x
MMBench-V1.1 (en test)
7299
478.85
93.26
5.13x
SEEDBench
14232
525.35
82.73
6.35x
ScienceQA (test)
2017
487.66
52.10
9.36x
Table 2. Statistics of evaluation benchmarks. Token counts are obtained by tokenizing queries with MolmoE-1B.
Model
Quantization Method
Precision
Model Size (GB)
MMMU
MMBench
SEED
SciQA
RWQA
MME-RW
AI2D
OCR
MolmoE-1B
-
BF16
13.45
32.22
59.48
68.57
34.80
58.69
33.87
73.41
55.0
Uniform (Full-Model)
INT8
7.20
32.00
59.74
68.67
34.61
58.95
33.35
73.77
55.2
INT4
4.08
31.11
58.18
68.06
34.41
56.73
32.57
72.96
54.6
Uniform (Expert-Only)
INT4
4.45
31.89
58.65
68.19
34.31
58.30
32.31
72.99
54.9
INT2
2.95
30.33
51.48
65.45
30.09
56.34
27.88
67.91
49.9
Table 3. Main comparison results of MolmoE-1B and DeepSeek-VL2-Tiny across eight multimodal benchmarks. The best results among mixed-precision quantization methods are highlighted in bold. Abbreviations: SEED: SEEDBench, SciQA: ScienceQA, RWQA: RealWorldQA, MME-RW: MME-RealWorld, OCR: OCRBench.
Figure 6. Accuracy-compression trade-off for MoMoVL variants and the skewed distribution of the combined importance metric. (a-b) Benchmark score as the INT4 threshold increases on (a) MolmoE-1B and (b) DeepSeek-VL2-Tiny. (c) Reverse Lorenz curve for MolmoE-1B with experts sorted in descending order of combined importance. Metrics computed from Top-K Weight and Top-K Index show Gini coefficients of 0.39 and 0.36, respectively. Accuracy-compression trade-off for two models and the distribution of activation statistics Both trade-offs for MolmoE and DeepSeek demonstrate that \moIresults in higher accuracy when given the same INT4 threshold. This is because activation frequency shows a more balanced distribution than gating weights.
Variant
Model Size (GB)
MMMU
MMBench
SEEDBench
w/o Visual (W)
3.43
30.00
55.22
67.33
w/o Visual (I)
3.35
30.89
55.79
67.12
w/o Modality (W)
3.27
30.33
53.71
66.43
w/o Modality (I)
3.21
29.56
53.35
66.29
w/o Scaling (W)
3.39
31.11
55.43
67.22
Table 4. Ablation study of modality-aware expert profile and sensitivity-based scaling on MolmoE-1B, with variants using Top-K Weight ( W ) or Top-K Index ( I ) as importance metric.
Method
Number of Samples
MMMU
MMBench
SEEDBench
MoMoVL -Weight
32
31.33
56.00
67.24
64
30.67
56.16
67.25
128 (default)
30.67
56.31
67.37
1000
31.00
56.52
67.39
MoMoVL -Index
32
28.89
55.74
66.96
Table 5. Effect of calibration set size on MoMoVL for MolmoE-1B. Calibration sets are hierarchically nested (i.e., C32⊂C64⊂C128⊂C1000 ) and generated via a fixed random seed.
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · 3Zhongguancun Academy