The rise of the agentic web has been accompanied by growing demand for on-device Vision-Language Models (VLMs) that perform diverse cross-modal web tasks with enhanced privacy and reduced latency. Mixture-of-Experts (MoE) models are widely adopted in modern VLMs for compute-efficient scalability, yet their multi-expert architecture poses deployment challenges due to substantial memory demands. Mixed-precision quantization is an effective compression technique that reduces bit-widths while adapting to the heterogeneous hardware budgets of edge devices, particularly well-suited for MoE due to its inherent structural sparsity, which causes imbalanced importance among experts. While prior work has applied quantization for efficient MoE inference at the edge, we point out that in MoE-based VLMs, the modality-specific characteristics of visual and text tokens further impact expert quantization. To address this issue, we present MoMoVL, a modality-aware adaptive compression framework for MoE-based VLMs that brings vision-language inference within reach of on-device resource budgets. Motivated by our empirical findings on modality-induced disparities across experts, MoMoVL jointly exploits the per-expert activation profile and per-modality quantization sensitivity to guide bit-width assignment. Experimental results show that MoMoVL improves the accuracy-compression trade-off relative to state-of-the-art mixed-precision baselines and reduces memory footprint by up to 15.44% with a modest drop in accuracy over its uniform-precision baseline. Our code and data are publicly available at https://github.com/sco-edge/MoMoVL.
Figures & tables
Figure 1. Architectural overview of an MoE-based VLM. Visual tokens and text tokens are processed with shared and routed experts using top-k routing. Overview of MoE-based VLM Input embeddings comprising visual and text tokens are processed with their routed experts based on gating weights, along with the shared experts that are always activated.
Figure 2. Modality-specific expert activation in MolmoE-1B on the COCO caption dataset. (a-b) Each cell shows the percentage of visual/text tokens routed to a given expert (summing to 800 % per layer due to top-8 routing); darker colors indicate more frequent activations. (c) Difference in percentage points; red and blue mark visual- and text-centric experts, respectively. All three panels share one color scale (0-61.5 %, ± 61.5 %p in (c)); the first 16 of 64 experts in native index order are shown. Three heatmaps of modality-specific expert activation. Expert activation pattern of MolmoE-1B is shown per-modality. Leftmost and center pane each show visual- and text-centric expert's activation frequency, which are distinct due to their modality-specific characteristics. The rightmost figure shows the subtraction of the two.
Figure 3. The trend of benchmark scores when increasing the proportion of experts quantized to INT4 in MolmoE-1B. Visual- and text-centric experts are selected according to their importance when processing visual tokens and text tokens, respectively. Baseline: MolmoE-1B in BF16 (59.48). Line plot with orange triangles (Text-Centric), blue circles (Visual-Centric), and a horizontal red dotted line (BF16 baseline) Quantizing text-centric experts in INT4 shows better accuracy overall than quantizing visual-centric experts in INT4. Setting INT4 proportion to 30% shows the largest gap of 2.65.
Figure 4. MoMoVL architecture: A workflow overview of an adaptive, modality-aware mixed-precision quantization framework for MoE-based VLMs. Workflow overview of MoMoVL At the first phase, per-expert activation statistics and per-modality quantization sensitivity are measured. At the second phase, the allocator assigns bit-width to each expert based on the collected offline statistics and the provided hardware budgets. At the third phase, mixed-precision quantization is performed on the target model based on the quantization configuration generated at the second phase.
Figure 5. Per-layer ratio of summed absolute gradients of text tokens over visual tokens in MolmoE-1B on the COCO caption dataset. Visual gradients vanish at the last layer due to loss masking and the absence of subsequent attention. Trend of gradient ratio, text over visual, across layers A line plot with blue circles is shown. Ranging from a minimum of 3.96x at Layer 4, then increasing up to Layer 14. The gradient ratio reaches infinity at Layer 15 as visual gradients become 0.
Metric
MolmoE-1B
DeepSeek-VL2-Tiny
Total Parameters
7.2B
3.4B
Expert Parameters
6.4B (89%)
2.5B (74%)
Experts per Layer (Routed / Shared)
64 / 0
64 / 2
Top-K Routing
8
6
Number of MoE Layers
16
11
Table 1. Configuration of evaluated MoE-based VLMs.
Benchmark
No. of Queries
No. of Tokens ( μ )
Token Ratio (Visual/Text)
Visual
Text
MMMU (val)
900
831.50
129.77
6.41x
MMBench-V1.1 (en test)
7299
478.85
93.26
5.13x
SEEDBench
14232
525.35
82.73
6.35x
ScienceQA (test)
2017
487.66
52.10
9.36x
Table 2. Statistics of evaluation benchmarks. Token counts are obtained by tokenizing queries with MolmoE-1B.
Model
Quantization Method
Precision
Model Size (GB)
MMMU
MMBench
SEED
SciQA
RWQA
MME-RW
AI2D
OCR
MolmoE-1B
-
BF16
13.45
32.22
59.48
68.57
34.80
58.69
33.87
73.41
55.0
Uniform (Full-Model)
INT8
7.20
32.00
59.74
68.67
34.61
58.95
33.35
73.77
55.2
INT4
4.08
31.11
58.18
68.06
34.41
56.73
32.57
72.96
54.6
Uniform (Expert-Only)
INT4
4.45
31.89
58.65
68.19
34.31
58.30
32.31
72.99
54.9
INT2
2.95
30.33
51.48
65.45
30.09
56.34
27.88
67.91
49.9
Table 3. Main comparison results of MolmoE-1B and DeepSeek-VL2-Tiny across eight multimodal benchmarks. The best results among mixed-precision quantization methods are highlighted in bold. Abbreviations: SEED: SEEDBench, SciQA: ScienceQA, RWQA: RealWorldQA, MME-RW: MME-RealWorld, OCR: OCRBench.
Figure 6. Accuracy-compression trade-off for MoMoVL variants and the skewed distribution of the combined importance metric. (a-b) Benchmark score as the INT4 threshold increases on (a) MolmoE-1B and (b) DeepSeek-VL2-Tiny. (c) Reverse Lorenz curve for MolmoE-1B with experts sorted in descending order of combined importance. Metrics computed from Top-K Weight and Top-K Index show Gini coefficients of 0.39 and 0.36, respectively. Accuracy-compression trade-off for two models and the distribution of activation statistics Both trade-offs for MolmoE and DeepSeek demonstrate that \moIresults in higher accuracy when given the same INT4 threshold. This is because activation frequency shows a more balanced distribution than gating weights.
Variant
Model Size (GB)
MMMU
MMBench
SEEDBench
w/o Visual (W)
3.43
30.00
55.22
67.33
w/o Visual (I)
3.35
30.89
55.79
67.12
w/o Modality (W)
3.27
30.33
53.71
66.43
w/o Modality (I)
3.21
29.56
53.35
66.29
w/o Scaling (W)
3.39
31.11
55.43
67.22
Table 4. Ablation study of modality-aware expert profile and sensitivity-based scaling on MolmoE-1B, with variants using Top-K Weight ( W ) or Top-K Index ( I ) as importance metric.
Method
Number of Samples
MMMU
MMBench
SEEDBench
MoMoVL -Weight
32
31.33
56.00
67.24
64
30.67
56.16
67.25
128 (default)
30.67
56.31
67.37
1000
31.00
56.52
67.39
MoMoVL -Index
32
28.89
55.74
66.96
Table 5. Effect of calibration set size on MoMoVL for MolmoE-1B. Calibration sets are hierarchically nested (i.e., C32⊂C64⊂C128⊂C1000 ) and generated via a fixed random seed.
Deploying vision-language models (VLMs) on mobile devices is challenging due to their significant memory and compute requirements. We present a framework for quantizing VLMs for efficient inference on resource-constrained hardware. Our approach combines a quantization pipeline that uses the model itself to generate training data and does not require access to the training setup, with a novel 2.7-bit-per-parameter format supporting efficient execution on Arm CPUs. We validate our approach by compressing the Llama 3.2 11B Vision Instruct model to 3.7 GB with 8-bit activations, preserving strong performance on a set of standard visual question answering tasks.
Mixture-of-Experts Multimodal Large Language Models (MoE-MLLMs) offer remarkable performance but incur prohibitive GPU memory costs, making compression essential. Among PTQ methods, expert-level mixed-precision quantization has proven effective for MoE-LLMs, yet suffers notable degradation on MoE-MLLMs due to two overlooked biases in expert importance estimation. (1) At the cross-modal level, the numerical dominance of vision tokens causes expert selection frequency to be dominated by vision tokens, masking experts that are critical to the text modality; (2) at the intra-vision level, the large proportion of redundant vision tokens further skew frequency statistics, obscuring experts critical for informative visual content. To bridge gaps, we propose MODE, a modality-decomposed expert-level mixed-precision quantization framework for MoE-MLLMs that decomposes expert selection frequency by modality, filters redundant vision tokens to obtain denoised visual frequency, and further evaluates quantization sensitivity per modality as a complementary signal to frequency-based estimation. These signals are integrated into an Integer Linear Programming formulation to assign per-expert bit-widths under a given budget. Extensive experiments show that MODE is particularly well-suited for MoE-MLLMs, limiting average performance loss to within 2.9% at W3A16, with larger gains at the extreme 2-bit setting.
Yuanteng Chen, Peisong Wang, Zhilei Liu +9
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · 3Zhongguancun Academy
Large-scale vision-language mixture-of-experts (VL-MoE) models provide strong multimodal capability, but efficient deployment on memory-constrained platforms remains difficult. Existing MoE offloading systems are largely designed for text-centric workloads and become much less effective for visual-heavy inputs, where large numbers of visual tokens induce broader and less predictable expert accesses. We present VisMMoE, a VL-MoE offloading system built on a single systems insight: pruning redundant visual tokens can improve offloading not only by reducing computation, but also by reshaping expert demand. We refer to this effect as visual-expert affinity: token pruning makes expert accesses more concentrated within layers and more stable across layers, producing a smaller and more predictable expert working set. Guided by this insight, VisMMoE combines affinity-aware token compression, lookahead expert prediction, and cache/pipeline orchestration to improve expert locality and prefetch effectiveness under tight memory budgets. We implement VisMMoE on multiple frameworks and evaluate it on representative VL-MoE models and benchmarks. VisMMoE improves end-to-end inference performance by up to 2.68x and 1.61x, respectively, over strong baselines for today's VL-MoE deployments while maintaining competitive accuracy.
Cheng Xu, Xiaofeng Hou, Jiacheng Liu +1
Shanghai Jiao Tong University Minhang Qu, Shanghai, China