Vision encoders are a critical component of vision-language models, and scaling their capacity effectively improves performance. However, dense scaling increases compute cost and inference latency. Mixture-of-Experts (MoE) architectures offer a compelling alternative, having enabled efficient scaling in LLMs, yet the MoE design space for CLIP-style vision encoders remains underexplored at State-of-the-Art (SOTA) levels. In this work, we systematically study MoE designs for vision encoder scaling and find that fine-grained MoE topologies yield substantial gains over both dense and standard MoE counterparts. We further propose an auxiliary-loss-free balancing variant for better expert utilization, and design a specialized MoE kernel to mitigate inference latency overhead. To enhance video capabilities while preserving image knowledge, we introduce frame-level distillation paired with a novel freezing mechanism. We pretrain a series of Mixture-of-Experts Vision Encoders (MoE-ViE) across a range of sizes, all consistently outperforming their dense counterparts. Our largest model matches the zero-shot performance of a SOTA encoder 1.7x its size at 76% of its latency. When aligned with an LLM, MoE-ViE surpasses all compared encoders on image and video benchmarks, including those with up to 5x more activated parameters. Code is available at https://github.com/facebookresearch/moe_vie.
Figures & tables
Figure 1: Illustration of MoE-ViE architecture and training pipeline. MoE-ViE adopts a fine-grained MoE design with optimized MoE kernels for latency reduction. We first perform contrastive pretraining on large-scale image-text pairs. During video finetuning, we introduce frame-level distillation and expert freezing to preserve the pretrained image understanding capabilities.
General Classification
Image Retrieval
Model
Total params
Activated params
Avg. Class.
ImageNet Val [ 22 ]
ImageNet v2 [ 60 ]
ObjectNet [ 4 ]
ImageNet Adversarial [ 33 ]
ImageNet Renditions [ 32 ]
Avg. Retrieval
COCO T → I [ 46 ]
COCO I → T [ 46 ]
Flickr30k T → I [ 88 ]
Flickr30k I → T [ 88 ]
ViT-B/32 dense
0.1B
0.1B
55.8
67.5
59.3
50.6
27.8
73.7
58.4
36.3
54.4
63.3
79.6
ViT-B/32 conventional MoE
0.5B
0.1B
59.9
70.6
62.3
54.3
34.7
77.7
59.7
37.8
56.3
64.4
80.2
MoE-ViE-B/32
0.5B
0.1B
63.2
72.9
64.9
57.7
40.4
80.0
61.4
38.9
56.6
66.5
83.5
ViT-L/16
0.3B
0.3B
78.0
79.7
72.6
74.9
72.4
90.5
68.9
45.8
63.8
75.9
90.3
ViT-L/16 conventional MoE
1.7B
0.3B
78.5
80.2
72.9
73.5
74.9
90.8
69.0
45.5
63.4
76.1
90.8
Table 1: Comparison of different MoE designs in vision encoders.
Figure 2: MoE-ViE vs. dense models at multiple scales. MoE-ViE consistently outperforms their dense counterparts under the same compute budget.
Video Classification
Video Retrieval
Model
Total params
Activated params
Resolution
Avg. Class.
K400 [ 38 ]
K600 [ 38 ]
UCF101 [ 68 ]
HMDB [ 41 ]
Avg. Retrieval
MSR-VTT T → V [ 84 ]
MSR-VTT V → T [ 84 ]
SigLIP2-B/16 [ 75 ]
0.1B
0.1B
224
59.5
58.7
55.0
82.0
42.3
34.3
38.5
30.1
PE core B/16 [ 5 ]
0.1B
0.1B
224
65.9
65.6
65.1
84.6
48.2
47.5
47.6
47.3
MoE-ViE-B/16
0.5B
0.1B
224
67.5
68.3
65.6
84.5
51.5
47.6
47.9
47.2
SigLIP2-L/16 [ 75 ]
0.3B
0.3B
384
66.0
65.3
62.5
86.7
49.3
36.5
41.5
31.4
PE core L/14 [ 5 ]
0.3B
0.3B
336
72.9
73.4
72.7
87.1
58.5
50.2
50.3
50.1
Table 2: Comparison of zero-shot video benchmarks between MoE-ViE and other previous SOTA vision encoders in the literature.
General Image Classification
Image Retrieval
Model
Total params
Activated params
Resolution
Avg. Class.
ImageNet Val [ 22 ]
ImageNet v2 [ 60 ]
ObjectNet [ 4 ]
ImageNet Adversarial [ 33 ]
ImageNet Renditions [ 32 ]
Avg. Retrieval
COCO T → I [ 46 ]
COCO I → T [ 46 ]
Flickr30k T → I [ 88 ]
Flickr30k I → T [ 88 ]
LIMOE-B/16 [ 54 ]
0.5B
0.1B
224
-
73.7
-
-
-
-
-
36.2
51.3
-
-
CLIP-UP-B/16 [ 79 ]
0.5B
0.2B
224
-
76.9
-
-
-
-
74.2
52.1
71.5
80.9
92.3
SigLIP2-B/16 [ 75 ]
0.1B
0.1B
224
74.0
78.2
71.4
73.6
55.0
91.7
73.7
52.1
68.9
80.7
93.0
PE core B/16 [ 5 ]
0.1B
0.1B
224
74.6
78.4
71.7
71.9
62.4
88.7
74.3
50.9
71.0
80.8
94.4
MoE-ViE-B/16
0.5B
0.1B
224
76.8
79.3
72.5
74.4
68.0
89.9
74.4
52.1
70.7
80.9
93.9
Table 3: Comparison of zero-shot image classification and retrieval between MoE-ViE and other SOTA vision encoders in the literature.
Fine-grained Img Classification
OCR
Model
Total params
Activated params
Resolution
Avg. Class.
Food101 [ 6 ]
Flowers [ 56 ]
Country211 [ 73 ]
Aircrafts [ 49 ]
Cars [ 40 ]
Average OCR
Text Cap T → I [ 65 ]
Text Cap I → T [ 65 ]
SigLIP2-B/16 [ 75 ]
0.1B
0.1B
224
69.2
92.8
85.7
19.2
54.8
93.4
71.5
72.3
70.7
PE core B/16 [ 5 ]
0.1B
0.1B
224
71.7
92.5
86.5
30.5
57.0
92.1
71.6
72.3
70.9
MoE-ViE-B/16
0.5B
0.1B
224
73.5
94.2
87.3
34.9
58.2
93.1
72.5
72.7
72.3
SigLIP2-L/16 [ 75 ]
0.3B
0.3B
384
76.1
96.1
90.0
31.6
67.0
95.8
79.2
80.2
78.2
PE core L/14 [ 5 ]
0.3B
0.3B
336
78.1
96.2
87.2
45.6
67.8
93.7
79.2
79.8
78.5
Table 4: Comparison of zero-shot fine-grained image classification and OCR benchmarks between MoE-ViE and SOTA dense vision encoders. We re-evaluated part of the OCR results for SigLIP2 and PE if not reported in [ 75 ] and [ 5 ] .
Image
Video
Captioning
Model
Activated params
Avg. Images
AI2D [ 39 ]
TextVQA [ 67 ]
ChartQA [ 51 ]
DocVQA [ 53 ]
Info. QA [ 52 ]
Avg. Video
VideoMME [ 27 ]
MVBench [ 44 ]
EgoSchema [ 50 ]
Avg. Cap.
COCO [ 46 ]
NoCaps [ 2 ]
MetaCLIP-G/14 [ 83 ]
1.8B
61.3
72.8
65.4
68.1
61.3
39.1
45.4
46.5
44.7
45.0
125.5
134.4
116.5
SigLIP2-g-opt/16 [ 75 ]
1.1B
59.0
72.4
70.3
63.1
55.3
34.0
49.5
46.2
48.5
53.8
129.1
137.8
120.3
PE core G/14 [ 5 ]
1.9B
69.2
69.7
74.3
73.4
81.2
47.6
48.6
46.0
48.7
51.2
123.7
134.5
112.9
InternViT2.5/14 [ 12 ]
5.5B
68.1
72.9
71.3
74.6
74.3
47.6
48.7
46.0
49.6
50.6
123.0
132.5
113.5
AIMv2 3B/14 [ 26 ]
2.7B
69.8
72.2
79.2
73.0
78.2
46.5
49.7
49.6
49.9
49.6
130.6
139.7
121.5
Table 5: Alignment comparison across models. Columns are grouped by task type. Llama 3.1 Instruct 8B [ 29 ] is used as the base LLM, and we use a maximum of 4 tiles.
Image
Video
Captioning
Model
Activated params
Avg. Images
AI2D [ 39 ]
TextVQA [ 67 ]
ChartQA [ 51 ]
DocVQA [ 53 ]
Info. QA [ 52 ]
Avg. Video
VideoMME [ 27 ]
MVBench [ 44 ]
EgoSchema [ 50 ]
Avg. Cap.
COCO [ 46 ]
NoCaps [ 2 ]
SigLIP2-g-opt/16 [ 75 ]
1.1B
62.7
75.2
70.3
71.0
60.4
36.7
55.6
52.0
52.8
62.0
130.1
139.0
121.1
PE core G/14 [ 5 ]
1.9B
65.4
72.9
67.9
75.9
68.8
41.6
54.1
48.7
52.9
60.8
123.8
135.2
112.3
InternViT2.5/14 [ 12 ]
5.5B
65.4
73.6
70.1
78.2
65.3
39.6
52.7
50.3
51.1
56.6
128.5
138.4
118.6
AIMv2 3B/14 [ 26 ]
2.7B
67.6
75.2
74.2
76.7
70.5
41.4
52.7
45.9
51.4
60.8
130.6
139.2
122.0
MoE-ViE-H/14
1.1B
74.0
86.5
72.3
73.0
84.1
54.0
59.2
53.4
61.5
62.8
129.7
138.6
120.8
Table 6: Alignment comparison across models. Columns are grouped by task type. Qwen 2.5 VL 7B is used as the base LLM, and we use a maximum of 4 tiles.
Activated Parameters
Total Parameters
Latency (ms)
Model
bsz =16
32
64
128
SigLIP2-g-opt/16 [ 75 ]
1.1B
1.1B
76.39
141.78
275.94
549.62
PE core G/14 [ 5 ]
1.9B
1.9B
101.21
188.76
366.23
708.62
MoE-ViE-H/14 (Vanilla Implementation)
1.1B
3.5B
318.76
448.93
740.84
1306.81
MoE-ViE-H/14 (Optimized Kernel)
1.1B
3.5B
82.59
145.82
276.87
544.96
Table 7: Performance comparison between MoE-ViE and dense vision encoders, across different inference batch sizes ( bsz ). Due to difference in patch size, we report latency based on the same number of input tokens (# tokens = 576) for a fair comparison.
General Classification
Image Retrieval
Auxiliary loss
Avg. Class.
ImageNet Val [ 22 ]
ImageNet v2 [ 60 ]
ObjectNet [ 4 ]
ImageNet Adversarial [ 33 ]
ImageNet Renditions [ 32 ]
Avg. Retrieval
COCO T → I [ 46 ]
COCO I → T [ 46 ]
Flickr30k T → I [ 88 ]
Flickr30k I → T [ 88 ]
Dense model
55.8
67.5
59.3
50.6
27.8
73.7
58.4
36.3
54.4
63.3
79.6
Importance and load loss [ 61 ]
59.9
70.6
62.3
54.3
34.7
77.7
59.7
37.8
56.3
64.4
80.2
Entropy loss [ 54 ]
59.9
70.7
62.2
53.9
35.5
77.1
60.0
38.5
56.4
65.0
80.1
Loss-free [ 77 ]
62.6
72.5
64.4
56.4
40.0
79.9
60.9
39.2
56.8
65.3
82.5
Ours
63.2
72.9
64.9
57.7
40.4
80.0
61.4
38.9
56.6
66.5
83.5
Table 8: Comparison of different auxiliary losses for training CLIP vision encoders.
Figure 3: Ablations of different video finetuning methods. (a) Our proposed distillation and freezing techniques outperform finetuning with vanilla video data, or mixed image and video data; (b) We observe significant degradation if we apply frame-level distillation without freezing the text tower; (c) The combination of the proposed distillation and freezing methods yields the best results.
Figure 5: MoE Expert activations in layer 20 of MoE-ViE-H/14. Experts specialize for distinct, semantically meaningful, aspects of the image.
Figure 6: Expert activation across all data for a given task, final layers of MoE-ViE-H/14.
Benchmark
L0–5
L6–15
L16–25
L26–30
Uniform
VTAB Flowers [ 56 ]
3.20
3.17
2.83
2.30
3.47
FGVC Aircraft [ 49 ]
3.19
3.26
2.95
2.56
3.47
ImageNetV2 [ 60 ]
3.34
3.31
3.30
3.21
3.47
Flickr30k [ 88 ]
3.28
3.30
3.23
3.07
3.47
COCO [ 46 ]
3.30
3.32
3.30
3.18
3.5
Table 9: Routing entropy averaged over layer ranges.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
General Classification
Image Retrieval
# of shared experts
Avg. Class.
ImageNet Val [ 22 ]
ImageNet v2 [ 60 ]
ObjectNet [ 4 ]
ImageNet Adversarial [ 33 ]
ImageNet Renditions [ 32 ]
Avg. Retrieval
COCO T → I [ 46 ]
COCO I → T [ 46 ]
Flickr30k T → I [ 88 ]
Flickr30k I → T [ 88 ]
0
62.1
72.5
64.2
56.1
38.4
79.5
61.1
38.8
56.4
66.2
83.0
1
63.2
72.9
64.9
57.7
40.4
80.0
61.4
38.9
56.6
66.5
83.5
2
63.0
73.2
64.5
57.9
39.7
79.9
61.3
39.1
56.4
66.3
83.6
4
62.5
72.6
64.2
57.2
38.8
79.5
60.2
37.5
56.5
65.0
81.7
Appendix
Table 10: Ablation on the number of shared experts.
General Classification
Image Retrieval
λ
Avg. Class.
ImageNet Val [ 22 ]
ImageNet v2 [ 60 ]
ObjectNet [ 4 ]
ImageNet Adversarial [ 33 ]
ImageNet Renditions [ 32 ]
Avg. Retrieval
COCO T → I [ 46 ]
COCO I → T [ 46 ]
Flickr30k T → I [ 88 ]
Flickr30k I → T [ 88 ]
10
63.0
72.5
64.7
57.4
40.5
80.1
61.3
38.6
56.2
66.8
83.5
1
63.2
72.9
64.9
57.7
40.4
80.0
61.4
38.9
56.6
66.5
83.5
0.1
63.1
73.0
64.5
57.8
40.1
79.8
61.1
39.1
56.3
66.3
82.8
0.01
62.9
72.6
64.5
57.2
40.4
80.0
61.1
38.6
56.4
66.7
83.0
Appendix
Table 11: Ablation on the weight of shared experts.
General Classification
Image Retrieval
k
Avg. Class.
ImageNet Val [ 22 ]
ImageNet v2 [ 60 ]
ObjectNet [ 4 ]
ImageNet Adversarial [ 33 ]
ImageNet Renditions [ 32 ]
Avg. Retrieval
COCO T → I [ 46 ]
COCO I → T [ 46 ]
Flickr30k T → I [ 88 ]
Flickr30k I → T [ 88 ]
4
62.4
72.6
64.3
56.9
38.0
80.0
61.0
38.4
56.0
66.3
83.2
8
63.2
72.9
64.9
57.7
40.4
80.0
61.4
38.9
56.6
66.5
83.5
16
63.1
73.0
64.8
57.5
40.3
79.8
61.4
39.1
56.5
66.3
83.6
Appendix
Table 12: Ablations of top- k .
General Classification
Image Retrieval
c
Avg. Class.
ImageNet Val [ 22 ]
ImageNet v2 [ 60 ]
ObjectNet [ 4 ]
ImageNet Adversarial [ 33 ]
ImageNet Renditions [ 32 ]
Avg. Retrieval
COCO T → I [ 46 ]
COCO I → T [ 46 ]
Flickr30k T → I [ 88 ]
Flickr30k I → T [ 88 ]
1
59.9
70.6
62.3
54.3
34.7
77.7
59.7
37.8
56.3
64.4
80.2
2
61.0
71.1
62.6
55.0
37.2
78.9
60.3
38.0
56.4
64.9
82.0
4
63.2
72.9
64.9
57.7
40.4
80.0
61.4
38.9
56.6
66.5
83.5
8
62.8
72.7
64.2
57.9
39.6
79.5
61.0
39.0
56.1
65.9
83.1
Appendix
Table 13: Ablations of expert width factor c .
Figure 7: Image classification accuracy by scaling the number of experts from 16 to 128, while keeping 8 activated experts.
Scale
B
L
H
Encoder
vision tower
text tower
vision tower
text tower
vision tower
text tower
Params
0.46B
0.30B
1.67B
0.30B
3.50B
0.81B
(0.10B activated)
(0.31B activated)
(1.06B activated)
Width
768
1024
1024
1024
1280
1408
Depth
12
24
24
24
32
34
MLP
512 × 32
2730
684 × 32
2736
854 × 32
3754
Appendix
Table 14: Configurations of MoE-ViE models.
Model
Active Params
ImageNet-1K
DINOv2-g [ 58 ]
1.1B
86.39
SigLIP2-g-opt/16 [ 75 ]
1.1B
88.32
PE core G/14 [ 5 ]
1.9B
88.80
InternViT-6B [ 13 ]
5.5B
88.44
MoE-ViE-H/14
1.1B
88.85
Appendix
Table 15: Linear probing results on ImageNet-1K.
S1
S2a
S2b
S3
Trainable Params
Projector
LLM + Proj.
All
All
Steps
8,000
1,000
34,000
15,000
Global Batch Size
2,048
512
512
512
Total Samples
16.4M
0.5M
17.4M
7.7M
Learning Rate
10−4
4×10−5
4×10−5
10−5
Max Seq. Length
1,280
6,144
6,144
11,520
Appendix
Table 16: Training configuration overview across stages. Stage 2a is our 1k sample VE freeze step described in Section 4.2 .
Image
Video
Captioning
Model
Training scale
Active params
Avg. Images
AI2D [ 39 ]
TextVQA [ 67 ]
ChartQA [ 51 ]
DocVQA [ 53 ]
Info. QA [ 52 ]
Avg. Video
VideoMME [ 27 ]
MVBench [ 44 ]
EgoSchema [ 50 ]
Avg. Cap.
COCO [ 46 ]
NoCaps [ 2 ]
SigLIP2-g-opt/16 [ 75 ]
–
1.1B
59.0
72.4
70.3
63.1
55.3
34.0
49.5
46.2
48.5
53.8
129.1
137.8
120.3
PE core G/14 [ 5 ]
–
1.9B
69.2
69.7
74.3
73.4
81.2
47.6
48.6
46.0
48.7
51.2
123.7
134.5
112.9
InternViT2.5/14 [ 12 ]
–
5.5B
68.1
72.9
71.3
74.6
74.3
47.6
48.7
46.0
49.6
50.6
123.0
132.5
113.5
AIMv2 3B/14 [ 26 ]
–
2.7B
69.8
72.2
79.2
73.0
78.2
46.5
49.7
49.6
49.9
49.6
130.6
139.7
121.5
MoE-ViE-H/14
14M
1.1B
75.2
84.9
75.7
74.2
86.1
55.0
58.9
52.0
63.6
61.2
127.5
135.4
119.6
Appendix
Table 17: Data scaling experiments with MoE-ViE. Columns are grouped by task type. We use a maximum of 4 tiles.
Model
Total params
Active params
Batch size
# of tiles
Latency (ms)
SigLIP2-g-opt/16 [ 75 ]
1.1B
1.1B
2
4
432.8
PE core G/14 [ 5 ]
1.9B
1.9B
2
4
493.2
MoE-ViE-H/14
3.5B
1.1B
2
4
433.3
SigLIP2-g-opt/16 [ 75 ]
1.1B
1.1B
2
16
523.4
PE core G/14 [ 5 ]
1.9B
1.9B
2
16
587.4
MoE-ViE-H/14
3.5B
1.1B
2
16
520.7
Appendix
Table 18: End-to-end latency with different vision encoders.
Large-scale vision-language mixture-of-experts (VL-MoE) models provide strong multimodal capability, but efficient deployment on memory-constrained platforms remains difficult. Existing MoE offloading systems are largely designed for text-centric workloads and become much less effective for visual-heavy inputs, where large numbers of visual tokens induce broader and less predictable expert accesses. We present VisMMoE, a VL-MoE offloading system built on a single systems insight: pruning redundant visual tokens can improve offloading not only by reducing computation, but also by reshaping expert demand. We refer to this effect as visual-expert affinity: token pruning makes expert accesses more concentrated within layers and more stable across layers, producing a smaller and more predictable expert working set. Guided by this insight, VisMMoE combines affinity-aware token compression, lookahead expert prediction, and cache/pipeline orchestration to improve expert locality and prefetch effectiveness under tight memory budgets. We implement VisMMoE on multiple frameworks and evaluate it on representative VL-MoE models and benchmarks. VisMMoE improves end-to-end inference performance by up to 2.68x and 1.61x, respectively, over strong baselines for today's VL-MoE deployments while maintaining competitive accuracy.
Cheng Xu, Xiaofeng Hou, Jiacheng Liu +1
Shanghai Jiao Tong University Minhang Qu, Shanghai, China
Multimodal mixture-of-experts (MoE) models combine sparse expert activation with visual-language capabilities, yet their inference remains costly because long visual-token sequences repeatedly incur attention, routing, dispatch, and expert-MLP computation. Existing methods typically compress either the token or expert dimension, leaving redundancy along the other. Our analysis reveals two complementary regularities: the depth required for visual propagation varies across inputs, while text-token routing exhibits concentrated and recurrent expert-importance patterns. Based on these observations, we propose DecoMoE, a two-dimensional structured compression framework that decouples visual propagation from expert computation. The Sample-Adaptive Visual Boundary (SAVB) predicts an input-dependent visual-exit layer at which the visual-token block is removed. The Routing-Calibrated Expert Prefix (RCEP) reorders experts offline using text-token routed mass and, from this predicted exit layer onward, retains at each MoE layer the shortest contiguous prefix covering a target routed-mass fraction. We evaluate DecoMoE on Qwen3-VL-MoE and InternVL3.5-30B-A3B across six benchmarks. On Qwen3-VL-MoE, DecoMoE retains 97.91% of dense-baseline performance while reducing computation from 27.06 to 16.73 TFLOPs and latency from 0.44 to 0.26 seconds, yielding a 1.69x speedup. Code will be available at https://github.com/ShawnTan86/DecoMoE.
Xudong Tan, Peng Ye, Ming Xie +4
Fudan University · The Chinese University of Hong Kong
We present CLEAR-MoE, a four-phase post-training pipeline that converts a frozen pretrained Vision Transformer (ViT) into a sparse Mixture-of-Experts (MoE) model without updating backbone weights. The pipeline (i) scores feed-forward network (FFN) layers by sparsity, clusterability, and output sensitivity; (ii) decomposes selected layers into a shared low-rank SVD basis and per-cluster residual experts using k-means clustering; (iii) trains lightweight routers supervised by cluster labels; and (iv) dispatches tokens through pluggable CUDA backends. On Imagenette with DeiT-Small, CLEAR-MoE retains 99.9% of the dense model's accuracy (86.70 +/- 0.02% versus 86.73%). Extensive ablation studies reveal a consistent empirical finding: the shared SVD basis is the primary factor responsible for preserving accuracy. Random routing, learned routing, and three different router architectures produce nearly identical performance, with accuracy varying by at most 0.06 percentage points (86.62%-86.68%). Accuracy also remains stable across different SVD ranks, expert counts (2-8), calibration set sizes (50-500), and random seeds. This behavior generalizes across five ViT backbones (DeiT-Tiny, DeiT-Small, DeiT-Base, ViT-Small, and ViT-Base), covering models from 5.7M to 86.6M parameters, with accuracy differences <= 0.10 percentage points from their dense counterparts. On a GTX 960 GPU, routing and scatter-gather overhead make the CLEAR-MoE FFN 1.3-1.7x slower than the dense implementation. A dispatch microbenchmark further shows that routing is an order of magnitude more memory-bound than expert matrix multiplications, identifying fused dispatch kernels as a promising direction for future optimization.
Md Irtiza Hossain, Humaira Ayesha, Junaid Ahmed Sifat