MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understanding

Bonan Zhang; Shiyu Dong; Quan Hung Tran; Katharina Gschwind; Shuqi Yang; Sijia Chen; Adel Ahmadyan; Seungwhan Moon

cs.CVAug 18, 2026

MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understanding

Authors: Bonan Zhang, Shiyu Dong, Quan Hung Tran, Katharina Gschwind, Shuqi Yang, Sijia Chen, Adel Ahmadyan, Seungwhan Moon,

Organizations: Meta · Work done while at Meta

Abstract

Vision encoders are a critical component of vision-language models, and scaling their capacity effectively improves performance. However, dense scaling increases compute cost and inference latency. Mixture-of-Experts (MoE) architectures offer a compelling alternative, having enabled efficient scaling in LLMs, yet the MoE design space for CLIP-style vision encoders remains underexplored at State-of-the-Art (SOTA) levels. In this work, we systematically study MoE designs for vision encoder scaling and find that fine-grained MoE topologies yield substantial gains over both dense and standard MoE counterparts. We further propose an auxiliary-loss-free balancing variant for better expert utilization, and design a specialized MoE kernel to mitigate inference latency overhead. To enhance video capabilities while preserving image knowledge, we introduce frame-level distillation paired with a novel freezing mechanism. We pretrain a series of Mixture-of-Experts Vision Encoders (MoE-ViE) across a range of sizes, all consistently outperforming their dense counterparts. Our largest model matches the zero-shot performance of a SOTA encoder 1.7x its size at 76% of its latency. When aligned with an LLM, MoE-ViE surpasses all compared encoders on image and video benchmarks, including those with up to 5x more activated parameters. Code is available at https://github.com/facebookresearch/moe_vie.

Figures & tables

Figure 1: Illustration of MoE-ViE architecture and training pipeline. MoE-ViE adopts a fine-grained MoE design with optimized MoE kernels for latency reduction. We first perform contrastive pretraining on large-scale image-text pairs. During video finetuning, we introduce frame-level distillation and expert freezing to preserve the pretrained image understanding capabilities.

			General Classification						Image Retrieval
Model	Total params	Activated params	Avg. Class.	ImageNet Val [ 22 ]	ImageNet v2 [ 60 ]	ObjectNet [ 4 ]	ImageNet Adversarial [ 33 ]	ImageNet Renditions [ 32 ]	Avg. Retrieval	COCO T $\rightarrow$ I [ 46 ]	COCO I $\rightarrow$ T [ 46 ]	Flickr30k T $\rightarrow$ I [ 88 ]	Flickr30k I $\rightarrow$ T [ 88 ]
ViT-B/32 dense	0.1B	0.1B	55.8	67.5	59.3	50.6	27.8	73.7	58.4	36.3	54.4	63.3	79.6
ViT-B/32 conventional MoE	0.5B	0.1B	59.9	70.6	62.3	54.3	34.7	77.7	59.7	37.8	56.3	64.4	80.2
MoE-ViE-B/32	0.5B	0.1B	63.2	72.9	64.9	57.7	40.4	80.0	61.4	38.9	56.6	66.5	83.5
ViT-L/16	0.3B	0.3B	78.0	79.7	72.6	74.9	72.4	90.5	68.9	45.8	63.8	75.9	90.3
ViT-L/16 conventional MoE	1.7B	0.3B	78.5	80.2	72.9	73.5	74.9	90.8	69.0	45.5	63.4	76.1	90.8

Table 1: Comparison of different MoE designs in vision encoders.

Figure 2: MoE-ViE vs. dense models at multiple scales. MoE-ViE consistently outperforms their dense counterparts under the same compute budget.

				Video Classification					Video Retrieval
Model	Total params	Activated params	Resolution	Avg. Class.	K400 [ 38 ]	K600 [ 38 ]	UCF101 [ 68 ]	HMDB [ 41 ]	Avg. Retrieval	MSR-VTT T $\rightarrow$ V [ 84 ]	MSR-VTT V $\rightarrow$ T [ 84 ]
SigLIP2-B/16 [ 75 ]	0.1B	0.1B	$224$	59.5	58.7	55.0	82.0	42.3	34.3	38.5	30.1
PE core B/16 [ 5 ]	0.1B	0.1B	$224$	65.9	65.6	65.1	84.6	48.2	47.5	47.6	47.3
MoE-ViE-B/16	0.5B	0.1B	$224$	67.5	68.3	65.6	84.5	51.5	47.6	47.9	47.2
SigLIP2-L/16 [ 75 ]	0.3B	0.3B	$384$	66.0	65.3	62.5	86.7	49.3	36.5	41.5	31.4
PE core L/14 [ 5 ]	0.3B	0.3B	$336$	72.9	73.4	72.7	87.1	58.5	50.2	50.3	50.1

Table 2: Comparison of zero-shot video benchmarks between MoE-ViE and other previous SOTA vision encoders in the literature.

				General Image Classification						Image Retrieval
Model	Total params	Activated params	Resolution	Avg. Class.	ImageNet Val [ 22 ]	ImageNet v2 [ 60 ]	ObjectNet [ 4 ]	ImageNet Adversarial [ 33 ]	ImageNet Renditions [ 32 ]	Avg. Retrieval	COCO T $\rightarrow$ I [ 46 ]	COCO I $\rightarrow$ T [ 46 ]	Flickr30k T $\rightarrow$ I [ 88 ]	Flickr30k I $\rightarrow$ T [ 88 ]
LIMOE-B/16 [ 54 ]	0.5B	0.1B	$224$	-	73.7	-	-	-	-	-	36.2	51.3	-	-
CLIP-UP-B/16 [ 79 ]	0.5B	0.2B	$224$	-	76.9	-	-	-	-	74.2	52.1	71.5	80.9	92.3
SigLIP2-B/16 [ 75 ]	0.1B	0.1B	$224$	74.0	78.2	71.4	73.6	55.0	91.7	73.7	52.1	68.9	80.7	93.0
PE core B/16 [ 5 ]	0.1B	0.1B	$224$	74.6	78.4	71.7	71.9	62.4	88.7	74.3	50.9	71.0	80.8	94.4
MoE-ViE-B/16	0.5B	0.1B	$224$	76.8	79.3	72.5	74.4	68.0	89.9	74.4	52.1	70.7	80.9	93.9

Table 3: Comparison of zero-shot image classification and retrieval between MoE-ViE and other SOTA vision encoders in the literature.

				Fine-grained Img Classification						OCR
Model	Total params	Activated params	Resolution	Avg. Class.	Food101 [ 6 ]	Flowers [ 56 ]	Country211 [ 73 ]	Aircrafts [ 49 ]	Cars [ 40 ]	Average OCR	Text Cap T $\rightarrow$ I [ 65 ]	Text Cap I $\rightarrow$ T [ 65 ]
SigLIP2-B/16 [ 75 ]	0.1B	0.1B	$224$	69.2	92.8	85.7	19.2	54.8	93.4	71.5	72.3	70.7
PE core B/16 [ 5 ]	0.1B	0.1B	$224$	71.7	92.5	86.5	30.5	57.0	92.1	71.6	72.3	70.9
MoE-ViE-B/16	0.5B	0.1B	$224$	73.5	94.2	87.3	34.9	58.2	93.1	72.5	72.7	72.3
SigLIP2-L/16 [ 75 ]	0.3B	0.3B	$384$	76.1	96.1	90.0	31.6	67.0	95.8	79.2	80.2	78.2
PE core L/14 [ 5 ]	0.3B	0.3B	$336$	78.1	96.2	87.2	45.6	67.8	93.7	79.2	79.8	78.5

Table 4: Comparison of zero-shot fine-grained image classification and OCR benchmarks between MoE-ViE and SOTA dense vision encoders. We re-evaluated part of the OCR results for SigLIP2 and PE if not reported in [ 75 ] and [ 5 ] .

		Image						Video				Captioning
Model	Activated params	Avg. Images	AI2D [ 39 ]	TextVQA [ 67 ]	ChartQA [ 51 ]	DocVQA [ 53 ]	Info. QA [ 52 ]	Avg. Video	VideoMME [ 27 ]	MVBench [ 44 ]	EgoSchema [ 50 ]	Avg. Cap.	COCO [ 46 ]	NoCaps [ 2 ]
MetaCLIP-G/14 [ 83 ]	1.8B	61.3	72.8	65.4	68.1	61.3	39.1	45.4	46.5	44.7	45.0	125.5	134.4	116.5
SigLIP2-g-opt/16 [ 75 ]	1.1B	59.0	72.4	70.3	63.1	55.3	34.0	49.5	46.2	48.5	53.8	129.1	137.8	120.3
PE core G/14 [ 5 ]	1.9B	69.2	69.7	74.3	73.4	81.2	47.6	48.6	46.0	48.7	51.2	123.7	134.5	112.9
InternViT2.5/14 [ 12 ]	5.5B	68.1	72.9	71.3	74.6	74.3	47.6	48.7	46.0	49.6	50.6	123.0	132.5	113.5
AIMv2 3B/14 [ 26 ]	2.7B	69.8	72.2	79.2	73.0	78.2	46.5	49.7	49.6	49.9	49.6	130.6	139.7	121.5

Table 5: Alignment comparison across models. Columns are grouped by task type. Llama 3.1 Instruct 8B [ 29 ] is used as the base LLM, and we use a maximum of 4 tiles.

		Image						Video				Captioning
Model	Activated params	Avg. Images	AI2D [ 39 ]	TextVQA [ 67 ]	ChartQA [ 51 ]	DocVQA [ 53 ]	Info. QA [ 52 ]	Avg. Video	VideoMME [ 27 ]	MVBench [ 44 ]	EgoSchema [ 50 ]	Avg. Cap.	COCO [ 46 ]	NoCaps [ 2 ]
SigLIP2-g-opt/16 [ 75 ]	1.1B	62.7	75.2	70.3	71.0	60.4	36.7	55.6	52.0	52.8	62.0	130.1	139.0	121.1
PE core G/14 [ 5 ]	1.9B	65.4	72.9	67.9	75.9	68.8	41.6	54.1	48.7	52.9	60.8	123.8	135.2	112.3
InternViT2.5/14 [ 12 ]	5.5B	65.4	73.6	70.1	78.2	65.3	39.6	52.7	50.3	51.1	56.6	128.5	138.4	118.6
AIMv2 3B/14 [ 26 ]	2.7B	67.6	75.2	74.2	76.7	70.5	41.4	52.7	45.9	51.4	60.8	130.6	139.2	122.0
MoE-ViE-H/14	1.1B	74.0	86.5	72.3	73.0	84.1	54.0	59.2	53.4	61.5	62.8	129.7	138.6	120.8

Table 6: Alignment comparison across models. Columns are grouped by task type. Qwen 2.5 VL 7B is used as the base LLM, and we use a maximum of 4 tiles.

	Activated Parameters	Total Parameters	Latency (ms)
Model	Activated Parameters	Total Parameters	bsz =16	32	64	128
SigLIP2-g-opt/16 [ 75 ]	1.1B	1.1B	76.39	141.78	275.94	549.62
PE core G/14 [ 5 ]	1.9B	1.9B	101.21	188.76	366.23	708.62
MoE-ViE-H/14 (Vanilla Implementation)	1.1B	3.5B	318.76	448.93	740.84	1306.81
MoE-ViE-H/14 (Optimized Kernel)	1.1B	3.5B	82.59	145.82	276.87	544.96

Table 7: Performance comparison between MoE-ViE and dense vision encoders, across different inference batch sizes ( bsz ). Due to difference in patch size, we report latency based on the same number of input tokens (# tokens = 576) for a fair comparison.

	General Classification						Image Retrieval
Auxiliary loss	Avg. Class.	ImageNet Val [ 22 ]	ImageNet v2 [ 60 ]	ObjectNet [ 4 ]	ImageNet Adversarial [ 33 ]	ImageNet Renditions [ 32 ]	Avg. Retrieval	COCO T $\rightarrow$ I [ 46 ]	COCO I $\rightarrow$ T [ 46 ]	Flickr30k T $\rightarrow$ I [ 88 ]	Flickr30k I $\rightarrow$ T [ 88 ]
Dense model	55.8	67.5	59.3	50.6	27.8	73.7	58.4	36.3	54.4	63.3	79.6
Importance and load loss [ 61 ]	59.9	70.6	62.3	54.3	34.7	77.7	59.7	37.8	56.3	64.4	80.2
Entropy loss [ 54 ]	59.9	70.7	62.2	53.9	35.5	77.1	60.0	38.5	56.4	65.0	80.1
Loss-free [ 77 ]	62.6	72.5	64.4	56.4	40.0	79.9	60.9	39.2	56.8	65.3	82.5
Ours	63.2	72.9	64.9	57.7	40.4	80.0	61.4	38.9	56.6	66.5	83.5

Table 8: Comparison of different auxiliary losses for training CLIP vision encoders.

Panel of Figure 3 — Figure 3: Ablations of different video finetuning methods. (a) Our proposed distillation and freezing techniques outperform finetuning with vanilla video data, or mixed image and video data; (b) We observe significant degradation if we apply frame-level distillation without freezing the text tower; (c) The combination of the proposed distillation and freezing methods yields the best results.

Panel of Figure 5 — Figure 5: MoE Expert activations in layer 20 of MoE-ViE-H/14. Experts specialize for distinct, semantically meaningful, aspects of the image.

Panel of Figure 6 — Figure 6: Expert activation across all data for a given task, final layers of MoE-ViE-H/14.

Benchmark	L0–5	L6–15	L16–25	L26–30	Uniform
VTAB Flowers [ 56 ]	3.20	3.17	2.83	2.30	3.47
FGVC Aircraft [ 49 ]	3.19	3.26	2.95	2.56	3.47
ImageNetV2 [ 60 ]	3.34	3.31	3.30	3.21	3.47
Flickr30k [ 88 ]	3.28	3.30	3.23	3.07	3.47
COCO [ 46 ]	3.30	3.32	3.30	3.18	3.5

Table 9: Routing entropy averaged over layer ranges.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

	General Classification						Image Retrieval
# of shared experts	Avg. Class.	ImageNet Val [ 22 ]	ImageNet v2 [ 60 ]	ObjectNet [ 4 ]	ImageNet Adversarial [ 33 ]	ImageNet Renditions [ 32 ]	Avg. Retrieval	COCO T $\rightarrow$ I [ 46 ]	COCO I $\rightarrow$ T [ 46 ]	Flickr30k T $\rightarrow$ I [ 88 ]	Flickr30k I $\rightarrow$ T [ 88 ]
0	62.1	72.5	64.2	56.1	38.4	79.5	61.1	38.8	56.4	66.2	83.0
1	63.2	72.9	64.9	57.7	40.4	80.0	61.4	38.9	56.6	66.5	83.5
2	63.0	73.2	64.5	57.9	39.7	79.9	61.3	39.1	56.4	66.3	83.6
4	62.5	72.6	64.2	57.2	38.8	79.5	60.2	37.5	56.5	65.0	81.7

Appendix

Table 10: Ablation on the number of shared experts.

	General Classification						Image Retrieval
$\lambda$	Avg. Class.	ImageNet Val [ 22 ]	ImageNet v2 [ 60 ]	ObjectNet [ 4 ]	ImageNet Adversarial [ 33 ]	ImageNet Renditions [ 32 ]	Avg. Retrieval	COCO T $\rightarrow$ I [ 46 ]	COCO I $\rightarrow$ T [ 46 ]	Flickr30k T $\rightarrow$ I [ 88 ]	Flickr30k I $\rightarrow$ T [ 88 ]
10	63.0	72.5	64.7	57.4	40.5	80.1	61.3	38.6	56.2	66.8	83.5
1	63.2	72.9	64.9	57.7	40.4	80.0	61.4	38.9	56.6	66.5	83.5
0.1	63.1	73.0	64.5	57.8	40.1	79.8	61.1	39.1	56.3	66.3	82.8
0.01	62.9	72.6	64.5	57.2	40.4	80.0	61.1	38.6	56.4	66.7	83.0

Appendix

Table 11: Ablation on the weight of shared experts.

	General Classification						Image Retrieval
$k$	Avg. Class.	ImageNet Val [ 22 ]	ImageNet v2 [ 60 ]	ObjectNet [ 4 ]	ImageNet Adversarial [ 33 ]	ImageNet Renditions [ 32 ]	Avg. Retrieval	COCO T $\rightarrow$ I [ 46 ]	COCO I $\rightarrow$ T [ 46 ]	Flickr30k T $\rightarrow$ I [ 88 ]	Flickr30k I $\rightarrow$ T [ 88 ]
4	62.4	72.6	64.3	56.9	38.0	80.0	61.0	38.4	56.0	66.3	83.2
8	63.2	72.9	64.9	57.7	40.4	80.0	61.4	38.9	56.6	66.5	83.5
16	63.1	73.0	64.8	57.5	40.3	79.8	61.4	39.1	56.5	66.3	83.6

Appendix

Table 12: Ablations of top-

k

	General Classification						Image Retrieval
$c$	Avg. Class.	ImageNet Val [ 22 ]	ImageNet v2 [ 60 ]	ObjectNet [ 4 ]	ImageNet Adversarial [ 33 ]	ImageNet Renditions [ 32 ]	Avg. Retrieval	COCO T $\rightarrow$ I [ 46 ]	COCO I $\rightarrow$ T [ 46 ]	Flickr30k T $\rightarrow$ I [ 88 ]	Flickr30k I $\rightarrow$ T [ 88 ]
1	59.9	70.6	62.3	54.3	34.7	77.7	59.7	37.8	56.3	64.4	80.2
2	61.0	71.1	62.6	55.0	37.2	78.9	60.3	38.0	56.4	64.9	82.0
4	63.2	72.9	64.9	57.7	40.4	80.0	61.4	38.9	56.6	66.5	83.5
8	62.8	72.7	64.2	57.9	39.6	79.5	61.0	39.0	56.1	65.9	83.1

Appendix

Table 13: Ablations of expert width factor

c

Panel of Figure 7 — Figure 7: Image classification accuracy by scaling the number of experts from 16 to 128, while keeping 8 activated experts.

Scale	B		L		H
Encoder	vision tower	text tower	vision tower	text tower	vision tower	text tower
Params	0.46B	0.30B	1.67B	0.30B	3.50B	0.81B
Params	(0.10B activated)	0.30B	(0.31B activated)	0.30B	(1.06B activated)	0.81B
Width	768	1024	1024	1024	1280	1408
Depth	12	24	24	24	32	34
MLP	512 $\times$ 32	2730	684 $\times$ 32	2736	854 $\times$ 32	3754

Appendix

Table 14: Configurations of MoE-ViE models.

Model	Active Params	ImageNet-1K
DINOv2-g [ 58 ]	1.1B	86.39
SigLIP2-g-opt/16 [ 75 ]	1.1B	88.32
PE core G/14 [ 5 ]	1.9B	88.80
InternViT-6B [ 13 ]	5.5B	88.44
MoE-ViE-H/14	1.1B	88.85

Appendix

Table 15: Linear probing results on ImageNet-1K.

	S1	S2a	S2b	S3
Trainable Params	Projector	LLM + Proj.	All	All
Steps	8,000	1,000	34,000	15,000
Global Batch Size	2,048	512	512	512
Total Samples	16.4M	0.5M	17.4M	7.7M
Learning Rate	$10^{-4}$	$4{\times}10^{-5}$	$4{\times}10^{-5}$	$10^{-5}$
Max Seq. Length	1,280	6,144	6,144	11,520

Appendix

Table 16: Training configuration overview across stages. Stage 2a is our 1k sample VE freeze step described in Section 4.2 .

			Image						Video				Captioning
Model	Training scale	Active params	Avg. Images	AI2D [ 39 ]	TextVQA [ 67 ]	ChartQA [ 51 ]	DocVQA [ 53 ]	Info. QA [ 52 ]	Avg. Video	VideoMME [ 27 ]	MVBench [ 44 ]	EgoSchema [ 50 ]	Avg. Cap.	COCO [ 46 ]	NoCaps [ 2 ]
SigLIP2-g-opt/16 [ 75 ]	–	1.1B	59.0	72.4	70.3	63.1	55.3	34.0	49.5	46.2	48.5	53.8	129.1	137.8	120.3
PE core G/14 [ 5 ]	–	1.9B	69.2	69.7	74.3	73.4	81.2	47.6	48.6	46.0	48.7	51.2	123.7	134.5	112.9
InternViT2.5/14 [ 12 ]	–	5.5B	68.1	72.9	71.3	74.6	74.3	47.6	48.7	46.0	49.6	50.6	123.0	132.5	113.5
AIMv2 3B/14 [ 26 ]	–	2.7B	69.8	72.2	79.2	73.0	78.2	46.5	49.7	49.6	49.9	49.6	130.6	139.7	121.5
MoE-ViE-H/14	14M	1.1B	75.2	84.9	75.7	74.2	86.1	55.0	58.9	52.0	63.6	61.2	127.5	135.4	119.6

Appendix

Table 17: Data scaling experiments with MoE-ViE. Columns are grouped by task type. We use a maximum of 4 tiles.

Model	Total params	Active params	Batch size	# of tiles	Latency (ms)
SigLIP2-g-opt/16 [ 75 ]	1.1B	1.1B	2	4	432.8
PE core G/14 [ 5 ]	1.9B	1.9B	2	4	493.2
MoE-ViE-H/14	3.5B	1.1B	2	4	433.3
SigLIP2-g-opt/16 [ 75 ]	1.1B	1.1B	2	16	523.4
PE core G/14 [ 5 ]	1.9B	1.9B	2	16	587.4
MoE-ViE-H/14	3.5B	1.1B	2	16	520.7

Appendix

Table 18: End-to-end latency with different vision encoders.

Efficient ViTs

Efficient Vision Transformers reduce the compute, memory, energy, or latency needed to process images and video while retaining useful accuracy.

See all 239 papers

VisMMOE: Exploiting Visual-Expert Affinity for Efficient Visual-Language MoE Offloading
May 7, 2026Cheng Xu, Xiaofeng Hou, Jiacheng Liu +1Expert Offloading Vision-Language Models
Shanghai Jiao Tong University Minhang Qu, Shanghai, China
DecoMoE: Decoupling Visual Propagation and Expert Computation for Efficient Multimodal MoE Inference
Sep 30, 2026Xudong Tan, Peng Ye, Ming Xie +4Mixture-of-Experts Pruning Efficient Multimodal Inference
Fudan University · The Chinese University of Hong Kong
CLEAR-MoE: Shared-Basis Expert Extraction from Frozen Vision Transformers via Calibration-Driven Layer Selection
Jun 26, 2026Md Irtiza Hossain, Humaira Ayesha, Junaid Ahmed SifatVision Transformer Low-Rank Matrix Decomposition
Brac University

May 7, 2026J/K move · Enter open · S save

VisMMOE: Exploiting Visual-Expert Affinity for Efficient Visual-Language MoE Offloading

Cheng Xu, Xiaofeng Hou, Jiacheng Liu, Chao Li

Shanghai Jiao Tong University Minhang Qu, Shanghai, China

Expert Offloading Vision-Language Models Efficient VLM Inference Mixture-of-Experts Inference Visual Token Compression

Open paper AskPDF arXiv

Large-scale vision-language mixture-of-experts (VL-MoE) models provide strong multimodal capability, but efficient deployment on memory-constrained platforms remains difficult. Existing MoE offloading systems are largely designed for text-centric workloads and become much less effective for visual-heavy inputs, where large numbers of visual tokens induce broader and less predictable expert accesses. We present VisMMoE, a VL-MoE offloading system built on a single systems insight: pruning redundant visual tokens can improve offloading not only by reducing computation, but also by reshaping expert demand. We refer to this effect as visual-expert affinity: token pruning makes expert accesses more concentrated within layers and more stable across layers, producing a smaller and more predictable expert working set. Guided by this insight, VisMMoE combines affinity-aware token compression, lookahead expert prediction, and cache/pipeline orchestration to improve expert locality and prefetch effectiveness under tight memory budgets. We implement VisMMoE on multiple frameworks and evaluate it on representative VL-MoE models and benchmarks. VisMMoE improves end-to-end inference performance by up to 2.68x and 1.61x, respectively, over strong baselines for today's VL-MoE deployments while maintaining competitive accuracy.

Abstract

Efficient ViTs

Explore similar work

VisMMOE: Exploiting Visual-Expert Affinity for Efficient Visual-Language MoE Offloading

DecoMoE: Decoupling Visual Propagation and Expert Computation for Efficient Multimodal MoE Inference

CLEAR-MoE: Shared-Basis Expert Extraction from Frozen Vision Transformers via Calibration-Driven Layer Selection