OmniMoE-VL: A Sparse Vision-Language Model with Coupled Visual-Depth Routing

Long Qian; Bingke Zhu; Jiaqi Wei; Yingying Chen; Jinqiao Wang

cs.CVSep 26, 2026

OmniMoE-VL: A Sparse Vision-Language Model with Coupled Visual-Depth Routing

Authors: Long Qian, Bingke Zhu, Jiaqi Wei, Yingying Chen, Jinqiao Wang

Organizations: Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences · School of Future Technology, University of Chinese Academy of Sciences · School of Engineering, Cardiff University · Wuhan AI Research

Abstract

Vision-language models (VLMs) increasingly use sparse mixture-of-experts (MoE) to scale language-side computation, yet visual information is typically routed only after passing through a fixed cross-modal interface. This leaves an important decision unresolved: which intermediate visual representations should be exposed to language computation for a given question? We introduce OmniMoE-VL, a sparse VLM with a coupled visual-depth routed projector. For each image-prompt pair, the projector selects a sparse set of intermediate visual depths and reuses the resulting global preference to guide both local patch fusion and dynamic visual injection into the language model. This design enables question-dependent visual access while preserving the native visual-token sequence, and complements token-level expert routing in the vision and language stacks. Across eight image-based benchmarks, OmniMoE-VL achieves an average score of 85.9 with 28B total and 9B activated parameters. Controlled comparisons show that the routed visual interface provides the dominant architectural gain, while matched route and component controls, same-image route analysis, and route interventions further support the value of coupling and question-conditioned visual access.

Figures & tables

Figure 1: Question-dependent visual evidence motivates depth selection. The first question requires label recognition and spatial context to identify the shampoo and its right-hand neighbor; the second requires the label on the rightmost bottle. A fixed interface supplies the same visual source to language experts across questions. Prompt-conditioned depth selection combines intermediate streams according to the evidence needed, connecting fine detail, object context, and spatial relations. Colors track the contributing visual streams.

Figure 2: OmniMoE-VL couples visual-depth selection with fusion and language-layer updates. (a) The model routes ten projected visual streams through one global depth distribution $w$ . (b) Sixteen softly weighted prototypes form eight queries for top-4 depth selection. (c) The route supplies a prior for group-local weights $g_{b}$ , which fuse aligned features into 576 visual tokens. (d) The same $w$ scales rank-32 updates at fixed language layers $a(s)=s$ . Solid arrows carry features; dashed peach arrows carry shared routing weights.

Config	Stage I	Stage II	Stage III	Stage IV
Vision MoE (experts / top- $k$ )	-	-	8 / 2	8 / 1
LLM MoE (experts / top- $k$ )	-	8 / 2	8 / 2	8 / 2
Projector MoE (scales / top- $k$ )	10 / 4	10 / 4	10 / 4	10 / 4
Trainable MoE params	Projector only	LLM only	Vision only	Projector + LLM + Vision
Data	OMVL-16M-Alignment	OMVL-16M-LLM	OMVL-16M-Vision	OMVL-16M-Joint
Image resolution	$384\times 384$

Table 1: Training configs for OmniMoE-VL .

Dense models
Methods	# Params	# Act.	Benchmarks								Avg.
Methods	# Params	# Act.	MMBench	MMVet	$VQA^{T}$	$VQA^{D}$	POPE	MMMU	OCRBench	ChartQA	Avg.
Qwen3-VL-8B-Instruct	8B	8B	85.0	-	82.9 †	-	-	69.6	89.6	89.6	83.3 (5)
GLM-4.6V-Flash	9B	9B	86.9	-	-	-	-	71.1	84.7	-	80.9 (3)
Qwen2.5-VL-7B	7B	7B	83.5	67.1	84.9	95.7	86.4	58.6	86.4	87.3	81.2 (8)
Sparse models

Table 2: Eight-benchmark comparison. Parameters are in billions;

VQA^{T}

VQA^{D}

denote TextVQA/DocVQA. Avg. uses listed scores; superscripts give their count.

\dagger

marks independent evaluations (Appendix B.4 ). Bold/underlined scores denote first/second place.

Method	Avg.
MoE-LLaVA ( Lin et al., 2026 )	61.7
EvoMoE ( Jing et al., 2025 )	69.0
OmniMoE-VL (w/o. Visual stack MoE)	83.7
Full OmniMoE-VL	84.1

Table 3: Public-recipe results and matched projector controls. Left: comparison scores (benchmark coverage in Appendix B.3 ). Right: shared versus independent routes at matched parameter count, followed by component replacements. Visual experts are disabled throughout (b).

Variant	V	P	L	Avg.
LLM-only MoE	✗	✗	✓	$80.3^{\ddagger}$
LLM + Projector MoE	✗	✓	✓	85.2
Full OmniMoE-VL	✓	✓	✓	85.9

Table 4: Architectural ablations. (a) Stack placement across eight benchmarks: V/P/L denote vision/projector/language experts;

\ddagger

marks the aggregate-derived mean (Appendix B.2 ). (b) Projector removals under the component protocol: Q/R/I denote query control, depth routing, and visual injection.

Comparison	Images	Targets	Qwen3-VL: $D_{\mathrm{prof}}$			OmniMoE-VL: $D_{\mathrm{TV}}$
Comparison	Images	Targets	Different	Paraphrase	$\Delta D$	Different	Paraphrase	$\Delta D$
Fixed target	283	322	0.257	0.171	0.086	0.301	0.162	0.139
+ matched structural type	136	174	0.251	0.169	0.083	0.282	0.159	0.123
+ matched answer length	193	214	0.253	0.193	0.060	0.264	0.171	0.093
+ identical reference answer	44	53	0.137	0.089	0.049	0.217	0.145	0.072
Route intervention: reference-answer $\Delta$ NLL (nats/token)

Table 5: Question-conditioned visual access. For same-image comparisons, Qwen3-VL depth-profile distances are measured in nats/token, while OmniMoE-VL route distances use total variation (TV).

\Delta D=D_{\mathrm{different}}-D_{\mathrm{paraphrase}}

. The lower block reports route interventions in OmniMoE-VL while holding the image, target question, and non-route inputs fixed.

Variant	Clean	Shifted	Drop
LLM-only MoE	81.3	77.4	4.8%
LLM + Projector MoE	87.0	85.8	1.4%
Full OmniMoE-VL	87.6	86.8	0.9%

Table 6: Robustness and deployment. (a) Clean/degraded scores under down-sampling and blur; drop is relative to clean accuracy. (b) Peak VRAM and decoding throughput on two H800 GPUs, BF16,

384\times 384

images (Appendix B.5 ).

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Item	Setting
Vision backbone	SigLIP2-Giant, patch size 16, input $384\times 384$
Language backbone	Qwen3-8B; hidden width $d=4096$
Visual depth taps	$1,5,9,13,17,21,25,29,33,37$ (one-based)
Vision parameter experts	8 experts, top-1, even-indexed blocks
Language parameter experts	8 experts, top-2, even-indexed blocks
Query bank	$Q=16$ ; probability floor $10^{-4}$

Appendix

Table 7: Current full-model implementation. Visual-depth routing and parameter-expert routing operate on different axes.

Route	Unit	Candidates	Role
$p$	Prompt	16 queries	Query participation
$w$	Image–prompt pair	10 depths, top-4	Shared global depth selection
$g_{b}$	Patch group	10 depths, top-4	Local feature fusion
Vision experts	Token	8 experts, top-1	Visual feature processing
Language experts	Token	8 experts, top-2	Language-state processing

Appendix

Table 8: Routing objects and the decisions they control. Query participation is soft; visual-depth and parameter-expert selection use top-

k

routes.

Panel of Figure 3 — Figure 3: Expert and visual-depth utilization on held-out data. Top: language-stack expert frequencies. Middle: vision-stack expert frequencies. Bottom: average global visual-depth weights. Layer axes enumerate the MoE blocks, and the depth labels S0–S9 enumerate the ten streams. E0–E7 identify parameter experts.

Explore similar work

VisMMOE: Exploiting Visual-Expert Affinity for Efficient Visual-Language MoE Offloading
May 7, 2026Cheng Xu, Xiaofeng Hou, Jiacheng Liu +1Expert Offloading Vision-Language Models
Shanghai Jiao Tong University Minhang Qu, Shanghai, China
DecoMoE: Decoupling Visual Propagation and Expert Computation for Efficient Multimodal MoE Inference
Sep 30, 2026Xudong Tan, Peng Ye, Ming Xie +4Mixture-of-Experts Pruning Efficient Multimodal Inference
Fudan University · The Chinese University of Hong Kong
SMoES: Soft Modality-Guided Expert Specialization in MoE-VLMs
Apr 27, 2026Zi-Hao Bo, Yaqian Li, Anzhou Hou +6Vision-Language Models Efficient VLM Inference
Li Auto Inc.

May 7, 2026J/K move · Enter open · S save

VisMMOE: Exploiting Visual-Expert Affinity for Efficient Visual-Language MoE Offloading

Cheng Xu, Xiaofeng Hou, Jiacheng Liu, Chao Li

Shanghai Jiao Tong University Minhang Qu, Shanghai, China

Expert Offloading Vision-Language Models Efficient VLM Inference Mixture-of-Experts Inference Visual Token Compression

Open paper AskPDF arXiv

Large-scale vision-language mixture-of-experts (VL-MoE) models provide strong multimodal capability, but efficient deployment on memory-constrained platforms remains difficult. Existing MoE offloading systems are largely designed for text-centric workloads and become much less effective for visual-heavy inputs, where large numbers of visual tokens induce broader and less predictable expert accesses. We present VisMMoE, a VL-MoE offloading system built on a single systems insight: pruning redundant visual tokens can improve offloading not only by reducing computation, but also by reshaping expert demand. We refer to this effect as visual-expert affinity: token pruning makes expert accesses more concentrated within layers and more stable across layers, producing a smaller and more predictable expert working set. Guided by this insight, VisMMoE combines affinity-aware token compression, lookahead expert prediction, and cache/pipeline orchestration to improve expert locality and prefetch effectiveness under tight memory budgets. We implement VisMMoE on multiple frameworks and evaluate it on representative VL-MoE models and benchmarks. VisMMoE improves end-to-end inference performance by up to 2.68x and 1.61x, respectively, over strong baselines for today's VL-MoE deployments while maintaining competitive accuracy.

Abstract

Explore similar work

VisMMOE: Exploiting Visual-Expert Affinity for Efficient Visual-Language MoE Offloading

DecoMoE: Decoupling Visual Propagation and Expert Computation for Efficient Multimodal MoE Inference

SMoES: Soft Modality-Guided Expert Specialization in MoE-VLMs