MHR-Router: Multimodal Heterogeneous Retrieval-Enhanced Adaptive Routing for VLM Inference

Wenjuan Zhu; Zheming Yang; Yunqing Hu; Qi Guo

cs.LGOct 9, 2026

MHR-Router: Multimodal Heterogeneous Retrieval-Enhanced Adaptive Routing for VLM Inference

Authors: Wenjuan Zhu, Zheming Yang, Yunqing Hu, Qi Guo

Organizations: Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China

Abstract

Vision-language models (VLMs) exhibit complementary capabilities, yet selecting the most suitable expert for a given image-question query remains challenging due to heterogeneous multimodal inputs and varying expert strengths. Existing routing approaches rely on text-based cues or static model profiles, failing to capture fine-grained visual evidence and expert-specific failure patterns. To address this, we propose MHR-Router, a multimodal heterogeneous retrieval-enhanced adaptive routing framework for VLM inference. It accurately dispatches each multimodal query to the most suitable VLM expert via local retrieval evidence. MHR-Router consists of two stages: (1) Multimodal Routing Representation, which fuses frozen multimodal features into a comprehensive query representation; and (2) Heterogeneous Retrieval-Enhanced Adaptive Routing, which retrieves expert-specific success and failure memories as local evidence and employs a shared scorer to adaptively select the optimal expert. Experiments show that MHR-Router effectively achieves heterogeneous modality routing and consistently improves accuracy over traditional baselines.

Figures & tables

Figure 1: Comparison between text-only LLM routing and multimodal VLM routing. (a) A text-only router selects an LLM expert primarily from prompt semantics. (b) A multimodal router selects a VLM expert by jointly considering visual evidence and question intent.

Figure 2: Overview of MHR-Router. (a) Frozen multimodal features form the routing representation. (b) Expert-specific success and failure memories provide local retrieval evidence for a shared scorer. The selected expert receives the original query.

Figure 3: Overall and ability-wise routing accuracy across input modalities and representation sources.

Method	Acc. (%)	Gain (pp)
Random Router	51.21	-13.00
Best Single (dev-selected)	64.21	+0.00
Per-ability Dev Selection †	67.70	+3.48
Per-task Dev Selection †	67.70	+3.48
Multimodal Query Router	$70.36{\pm}0.38$	+6.14
MHR-Router (Ours)	$\mathbf{71.58{\pm}0.02}$	$\mathbf{+7.36}$

Table 1: Main routing accuracy and gains over the development-selected Best Single baseline.

Ability	$N$	MHR-Router	Dev-best	Gain (pp)
Counting	480	$96.88{\pm}0.00$	96.88	+0.00
Defect anomaly	480	$27.71{\pm}0.75$	19.17	+8.54
Fine-grained recog.	480	$72.43{\pm}0.43$	50.83	+21.60
Object recog.	480	$11.39{\pm}0.32$	11.04	+0.35
OCR understanding	467	$77.02{\pm}2.55$	79.44	-2.43
Small-object detection	480	$98.33{\pm}0.00$	97.71	+0.62

Table 2: Ability-wise routing accuracy relative to the development-selected expert for each ability.

Figure 4: Parameter sensitivity analysis of MHR-Router.

Multimodal Model Routing

See all 14 papers

LatentRouter: Can We Choose the Right Multimodal Model Before Seeing Its Answer?
May 11, 2026Xueqi Cheng, Yushun DongEfficient Multimodal Inference Multimodal Large Language Models
Department of Computer Science Florida State University University
An Effective Router for Vision-Language Model Selection
Jun 8, 2026Can Wang, Shengwei Wang, Bolin Zhang +2Vision-Language Models
Harbin Institute of Technology Harbin, China · Harbin Institute of Technology Shandong Key Laboratory of Digital Service Computing Technology and Systems Harbin, China
MoE-GRPO: Optimizing Mixture-of-Experts via Reinforcement Learning in Vision-Language Models
Mar 26, 2026Dohwan Ko, Jinyoung Park, Seoung Choi +3Expert Routing Mixture-of-Experts Models
Korea University · KAIST

May 11, 2026J/K move · Enter open · S save

LatentRouter: Can We Choose the Right Multimodal Model Before Seeing Its Answer?

Xueqi Cheng, Yushun Dong

Department of Computer Science Florida State University University

Efficient Multimodal Inference Multimodal Large Language Models LLM Routing Multimodal Foundation Models Multimodal Understanding Multimodal Model Routing

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) have heterogeneous strengths across OCR, chart understanding, spatial reasoning, visual question answering, cost, and latency. Effective MLLM routing therefore requires more than estimating query difficulty: a router must match the multimodal requirements of the current image-question input with the capabilities of each candidate model. We propose LatentRouter, a router that formulates MLLM routing as counterfactual multimodal utility prediction. Given an image-question query, LatentRouter extracts learned multimodal routing capsules, represents each candidate MLLM with a model capability token, and performs latent communication between these states to estimate how each model would perform if selected. A distributional outcome head predicts model-specific counterfactual quality, while a bounded capsule correction refines close decisions without allowing residual signals to dominate the prediction. The resulting utility-based policy supports performance-oriented and performance-cost routing, and handles changing candidate pools through shared per-model scoring with availability masking. Experiments on MMR-Bench and VL-RouterBench show that LatentRouter outperforms fixed-model, feature-level, and learned-router baselines. Additional analyses show that the gains are strongest on multimodal task groups where model choice depends on visual, layout-sensitive, or reasoning-oriented requirements, and that latent communication is the main contributor to the improvement. The code is available at: https://github.com/LabRAI/LatentRouter.

Abstract

Multimodal Model Routing

Explore similar work

LatentRouter: Can We Choose the Right Multimodal Model Before Seeing Its Answer?

An Effective Router for Vision-Language Model Selection

MoE-GRPO: Optimizing Mixture-of-Experts via Reinforcement Learning in Vision-Language Models