This paper studies dynamic model routing under uncertainty in multimodal settings, where a system must select and potentially compose multiple models to answer a query. Standard reinforcement learning approaches for routing often suffer from premature convergence to suboptimal models due to noisy rewards and insufficient exploration. We propose a unified vision-language framework (VL-Scout) for efficient multimodal and multi-step reasoning with turn-based reinforcement fusion. First, VL-Scout combines Bayesian priors, policy gradient optimization, and sequential decision-making to enable structured exploration and adaptive inference. A context-dependent Dirichlet belief model allows for epistemic uncertainty over model quality and is updated online based on observed rewards and context-specific evidence, enabling Thompson-style exploration while serving as a prior for regularization. Second, a routing policy is trained via a KL-regularized REINFORCE objective by maximizing a reward-weighted evidence lower bound, where the policy acts as an amortized posterior, and the Dirichlet belief serves as a prior. Third, we introduce a fusion policy trained with Group Relative Policy Optimization (GRPO) or Reinforcement Learning with Verifiable Rewards (RLVR), which operates over intermediate model rationales and selects between two actions: CONTINUE or ANSWER, terminating with a generated output. We evaluate VL-Scout on four representative multi-modal benchmarks (MMMU, MMMU-Pro, OKVQA, and RealWorldQA), compared against 11 representative routing and policy optimization baselines. Our method provides consistent performance improvements while reducing premature policy collapse. These results highlight the importance of integrating uncertainty-aware priors with both amortized policy learning and sequential fusion for robust and efficient model selection in discrete action spaces
Figures & tables
Figure 1: Overview of VL-Scout two stage ensemble by reinforcement learning agents. The step sequence number is used to annotate the order of the workflow steps of VL-Scout.
Figure 2: Example illustration of VL-Scout reasoning. Example-1 reaches the correct answer after evaluating the first model’s output. Example-2 makes the correct decision after consulting the second model. Example-3 continues consulting models until it finds a correct rationale and answer. Example-4 makes correct decision when all consulted models are incorrect. Detailed responses in Appendix- G .
Figure 3: The left shows an ablation study on routing and fusion in the MMMU dataset. The last two plots show how the prior affects the convergence on the stochastic model selection.
Figure 4: All candidate ensemble teams from the model pool of size K=11 are plotted with their focal diversity scores ( Tekin et al., 2024 ) , Fleiss Kappa, and Plurality Voting Accuracy using two popular datasets. We use cubic interpolation with RBF smoothing to create a surface, and the dark red represents a higher performance score. Each dot represents the explored ensemble team by the routing models REINFORCE and REINFORCE + PRIOR, colored by their appearances during testing.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Description
spatial_and_visual
Where, position, layout, scene understanding
counting_and_quantity
How many, amounts, numbers
text_and_document
OCR, reading, documents, charts
science_and_math
Physics, chemistry, biology, math, engineering
humanities_and_arts
History, literature, music, art, philosophy
social_and_applied
Economics, law, medicine, psychology, business
Appendix
Table 3: Question Categories and Descriptions
Prompt
Description
Content
VQA System Prompt
Base prompt. Instructs the model to output the final answer only, no reasoning.
You are a visual question answering assistant.
For every question, first write a short rationale
describing what you observe and how you reason, then give
your final answer. Always use this format:
Rationale: [your reasoning]
Answer: [your final answer]]
Appendix
Table 4: Prompt Configurations
Question: Is the closest car closer to us than the pedestrian?
Model
Rationale
Answer
Decision
Correct?
Qwen (Model 1)
The pedestrian is visible on the sidewalk to the right, standing near a tree and some snow piles. The closest car is further down the street, behind the pedestrian. Since the pedestrian is on the sidewalk and the car is on the road and further away, the pedestrian is closer to the viewer than the car.
Yes
CONTINUE
×
InternVL (Model 2)
The closest car is on the left side of the road, while the pedestrian is on the right side. By comparing their relative positions to the camera’s viewpoint, the car appears closer to the viewer than the pedestrian.
Yes
CONTINUE
×
Gemma (Model 3)
The image shows a street scene. The closest car is driving down the middle of the road, relatively far away. The pedestrian is walking on the sidewalk on the right side, and is much closer to the camera than the car. Therefore, the pedestrian is closer to us than the closest car.
1
ANSWER: No
✓
Ground Truth
No
✓
Appendix
Table 5: Model Rationales and Answers for Example-4 for Figure- 2
Question: Is there a stop sign?
Model
Rationale
Answer
Decision
Correct?
InternVL (Model 1)
To determine if there is a stop sign, I examine the street for any octagonal signs with a red border and white background. Upon reviewing the image, I see several signs along the street, but none match the description of a stop sign. The signs visible are either rectangular or do not have the characteristic red border and white center.
No
CONTINUE
✓
Qwen (Model 2)
I can see a white SUV parked on the left side of the street, and behind it, there appears to be a stop sign mounted on a pole near the sidewalk. The sign is partially obscured by the SUV and the angle of the photo, but its shape and position are consistent with a stop sign. Therefore, based on the visual evidence, there is at least one stop sign present.
Yes
CONTINUE
×
Gemma (Model 3)
I see a street with cars parked on the side and yellow crosswalk markings. I do not see any stop signs visible in the image.
0
ANSWER: No
✓
Ground Truth
No
✓
Appendix
Table 6: Model Rationales and Answers Example-1 for Figure- 2
Adaptive model routing selects models or specialized experts per input to balance prediction quality, computational cost, latency, and resource constraints.
Large multimodal models have achieved strong reasoning on complex visual tasks, but their inference efficiency is often restricted by long chains of thought. A promising solution is to pair a small draft model with a large target model, enabling cooperative inference employing a routing signal that adaptively routes queries to either the draft or target model based on their difficulties for optimal efficiency and accuracy. Yet, the remaining bottleneck is to establish a reliable query difficulty signal under multimodal settings. Existing approaches designed for language models either rely on post-hoc token probabilities, which fall short in multimodal scenarios, or depend on supervised fine-tuning, which is a data-sensitive strategy. Both paradigms perform routing only after a complete output, and ignore whether the target model can actually solve the routed instances. To address this, we propose PRP, a Proactive Routing Paradigm that enables early decision-making by jointly evaluating the competence of both the draft and target models. Our Draft Rating Learning (DRL) equips the draft model with an internal confidence estimator, while Joint Rating Learning (JRL) predicts how well the target model can handle a given query, thereby prioritizing the allocation of samples it excels at rather than the hardest ones. These ratings enable fine-grained, instance-level Proactive Routing and substantially accelerate inference without compromising overall performance. Extensive experiments across multiple multimodal reasoning benchmarks validate our effectiveness and efficiency.
Yinan Zhou, Haokun Lin, Yichen Wu +7
Xi’an Jiaotong University · ARC Lab, Tencent IEG · City University of Hong Kong +3
Group-based reinforcement learning methods for multimodal large language models typically rely on trajectory-level credit assignment that applies a single advantage to all tokens in a response. However, multimodal reasoning involves substantially higher perceptual uncertainty than text-only settings, where the model must repeatedly re-examine visual information to verify intermediate interpretations, and different visual groundings can lead to divergent reasoning paths, making such uniform credit assignment particularly inadequate and causing relative advantages to progressively degenerate toward zero. To address these challenges, we propose Multi-Branch Policy Optimization (MBPO), a tree-based framework that constructs reasoning trees at vision-language decision boundaries, enabling sibling branches to explore diverse visual hypotheses and assigning segment-level credit through branch-relative advantages. We further introduce a temporal replay buffer to reuse informative segments while controlling policy staleness. Experiments on several multimodal reasoning benchmarks show that MBPO outperforms representative baselines, improving both learning signal quality and optimization efficiency. The code is publicly available at https://github.com/ShuaiLyu0110/MBPO.
Shuai Lyu, Yuning Gong, Ruiling Gao +7
Beijing University of Posts and Telecommunications Beijing, China · Sichuan University Chengdu, China · Shanghai University Shanghai, China +1
Multimodal large language models (MLLMs) have heterogeneous strengths across OCR, chart understanding, spatial reasoning, visual question answering, cost, and latency. Effective MLLM routing therefore requires more than estimating query difficulty: a router must match the multimodal requirements of the current image-question input with the capabilities of each candidate model. We propose LatentRouter, a router that formulates MLLM routing as counterfactual multimodal utility prediction. Given an image-question query, LatentRouter extracts learned multimodal routing capsules, represents each candidate MLLM with a model capability token, and performs latent communication between these states to estimate how each model would perform if selected. A distributional outcome head predicts model-specific counterfactual quality, while a bounded capsule correction refines close decisions without allowing residual signals to dominate the prediction. The resulting utility-based policy supports performance-oriented and performance-cost routing, and handles changing candidate pools through shared per-model scoring with availability masking. Experiments on MMR-Bench and VL-RouterBench show that LatentRouter outperforms fixed-model, feature-level, and learned-router baselines. Additional analyses show that the gains are strongest on multimodal task groups where model choice depends on visual, layout-sensitive, or reasoning-oriented requirements, and that latent communication is the main contributor to the improvement. The code is available at: https://github.com/LabRAI/LatentRouter.
Xueqi Cheng, Yushun Dong
Department of Computer Science Florida State University University