Alliance of Stochastic Routing and Reinforcement Fusion for Multi-Model Reasoning

Selim Furkan Tekin; Ling Liu

cs.LGOct 16, 2026

Alliance of Stochastic Routing and Reinforcement Fusion for Multi-Model Reasoning

Authors: Selim Furkan Tekin, Ling Liu

Organizations: Georgia Institute of Technology

Abstract

This paper studies dynamic model routing under uncertainty in multimodal settings, where a system must select and potentially compose multiple models to answer a query. Standard reinforcement learning approaches for routing often suffer from premature convergence to suboptimal models due to noisy rewards and insufficient exploration. We propose a unified vision-language framework (VL-Scout) for efficient multimodal and multi-step reasoning with turn-based reinforcement fusion. First, VL-Scout combines Bayesian priors, policy gradient optimization, and sequential decision-making to enable structured exploration and adaptive inference. A context-dependent Dirichlet belief model allows for epistemic uncertainty over model quality and is updated online based on observed rewards and context-specific evidence, enabling Thompson-style exploration while serving as a prior for regularization. Second, a routing policy is trained via a KL-regularized REINFORCE objective by maximizing a reward-weighted evidence lower bound, where the policy acts as an amortized posterior, and the Dirichlet belief serves as a prior. Third, we introduce a fusion policy trained with Group Relative Policy Optimization (GRPO) or Reinforcement Learning with Verifiable Rewards (RLVR), which operates over intermediate model rationales and selects between two actions: CONTINUE or ANSWER, terminating with a generated output. We evaluate VL-Scout on four representative multi-modal benchmarks (MMMU, MMMU-Pro, OKVQA, and RealWorldQA), compared against 11 representative routing and policy optimization baselines. Our method provides consistent performance improvements while reducing premature policy collapse. These results highlight the importance of integrating uncertainty-aware priors with both amortized policy learning and sequential fusion for robust and efficient model selection in discrete action spaces

Figures & tables

Figure 1: Overview of VL-Scout two stage ensemble by reinforcement learning agents. The step sequence number is used to annotate the order of the workflow steps of VL-Scout.

Figure 2: Example illustration of VL-Scout reasoning. Example-1 reaches the correct answer after evaluating the first model’s output. Example-2 makes the correct decision after consulting the second model. Example-3 continues consulting models until it finds a correct rationale and answer. Example-4 makes correct decision when all consulted models are incorrect. Detailed responses in Appendix- G .

Figure 3: The left shows an ablation study on routing and fusion in the MMMU dataset. The last two plots show how the prior affects the convergence on the stochastic model selection.

Figure 4: All candidate ensemble teams from the model pool of size $K=11$ are plotted with their focal diversity scores ( Tekin et al., 2024 ) , Fleiss Kappa, and Plurality Voting Accuracy using two popular datasets. We use cubic interpolation with RBF smoothing to create a surface, and the dark red represents a higher performance score. Each dot represents the explored ensemble team by the routing models REINFORCE and REINFORCE + PRIOR, colored by their appearances during testing.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Description
spatial_and_visual	Where, position, layout, scene understanding
counting_and_quantity	How many, amounts, numbers
text_and_document	OCR, reading, documents, charts
science_and_math	Physics, chemistry, biology, math, engineering
humanities_and_arts	History, literature, music, art, philosophy
social_and_applied	Economics, law, medicine, psychology, business

Appendix

Table 3: Question Categories and Descriptions

Prompt	Description	Content
VQA System Prompt	Base prompt. Instructs the model to output the final answer only, no reasoning.	You are a visual question answering assistant.
		For every question, first write a short rationale
		describing what you observe and how you reason, then give
		your final answer. Always use this format:
		Rationale: [your reasoning]
		Answer: [your final answer]]

Appendix

Table 4: Prompt Configurations

Question: Is the closest car closer to us than the pedestrian?
Model	Rationale	Answer	Decision	Correct?
Qwen (Model 1)	The pedestrian is visible on the sidewalk to the right, standing near a tree and some snow piles. The closest car is further down the street, behind the pedestrian. Since the pedestrian is on the sidewalk and the car is on the road and further away, the pedestrian is closer to the viewer than the car.	Yes	CONTINUE	$\times$
InternVL (Model 2)	The closest car is on the left side of the road, while the pedestrian is on the right side. By comparing their relative positions to the camera’s viewpoint, the car appears closer to the viewer than the pedestrian.	Yes	CONTINUE	$\times$
Gemma (Model 3)	The image shows a street scene. The closest car is driving down the middle of the road, relatively far away. The pedestrian is walking on the sidewalk on the right side, and is much closer to the camera than the car. Therefore, the pedestrian is closer to us than the closest car.	1	ANSWER: No	✓
Ground Truth		No		✓

Appendix

Table 5: Model Rationales and Answers for Example-4 for Figure- 2

Question: Is there a stop sign?
Model	Rationale	Answer	Decision	Correct?
InternVL (Model 1)	To determine if there is a stop sign, I examine the street for any octagonal signs with a red border and white background. Upon reviewing the image, I see several signs along the street, but none match the description of a stop sign. The signs visible are either rectangular or do not have the characteristic red border and white center.	No	CONTINUE	✓
Qwen (Model 2)	I can see a white SUV parked on the left side of the street, and behind it, there appears to be a stop sign mounted on a pole near the sidewalk. The sign is partially obscured by the SUV and the angle of the photo, but its shape and position are consistent with a stop sign. Therefore, based on the visual evidence, there is at least one stop sign present.	Yes	CONTINUE	$\times$
Gemma (Model 3)	I see a street with cars parked on the side and yellow crosswalk markings. I do not see any stop signs visible in the image.	0	ANSWER: No	✓
Ground Truth		No		✓

Appendix

Table 6: Model Rationales and Answers Example-1 for Figure- 2

Adaptive Model Routing

Adaptive model routing selects models or specialized experts per input to balance prediction quality, computational cost, latency, and resource constraints.

See all 87 papers

Before Thinking, Learn to Decide: Proactive Routing for Efficient Visual Reasoning
Jun 29, 2026Yinan Zhou, Haokun Lin, Yichen Wu +7Efficient Multimodal Inference Adaptive Model Routing
Xi’an Jiaotong University · ARC Lab, Tencent IEG · City University of Hong Kong +3
Multi-Branch Policy Optimization for Multimodal Large Language Models
Aug 5, 2026Shuai Lyu, Yuning Gong, Ruiling Gao +7Credit Assignment in RL Multimodal Large Language Models
Beijing University of Posts and Telecommunications Beijing, China · Sichuan University Chengdu, China · Shanghai University Shanghai, China +1
LatentRouter: Can We Choose the Right Multimodal Model Before Seeing Its Answer?
May 11, 2026Xueqi Cheng, Yushun DongEfficient Multimodal Inference Multimodal Large Language Models
Department of Computer Science Florida State University University

Jun 29, 2026J/K move · Enter open · S save

Before Thinking, Learn to Decide: Proactive Routing for Efficient Visual Reasoning

Yinan Zhou, Haokun Lin, Yichen Wu, Yuxin Chen+6

Xi’an Jiaotong University · ARC Lab, Tencent IEG · City University of Hong Kong+3

Efficient Multimodal Inference Adaptive Model Routing Confidence Estimation Multimodal Reasoning

Open paper AskPDF arXiv

Large multimodal models have achieved strong reasoning on complex visual tasks, but their inference efficiency is often restricted by long chains of thought. A promising solution is to pair a small draft model with a large target model, enabling cooperative inference employing a routing signal that adaptively routes queries to either the draft or target model based on their difficulties for optimal efficiency and accuracy. Yet, the remaining bottleneck is to establish a reliable query difficulty signal under multimodal settings. Existing approaches designed for language models either rely on post-hoc token probabilities, which fall short in multimodal scenarios, or depend on supervised fine-tuning, which is a data-sensitive strategy. Both paradigms perform routing only after a complete output, and ignore whether the target model can actually solve the routed instances. To address this, we propose PRP, a Proactive Routing Paradigm that enables early decision-making by jointly evaluating the competence of both the draft and target models. Our Draft Rating Learning (DRL) equips the draft model with an internal confidence estimator, while Joint Rating Learning (JRL) predicts how well the target model can handle a given query, thereby prioritizing the allocation of samples it excels at rather than the hardest ones. These ratings enable fine-grained, instance-level Proactive Routing and substantially accelerate inference without compromising overall performance. Extensive experiments across multiple multimodal reasoning benchmarks validate our effectiveness and efficiency.

Abstract

Adaptive Model Routing

Explore similar work

Before Thinking, Learn to Decide: Proactive Routing for Efficient Visual Reasoning

Multi-Branch Policy Optimization for Multimodal Large Language Models

LatentRouter: Can We Choose the Right Multimodal Model Before Seeing Its Answer?