MRC-MMAgent: Reinforcement Learning for Multi-Role Collaboration in Multimodal Tool-Using Agents

Shengqin Wang; Jie Jin; Xiaofan Li; Wentao Yan; Yu Cheng; Qingsong HuYihang Chen; Weilin Luo; Jingyu Gong

cs.CVOct 29, 2026

MRC-MMAgent: Reinforcement Learning for Multi-Role Collaboration in Multimodal Tool-Using Agents

Authors: Shengqin Wang, Jie Jin, Xiaofan Li, Wentao Yan, Yu Cheng, Qingsong HuYihang Chen, Weilin Luo, Jingyu Gong,

Abstract

Multimodal tool-using agents augment vision-language models with access to external information and capabilities, enabling them to solve knowledge-intensive and visually complex tasks that cannot be addressed using visual context alone. However, existing reinforcement learning pipelines largely train single-role agents to search and answer using terminal rewards, overlooking how humans inspect retrieved evidence, reformulate queries, and revise conclusions during search. Consequently, they fail to optimize this self-corrective process over complete interaction trajectories. We propose MRC-MMAgent, a multi-role collaborative reinforcement learning framework that incorporates trajectory assessment and continuation directly into the executable agent loop. First, the framework establishes a multi-role collaborative loop that integrates solving, inspection, and continuation, enabling the agent to acquire evidence through tools, evaluate the preceding trajectory, and resume problem solving when the current answer is insufficiently supported. Second, we introduce a role-coupled learning objective that combines task success, interaction validity, and successful correction with full-trajectory policy optimization across the decisions generated by all roles. Extensive experiments show that, using only the original baseline training set without constructing additional datasets, our method consistently outperforms the corresponding base models and achieves leading or highly competitive performance across benchmarks.

Figures & tables

Figure 1: Performance comparison of agents across different scales. Each radar chart reports results on the same five evaluation datasets.

Figure 2: Overview of the proposed multi-role collaborative training framework for multimodal tasks. The Solver iteratively tackles the initial query, while the Inspector analyzes visual-textual trajectories to generate suggestions. Failed agent loops are re-executed based on historical trajectories to ensure convergence.

Figure 3: Case Study. The left panel shows a failure case where the current agent loop produces an incorrect answer. The right panel shows that after the inspector reviews the visual-textual trajectory, it detects the failure; the solver then resumes from the accumulated context and successfully corrects the answer.

Direct Answer
Method	Avg.	FVQA-Test	InfoSeek	SimpleVQA	LiveVQA	MMSearch
Qwen2.5-VL-7B-Instruct	26.7	26.3	32.0	47.9	19.6	7.6
Qwen3-VL-8B-Instruct	25.0	24.2	23.2	42.9	23.2	11.7
Qwen3-VL-32B-Instruct	31.1	32.2	29.0	45.9	31.6	17.0
GPT-5	50.0	54.4	54.2	61.7	44.4	35.1
Claude-4-Sonnet	40.1	35.3	57.3	50.9	38.5	18.7

Table 1: Accuracy comparison on five search-intensive benchmarks. Unlike competing trained agents that use additionally constructed training data, MRC-MMAgent is trained solely on the original baseline (MMSearch-R1) training set.

Direct Answer
Model	V* Bench	4K	8K
Gemini-2.5-Pro	83.8	87.3	85.4
GPT-4o	67.5	65.0	59.6
LLaVA-onevision	75.4	63.0	59.8
Qwen3-VL-8B-Instruct	86.4	78.9	74.6
Agentic Model

Table 2: Accuracy on image reasoning benchmarks. 4K and 8K denote HR-Bench 4K and HR-Bench 8K.

Method	FVQA-Test	MMSearch
GRPO	58.3	50.9
ReflectAgent	58.6	52.3
Solver-Inspector(two-stage RL)	59.1	56.0
Ours w/o role-coupled reward	62.3	60.2
Ours	64.0	62.0

Table 3: The ablation of different inspect and solve workflow with 7B.

Figure 4: Pie chart visualizations comparing ablation results across model sizes, training approaches, and benchmarks. Orange indicates REJECT, and green indicates PASS.

Dataset	Training	Finalize	Continue
MMSearch	Offline	69.0	31.0
MMSearch	Online	90.1	9.9
FVQA	Offline	89.8	10.2
FVQA	Online	93.9	6.1
FVQA-7B	Offline	34.1	65.9
FVQA-7B	Online	39.6	60.4

Table 4: Ablation studies on model scales, training methods, and benchmarks. Finalize indicates that the inspection is passed and the task is completed. Continue indicates that issues exist in the current trajectory and further problem-solving is required. Offline uses the local knowledge base Wiki25, while online search uses the more comprehensive Google Search.

Setting	Solver	Inspector	Solver1	Solver2	Err.	Acc
Config. 1	$\checkmark$	$\checkmark$	57.2	72.6	5.3	64.0
Config. 2	$\checkmark$	$\times$	54.3	72.3	10.4	62.2
Config. 3	$\times$	$\checkmark$	41.1	53.7	14.7	52.9
Config. 4	$\times$	$\times$	18.4	52.9	22.6	51.1

Table 5: Functional ablation under different role-training settings with 7B.

\checkmark

indicates that the corresponding role is trained, and

\times

indicates that it is frozen.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Method	No extra SFT trajectories	Trainable control inspector	Reference-free inspection	Inspection-triggered continuation	Full-trajectory coupled update
Look-Back	$\times$	$\times$	$\times$	$\times$	$\times$
Klissarov et al.(Google)	$\times$	$\triangle$	$\times$	$\checkmark$	$\times$
VSearcher	$\times$	$\times$	$\times$	$\times$	$\times$
PyVision-RL	$\triangle$	$\times$	$\times$	$\times$	$\times$
Tool-Star	$\times$	$\triangle$	$\triangle$	$\times$	$\times$
VideoChat-M1	$\triangle$	$\times$	$\times$	$\times$	$\triangle$

Appendix

Table 6: Structural comparison with representative reflective, multimodal tool-use, and multi-role RL methods.

\checkmark

: explicit support;

\triangle

: partial or indirect support;

\times

: absent or not a primary training objective. Reference-free inspection means that the inspection decision does not use a teacher critique or ground-truth answer. Full-trajectory coupled update means that one trajectory reward updates all policy-generated role tokens without post-hoc role-selective masking.

Panel of Figure 5 — Figure 5: Comparison of rewards and tool calls for our method and mask-based variant for role-selective optimization.

$N_{m}$	Accuracy	Avg. Tokens	Avg. Tool Calls
1	50.9	1717.5	2.5
2	50.3	2751.7	2.9
Baseline	45.6	1643.4	2.3

Appendix

Table 7: Ablation study on the number of continuation loops with 8B in offline.

Figure 6: Mean Acc on the number of continuation loops.

Method	Acc	Finalize	Continue
Config. 2 w/ think	62.2	31.5	68.5
Config. 2 w/o think	62.7	1.7	98.3
Ours	64.0	39.6	60.4

Appendix

Table 8: The abation of inspector’s action on FVQA-Test.

Setting	Acc	Step
0.2	64.7	100
0.3	63.6	100
0.1	63.1	100
Baseline	57.3	100

Appendix

Table 9: The abation of inspector’s action on FVQA-Test with 8B in offline.

Method	SFT	New Training Dataset	Base Dataset	Train-time Web Search	Test-time Web Search	ACC
MMSearch-R1	$\times$	$\times$	$\checkmark$	$\checkmark$	$\checkmark$	51.1
DeepEyes2-7B	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	–
MM-DeepResearch-7B	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	60.8
MRC-MMAgent-7B (ours)	$\times$	$\times$	$\checkmark$	$\times$	$\checkmark$	62.2
SenseNova-MARS-8B	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	64.6
OpenSearch-VL-8B	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	67.5

Appendix

Table 10: Comparison of training data, web-search usage, and accuracy among representative methods.

\checkmark

: used;

\triangle

: partially used;

\times

: not used. ACC denotes the average accuracy across the evaluated benchmarks.

Multimodal Agents

Multimodal agents perceive and reason across text, images, audio, and video to use tools, interact with interfaces, and act in the world.

See all 389 papers

Think, Look, and Revise: Inconsistency-Aware Visual Self-Correction in MLLMs
Aug 29, 2026Yu Cheng, Arushi Goel, Hakan BilenTool Use in VLMs Multimodal Large Language Models
University of Edinburgh · NVIDIA Research
PyVision-RL: Forging Open Agentic Vision Models via RL
Feb 24, 2026Shitian Zhao, Shaoheng Lin, Ming Li +4Adaptive Video Frame Sampling Multimodal Agents
Shanghai AI Lab · UMD · CUHK +3
Eliciting Self-Verification in Multimodal Reasoning Agents with Reinforcement Learning
Sep 7, 2026Vishwas Sathish, Viresh Ranjan, Xinliang Zhu +2Tool Use in VLMs Multimodal QA
University of Washington · Seattle, WA, USA · Amazon +1

Aug 29, 2026J/K move · Enter open · S save

Think, Look, and Revise: Inconsistency-Aware Visual Self-Correction in MLLMs

Yu Cheng, Arushi Goel, Hakan Bilen

University of Edinburgh · NVIDIA Research

Tool Use in VLMs Multimodal Large Language Models Multimodal Reasoning

Open paper AskPDF arXiv

Tool-augmented multimodal reasoning integrates external tools (e.g., object detection, depth estimation) into multimodal large language models (MLLMs) to address perceptual bottlenecks in complex visual tasks. However, existing approaches rarely verify tool outputs, limiting their ability to detect and recover from tool failures. We propose ReVISE, a framework that equips MLLMs with verification and dynamic error recovery for tool-augmented reasoning. ReVISE introduces (1) a curated training dataset that supervises reflective behaviors, enabling models to validate tool-derived evidence, reformulate queries when visual mismatches arise, and fall back to intrinsic grounding when external tools are unreliable; and (2) a reinforcement learning based targeted rewards that encourage internal reflection and penalize spatial misalignment. Experiments on several benchmarks demonstrate consistent improvements over existing methods, highlighting the importance of error detection and correction in tool-augmented multimodal reasoning.

Abstract

Multimodal Agents

Explore similar work

Think, Look, and Revise: Inconsistency-Aware Visual Self-Correction in MLLMs

PyVision-RL: Forging Open Agentic Vision Models via RL

Eliciting Self-Verification in Multimodal Reasoning Agents with Reinforcement Learning