MRC-MMAgent: Reinforcement Learning for Multi-Role Collaboration in Multimodal Tool-Using Agents

Shengqin Wang; Jie Jin; Xiaofan Li; Wentao Yan; Yu Cheng; Qingsong HuYihang Chen; Weilin Luo; Jingyu Gong

cs.CVOct 29, 2026

MRC-MMAgent: Reinforcement Learning for Multi-Role Collaboration in Multimodal Tool-Using Agents

Authors: Shengqin Wang, Jie Jin, Xiaofan Li, Wentao Yan, Yu Cheng, Qingsong HuYihang Chen, Weilin Luo, Jingyu Gong,

Abstract

Multimodal tool-using agents augment vision-language models with access to external information and capabilities, enabling them to solve knowledge-intensive and visually complex tasks that cannot be addressed using visual context alone. However, existing reinforcement learning pipelines largely train single-role agents to search and answer using terminal rewards, overlooking how humans inspect retrieved evidence, reformulate queries, and revise conclusions during search. Consequently, they fail to optimize this self-corrective process over complete interaction trajectories. We propose MRC-MMAgent, a multi-role collaborative reinforcement learning framework that incorporates trajectory assessment and continuation directly into the executable agent loop. First, the framework establishes a multi-role collaborative loop that integrates solving, inspection, and continuation, enabling the agent to acquire evidence through tools, evaluate the preceding trajectory, and resume problem solving when the current answer is insufficiently supported. Second, we introduce a role-coupled learning objective that combines task success, interaction validity, and successful correction with full-trajectory policy optimization across the decisions generated by all roles. Extensive experiments show that, using only the original baseline training set without constructing additional datasets, our method consistently outperforms the corresponding base models and achieves leading or highly competitive performance across benchmarks.

Figures & tables

Figure 1: Performance comparison of agents across different scales. Each radar chart reports results on the same five evaluation datasets.

Figure 2: Overview of the proposed multi-role collaborative training framework for multimodal tasks. The Solver iteratively tackles the initial query, while the Inspector analyzes visual-textual trajectories to generate suggestions. Failed agent loops are re-executed based on historical trajectories to ensure convergence.

Figure 3: Case Study. The left panel shows a failure case where the current agent loop produces an incorrect answer. The right panel shows that after the inspector reviews the visual-textual trajectory, it detects the failure; the solver then resumes from the accumulated context and successfully corrects the answer.

Direct Answer
Method	Avg.	FVQA-Test	InfoSeek	SimpleVQA	LiveVQA	MMSearch
Qwen2.5-VL-7B-Instruct	26.7	26.3	32.0	47.9	19.6	7.6
Qwen3-VL-8B-Instruct	25.0	24.2	23.2	42.9	23.2	11.7
Qwen3-VL-32B-Instruct	31.1	32.2	29.0	45.9	31.6	17.0
GPT-5	50.0	54.4	54.2	61.7	44.4	35.1
Claude-4-Sonnet	40.1	35.3	57.3	50.9	38.5	18.7

Table 1: Accuracy comparison on five search-intensive benchmarks. Unlike competing trained agents that use additionally constructed training data, MRC-MMAgent is trained solely on the original baseline (MMSearch-R1) training set.

Direct Answer
Model	V* Bench	4K	8K
Gemini-2.5-Pro	83.8	87.3	85.4
GPT-4o	67.5	65.0	59.6
LLaVA-onevision	75.4	63.0	59.8
Qwen3-VL-8B-Instruct	86.4	78.9	74.6
Agentic Model

Table 2: Accuracy on image reasoning benchmarks. 4K and 8K denote HR-Bench 4K and HR-Bench 8K.

Method	FVQA-Test	MMSearch
GRPO	58.3	50.9
ReflectAgent	58.6	52.3
Solver-Inspector(two-stage RL)	59.1	56.0
Ours w/o role-coupled reward	62.3	60.2
Ours	64.0	62.0

Table 3: The ablation of different inspect and solve workflow with 7B.

Figure 4: Pie chart visualizations comparing ablation results across model sizes, training approaches, and benchmarks. Orange indicates REJECT, and green indicates PASS.

Dataset	Training	Finalize	Continue
MMSearch	Offline	69.0	31.0
MMSearch	Online	90.1	9.9
FVQA	Offline	89.8	10.2
FVQA	Online	93.9	6.1
FVQA-7B	Offline	34.1	65.9
FVQA-7B	Online	39.6	60.4

Table 4: Ablation studies on model scales, training methods, and benchmarks. Finalize indicates that the inspection is passed and the task is completed. Continue indicates that issues exist in the current trajectory and further problem-solving is required. Offline uses the local knowledge base Wiki25, while online search uses the more comprehensive Google Search.

Setting	Solver	Inspector	Solver1	Solver2	Err.	Acc
Config. 1	$\checkmark$	$\checkmark$	57.2	72.6	5.3	64.0
Config. 2	$\checkmark$	$\times$	54.3	72.3	10.4	62.2
Config. 3	$\times$	$\checkmark$	41.1	53.7	14.7	52.9
Config. 4	$\times$	$\times$	18.4	52.9	22.6	51.1

Table 5: Functional ablation under different role-training settings with 7B.

\checkmark

indicates that the corresponding role is trained, and

\times

indicates that it is frozen.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Method	No extra SFT trajectories	Trainable control inspector	Reference-free inspection	Inspection-triggered continuation	Full-trajectory coupled update
Look-Back	$\times$	$\times$	$\times$	$\times$	$\times$
Klissarov et al.(Google)	$\times$	$\triangle$	$\times$	$\checkmark$	$\times$
VSearcher	$\times$	$\times$	$\times$	$\times$	$\times$
PyVision-RL	$\triangle$	$\times$	$\times$	$\times$	$\times$
Tool-Star	$\times$	$\triangle$	$\triangle$	$\times$	$\times$
VideoChat-M1	$\triangle$	$\times$	$\times$	$\times$	$\triangle$

Appendix

Table 6: Structural comparison with representative reflective, multimodal tool-use, and multi-role RL methods.

\checkmark

: explicit support;

\triangle

: partial or indirect support;

\times

: absent or not a primary training objective. Reference-free inspection means that the inspection decision does not use a teacher critique or ground-truth answer. Full-trajectory coupled update means that one trajectory reward updates all policy-generated role tokens without post-hoc role-selective masking.

Panel of Figure 5 — Figure 5: Comparison of rewards and tool calls for our method and mask-based variant for role-selective optimization.

$N_{m}$	Accuracy	Avg. Tokens	Avg. Tool Calls
1	50.9	1717.5	2.5
2	50.3	2751.7	2.9
Baseline	45.6	1643.4	2.3

Appendix

Table 7: Ablation study on the number of continuation loops with 8B in offline.

Figure 6: Mean Acc on the number of continuation loops.

Method	Acc	Finalize	Continue
Config. 2 w/ think	62.2	31.5	68.5
Config. 2 w/o think	62.7	1.7	98.3
Ours	64.0	39.6	60.4

Appendix

Table 8: The abation of inspector’s action on FVQA-Test.

Setting	Acc	Step
0.2	64.7	100
0.3	63.6	100
0.1	63.1	100
Baseline	57.3	100

Appendix

Table 9: The abation of inspector’s action on FVQA-Test with 8B in offline.

Method	SFT	New Training Dataset	Base Dataset	Train-time Web Search	Test-time Web Search	ACC
MMSearch-R1	$\times$	$\times$	$\checkmark$	$\checkmark$	$\checkmark$	51.1
DeepEyes2-7B	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	–
MM-DeepResearch-7B	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	60.8
MRC-MMAgent-7B (ours)	$\times$	$\times$	$\checkmark$	$\times$	$\checkmark$	62.2
SenseNova-MARS-8B	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	64.6
OpenSearch-VL-8B	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	67.5

Appendix

Table 10: Comparison of training data, web-search usage, and accuracy among representative methods.

\checkmark

: used;

\triangle

: partially used;

\times

: not used. ACC denotes the average accuracy across the evaluated benchmarks.

Multimodal Agents

Multimodal agents perceive and reason across text, images, audio, and video to use tools, interact with interfaces, and act in the world.

See all 387 papers

Aug 29, 2026

Think, Look, and Revise: Inconsistency-Aware Visual Self-Correction in MLLMs

Tool-augmented multimodal reasoning integrates external tools (e.g., object detection, depth estimation) into multimodal large language models (MLLMs) to address perceptual bottlenecks in complex visual tasks. However, existing approaches rarely verify tool outputs, limiting their ability to detect and recover from tool failures. We propose ReVISE, a framework that equips MLLMs with verification and dynamic error recovery for tool-augmented reasoning. ReVISE introduces (1) a curated training dataset that supervises reflective behaviors, enabling models to validate tool-derived evidence, reformulate queries when visual mismatches arise, and fall back to intrinsic grounding when external tools are unreliable; and (2) a reinforcement learning based targeted rewards that encourage internal reflection and penalize spatial misalignment. Experiments on several benchmarks demonstrate consistent improvements over existing methods, highlighting the importance of error detection and correction in tool-augmented multimodal reasoning.

Yu Cheng, Arushi Goel, Hakan Bilen

University of Edinburgh · NVIDIA Research

Tool Use in VLMs Multimodal Large Language Models Multimodal Reasoning

Ask PDF

Feb 24, 2026

PyVision-RL: Forging Open Agentic Vision Models via RL

Reinforcement learning for agentic multimodal models often suffers from interaction collapse, where models learn to reduce tool usage and multi-turn reasoning, limiting the benefits of agentic behavior. We introduce PyVision-RL, a reinforcement learning framework for open-weight multimodal models that stabilizes training and sustains interaction. Our approach combines an oversampling-filtering-ranking rollout strategy with an accumulative tool reward to prevent collapse and encourage multi-turn tool use. Using a unified training pipeline, we develop PyVision-Image and PyVision-Video for image and video understanding. For video reasoning, PyVision-Video employs on-demand context construction, selectively sampling task-relevant frames during reasoning to significantly reduce visual token usage. Experiments show strong performance and improved efficiency, demonstrating that sustained interaction and on-demand visual processing are critical for scalable multimodal agents.

Shitian Zhao, Shaoheng Lin, Ming Li +4

Shanghai AI Lab · UMD · CUHK +3

Adaptive Video Frame Sampling Multimodal Agents RL for VLMs Agentic RL Video Reasoning

Ask PDF

Sep 7, 2026

Eliciting Self-Verification in Multimodal Reasoning Agents with Reinforcement Learning

Reasoning agents increasingly rely on external tools such as web search to answer complex queries. Reinforcement learning (RL) finetuning algorithms such as GRPO have improved long-form reasoning in text-only language models, particularly for coding and mathematics. Reliable tool use in multimodal agents, however, remains challenging because models must interpret text and images while integrating noisy retrieved evidence, often under sparse outcome-level supervision without explicit verification signals. We present Self-Verification via Reinforcement Learning (SVRL), an RL-only finetuning framework that trains multimodal agents to verify and filter retrieved evidence within their own reasoning traces, reducing reliance on external verifiers at inference time. SVRL also introduces a search-aware penalty that discourages unnecessary tool calls and a query-diversity reward that encourages diverse, well-formed search queries, providing fine-grained feedback on when and what to search. Finetuning Qwen-2.5-VL-7B with SVRL on only 5{,}000 visual question answering examples yields consistent gains in multi-hop VQA generalization and tool efficiency across benchmarks. Overall, SVRL narrows the gap between compact agents and much larger proprietary models while requiring substantially lower training and inference cost.

Vishwas Sathish, Viresh Ranjan, Xinliang Zhu +2

University of Washington · Seattle, WA, USA · Amazon +1

Tool Use in VLMs Multimodal QA Multimodal Reasoning LLM Agent Verification RL for VLMs

Ask PDF

Abstract

Multimodal Agents

Explore similar work

Think, Look, and Revise: Inconsistency-Aware Visual Self-Correction in MLLMs

PyVision-RL: Forging Open Agentic Vision Models via RL

Eliciting Self-Verification in Multimodal Reasoning Agents with Reinforcement Learning