Organizations: National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · AI Business, Alibaba Group · School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, China · School of Intelligent Systems Engineering, Sun Yat-sen University, Shenzhen, China · School of Computing, National University of Singapore, Singapore, Singapore · School of Information Science and Technology, University of Science and Technology of China · School of Software Technology, Zhejiang University, China
Recent advancements in Multimodal Large Language Models (MLLMs) have incentivized models to ``think with images'' by actively invoking visual tools during multi-turn reasoning. The common Reinforcement Learning (RL) practice of relying on outcome-based rewards ignores the fact that textual plausibility often masks executive failure, meaning that models may exhibit intuitive textual reasoning while executing imprecise or irrelevant visual actions within their agentic reasoning trajectories. This reasoning-action discrepancy introduces noise that accumulates throughout the multi-turn reasoning process, severely degrading the model's multimodal reasoning capabilities and potentially leading to training collapse. In this paper, we introduce Multimodal Agentic Policy Optimization (MAPO), bridging the gap between textual reasoning and visual actions generated by models within their Multimodal Chain-of-Thought (MCoT). Specifically, MAPO mandates the model to generate explicit textual descriptions for the visual content obtained via tool usage. We then employ a novel advantage estimation that couples the semantic alignment between these descriptions and the actual observations with the task reward. Theoretical findings are provided to justify the rationale behind MAPO, which inherently reduces the variance of gradients, and extensive experiments demonstrate that our method achieves superior performance across multiple visual reasoning benchmarks.
Figures & tables
Figure 1: Left : Overview of the agentic “thinking with images” framework where the MLLM agent executes visual actions guided by textual reasoning. Right : Training curves of the semantic score demonstrating that MAPO significantly outperforms other RL methods.
Figure 2: Overview of MAPO. Our method bridges the reasoning-action gap by using a CLIP model to measure the semantic alignment between self-generated labels and observations. These signals are integrated into the advantage estimation to enforce process consistency.
Method
V*
HR-Bench
MME-Real-Lite
attribute
relative
overall
4K
8K
overall
GPT-5 ( OpenAI, 2025a )
70.0
73.4
71.3
74.6
72.0
73.3
55.3
Gemini 2.5 Pro ( Comanici et al., 2025 )
85.5
75.2
81.4
67.3
61.4
64.3
49.2
DeepEyes † ( Zheng et al., 2025b )
89.6
85.5
88.0
74.9
71.2
73.0
48.4
Thyme ∗ ( Zhang et al., 2025c )
83.5
80.3
82.2
77.0
72.0
74.5
55.2
Mini-o3 † ( Lai et al., 2025 )
90.4
88.2
89.3
75.4
72.8
74.1
49.4
Table 1: Main Results on real-world fine-grained visual reasoning benchmarks . The top two outcomes are bolded and underlined , respectively. Here, ∗ denotes results reported in the original paper, and † denotes results reproduced by ourselves through available model weights.
Figure 3: Visualization of Multimodal Reasoning Trajectories. We visualize the intermediate reasoning steps of MAPO ( left ) and GRPO ( right ) for the same query. While GRPO fails to execute the visual action implied by its text, MAPO successfully aligns the visual execution of MLLM agent with its textual reasoning, effectively bridging the reasoning-action gap.
Figure 4: Reward Curves of MAPO and GRPO. Unlike GRPO, which degrades in later stages, MAPO maintains stable learning, validating its scalability for multimodal agents large-scale training .
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Optimization
Global Batch Size
96
Mini Batch Size
96
Micro Batch Size (per GPU)
4
Optimizer
AdamW
Peak Learning Rate
1e-6
Appendix
Table 2: Hyperparameter settings for MAPO training.
The fundamental goal of agentic visual reasoning is to improve the success rate of multimodal large language models (MLLMs) on complex tasks. We rethink agentic visual reasoning through two key dimensions of tool use: Mode Adaptiveness and Tool Effect. Mode Adaptiveness characterizes whether an MLLM recognizes when tools are necessary and invokes them accordingly, avoiding unnecessary computational overhead while improving performance on problems requiring tool assistance. Tool Effect characterizes whether tools extend the model's capabilities on problems unsolvable through tool-free reasoning without introducing errors on problems it can already solve. Our analysis quantifies these properties and reveals that existing models exhibit limited Mode Adaptiveness, while tool-use gains on hard examples are largely offset by harm on easy ones. Motivated by these observations, we propose Beacon, a novel agentic visual reasoning model trained with supervised fine-tuning (SFT) and reinforcement learning (RL). Its RL stage combines Necessity-Aware Adaptive Reward and Hint-Guided Capability Expansion. Necessity-Aware Adaptive Reward encourages tool-free solutions when they succeed while preserving full reward for successful tool use when tool-free rollouts fail. Hint-Guided Capability Expansion uses verified, answer-free expert hints to recover learning signals from all-wrong rollout groups, aiming to extend tool-use capability on the hardest problems. Across 13 benchmarks, Beacon achieves the highest average score among the evaluated open-source models and ranks first on 11 benchmarks. On five diagnostic benchmarks, it improves the average tool-available accuracy over its tool-free accuracy by 1.96 points and achieves the largest tool-gain minus tool-harm score (+3.14 points). These results show Beacon's advanced performance, Mode Adaptiveness, and the net benefit of tool use.
Vision-language models with extended reasoning succeed on complex problems, but many real-world problems require external tools that internal reasoning alone often cannot resolve. Agentic reasoning therefore interleaves two behaviors with a structural asymmetry: thinking (the self-contained default) and tool use (a high-variance auxiliary acting). We refer to this asymmetry as the Thinking-Acting Gap. Under standard RL recipes like GRPO, the gap manifests as two diagnostic symptoms during training: tool use is attempted on only ~30% of rollouts, and when attempted, the tool-using rollouts within a group are all-wrong on ~40% of questions, suppressing the learning signal at the tool calls that needed it. We propose AXPO (Agent eXplorative Policy Optimization): for each all-wrong tool-using subgroup, AXPO fixes the thinking prefix and resamples the tool call and its continuation, paired with uncertainty-based prefix selection. Across nine multimodal benchmarks and three scales of Qwen3-VL-Thinking, SFT+AXPO outperforms SFT+GRPO at average (+1.8pp Pass@1 and +1.8pp Pass@4 at 8B on average) and 8B with SFT+AXPO surpasses the 32B Base on Pass@4 with 4 times fewer parameters.
Group-based reinforcement learning methods for multimodal large language models typically rely on trajectory-level credit assignment that applies a single advantage to all tokens in a response. However, multimodal reasoning involves substantially higher perceptual uncertainty than text-only settings, where the model must repeatedly re-examine visual information to verify intermediate interpretations, and different visual groundings can lead to divergent reasoning paths, making such uniform credit assignment particularly inadequate and causing relative advantages to progressively degenerate toward zero. To address these challenges, we propose Multi-Branch Policy Optimization (MBPO), a tree-based framework that constructs reasoning trees at vision-language decision boundaries, enabling sibling branches to explore diverse visual hypotheses and assigning segment-level credit through branch-relative advantages. We further introduce a temporal replay buffer to reuse informative segments while controlling policy staleness. Experiments on several multimodal reasoning benchmarks show that MBPO outperforms representative baselines, improving both learning signal quality and optimization efficiency. The code is publicly available at https://github.com/ShuaiLyu0110/MBPO.
Shuai Lyu, Yuning Gong, Ruiling Gao +7
Beijing University of Posts and Telecommunications Beijing, China · Sichuan University Chengdu, China · Shanghai University Shanghai, China +1