The advent of agentic multimodal models has empowered systems to actively interact with external environments. However, current agents suffer from a profound meta-cognitive deficit: they struggle to arbitrate between leveraging internal knowledge and querying external utilities. Consequently, they frequently fall prey to blind tool invocation, resorting to reflexive tool execution even when queries are resolvable from the raw visual context. This pathological behavior precipitates severe latency bottlenecks and injects extraneous noise that derails sound reasoning. Existing reinforcement learning protocols attempt to mitigate this via a scalarized reward that penalizes tool usage. Yet, this coupled formulation creates an irreconcilable optimization dilemma: an aggressive penalty suppresses essential tool use, whereas a mild penalty is entirely subsumed by the variance of the accuracy reward during advantage normalization, rendering it impotent against tool overuse. To transcend this bottleneck, we propose HDPO, a framework that reframes tool efficiency from a competing scalar objective to a strictly conditional one. By eschewing reward scalarization, HDPO maintains two orthogonal optimization channels: an accuracy channel that maximizes task correctness, and an efficiency channel that enforces execution economy exclusively within accurate trajectories via conditional advantage estimation. This decoupled architecture naturally induces a cognitive curriculum-compelling the agent to first master task resolution before refining its self-reliance. Extensive evaluations demonstrate that our resulting model, Metis, reduces tool invocations by orders of magnitude while simultaneously elevating reasoning accuracy.
Figures & tables
Figure 1: Comparison of tool-use efficiency and task performance. Existing methods rely heavily on tool calls, reflecting limited efficiency awareness. In contrast, our method uses tools far more selectively while achieving the best overall performance, showing that strong accuracy and high efficiency can be attained simultaneously.
Figure 2: Comparison between coupled-reward optimization and HDPO . Existing methods entangle accuracy and efficiency into a single reward signal, while HDPO decouples them into separate branches and combines them only at the final loss, enabling more strategic tool use.
Figure 3: Overview of Metis . A strategic multimodal reasoning agent that selectively invokes code execution, text search, and image search tools during multi-turn reasoning. Rather than invoking tools by default, Metis adaptively determines when tool interactions provide genuinely useful evidence, and otherwise reasons directly from the available context to obtain the final answer.
Models
Perception
Document
V* Bench
HR4K
HR8K
TreeBench
MME RealWorld
SEED2 PLUS
CharXiv(DQ)
CharXiv(RQ)
Open-Source Models
LLaVA-OneVision ( Li et al., 2024a )
75.4
63.0
59.8
37.3
57.4
65.4
-
-
InternVL3-8B ( Zhu et al., 2025 )
81.2
70.0
69.3
38.8
-
69.7
73.6
37.6
Qwen2.5-VL-7B-Instruct ( Bai et al., 2025b )
75.3
65.5
62.1
37.0
56.8
70.4
72.7
40.2
Qwen2.5-VL-32B-Instruct ( Bai et al., 2025b )
80.6
69.3
63.6
42.5
59.1
72.4
83.2
48.0
Table 1: Performance comparison on visual perception and document understanding benchmarks. Metis consistently outperforms existing open-source agentic models, demonstrating that strategic tool use enhances performance on high-resolution and complex document tasks.
Models
MathVista mini
MathVerse mini
WeMath
DynaMath
LogicVista
Avg.
Open-source Models
LLaVA-OneVision ( Li et al., 2024a )
58.6
19.3
20.9
-
33.3
-
Qwen-2.5-VL-7B-Instruct ( Bai et al., 2025b )
68.3
45.6
34.6
53.3
45.9
49.5
InternVL3-8B ( Zhu et al., 2025 )
71.6
39.8
37.1
-
44.1
-
Qwen3-VL-8B-Instruct ( Bai et al., 2025a )
76.3
61.3
38.8
65.5
54.9
59.4
Text-only Reasoning Models
Table 2: Performance comparison on multimodal reasoning benchmarks. By decoupling the efficiency penalty, Metis effectively leverages code execution for complex calculations, achieving state-of-the-art accuracy across all evaluated datasets.
Method
V* Bench
HRBench4K
HRBench8K
CharXiv(RQ)
MathVista mini
Qwen3-VL-8B-Instruct
86.4
78.9
74.6
46.3
76.3
+ standard GRPO ( wtool=0 )
88.7
81.0
79.2
51.0
76.9
+ HDPO ( wtool=0.10 )
88.0
83.5
81.0
52.7
77.4
+ HDPO ( wtool=0.15 )
91.1
83.5
82.0
54.1
78.0
+ HDPO ( wtool=0.20 )
87.4
82.5
80.5
51.5
77.2
Table 3: Ablation study on the optimization objective and efficiency weight. Setting wtool=0 reduces the optimization to standard GRPO (accuracy-only). HDPO consistently improves reasoning performance across all evaluated benchmarks, demonstrating that strategic tool parsimony acts as a catalyst for accuracy.
Figure 4: Direct reasoning from visual context. The query can be resolved through visual understanding and prior knowledge alone. Metis abstains from tool invocation and answers directly, exemplifying the meta-cognitive restraint instilled by HDPO .
Figure 5: Targeted code execution for fine-grained visual analysis. The question requires comparing curves in a specific subplot region that is difficult to resolve at the original image scale. Metis invokes code to crop and enlarge the relevant area, enabling precise identification of the curve behavior near the queried time step.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: System prompt used for metis. The prompt defines the available tools, their invocation formats, and decision guidelines, encouraging the model to answer directly whenever possible and to invoke external tools only when they provide genuinely useful information.
Figure 7: Direct reasoning from visual inspection. The on-screen text is clearly legible from the raw image. Metis correctly extracts the answer without invoking code execution or search tools, avoiding unnecessary computational overhead.
Figure 8: Strategic image search for visual identification. The artwork cannot be reliably identified from visual features alone. Metis invokes image search to match the visual content against external references, then retrieves the completion year from the search results.
Figure 9: Strategic text search for factual knowledge. While the monument is visually identifiable, the queried measurement (cella width) cannot be inferred from the image. Metis recognizes this epistemic gap and invokes text search to retrieve the precise factual information from external sources.
Multimodal agents have achieved notable progress on complex reasoning tasks through tool use, yet remain limited by two issues: statically predefined tool inventories fail to generalize to unseen scenarios, and indiscriminate tool invocation incurs redundant cost and noise-induced errors. We propose MetaForge, a multimodal agent framework that learns when to invoke tools and how to evolve its toolset on demand. MetaForge factorizes agentic behavior into four coupled stages: Decide (judging whether tool use is warranted), Retrieve (selecting suitable tools), Adapt (grounding tool parameters in task context), and Forge (synthesizing new skills online and recycling them into the tool library for reuse), forming a closed judge-retrieve-adapt-forge-recycle loop. A unified orchestration policy enables the agent to choose among answering directly, reusing existing tools, or forging new ones. We jointly optimize invocation necessity, retrieval accuracy, execution effectiveness, and forged-skill reusability via reinforcement learning, with an explicit invocation-cost penalty discouraging redundant calls. Across 12 benchmarks, MetaForge consistently surpasses 16 baselines in accuracy, efficiency, and generalization, validating a paradigm shift from static tool inventories to on-demand self-evolution.
Shouang Wei, Houcheng Min, Xinpeng Dong +8
East China Normal University · Zhejiang University · Tsinghua University +2
Tool-augmented reasoning has emerged as a promising direction for enhancing the reasoning capabilities of multimodal large language models (MLLMs). However, existing studies mainly focus on enabling models to perform tool invocation, while neglecting the necessity of invoking tools. We argue that tool usage is not always beneficial, as redundant or inappropriate invocations largely increase reasoning overhead and even mislead model predictions. To address this issue, we introduce AutoTool, a model that adaptively decides whether to invoke tools according to the characteristics of each query. Within a reinforcement learning framework, we design an explicit dual-mode reasoning strategy with mode-specific reward functions to guide the model toward producing accurate responses. Moreover, to prevent premature bias toward a single reasoning mode, AutoTool jointly explores and balances tool-assisted and text-centric reasoning throughout training, and promotes free exploration in later stages. Extensive experiments demonstrate that AutoTool exhibits outstanding performance and high efficiency, yielding a 21.8% accuracy gain on V* benchmark compared to the base model, and a 44.9% improvement in efficiency over existing tool-augmented methods on POPE benchmark. Code is available at https://github.com/MQinghe/AutoTool.
Qinghe Ma, Zhen Zhao, Yiming Wu +3
The State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China · Shanghai Artificial Intelligence Laboratory +2
The fundamental goal of agentic visual reasoning is to improve the success rate of multimodal large language models (MLLMs) on complex tasks. We rethink agentic visual reasoning through two key dimensions of tool use: Mode Adaptiveness and Tool Effect. Mode Adaptiveness characterizes whether an MLLM recognizes when tools are necessary and invokes them accordingly, avoiding unnecessary computational overhead while improving performance on problems requiring tool assistance. Tool Effect characterizes whether tools extend the model's capabilities on problems unsolvable through tool-free reasoning without introducing errors on problems it can already solve. Our analysis quantifies these properties and reveals that existing models exhibit limited Mode Adaptiveness, while tool-use gains on hard examples are largely offset by harm on easy ones. Motivated by these observations, we propose Beacon, a novel agentic visual reasoning model trained with supervised fine-tuning (SFT) and reinforcement learning (RL). Its RL stage combines Necessity-Aware Adaptive Reward and Hint-Guided Capability Expansion. Necessity-Aware Adaptive Reward encourages tool-free solutions when they succeed while preserving full reward for successful tool use when tool-free rollouts fail. Hint-Guided Capability Expansion uses verified, answer-free expert hints to recover learning signals from all-wrong rollout groups, aiming to extend tool-use capability on the hardest problems. Across 13 benchmarks, Beacon achieves the highest average score among the evaluated open-source models and ranks first on 11 benchmarks. On five diagnostic benchmarks, it improves the average tool-available accuracy over its tool-free accuracy by 1.96 points and achieves the largest tool-gain minus tool-harm score (+3.14 points). These results show Beacon's advanced performance, Mode Adaptiveness, and the net benefit of tool use.