Act Wisely: Cultivating Meta-Cognitive Tool Use in Agentic Multimodal Models

Shilin Yan; Jintao Tong; Hongwei Xue; Xiaojun Tang; Yangyang Wang; Kunyu Shi; Guannan Zhang; Ruixuan Li

cs.CVApr 9, 2026

Act Wisely: Cultivating Meta-Cognitive Tool Use in Agentic Multimodal Models

Authors: Shilin Yan, Jintao Tong, Hongwei Xue, Xiaojun Tang, Yangyang Wang, Kunyu Shi, Guannan Zhang, Ruixuan Li, +1 more

Organizations: Accio Team, Alibaba Group · Huazhong University of Science and Technology

Abstract

The advent of agentic multimodal models has empowered systems to actively interact with external environments. However, current agents suffer from a profound meta-cognitive deficit: they struggle to arbitrate between leveraging internal knowledge and querying external utilities. Consequently, they frequently fall prey to blind tool invocation, resorting to reflexive tool execution even when queries are resolvable from the raw visual context. This pathological behavior precipitates severe latency bottlenecks and injects extraneous noise that derails sound reasoning. Existing reinforcement learning protocols attempt to mitigate this via a scalarized reward that penalizes tool usage. Yet, this coupled formulation creates an irreconcilable optimization dilemma: an aggressive penalty suppresses essential tool use, whereas a mild penalty is entirely subsumed by the variance of the accuracy reward during advantage normalization, rendering it impotent against tool overuse. To transcend this bottleneck, we propose HDPO, a framework that reframes tool efficiency from a competing scalar objective to a strictly conditional one. By eschewing reward scalarization, HDPO maintains two orthogonal optimization channels: an accuracy channel that maximizes task correctness, and an efficiency channel that enforces execution economy exclusively within accurate trajectories via conditional advantage estimation. This decoupled architecture naturally induces a cognitive curriculum-compelling the agent to first master task resolution before refining its self-reliance. Extensive evaluations demonstrate that our resulting model, Metis, reduces tool invocations by orders of magnitude while simultaneously elevating reasoning accuracy.

Figures & tables

Figure 1: Comparison of tool-use efficiency and task performance. Existing methods rely heavily on tool calls, reflecting limited efficiency awareness. In contrast, our method uses tools far more selectively while achieving the best overall performance, showing that strong accuracy and high efficiency can be attained simultaneously.

Figure 2: Comparison between coupled-reward optimization and HDPO . Existing methods entangle accuracy and efficiency into a single reward signal, while HDPO decouples them into separate branches and combines them only at the final loss, enabling more strategic tool use.

Figure 3: Overview of Metis . A strategic multimodal reasoning agent that selectively invokes code execution, text search, and image search tools during multi-turn reasoning. Rather than invoking tools by default, Metis adaptively determines when tool interactions provide genuinely useful evidence, and otherwise reasons directly from the available context to obtain the final answer.

Models	Perception					Document
Models	V* Bench	HR4K	HR8K	TreeBench	MME RealWorld	SEED2 PLUS	CharXiv(DQ)	CharXiv(RQ)
Open-Source Models
LLaVA-OneVision ( Li et al., 2024a )	75.4	63.0	59.8	37.3	57.4	65.4	-	-
InternVL3-8B ( Zhu et al., 2025 )	81.2	70.0	69.3	38.8	-	69.7	73.6	37.6
Qwen2.5-VL-7B-Instruct ( Bai et al., 2025b )	75.3	65.5	62.1	37.0	56.8	70.4	72.7	40.2
Qwen2.5-VL-32B-Instruct ( Bai et al., 2025b )	80.6	69.3	63.6	42.5	59.1	72.4	83.2	48.0

Table 1: Performance comparison on visual perception and document understanding benchmarks. Metis consistently outperforms existing open-source agentic models, demonstrating that strategic tool use enhances performance on high-resolution and complex document tasks.

Models	MathVista ${}_{\text{mini}}$	MathVerse ${}_{\text{mini}}$	WeMath	DynaMath	LogicVista	Avg.
Open-source Models
LLaVA-OneVision ( Li et al., 2024a )	58.6	19.3	20.9	-	33.3	-
Qwen-2.5-VL-7B-Instruct ( Bai et al., 2025b )	68.3	45.6	34.6	53.3	45.9	49.5
InternVL3-8B ( Zhu et al., 2025 )	71.6	39.8	37.1	-	44.1	-
Qwen3-VL-8B-Instruct ( Bai et al., 2025a )	76.3	61.3	38.8	65.5	54.9	59.4
Text-only Reasoning Models

Table 2: Performance comparison on multimodal reasoning benchmarks. By decoupling the efficiency penalty, Metis effectively leverages code execution for complex calculations, achieving state-of-the-art accuracy across all evaluated datasets.

Method	V* Bench	HRBench4K	HRBench8K	CharXiv(RQ)	MathVista ${}_{\text{mini}}$
Qwen3-VL-8B-Instruct	86.4	78.9	74.6	46.3	76.3
+ standard GRPO ( $w_{\mathrm{tool}}{=}0$ )	88.7	81.0	79.2	51.0	76.9
+ HDPO ( $w_{\mathrm{tool}}{=}0.10$ )	88.0	83.5	81.0	52.7	77.4
+ HDPO ( $w_{\mathrm{tool}}{=}0.15$ )	91.1	83.5	82.0	54.1	78.0
+ HDPO ( $w_{\mathrm{tool}}{=}0.20$ )	87.4	82.5	80.5	51.5	77.2

Table 3: Ablation study on the optimization objective and efficiency weight. Setting

w_{\mathrm{tool}}{=}0

reduces the optimization to standard GRPO (accuracy-only). HDPO consistently improves reasoning performance across all evaluated benchmarks, demonstrating that strategic tool parsimony acts as a catalyst for accuracy.

Figure 4: Direct reasoning from visual context. The query can be resolved through visual understanding and prior knowledge alone. Metis abstains from tool invocation and answers directly, exemplifying the meta-cognitive restraint instilled by HDPO .

Figure 5: Targeted code execution for fine-grained visual analysis. The question requires comparing curves in a specific subplot region that is difficult to resolve at the original image scale. Metis invokes code to crop and enlarge the relevant area, enabling precise identification of the curve behavior near the queried time step.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: System prompt used for metis. The prompt defines the available tools, their invocation formats, and decision guidelines, encouraging the model to answer directly whenever possible and to invoke external tools only when they provide genuinely useful information.

Figure 7: Direct reasoning from visual inspection. The on-screen text is clearly legible from the raw image. Metis correctly extracts the answer without invoking code execution or search tools, avoiding unnecessary computational overhead.

Figure 8: Strategic image search for visual identification. The artwork cannot be reliably identified from visual features alone. Metis invokes image search to match the visual content against external references, then retrieves the completion year from the search results.

Figure 9: Strategic text search for factual knowledge. While the monument is visually identifiable, the queried measurement (cella width) cannot be inferred from the image. Metis recognizes this epistemic gap and invokes text search to retrieve the precise factual information from external sources.

Explore similar work

MetaForge: A Self-Evolving Multimodal Agent that Retrieves, Adapts, and Forges Tools On Demand
Jun 1, 2026Shouang Wei, Houcheng Min, Xinpeng Dong +8LLM Agent Skill Learning Multimodal Agents
East China Normal University · Zhejiang University · Tsinghua University +2
Are Tools Always Beneficial? Learning to Invoke Tools Adaptively for Dual-Mode Multimodal LLM Reasoning
May 19, 2026Qinghe Ma, Zhen Zhao, Yiming Wu +3Multimodal Large Language Models Multimodal Reasoning
The State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China · Shanghai Artificial Intelligence Laboratory +2
Beacon: Knowing When and How to Perform Agentic Visual Reasoning
Jul 30, 2026Qixun Wang, Yang Shi, Letian Cheng +11Tool Use in VLMs RL for VLMs
Peking University · Kling Team · HKUST(GZ) +3

Jun 1, 2026J/K move · Enter open · S save

MetaForge: A Self-Evolving Multimodal Agent that Retrieves, Adapts, and Forges Tools On Demand

Shouang Wei, Houcheng Min, Xinpeng Dong, Xin Lin+7

East China Normal University · Zhejiang University · Tsinghua University+2

LLM Agent Skill Learning Multimodal Agents Self-Evolving Agents RL for Tool Use Multimodal Tool Use

Open paper AskPDF arXiv

Multimodal agents have achieved notable progress on complex reasoning tasks through tool use, yet remain limited by two issues: statically predefined tool inventories fail to generalize to unseen scenarios, and indiscriminate tool invocation incurs redundant cost and noise-induced errors. We propose MetaForge, a multimodal agent framework that learns when to invoke tools and how to evolve its toolset on demand. MetaForge factorizes agentic behavior into four coupled stages: Decide (judging whether tool use is warranted), Retrieve (selecting suitable tools), Adapt (grounding tool parameters in task context), and Forge (synthesizing new skills online and recycling them into the tool library for reuse), forming a closed judge-retrieve-adapt-forge-recycle loop. A unified orchestration policy enables the agent to choose among answering directly, reusing existing tools, or forging new ones. We jointly optimize invocation necessity, retrieval accuracy, execution effectiveness, and forged-skill reusability via reinforcement learning, with an explicit invocation-cost penalty discouraging redundant calls. Across 12 benchmarks, MetaForge consistently surpasses 16 baselines in accuracy, efficiency, and generalization, validating a paradigm shift from static tool inventories to on-demand self-evolution.

Abstract

Explore similar work

MetaForge: A Self-Evolving Multimodal Agent that Retrieves, Adapts, and Forges Tools On Demand

Are Tools Always Beneficial? Learning to Invoke Tools Adaptively for Dual-Mode Multimodal LLM Reasoning

Beacon: Knowing When and How to Perform Agentic Visual Reasoning