Recent advancements extend Multimodal Large Language Models (MLLMs) beyond standard visual question answering to utilizing external tools for advanced visual tasks. Despite this progress, precisely executing and effectively composing diverse tools for complex tasks remain persistent bottleneck. Constrained by sparse tool-sets and simple tool-use trajectories, existing benchmarks fail to capture complex and diverse tool interactions, falling short in evaluating model performance under practical, real-world conditions. To bridge this gap, we introduce VisualToolChain-Bench(VTC-Bench), a comprehensive benchmark designed to evaluate tool-use proficiency in MLLMs. To align with realistic computer vision pipelines, our framework features 32 diverse OpenCV-based visual operations. This rich tool-set enables extensive combinations, allowing VTC-Bench to rigorously assess multi-tool composition and long-horizon, multi-step plan execution. For precise evaluation, we provide 680 curated problems structured across a nine-category cognitive hierarchy, each with ground-truth execution trajectories. Extensive experiments on 19 leading MLLMs reveal critical limitations in current models' visual agentic capabilities. Specifically, models struggle to adapt to diverse tool-sets and generalize to unseen operations, with the leading model Gemini-3.0-Pro only achieving 51% on our benchmark. Furthermore, multi-tool composition remains a persistent challenge. When facing complex tasks, models struggle to formulate efficient execution plans, relying heavily on a narrow, suboptimal subset of familiar functions rather than selecting the optimal tools. By identifying these fundamental challenges, VTC-Bench establishes a rigorous baseline to guide the development of more generalized visual agentic models.
Figures & tables
Figure 1: Overview of VTC-Bench and the agentic reasoning workflow. The top panel illustrates the framework architecture, featuring a hierarchical task taxonomy and an MLLM-driven toolkit. The bottom panel exemplifies the multi-stage reasoning trajectory, evolving from visual perception enhancement to compositional reasoning.
Benchmark
Paradigm
#QA
#Tasks
#Tools
RT
MTC
LHC
SFD
V* [ 43 ]
Static
191
2
0
✗
✗
✗
✗
HRBench [ 42 ]
Static
200
6
0
✗
✗
✗
✗
GTA [ 40 ]
Code
229
3
14
✓
✓
✗
✗
TIR-Bench [ 17 ]
Code
1.2K
13
1/Code
✗
✗
✗
✗
Agent-X [ 1 ]
Code
828
5
14
✓
✓
✗
✗
VisualToolBench [ 10 ]
Code
1.2K
5
6
✓
✓
✗
✗
Table 1: Comparison of VTC-Bench with representative multimodal benchmarks. Compared to existing benchmarks, our work encompasses 32 diverse tools and supports dual interaction paradigms (Code & Interface). RT : Reference tool-call trajectory provided for analysis. MTC : Multi-tool composition for a single task. LHC : Long-horizon calling with extended execution steps. Tasks necessitate extensive sequential tool invocations. SFD : Strict Functional Dependency where prior outputs are mandatory inputs for subsequent tools. As exemplified in Fig. 8(a) , pre-processing serves as a mandatory prerequisite for accurate downstream tool execution.
Figure 2: Overview of VTC-Bench. Our benchmark’s progressive design structures 9 VQA tasks into a three-tier cognitive hierarchy, evaluating MLLMs’ multi-tool orchestration from basic visual recovery to high-level logical deduction.Each task provides a reference toolchain to enable fine-grained diagnostic analysis.
Figure 4
Task Category
Data Source
QA Construction
Samples
Answer Type
Attention Focusing
HRBench [ 42 ] & Web
Human-annotated
45
Open-ended
Chart
ChartQA [ 24 ] & Web
Human-annotated
100
Single-choice
Color
ColorBench [ 19 ] & Web
Human-annotated
90
Single-choice
Counting
Kaggle & Web
Human-annotated
85
Single-choice
Math
Web & Manual Construction
Human-annotated
110
Single-choice
Measurement
Web
Human-annotated
105
Single-choice
Table 3: Overview of 9 distinct tasks in our benchmark. All tasks feature human-annotated questions and single-choice answers to ensure rigorous evaluation.
Model
Size
Setting
Overall
Delta
OCR
Attn.
Rest.
Chart
Meas.
Count.
Math
Spat.
Color
Category 1: Proprietary Tool-use Models
GPT-o3
—
Base
31.91
20.00
26.67
32.00
30.00
38.10
35.29
28.18
35.56
35.56
Code
36.62
+4.71
28.00
31.11
34.00
40.00
38.10
34.12
30.91
51.11
42.22
Inter.
36.76
+4.85
28.00
24.44
34.00
39.00
35.24
40.00
34.55
44.44
44.44
GPT-o4-mini
—
Base
31.18
18.00
13.33
30.00
29.00
33.33
44.71
30.91
42.22
30.00
Code
33.68
+2.50
12.00
11.11
30.00
41.00
31.43
35.29
40.00
46.67
37.78
Table 4: Comprehensive evaluation results. Bold and Underlined indicates the highest value in each column. For Overall and Delta columns: Green indicates improvement and Red indicates degradation compared to base. For subcategories, Yellow indicates an improvement of more than 10% compared to base.
Figure 4: Overview of workflows for two tool-use protocols of LLM agents during evaluation. The agent iteratively interacts with tools within the Max Round constraint, ultimately synthesizing tool execution results to generate the final answer.
Model
TCR (%)
APR (%)
All Calls
Effective Calls
Eff. (%)
Avg. T.
Uniq. T.
MAE
Avg. T.
Uniq. T.
MAE
DeepEyes
66.32
29.26
1.90
1.04
4.09
1.21
0.90
3.97
63.68
Qwen3-VL-235B
86.62
38.09
2.19
1.77
3.02
1.51
1.47
3.36
68.95
GPT-5.2
94.26
40.74
13.23
4.08
9.96
2.22
1.97
3.01
16.78
GPT-o3
93.68
36.76
9.77
3.47
6.98
1.68
1.51
3.32
17.20
GPT-o4-mini
76.62
32.50
3.63
1.77
3.63
1.04
1.02
3.76
28.65
Table 5: Statistical analysis of tool invocation performance. Beyond the Tool-calling Rate(TCR) and Average Pass Rate (APR), we provide a fine-grained evaluation categorized by total attempted tool calls (All Calls) and the effective toolchain (Effective Calls). For both categories, we detail the average number of calls (Avg. T.), the number of unique tools used (Uniq. T.), and the Mean Absolute Error (MAE) from the ground-truth length. Finally, the Tool Usage Efficiency ( Efftool , denoted as Eff.) represents the ratio of effective steps to total attempted steps.
Figure 5: Relationships among Tool Call Rate, Tool Call Efficiency, and APR. The panels illustrate APR vs. Tool Call Rate (left), APR vs. Tool Call Efficiency (middle), and Tool Call Rate vs. Tool Call Efficiency (right).
Figure 6: Distribution of tool utilization across different models. The usage patterns remain largely consistent across models, with the most frequently invoked tools including Zoom In, Inrange Color, Rotate, Histogram, and Connected Components .
Figure 7: Comprehensive analysis for Gemini-3.0-Flash. (a) Correlation between sample density and APR. (b) Distributional gap between Ground Truth and Model. Both metrics are stratified by chain length (left) and tool diversity (right).
Figure 8: Qualitative analysis of failure modes in tool utilization. (a) The agent mistakenly invokes Draw Circle and Draw Line based on unfounded intuition, leading to a speculative answer. (b) The agent blindly adopts preliminary tool outputs without visual validation, bypassing essential verification steps.
Figure 9: Impact of system prompts and prompt information density on model APR. We evaluate the performance of three distinct models, under four prompting configurations: Direct, Weak Prompt, Strong Prompt, and Strong Prompt + GT Tools.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Tool Name
Description
Geometry
Resize
Adjusts the image dimensions to a specified size.
Rotate
Rotates the image by a specified angle.
Translate
Shifts the image location along the x or y axis.
Flip
Mirrors the image horizontally, vertically, or both.
Crop
Extracts a specific rectangular region of interest (ROI).
Appendix
Table 6: Detailed taxonomy and definitions of the 35 OpenCV-based tools utilized in our benchmark. The tools are categorized into four logical groups: Geometry, Enhancement, Feature Extraction, and Drawing.
Figure 10: Category-level distribution difference compared with ground truth for Gemini-3.0-Flash.
Figure 11: Supplementary examples of Attention Focusing tasks.
Figure 12: Supplementary examples of Chart tasks.
Figure 13: Supplementary examples of Color tasks.
Figure 14: Supplementary examples of Counting tasks.
Figure 15: Supplementary examples of Math tasks.
Figure 16: Supplementary examples of Measure tasks.
Figure 17: Supplementary examples of OCR tasks.
Figure 18: Supplementary examples of Perceptual Restoration tasks.
Figure 19: Supplementary examples of Spatial Reasoning tasks.
Figure 20: A case study of the reasoning process for Perceptual Restoration tasks.
Figure 21: A case study of the reasoning process for Measure tasks.
Agentic multimodal reasoning extends passive image understanding by allowing VLMs to actively acquire and refine visual evidence through visual tool interactions. Effective visual tool use requires three capabilities: grounding tool calls in visual context, composing tools across multiple steps, and adapting reasoning to tool-returned observations. However, existing approaches largely focus on grounding within fixed tool spaces and rigid invocation patterns, leaving composition and adaptation insufficiently addressed. We present VC-Tooler, which learns visual tool use as a compositional and adaptive capability. To this end, we first build a trajectory bank through a hierarchical synthesis pipeline covering three capability levels: single-tool grounding, multi-tool composition, and diverse tool contexts and interfaces. We then train the model in two stages: a supervised cold start that establishes these capabilities, followed by reinforcement learning that encourages accurate, efficient, and context-aware visual tool use. VC-Tooler achieves state-of-the-art performance among open-source models on both general-purpose and agentic benchmarks, including 95.8% on V* and 35.3% on VTC-Bench, and shows promising transfer under richer tool settings at inference time. Project page: https://w1zheng.github.io/VC-Tooler
We introduce MM-ToolSandBox, a benchmark and evaluation framework for visually grounded tool-calling agents. The framework provides a stateful execution environment spanning 500+ tools across 16 application domains, supporting multi-image, multi-turn tasks where agents must ground progressively arriving visual inputs into executable tool calls while handling realistic conversational phenomena (goal revisions, error corrections, state mutations). An automated scenario generation pipeline produces diverse, visually grounded scenarios through information-flow-guided planning and multi-stage quality filtering, yielding 258 human-verified nominal scenarios and 50 variants targeting interactive UI applications. Evaluating 12 state-of-the-art models, from 4B open-weight to frontier proprietary systems, shows that current models still lack robust visual tool-calling capability: even the best model achieves below 50% success rate. Our failure analysis further reveals that visual precision, not only planning, is a primary bottleneck for capable models: 53% of failures stem from incorrect information extraction from images despite otherwise correct task workflows. A planning-to-precision crossover emerges with scale: smaller models fail at deciding what to do, while larger models fail at perceiving what they see, suggesting fundamentally different research directions for improving models at different capability levels. The framework and the benchmark are publicly available at https://github.com/apple/ml-mmtoolsandbox
Multimodal Large Language Models (MLLMs) are evolving from passive observers into active agents, solving problems through Visual Expansion (invoking visual tools) and Knowledge Expansion (open-web search). However, existing evaluations fall short: they lack flexible tool integration, test visual and search tools separately, and evaluate primarily by final answers. Consequently, they cannot verify if tools were actually invoked, applied correctly, or used efficiently. To address this, we introduce Agentic-MME, a process-verified benchmark for Multimodal Agentic Capabilities. It contains 418 real-world tasks across 6 domains and 3 difficulty levels to evaluate capability synergy, featuring over 2,000 stepwise checkpoints that average 10+ person-hours of manual annotation per task. Each task includes a unified evaluation framework supporting sandboxed code and APIs, alongside a human reference trajectory annotated with stepwise checkpoints along dual-axis: S-axis and V-axis. To enable true process-level verification, we audit fine-grained intermediate states rather than just final answers, and quantify efficiency via an overthinking metric relative to human trajectories. Experimental results show the best model, Gemini3-pro, achieves 56.3% overall accuracy, which falls significantly to 23.0% on Level-3 tasks, underscoring the difficulty of real-world multimodal agentic problem solving.