Recent advancements extend Multimodal Large Language Models (MLLMs) beyond standard visual question answering to utilizing external tools for advanced visual tasks. Despite this progress, precisely executing and effectively composing diverse tools for complex tasks remain persistent bottleneck. Constrained by sparse tool-sets and simple tool-use trajectories, existing benchmarks fail to capture complex and diverse tool interactions, falling short in evaluating model performance under practical, real-world conditions. To bridge this gap, we introduce VisualToolChain-Bench(VTC-Bench), a comprehensive benchmark designed to evaluate tool-use proficiency in MLLMs. To align with realistic computer vision pipelines, our framework features 32 diverse OpenCV-based visual operations. This rich tool-set enables extensive combinations, allowing VTC-Bench to rigorously assess multi-tool composition and long-horizon, multi-step plan execution. For precise evaluation, we provide 680 curated problems structured across a nine-category cognitive hierarchy, each with ground-truth execution trajectories. Extensive experiments on 19 leading MLLMs reveal critical limitations in current models' visual agentic capabilities. Specifically, models struggle to adapt to diverse tool-sets and generalize to unseen operations, with the leading model Gemini-3.0-Pro only achieving 51% on our benchmark. Furthermore, multi-tool composition remains a persistent challenge. When facing complex tasks, models struggle to formulate efficient execution plans, relying heavily on a narrow, suboptimal subset of familiar functions rather than selecting the optimal tools. By identifying these fundamental challenges, VTC-Bench establishes a rigorous baseline to guide the development of more generalized visual agentic models.
Figures & tables
Figure 1: Overview of VTC-Bench and the agentic reasoning workflow. The top panel illustrates the framework architecture, featuring a hierarchical task taxonomy and an MLLM-driven toolkit. The bottom panel exemplifies the multi-stage reasoning trajectory, evolving from visual perception enhancement to compositional reasoning.
Benchmark
Paradigm
#QA
#Tasks
#Tools
RT
MTC
LHC
SFD
V* [ 43 ]
Static
191
2
0
✗
✗
✗
✗
HRBench [ 42 ]
Static
200
6
0
✗
✗
✗
✗
GTA [ 40 ]
Code
229
3
14
✓
✓
✗
✗
TIR-Bench [ 17 ]
Code
1.2K
13
1/Code
✗
✗
✗
✗
Agent-X [ 1 ]
Code
828
5
14
✓
✓
✗
✗
VisualToolBench [ 10 ]
Code
1.2K
5
6
✓
✓
✗
✗
Table 1: Comparison of VTC-Bench with representative multimodal benchmarks. Compared to existing benchmarks, our work encompasses 32 diverse tools and supports dual interaction paradigms (Code & Interface). RT : Reference tool-call trajectory provided for analysis. MTC : Multi-tool composition for a single task. LHC : Long-horizon calling with extended execution steps. Tasks necessitate extensive sequential tool invocations. SFD : Strict Functional Dependency where prior outputs are mandatory inputs for subsequent tools. As exemplified in Fig. 8(a) , pre-processing serves as a mandatory prerequisite for accurate downstream tool execution.
Figure 2: Overview of VTC-Bench. Our benchmark’s progressive design structures 9 VQA tasks into a three-tier cognitive hierarchy, evaluating MLLMs’ multi-tool orchestration from basic visual recovery to high-level logical deduction.Each task provides a reference toolchain to enable fine-grained diagnostic analysis.
Figure 4
Task Category
Data Source
QA Construction
Samples
Answer Type
Attention Focusing
HRBench [ 42 ] & Web
Human-annotated
45
Open-ended
Chart
ChartQA [ 24 ] & Web
Human-annotated
100
Single-choice
Color
ColorBench [ 19 ] & Web
Human-annotated
90
Single-choice
Counting
Kaggle & Web
Human-annotated
85
Single-choice
Math
Web & Manual Construction
Human-annotated
110
Single-choice
Measurement
Web
Human-annotated
105
Single-choice
Table 3: Overview of 9 distinct tasks in our benchmark. All tasks feature human-annotated questions and single-choice answers to ensure rigorous evaluation.
Model
Size
Setting
Overall
Delta
OCR
Attn.
Rest.
Chart
Meas.
Count.
Math
Spat.
Color
Category 1: Proprietary Tool-use Models
GPT-o3
—
Base
31.91
20.00
26.67
32.00
30.00
38.10
35.29
28.18
35.56
35.56
Code
36.62
+4.71
28.00
31.11
34.00
40.00
38.10
34.12
30.91
51.11
42.22
Inter.
36.76
+4.85
28.00
24.44
34.00
39.00
35.24
40.00
34.55
44.44
44.44
GPT-o4-mini
—
Base
31.18
18.00
13.33
30.00
29.00
33.33
44.71
30.91
42.22
30.00
Code
33.68
+2.50
12.00
11.11
30.00
41.00
31.43
35.29
40.00
46.67
37.78
Table 4: Comprehensive evaluation results. Bold and Underlined indicates the highest value in each column. For Overall and Delta columns: Green indicates improvement and Red indicates degradation compared to base. For subcategories, Yellow indicates an improvement of more than 10% compared to base.
Figure 4: Overview of workflows for two tool-use protocols of LLM agents during evaluation. The agent iteratively interacts with tools within the Max Round constraint, ultimately synthesizing tool execution results to generate the final answer.
Model
TCR (%)
APR (%)
All Calls
Effective Calls
Eff. (%)
Avg. T.
Uniq. T.
MAE
Avg. T.
Uniq. T.
MAE
DeepEyes
66.32
29.26
1.90
1.04
4.09
1.21
0.90
3.97
63.68
Qwen3-VL-235B
86.62
38.09
2.19
1.77
3.02
1.51
1.47
3.36
68.95
GPT-5.2
94.26
40.74
13.23
4.08
9.96
2.22
1.97
3.01
16.78
GPT-o3
93.68
36.76
9.77
3.47
6.98
1.68
1.51
3.32
17.20
GPT-o4-mini
76.62
32.50
3.63
1.77
3.63
1.04
1.02
3.76
28.65
Table 5: Statistical analysis of tool invocation performance. Beyond the Tool-calling Rate(TCR) and Average Pass Rate (APR), we provide a fine-grained evaluation categorized by total attempted tool calls (All Calls) and the effective toolchain (Effective Calls). For both categories, we detail the average number of calls (Avg. T.), the number of unique tools used (Uniq. T.), and the Mean Absolute Error (MAE) from the ground-truth length. Finally, the Tool Usage Efficiency ( Efftool , denoted as Eff.) represents the ratio of effective steps to total attempted steps.
Figure 5: Relationships among Tool Call Rate, Tool Call Efficiency, and APR. The panels illustrate APR vs. Tool Call Rate (left), APR vs. Tool Call Efficiency (middle), and Tool Call Rate vs. Tool Call Efficiency (right).
Figure 6: Distribution of tool utilization across different models. The usage patterns remain largely consistent across models, with the most frequently invoked tools including Zoom In, Inrange Color, Rotate, Histogram, and Connected Components .
Figure 7: Comprehensive analysis for Gemini-3.0-Flash. (a) Correlation between sample density and APR. (b) Distributional gap between Ground Truth and Model. Both metrics are stratified by chain length (left) and tool diversity (right).
Figure 8: Qualitative analysis of failure modes in tool utilization. (a) The agent mistakenly invokes Draw Circle and Draw Line based on unfounded intuition, leading to a speculative answer. (b) The agent blindly adopts preliminary tool outputs without visual validation, bypassing essential verification steps.
Figure 9: Impact of system prompts and prompt information density on model APR. We evaluate the performance of three distinct models, under four prompting configurations: Direct, Weak Prompt, Strong Prompt, and Strong Prompt + GT Tools.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Tool Name
Description
Geometry
Resize
Adjusts the image dimensions to a specified size.
Rotate
Rotates the image by a specified angle.
Translate
Shifts the image location along the x or y axis.
Flip
Mirrors the image horizontally, vertically, or both.
Crop
Extracts a specific rectangular region of interest (ROI).
Appendix
Table 6: Detailed taxonomy and definitions of the 35 OpenCV-based tools utilized in our benchmark. The tools are categorized into four logical groups: Geometry, Enhancement, Feature Extraction, and Drawing.
Figure 10: Category-level distribution difference compared with ground truth for Gemini-3.0-Flash.
Figure 11: Supplementary examples of Attention Focusing tasks.
Figure 12: Supplementary examples of Chart tasks.
Figure 13: Supplementary examples of Color tasks.
Figure 14: Supplementary examples of Counting tasks.
Figure 15: Supplementary examples of Math tasks.
Figure 16: Supplementary examples of Measure tasks.
Figure 17: Supplementary examples of OCR tasks.
Figure 18: Supplementary examples of Perceptual Restoration tasks.
Figure 19: Supplementary examples of Spatial Reasoning tasks.
Figure 20: A case study of the reasoning process for Perceptual Restoration tasks.
Figure 21: A case study of the reasoning process for Measure tasks.