Multimodal Large Language Models (MLLMs) are evolving from passive observers into active agents, solving problems through Visual Expansion (invoking visual tools) and Knowledge Expansion (open-web search). However, existing evaluations fall short: they lack flexible tool integration, test visual and search tools separately, and evaluate primarily by final answers. Consequently, they cannot verify if tools were actually invoked, applied correctly, or used efficiently. To address this, we introduce Agentic-MME, a process-verified benchmark for Multimodal Agentic Capabilities. It contains 418 real-world tasks across 6 domains and 3 difficulty levels to evaluate capability synergy, featuring over 2,000 stepwise checkpoints that average 10+ person-hours of manual annotation per task. Each task includes a unified evaluation framework supporting sandboxed code and APIs, alongside a human reference trajectory annotated with stepwise checkpoints along dual-axis: S-axis and V-axis. To enable true process-level verification, we audit fine-grained intermediate states rather than just final answers, and quantify efficiency via an overthinking metric relative to human trajectories. Experimental results show the best model, Gemini3-pro, achieves 56.3% overall accuracy, which falls significantly to 23.0% on Level-3 tasks, underscoring the difficulty of real-world multimodal agentic problem solving.
Figures & tables
Figure 1: Case Studies in Agentic-MME across three difficulty levels. The examples highlight the benchmark’s escalating complexity, evolving from isolated visual operations in Level 1 to deeply synergistic, multi-round visual and knowledge workflows in Level 3.
Benchmark
Capabilities
Evaluation Protocol
Levels
Img Tools
Search Core
Process Verified
Unified Code+Tool
Efficiency Metric
GTA ( Wang et al., 2024 )
✓
✗
✗
✗
✗
✗
m&m’s ( Ma et al., 2024 )
✓
✗
✗
✗
✗
✗
TIR-Bench ( Li et al., 2025a )
✓
✗
✗
✗
✗
✗
VisToolBench ( Guo et al., 2025 )
✓
✗
✗
✗
✗
✗
MMSearch ( Jiang et al., 2024 )
✗
✓
✗
✗
✗
✗
Table 1: Comparison with representative benchmarks. ImgTools : dedicated visual manipulation tools; SearchCore : open-web search is the core task; ProcessVerified : explicit process verification; Unified : protocol supports both code & function-calling for visual tools; Efficiency : cost/efficiency metric; Levels : explicit difficulty levels splits. ( ⚫ : partly supported.)
Figure 2: Data Collection and Annotation pipeline, including image sourcing, backward drafting, granular step-wise annotation, and quality assurance.
Figure 3: Overview of Agentic-MME Dataset Statistics. The benchmark exhibits broad domain and semantic diversity, with increasing tool calls and checkpoints reflecting the escalating demand for long-horizon reasoning across difficulty levels.
Task Difficulty
Share
Ckpts / Task
Tools / Task
Level 1 (Easy)
48.6%
2.89
1.21
Level 2 (Mid)
32.1%
4.64
2.42
Level 3 (Hard)
19.4%
6.67
4.07
Key Property
Value
Key Property
Value
Total Images / Tools
430 / 899
Small Cues ( <10% )
226 (43.1%)
Domains / Sub
6 / 35
Avg. Cue Area
35.8%
Table 2: Dataset Statistics of Agentic-MME. textbfDataset Statistics of Agentic-MME. Specifically, Total Images / Tools denotes the unique visual inputs and tool invocations collected from human trajectories, while PerDescri. Len indicates the average token of the intent description per checkpoint.
Model
Mode
Overall
Level 1
Level 2
Level 3
Acc
S
V
Acc
S
Vto
Vtr
Acc
S
Vto
Vtr
Acc
S
Vto
Vtr
Human
–
93.8
–
–
99.0
–
–
–
92.6
–
–
–
82.3
–
–
–
No Tooluse Models
Gemini 3 pro-preview
–
30.2
–
–
42.9
–
–
–
24.6
–
–
–
7.5
–
–
–
Open-Source Models
Qwen 3 Vl-235B
Gen
28.5
20.8
17.3
43.9
28.6
17.8
18.2
23.3
33.0
20.0
19.3
2.5
10.1
12.1
15.2
Table 3: Comprehensive evaluation results on the Agentic-MME benchmark. We evaluate tool-augmented models under two distinct interaction interfaces: Gen (Code Generation mode, where models write sandboxed Python for visual transforms) and Atm (Atomic mode, interacting via structured function-calling APIs). Within each category, models are sorted in descending order based on their highest (Acc) across all modes. The top model in each category is highlighted with a small icon.
Gemini 3 Flash
Qwen 3 VL-235B
Setting
Overall
L1
L2
L3
Overall
L1
L2
L3
Perception-only
39.82
55.84
30.10
15.24
31.94
48.44
23.13
7.41
Image-only
47.61
64.58
38.90
18.79
36.93
57.67
25.58
6.25
Search-only
46.68
65.62
34.33
22.22
37.10
52.08
31.34
11.11
Full (Img+Search)
52.24
67.60
43.77
27.19
42.85
58.82
34.27
19.23
Table 4: Tool availability analysis. We reprot Acc in different level tasks
Gemini 3 Flash
Qwen 3 VL-235B
Setting
Overall
L1
L2
L3
Overall
L1
L2
L3
Full (baseline)
52.24
67.60
43.77
27.19
42.85
58.82
34.27
19.23
+ Visual Cues
58.37
76.56
45.52
36.25
49.38
72.11
38.17
13.75
+ Stepwise Guidance
76.21
89.58
71.94
51.25
72.80
88.59
66.67
46.91
Table 5: Oracle guidance study. We report acc in different level tasks.
Table 9
Figure 4: Fine-Grained Error Analysis. The heatmap illustrates the frequency of seven failure modes across different difficulty levels, averaged over both Code (Gen) and Atomic (Atm) execution modes. Darker colors denote higher frequencies.