Evaluation is shifting from static QA toward agentic settings where models act through external tools. We identify a critical yet underexplored capability within this space - dexterous visual tool use: fine-grained, closed-loop parameterized visual action in which models infer tool parameters from visual evidence, and those parameters directly govern the final result. Existing benchmarks cover web navigation, GUI operation, and software engineering, but rarely target this coupling between visual evidence and execution precision. We propose EASEL, a benchmark evaluating a controlled instance of dexterous visual tool use that adopts reference-guided visual reconstruction as its primary proxy task: the agent incrementally paints a canvas to match a reference image. EASEL additionally includes semantic tasks spanning region annotation, handwriting, and path planning. We further provide EASEL-Data, a 440k-sample two-stage curriculum dataset for trajectory supervision, and EASEL-9B to investigate its effect on this capability. Evaluation of 25 models reveals that current multimodal agents systematically struggle on EASEL. Reconstruction similarity bottlenecks at low levels (0.40-0.54), while trajectory diagnostics expose severe closed-loop instability - models typically saturate early or degrade post-peak. Semantic tasks reveal sharp capability boundaries in precision annotation and path planning. EASEL-9B, trained on EASEL-Data, surpasses the base model by a relative 6.3%, ranking third among all evaluated models.
Figures & tables
Figure 1: Approaches to visual task execution. Existing paradigms each avoid precise execution—through tolerance margins, understanding-only operation, or delegation to external models. EASEL requires the agent to directly produce parameterized actions through closed-loop refinement.
Figure 2: EASEL overview. (A) At each step, the agent observes reference R and current canvas Ct , outputs a parameterized brush_stroke action, and receives the updated canvas as feedback. (B) The full rollout is evaluated along two axes: Result Quality and Trajectory Quality.
Category
#Samples
Resolution
Budget
Programmatic
15
64 × 64
32
Spatial Geometry
40
128 × 128
96
Abstract/Cartoon
15
128 × 128
96
Natural Images
40
128 × 128
128
Semantic Tasks
5
64–512
64–128
Table 1: EASEL benchmark categories.
Figure 3: EASEL-Data construction pipeline. (1) Reference Pool: 11k reference images spanning four source domains. (2) Vector Trajectory Generation: a stroke-based rendering policy produces a 250-step vector trajectory per reference. (3) Two-stage SFT Construction: C1 densely covers early strokes (0–49, with full retention through stroke 16); C2 spans the full trajectory (0–249), yielding 440k next-action prediction samples. For 10% of references, a reasoning model annotates all corresponding trajectory samples with CoT supervision (44k CoT samples).
Control / Agent
Final Sim. ↑
Edge IoU ↑
Blank-white canvas
0.446
0.000
Reference mean-color fill
0.534
0.000
Reference dominant-color fill
0.531
0.000
Gemini 3.1 Pro (best MLLM)
0.535
0.053
Data policy, budget-aligned
0.665
0.281
Data policy, native (250-step)
0.715
0.563
Table 2: Reconstruction score anchors. Uniform fills score zero on Edge IoU; the data policy provides a reference point for attainable scores under two step-budget settings.
Model
Category Breakdown
Overall
Trajectory
Model
Prog.
Spatial
Abs.
Nat.
Final ↑
@50% ↑
AUC ↑
Best ↑
F-B Gap ↓
Gemini 3.1 Pro
0.598
0.579
0.521
0.472
0.535
0.532
0.534
0.566
0.032
Claude Opus 4.7
0.469
0.531
0.468
0.415
0.472
0.473
0.474
0.506
0.035
Claude Sonnet 4.6
0.439
0.496
0.475
0.390
0.447
0.450
0.451
0.478
0.031
GLM-5V-Turbo
0.367
0.495
0.430
0.382
0.428
0.431
0.433
0.487
0.060
GPT-5.5
0.434
0.467
0.430
0.380
0.426
0.436
0.440
0.499
0.073
Table 3: EASEL standard reconstruction results and trajectory diagnostics. Final is the primary ranking metric; @50%, AUC, Best, and F-B Gap are trajectory diagnostics defined in Section 3.3 . Models are grouped by family; EASEL-9B variants are listed separately as a curriculum ablation. Bold indicates the best result in each column; underline indicates the second best.
Figure 4: Average similarity over painting progress across reconstruction tasks. Most gains occur within the first 10% of the step budget, after which trajectories plateau or degrade.
Model
Result Quality
Trajectory
Model
Circle ↑
Apple ↑
Hello ↑
S-Maze ↑
H-Maze ↑
Maze Inv. ↓
Maze Undo
Non-Maze Undo
Avg. Submit (%)
Gemini 3.1 Pro
0.919
0.783
1.000
0.183
0.007
0.21
87
0
43.5
GPT-5.5
0.785
0.761
1.000
1.000
0.031
0.19
81
0
41.6
Claude Sonnet 4.6
0.521
0.491
0.200
0.157
0.047
0.93
0
1
48.7
Claude Opus 4.7
0.000
0.602
0.200
0.026
0.000
0.87
3
124
87.7
GLM-5V-Turbo
0.484
0.722
0.000
0.000
0.000
1.00
0
0
43.0
Table 4: Semantic task results and trajectory diagnostics. Circle and Apple are outline Dice scores; Hello is letter-level recognition accuracy; S-Maze and H-Maze are normalized path-progress scores. Maze Inv.: fraction of invalid actions on maze tasks; Maze Undo / Non-Maze Undo: undo counts on maze vs. outline+handwriting tasks; Avg. Submit%: mean of per-episode submit percentile (submit step ÷ episode budget ×100 ), with budget-exhausted episodes counted as 100%. Bold indicates best result per quality column and lowest Maze Inv.
Figure 5: Best results on each semantic task. Top row : starting canvas. Bottom row : model output (Gemini 3.1 Pro for Circle, Apple, and Hello; GPT-5.5 for S-Maze; Claude Sonnet 4.6 for H-Maze).
Real-world multimodal agents solve multi-step workflows grounded in visual evidence. For example, an agent can troubleshoot a device by linking a wiring photo to a schematic and validating the fix with online documentation, or plan a trip by interpreting a transit map and checking schedules under routing constraints. However, existing multimodal benchmarks mainly evaluate single-turn visual reasoning or specific tool skills, and they do not fully capture the realism, visual subtlety, and long-horizon tool use that practical agents require. We introduce AgentVista, a benchmark for generalist multimodal agents that spans 25 sub-domains across 7 categories, pairing realistic and detail-rich visual scenarios with natural hybrid tool use. Tasks require long-horizon tool interactions across modalities, including web search, image search, page navigation, and code-based operations for both image processing and general programming. Comprehensive evaluation of state-of-the-art models exposes significant gaps in their ability to carry out long-horizon multimodal tool use. Even the best model in our evaluation, Gemini-3-Pro with tools, achieves only 27.3% overall accuracy, and hard instances can require more than 25 tool-calling turns. We expect AgentVista to accelerate the development of more capable and reliable multimodal agents for realistic and ultra-challenging problem solving.
Zhaochen Su, Jincheng Gao, Hangyu Guo +10
Hong Kong University of Science and Technology · University of North Carolina at Chapel Hill · Zhejiang University +1
We introduce MM-ToolSandBox, a benchmark and evaluation framework for visually grounded tool-calling agents. The framework provides a stateful execution environment spanning 500+ tools across 16 application domains, supporting multi-image, multi-turn tasks where agents must ground progressively arriving visual inputs into executable tool calls while handling realistic conversational phenomena (goal revisions, error corrections, state mutations). An automated scenario generation pipeline produces diverse, visually grounded scenarios through information-flow-guided planning and multi-stage quality filtering, yielding 258 human-verified nominal scenarios and 50 variants targeting interactive UI applications. Evaluating 12 state-of-the-art models, from 4B open-weight to frontier proprietary systems, shows that current models still lack robust visual tool-calling capability: even the best model achieves below 50% success rate. Our failure analysis further reveals that visual precision, not only planning, is a primary bottleneck for capable models: 53% of failures stem from incorrect information extraction from images despite otherwise correct task workflows. A planning-to-precision crossover emerges with scale: smaller models fail at deciding what to do, while larger models fail at perceiving what they see, suggesting fundamentally different research directions for improving models at different capability levels. The framework and the benchmark are publicly available at https://github.com/apple/ml-mmtoolsandbox
Agentic multimodal reasoning extends passive image understanding by allowing VLMs to actively acquire and refine visual evidence through visual tool interactions. Effective visual tool use requires three capabilities: grounding tool calls in visual context, composing tools across multiple steps, and adapting reasoning to tool-returned observations. However, existing approaches largely focus on grounding within fixed tool spaces and rigid invocation patterns, leaving composition and adaptation insufficiently addressed. We present VC-Tooler, which learns visual tool use as a compositional and adaptive capability. To this end, we first build a trajectory bank through a hierarchical synthesis pipeline covering three capability levels: single-tool grounding, multi-tool composition, and diverse tool contexts and interfaces. We then train the model in two stages: a supervised cold start that establishes these capabilities, followed by reinforcement learning that encourages accurate, efficient, and context-aware visual tool use. VC-Tooler achieves state-of-the-art performance among open-source models on both general-purpose and agentic benchmarks, including 95.8% on V* and 35.3% on VTC-Bench, and shows promising transfer under richer tool settings at inference time. Project page: https://w1zheng.github.io/VC-Tooler