Evaluation is shifting from static QA toward agentic settings where models act through external tools. We identify a critical yet underexplored capability within this space - dexterous visual tool use: fine-grained, closed-loop parameterized visual action in which models infer tool parameters from visual evidence, and those parameters directly govern the final result. Existing benchmarks cover web navigation, GUI operation, and software engineering, but rarely target this coupling between visual evidence and execution precision. We propose EASEL, a benchmark evaluating a controlled instance of dexterous visual tool use that adopts reference-guided visual reconstruction as its primary proxy task: the agent incrementally paints a canvas to match a reference image. EASEL additionally includes semantic tasks spanning region annotation, handwriting, and path planning. We further provide EASEL-Data, a 440k-sample two-stage curriculum dataset for trajectory supervision, and EASEL-9B to investigate its effect on this capability. Evaluation of 25 models reveals that current multimodal agents systematically struggle on EASEL. Reconstruction similarity bottlenecks at low levels (0.40-0.54), while trajectory diagnostics expose severe closed-loop instability - models typically saturate early or degrade post-peak. Semantic tasks reveal sharp capability boundaries in precision annotation and path planning. EASEL-9B, trained on EASEL-Data, surpasses the base model by a relative 6.3%, ranking third among all evaluated models.
Figures & tables
Figure 1: Approaches to visual task execution. Existing paradigms each avoid precise execution—through tolerance margins, understanding-only operation, or delegation to external models. EASEL requires the agent to directly produce parameterized actions through closed-loop refinement.
Figure 2: EASEL overview. (A) At each step, the agent observes reference R and current canvas Ct , outputs a parameterized brush_stroke action, and receives the updated canvas as feedback. (B) The full rollout is evaluated along two axes: Result Quality and Trajectory Quality.
Category
#Samples
Resolution
Budget
Programmatic
15
64 × 64
32
Spatial Geometry
40
128 × 128
96
Abstract/Cartoon
15
128 × 128
96
Natural Images
40
128 × 128
128
Semantic Tasks
5
64–512
64–128
Table 1: EASEL benchmark categories.
Figure 3: EASEL-Data construction pipeline. (1) Reference Pool: 11k reference images spanning four source domains. (2) Vector Trajectory Generation: a stroke-based rendering policy produces a 250-step vector trajectory per reference. (3) Two-stage SFT Construction: C1 densely covers early strokes (0–49, with full retention through stroke 16); C2 spans the full trajectory (0–249), yielding 440k next-action prediction samples. For 10% of references, a reasoning model annotates all corresponding trajectory samples with CoT supervision (44k CoT samples).
Control / Agent
Final Sim. ↑
Edge IoU ↑
Blank-white canvas
0.446
0.000
Reference mean-color fill
0.534
0.000
Reference dominant-color fill
0.531
0.000
Gemini 3.1 Pro (best MLLM)
0.535
0.053
Data policy, budget-aligned
0.665
0.281
Data policy, native (250-step)
0.715
0.563
Table 2: Reconstruction score anchors. Uniform fills score zero on Edge IoU; the data policy provides a reference point for attainable scores under two step-budget settings.
Model
Category Breakdown
Overall
Trajectory
Model
Prog.
Spatial
Abs.
Nat.
Final ↑
@50% ↑
AUC ↑
Best ↑
F-B Gap ↓
Gemini 3.1 Pro
0.598
0.579
0.521
0.472
0.535
0.532
0.534
0.566
0.032
Claude Opus 4.7
0.469
0.531
0.468
0.415
0.472
0.473
0.474
0.506
0.035
Claude Sonnet 4.6
0.439
0.496
0.475
0.390
0.447
0.450
0.451
0.478
0.031
GLM-5V-Turbo
0.367
0.495
0.430
0.382
0.428
0.431
0.433
0.487
0.060
GPT-5.5
0.434
0.467
0.430
0.380
0.426
0.436
0.440
0.499
0.073
Table 3: EASEL standard reconstruction results and trajectory diagnostics. Final is the primary ranking metric; @50%, AUC, Best, and F-B Gap are trajectory diagnostics defined in Section 3.3 . Models are grouped by family; EASEL-9B variants are listed separately as a curriculum ablation. Bold indicates the best result in each column; underline indicates the second best.
Figure 4: Average similarity over painting progress across reconstruction tasks. Most gains occur within the first 10% of the step budget, after which trajectories plateau or degrade.
Model
Result Quality
Trajectory
Model
Circle ↑
Apple ↑
Hello ↑
S-Maze ↑
H-Maze ↑
Maze Inv. ↓
Maze Undo
Non-Maze Undo
Avg. Submit (%)
Gemini 3.1 Pro
0.919
0.783
1.000
0.183
0.007
0.21
87
0
43.5
GPT-5.5
0.785
0.761
1.000
1.000
0.031
0.19
81
0
41.6
Claude Sonnet 4.6
0.521
0.491
0.200
0.157
0.047
0.93
0
1
48.7
Claude Opus 4.7
0.000
0.602
0.200
0.026
0.000
0.87
3
124
87.7
GLM-5V-Turbo
0.484
0.722
0.000
0.000
0.000
1.00
0
0
43.0
Table 4: Semantic task results and trajectory diagnostics. Circle and Apple are outline Dice scores; Hello is letter-level recognition accuracy; S-Maze and H-Maze are normalized path-progress scores. Maze Inv.: fraction of invalid actions on maze tasks; Maze Undo / Non-Maze Undo: undo counts on maze vs. outline+handwriting tasks; Avg. Submit%: mean of per-episode submit percentile (submit step ÷ episode budget ×100 ), with budget-exhausted episodes counted as 100%. Bold indicates best result per quality column and lowest Maze Inv.
Figure 5: Best results on each semantic task. Top row : starting canvas. Bottom row : model output (Gemini 3.1 Pro for Circle, Apple, and Hello; GPT-5.5 for S-Maze; Claude Sonnet 4.6 for H-Maze).