Visual commissions require image agents to integrate heterogeneous source materials, preserve requirements across dependent stages, and produce coordinated deliverables. Existing evaluations examine specific capabilities and execution settings, leaving end-to-end commission completion less understood. We introduce Visual Commission Bench (VCB), comprising 600 tasks across 12 categories and three complementary tiers, from capability-focused tasks to complete commissions. VCB uses 5,039 requirement checks and separate visual quality ratings to assess content correctness, visual realization, constraint preservation, and delivery completeness without prescribing an internal workflow. We present V-Protocol, a reference agent that dynamically composes reusable task, cognitive, and expression modules at inference time. Task modules organize delivery, cognitive modules derive content, and expression modules guide visual realization. The modules exchange structured intermediate artifacts, while a shared execution harness tracks artifact revisions and dependencies, manages tool execution, and checks completion. With the same controller and image generator as the agent baselines, V-Protocol achieves 69.0% strict task success on VCB, compared with 42.0% for SCOPE. On Mind-Bench, strict accuracy increases from 39.2% to 45.4%.
Figures & tables
Figure 1: Gallery for V-protocol.
Figure 2: From image generation to visual commission completion. (a). An explicit prompt guides image synthesis. (b). A specified generation goal is supplemented with missing information through reasoning, search, and planning. (c). A commission and its source materials require the agent to determine content and design, preserve active constraints, and coordinate the requested deliverables. VCB evaluates content correctness, visual realization, constraint preservation, and delivery completeness. The panels illustrate task settings rather than mutually exclusive method categories.
Benchmark
# Tasks
Checks / Ratings
Primary focus
WISE
1,000
3-dim ratings
World-knowledge reasoning
Mind-Bench
500
1,633 checks
Reasoning and search
KnowGen
630
4-dim ratings
Search-grounded generation
Gen-Arena
300
2,533 checks
Entity and constraint satisfaction
IA-Bench 1
730
1,801 checks
Planning, reasoning, search, and memory
VCB (Ours)
600
5,039 checks + 4-dim ratings
Distributed-source integration, dependent subtask execution, and coordinated multi-artifact delivery
Table 1: Comparison of benchmark scale and evaluation focus. Checks denote task-specific verification items, while rating dimensions denote graded evaluation criteria. VCB broadens the evaluation toward end-to-end visual commission completion.
Figure 3: Overview of Visual Commission Benchmark (VCB). VCB contains 600 tasks across three tiers, 12 categories, and 45 subcategories. The S tier focuses on reasoning, planning, long-context extraction, and retrieval. The M tier combines capabilities through iterative editing, visual goal interpretation, diagnosis and repair, and evidence synthesis. The C tier evaluates integrated visual commissions through visual argumentation, example transfer, research visualization, and workspace design. Each panel presents an example input and output, with darker labels indicating the represented subcategories. Documents and workspaces are shown schematically, and the displayed outputs are illustrative rather than experimental results.
S
M
C
Overall
System
RP ↑
TP ↑
VQ ↑
RP ↑
TP ↑
VQ ↑
RP ↑
TP ↑
VQ ↑
RP ↑
TP ↑
VQ ↑
Direct
8.33
0.50
2.396
12.15
0.00
2.039
12.81
0.00
2.091
11.14
0.17
2.175
Mind-Brush
13.83
1.00
2.794
6.71
0.00
2.010
15.32
0.00
2.121
12.69
0.33
2.308
GEMS
41.20
16.00
3.639
10.49
0.00
2.236
34.69
8.00
2.337
30.92
8.00
2.737
SCOPE
84.36
55.00
3.658
77.74
39.50
4.097
82.10
31.50
4.192
81.78
42.00
3.982
V-Protocol
96.95
82.00
4.250
91.78
67.00
4.157
94.15
58.00
4.195
94.48
69.00
4.200
Table 2: Commission completion on VCB. Each tier reports ReqPass (RP), TaskPass (TP), and visual quality (VQ). RP and TP are percentages. VQ averages 1–5 ratings, with missing or undecodable required images assigned zero. Overall reports benchmark-wide results. V-Protocol achieves the highest scores, while complete commissions remain challenging even when visual quality is high.
Table 3: Results on WISE-Verified and Mind-Bench. V-Protocol achieves the highest overall scores among the compared systems, with a larger advantage over SCOPE on reasoning-driven Mind-Bench tasks. WISE-Verified reports binary WiScore, and Mind-Bench reports CSA. Knowledge and Reasoning denote the corresponding category averages. The first three rows report published references; the remaining rows use shared backends. Bold marks the best results within the shared-backend group.
Configuration
RP ↑
TP ↑
VQ ↑
w/o Runtime Enforcement
82.14
60.17
3.800
w/o Adaptive Module Selection
82.20
60.83
3.862
Full V-Protocol
94.48
69.00
4.200
Table 4: Component analysis on VCB.
Appendix figures & tables33 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Overview of V-Protocol. Public requests and source materials define a task specification, from which the agent composes an executable program using task (T), cognitive (C), and expression (E) modules. Colored tags indicate module assignments; the dashed branch illustrates a revisable future operation. The model determines content, design, and tool arguments, while the harness checks runtime conditions and records artifacts. Verification feedback informs edits, regeneration, or program revision. The bottom panels illustrate how changes to declared sources make dependency snapshots stale, while changes to an image or relevant criteria invalidate its verification reports. Private benchmark annotations are used only for external evaluation.
Configuration
RP ↑
TP ↑
VQ ↑
Adaptive selection, K=1
82.35
60.17
3.769
Adaptive selection, K=2
84.24
59.83
3.853
Adaptive selection, K=3
84.83
59.83
3.872
Full V-Protocol (reference)
94.48
69.00
4.200
Appendix
Table 5: Selected module-budget results. K caps the number of distinct modules loaded throughout a task across all three axes. RP and TP are percentages; VQ is measured on a 0–5 scale.
Benchmark
Method
Tasks
Tasks with sleep
Sleep records
Repeated commands
VCB
SCOPE
600
361
5,386
2,652
V-Protocol
600
0
0
31
WISE-Verified
SCOPE
1,000
519
5,136
1,325
V-Protocol
1,000
0
0
0
Mind-Bench
SCOPE
500
358
8,970
3,829
V-Protocol
500
0
0
0
Appendix
Table 6: Observed controller-issued shell operations. Each method has one retained log per task. Sleep counts records containing an explicit shell sleep ; repeated commands count occurrences beyond the first identical command within a task after whitespace normalization. These are execution-pattern statistics, not counts of all wasted work. Tool-internal waits are excluded.
Method
Input (k)
Output (k)
Total (k)
Median time (min)
P90 time (min)
SCOPE
4,895.7
54.7
4,950.4
18.45
57.01
V-Protocol
594.7
22.1
616.8
11.01
18.88
Appendix
Table 7: Recorded controller usage on VCB. Token counts are per-task means in thousands, with cached input included. Time measures the retained session’s event span. V-Protocol records lower token volumes and shorter median and P90 session durations in these historical runs.
Figure 6: S1: Reasoning. The example compares liquid depth, base pressure, and total force for equal liquid volumes in containers with different base areas.
Figure 7: S2: Planning. The example translates a request for a seaside studio into a spatial arrangement supporting both work and relaxation.
Figure 8: S3: Long-Context Extraction. The example extracts the requested record from a long dossier and presents the relevant fields in a visual summary.
Figure 9: S4: Retrieval. The example requires locating the correct product image and current specifications before constructing the visual output.
Figure 10: M1: Iterative Editing. Successive edits modify the subject and scene. A later instruction reverses one earlier change while retaining the other active edits.
Figure 11: M2: Visual Goals. The task instructions and spatial constraints are embedded in the input image. The output visualizes the inferred solution.
Figure 12: M3: Diagnosis and Repair. The example concerns an inconsistent mirror reflection. Repair must address the reflected object while preserving the surrounding scene.
Figure 13: M4: Evidence Synthesis. The example combines product specifications, prices, and validity records to construct an evidence-grounded comparison.
Figure 14: C1: Visual Argumentation. The example uses a constructed case and a comic narrative to explain why compression can prevent recovery of the original information.
Figure 15: C2: Example Transfer. Three paired demonstrations define coloring, ceramic material conversion, and miniature-worker composition. The illustrated sequence applies these operations to a new object.
Figure 16: C3: Scientific Visualization. The example integrates supplied papers into a visual comparison of FCN, U-Net, and DeepLab, including their approaches to recovering spatial detail.
Figure 17: C4: Workspace Design. The example combines conversation history and source files into a coordinated coffee-shop identity, poster, and banner.
Figure 18: Open workspace: earbuds. Inputs are shown above; outputs from GEMS, SCOPE, and V-Protocol (Ours) are shown below. Empty slots indicate missing deliverables.
Figure 19: Example transfer: bus-depot renewal. Comparison of topic adaptation and material transformation.
Figure 20: Diagnosis and repair: floating table. Comparison of physical-error detection and repair.
Figure 21: Paper evidence: video understanding. Comparison of evidence coverage and information organization across the supplied papers.
Figure 22: Qualitative gallery: visual identity. Typography, material, and composition in the V-Protocol identity illustration.
Figure 23: Qualitative gallery: brand and layout adaptation. Coordinated brand presentation across different page formats.
Figure 24: Qualitative gallery: character design. Character appearance, multiple views, and related presentation materials.
Modern image generation largely relies on specialized generative models that leave the process of image construction implicit in learned parameters. We explore an alternative: general-purpose coding agents that construct images through executable programs. Our study of zero-shot drawing on AgentDrawBench, a 200-prompt benchmark, reveals a distinctive capability profile. Agents excel at preserving structure and satisfying precise visual constraints, achieving competitive results in domains where compositional accuracy matters more than photorealistic appearance, while remaining limited in perceptual richness. Their visual expressiveness varies with the drawing interface, and a qualitative study on reconstruction shows how procedural skills extracted from human demonstrations support different artistic media. We then examine code as an external visual language of thought: a structured, executable representation that agents can construct and inspect. We test it in visual question answering task and find reconstructed visual code retains substantial information for question answering and, in some domains, supports higher accuracy than the original image. These abilities raise a further question: do agents draw on a shared internal representation of scene content beyond a particular drawing program? We explore an internal visual language-of-thought hypothesis, in which a shared, compositionally structured representation supports visual operations across drawing languages. Cross-harness drawings and images assembled from independent single-pixel queries exhibit related layouts and tones. Agents also maintain self-generated room arrangements across viewpoints after visible drawings are removed. These results demonstrate that the coding agent would maintain a consistent and coherent world inside its imagine even before the harness gets involved and the generation starts.
Hengguang Zhou, Yuanhao Ban, Jiaqi Feng +3
University of California, Los Angeles · Tsinghua University
Agentic image editors decompose complex requests into steps, but when a step fails, they mostly retry with the same narrow set of tools and a judge that scores each output by eye directly. We argue that such agents are bounded by two spaces: the action space of what the agent can do, and the observation space of what its judge can see. On our expert-annotated benchmark, a general-purpose editor yields no usable result in over 40% of the cases, and no judge is reliable across edit types. We present EditAgent, which expands both spaces. It routes each step to general editors and specialized methods, and attaches to each candidate a verification record. With these records, off-the-shelf VLM judges outperform finetuned reward models. Candidates are generated in parallel and judged in one listwise call: experts prefer this to sequential refinement in 78.5% of decided comparisons, at no more than half the judge calls. EditAgent improves two open-weight editors on three benchmarks and serves as a data engine for training stronger editors.
We introduce MM-ToolSandBox, a benchmark and evaluation framework for visually grounded tool-calling agents. The framework provides a stateful execution environment spanning 500+ tools across 16 application domains, supporting multi-image, multi-turn tasks where agents must ground progressively arriving visual inputs into executable tool calls while handling realistic conversational phenomena (goal revisions, error corrections, state mutations). An automated scenario generation pipeline produces diverse, visually grounded scenarios through information-flow-guided planning and multi-stage quality filtering, yielding 258 human-verified nominal scenarios and 50 variants targeting interactive UI applications. Evaluating 12 state-of-the-art models, from 4B open-weight to frontier proprietary systems, shows that current models still lack robust visual tool-calling capability: even the best model achieves below 50% success rate. Our failure analysis further reveals that visual precision, not only planning, is a primary bottleneck for capable models: 53% of failures stem from incorrect information extraction from images despite otherwise correct task workflows. A planning-to-precision crossover emerges with scale: smaller models fail at deciding what to do, while larger models fail at perceiving what they see, suggesting fundamentally different research directions for improving models at different capability levels. The framework and the benchmark are publicly available at https://github.com/apple/ml-mmtoolsandbox