cs.CVOct 29, 2026
SaveCan Image Agents Complete Visual Commissions? A Benchmark and Modular Framework
Organizations: Meituan · University of Science and Technology of China
Abstract
Visual commissions require image agents to integrate heterogeneous source materials, preserve requirements across dependent stages, and produce coordinated deliverables. Existing evaluations examine specific capabilities and execution settings, leaving end-to-end commission completion less understood. We introduce Visual Commission Bench (VCB), comprising 600 tasks across 12 categories and three complementary tiers, from capability-focused tasks to complete commissions. VCB uses 5,039 requirement checks and separate visual quality ratings to assess content correctness, visual realization, constraint preservation, and delivery completeness without prescribing an internal workflow. We present V-Protocol, a reference agent that dynamically composes reusable task, cognitive, and expression modules at inference time. Task modules organize delivery, cognitive modules derive content, and expression modules guide visual realization. The modules exchange structured intermediate artifacts, while a shared execution harness tracks artifact revisions and dependencies, manages tool execution, and checks completion. With the same controller and image generator as the agent baselines, V-Protocol achieves 69.0% strict task success on VCB, compared with 42.0% for SCOPE. On Mind-Bench, strict accuracy increases from 39.2% to 45.4%.
Figures & tables
Figure 1: Gallery for V-protocol.
Figure 2: From image generation to visual commission completion. (a). An explicit prompt guides image synthesis. (b). A specified generation goal is supplemented with missing information through reasoning, search, and planning. (c). A commission and its source materials require the agent to determine content and design, preserve active constraints, and coordinate the requested deliverables. VCB evaluates content correctness, visual realization, constraint preservation, and delivery completeness. The panels illustrate task settings rather than mutually exclusive method categories.
| Benchmark | # Tasks | Checks / Ratings | Primary focus |
|---|---|---|---|
| WISE | 1,000 | 3-dim ratings | World-knowledge reasoning |
| Mind-Bench | 500 | 1,633 checks | Reasoning and search |
| KnowGen | 630 | 4-dim ratings | Search-grounded generation |
| Gen-Arena | 300 | 2,533 checks | Entity and constraint satisfaction |
| IA-Bench 1 | 730 | 1,801 checks | Planning, reasoning, search, and memory |
| VCB (Ours) | 600 | 5,039 checks + 4-dim ratings | Distributed-source integration, dependent subtask execution, and coordinated multi-artifact delivery |
Table 1: Comparison of benchmark scale and evaluation focus. Checks denote task-specific verification items, while rating dimensions denote graded evaluation criteria. VCB broadens the evaluation toward end-to-end visual commission completion.
Figure 3: Overview of Visual Commission Benchmark (VCB). VCB contains 600 tasks across three tiers, 12 categories, and 45 subcategories. The S tier focuses on reasoning, planning, long-context extraction, and retrieval. The M tier combines capabilities through iterative editing, visual goal interpretation, diagnosis and repair, and evidence synthesis. The C tier evaluates integrated visual commissions through visual argumentation, example transfer, research visualization, and workspace design. Each panel presents an example input and output, with darker labels indicating the represented subcategories. Documents and workspaces are shown schematically, and the displayed outputs are illustrative rather than experimental results.
| S | M | C | Overall | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| System | RP | TP | VQ | RP | TP | VQ | RP | TP | VQ | RP | TP | VQ |
| Direct | 8.33 | 0.50 | 2.396 | 12.15 | 0.00 | 2.039 | 12.81 | 0.00 | 2.091 | 11.14 | 0.17 | 2.175 |
| Mind-Brush | 13.83 | 1.00 | 2.794 | 6.71 | 0.00 | 2.010 | 15.32 | 0.00 | 2.121 | 12.69 | 0.33 | 2.308 |
| GEMS | 41.20 | 16.00 | 3.639 | 10.49 | 0.00 | 2.236 | 34.69 | 8.00 | 2.337 | 30.92 | 8.00 | 2.737 |
| SCOPE | 84.36 | 55.00 | 3.658 | 77.74 | 39.50 | 4.097 | 82.10 | 31.50 | 4.192 | 81.78 | 42.00 | 3.982 |
| V-Protocol | 96.95 | 82.00 | 4.250 | 91.78 | 67.00 | 4.157 | 94.15 | 58.00 | 4.195 | 94.48 | 69.00 | 4.200 |
Table 2: Commission completion on VCB. Each tier reports ReqPass (RP), TaskPass (TP), and visual quality (VQ). RP and TP are percentages. VQ averages 1–5 ratings, with missing or undecodable required images assigned zero. Overall reports benchmark-wide results. V-Protocol achieves the highest scores, while complete commissions remain challenging even when visual quality is high.
Figure 4: Qualitative comparison. V-Protocol achives better visual commissions completion.
| WISE-Verified | Mind-Bench | |||||||||
| System | Culture | Time | Space | Biology | Physics | Chemistry | Overall | Knowledge | Reasoning | Overall |
| GPT-Image-1.5 | 0.8900 | 0.6917 | 0.8833 | 0.8000 | 0.7583 | 0.7750 | 0.8250 | 0.220 | 0.180 | 0.210 |
| Nano Banana Pro | 0.8975 | 0.8167 | 0.9333 | 0.8167 | 0.8667 | 0.8750 | 0.8760 | 0.400 | 0.440 | 0.410 |
| Qwen-Image-Agent | 0.9200 | 0.9167 | 0.9333 | 0.8333 | 0.8667 | 0.9000 | 0.9020 | 0.404 | 0.436 | 0.420 |
| Direct | 0.4800 | 0.4667 | 0.5583 | 0.3417 | 0.5417 | 0.3083 | 0.4580 | 0.032 | 0.052 | 0.042 |
| Mind-Brush | 0.6550 | 0.6417 | 0.6750 | 0.3750 | 0.5583 | 0.3500 | 0.5740 | 0.136 | 0.060 | 0.098 |
Table 3: Results on WISE-Verified and Mind-Bench. V-Protocol achieves the highest overall scores among the compared systems, with a larger advantage over SCOPE on reasoning-driven Mind-Bench tasks. WISE-Verified reports binary WiScore, and Mind-Bench reports CSA. Knowledge and Reasoning denote the corresponding category averages. The first three rows report published references; the remaining rows use shared backends. Bold marks the best results within the shared-backend group.
| Configuration | RP | TP | VQ |
|---|---|---|---|
| w/o Runtime Enforcement | 82.14 | 60.17 | 3.800 |
| w/o Adaptive Module Selection | 82.20 | 60.83 | 3.862 |
| Full V-Protocol | 94.48 | 69.00 | 4.200 |
Table 4: Component analysis on VCB.
Appendix figures & tables33 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Overview of V-Protocol. Public requests and source materials define a task specification, from which the agent composes an executable program using task (T), cognitive (C), and expression (E) modules. Colored tags indicate module assignments; the dashed branch illustrates a revisable future operation. The model determines content, design, and tool arguments, while the harness checks runtime conditions and records artifacts. Verification feedback informs edits, regeneration, or program revision. The bottom panels illustrate how changes to declared sources make dependency snapshots stale, while changes to an image or relevant criteria invalidate its verification reports. Private benchmark annotations are used only for external evaluation.
| Configuration | RP | TP | VQ |
|---|---|---|---|
| Adaptive selection, | 82.35 | 60.17 | 3.769 |
| Adaptive selection, | 84.24 | 59.83 | 3.853 |
| Adaptive selection, | 84.83 | 59.83 | 3.872 |
| Full V-Protocol (reference) | 94.48 | 69.00 | 4.200 |
Table 5: Selected module-budget results. caps the number of distinct modules loaded throughout a task across all three axes. RP and TP are percentages; VQ is measured on a 0–5 scale.
| Benchmark | Method | Tasks | Tasks with sleep | Sleep records | Repeated commands |
|---|---|---|---|---|---|
| VCB | SCOPE | 600 | 361 | 5,386 | 2,652 |
| V-Protocol | 600 | 0 | 0 | 31 | |
| WISE-Verified | SCOPE | 1,000 | 519 | 5,136 | 1,325 |
| V-Protocol | 1,000 | 0 | 0 | 0 | |
| Mind-Bench | SCOPE | 500 | 358 | 8,970 | 3,829 |
| V-Protocol | 500 | 0 | 0 | 0 |
Table 6: Observed controller-issued shell operations. Each method has one retained log per task. Sleep counts records containing an explicit shell sleep ; repeated commands count occurrences beyond the first identical command within a task after whitespace normalization. These are execution-pattern statistics, not counts of all wasted work. Tool-internal waits are excluded.
| Method | Input (k) | Output (k) | Total (k) | Median time (min) | P90 time (min) |
|---|---|---|---|---|---|
| SCOPE | 4,895.7 | 54.7 | 4,950.4 | 18.45 | 57.01 |
| V-Protocol | 594.7 | 22.1 | 616.8 | 11.01 | 18.88 |
Table 7: Recorded controller usage on VCB. Token counts are per-task means in thousands, with cached input included. Time measures the retained session’s event span. V-Protocol records lower token volumes and shorter median and P90 session durations in these historical runs.
Figure 6: S1: Reasoning. The example compares liquid depth, base pressure, and total force for equal liquid volumes in containers with different base areas.
Figure 7: S2: Planning. The example translates a request for a seaside studio into a spatial arrangement supporting both work and relaxation.
Figure 8: S3: Long-Context Extraction. The example extracts the requested record from a long dossier and presents the relevant fields in a visual summary.
Figure 9: S4: Retrieval. The example requires locating the correct product image and current specifications before constructing the visual output.
Figure 10: M1: Iterative Editing. Successive edits modify the subject and scene. A later instruction reverses one earlier change while retaining the other active edits.
Figure 11: M2: Visual Goals. The task instructions and spatial constraints are embedded in the input image. The output visualizes the inferred solution.
Figure 12: M3: Diagnosis and Repair. The example concerns an inconsistent mirror reflection. Repair must address the reflected object while preserving the surrounding scene.
Figure 13: M4: Evidence Synthesis. The example combines product specifications, prices, and validity records to construct an evidence-grounded comparison.
Figure 14: C1: Visual Argumentation. The example uses a constructed case and a comic narrative to explain why compression can prevent recovery of the original information.
Figure 15: C2: Example Transfer. Three paired demonstrations define coloring, ceramic material conversion, and miniature-worker composition. The illustrated sequence applies these operations to a new object.
Figure 16: C3: Scientific Visualization. The example integrates supplied papers into a visual comparison of FCN, U-Net, and DeepLab, including their approaches to recovering spatial detail.
Figure 17: C4: Workspace Design. The example combines conversation history and source files into a coordinated coffee-shop identity, poster, and banner.
Figure 18: Open workspace: earbuds. Inputs are shown above; outputs from GEMS, SCOPE, and V-Protocol (Ours) are shown below. Empty slots indicate missing deliverables.
Figure 19: Example transfer: bus-depot renewal. Comparison of topic adaptation and material transformation.
Figure 20: Diagnosis and repair: floating table. Comparison of physical-error detection and repair.
Figure 21: Paper evidence: video understanding. Comparison of evidence coverage and information organization across the supplied papers.
Figure 22: Qualitative gallery: visual identity. Typography, material, and composition in the V-Protocol identity illustration.
Figure 23: Qualitative gallery: brand and layout adaptation. Coordinated brand presentation across different page formats.
Figure 24: Qualitative gallery: character design. Character appearance, multiple views, and related presentation materials.
Figure 25: Qualitative gallery: poster design. Examples combining imagery, typography, and promotional composition.
Figure 26: Qualitative gallery: visual information. Examples organizing explanatory and promotional content through text and imagery.
Figure 27: Qualitative gallery: illustration and visual storytelling. Examples of stylized depiction and narrative composition.
Figure 28: Qualitative gallery: fashion and portraits. Examples of portrait treatment, styling, and editorial composition.
Figure 29: Qualitative gallery: landscapes and atmosphere. Examples varying scene composition, lighting, and environmental detail.
Figure 30: Qualitative gallery: imagined scenes. Examples combining familiar subjects with constructed visual settings.
Figure 31: Qualitative gallery: material and detail. Examples emphasizing surface appearance, texture, and object composition.
Figure 32: Qualitative gallery: motion and fantasy. Examples of dynamic poses and imaginative visual composition.
Figure 33: Qualitative gallery: people and characters. Examples of human and character depiction across different visual settings.
Figure 34: Qualitative gallery: culture and events. Examples drawing on cultural themes, seasonal imagery, and public events.
More in this topic
AI Agent Benchmarks
AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.
See all 1,503 papers