VisualArtifactSpec 1.0: Evaluating Visual Artifacts under Task Specifications
Organizations: Zhejiang University · Accio Team, Alibaba Group
Abstract
Visual artifact tasks test agents' ability to translate task specifications and source materials into outputs that meet practical needs. Evaluation across artifact families helps distinguish broadly applicable authoring capabilities from strengths confined to a particular medium. We introduce VisualArtifactSpec 1.0 (VAS), a benchmark of 211 creation and modification tasks spanning presentations, static graphics, animations and videos, and web interfaces. Each task specifies authoring objectives, source materials, constraints, and deliverables. Existing benchmarks provide limited support for systematically evaluating how authoring choices suit different audiences, purposes, and media across artifact families. VAS evaluates two aspects separately: Task Fulfillment verifies explicit requirements and applicable functionality, while Artifact Performance assesses Content, Organization, Visuals, and Delivery. Task-specific rating anchors instantiate shared criteria for the intended audience, purpose, and medium, with judgments grounded in source materials and artifact inspection records. We evaluate nine model-harness configurations on all 211 tasks. Overall scores range from 42.12 to 70.52, with Artifact Performance accounting for 73.4% of the gap between the highest- and lowest-scoring configurations.
Figures & tables
| Benchmark | Coverage | Task Fulfillment | Artifact Performance | ||||
| Instruction | Functional | Visual | Content | Audience | Medium | ||
| compliance | verification | presentation | accuracy | alignment | adaptation | ||
| StructBench ( Zhuo et al., 2026 ) | G | – | – | – | |||
| DrawAI-Bench ( Cao et al., 2026 ) | G | – | – | – | |||
| BizGenEval ( Li et al., 2026 ) | G | – | – | – | |||
| VAO-Bench ( Guo et al., 2026 ) | G | – | – | – | |||
| Family | Tasks | Share | Task forms |
|---|---|---|---|
| Presentations | 47 | 22.3% | Original decks (31); template adaptation (14); existing-deck revision (2) |
| Static graphics | 72 | 34.1% | Posters (8); infographics and visual summaries (24); banners (10); multi-page series (30) |
| Animations and videos | 32 | 15.2% | Animated graphics (8); animated videos (22); edited videos (2) |
| Web interfaces | 60 | 28.4% | Content browsing (22); data exploration (16); tools and simulations (13); stateful workflows (9) |
| Total | 211 | 100.0% | Four artifact families |
| Task | Intended context | Observation targets |
|---|---|---|
| P80 | Research-group slides | Within-page scanning order and clear stopping points at key conclusions across eight projected slides. |
| P99 | Public-facing mobile carousel | Entry points, density, and reading pace across five cards, including dense tables and image captions. |
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Purpose | P | G | M | W | Total |
|---|---|---|---|---|---|
| Research Communication | 18 | 6 | 0 | 0 | 24 |
| Decision Support | 14 | 2 | 0 | 19 | 35 |
| Conceptual Understanding | 9 | 21 | 26 | 5 | 61 |
| Practical Guidance | 3 | 14 | 3 | 4 | 24 |
| Promotion & Engagement | 3 | 16 | 2 | 6 | 27 |
| Public Information | 0 | 13 | 1 | 4 | 18 |
| ID | Criterion | Evaluation scope |
|---|---|---|
| Content | ||
| C1 | Conceptual expression | Accurate, understandable concepts, terminology, qualifications, and references for the intended audience. |
| C2 | Support for claims | How evidence, data, or observed processes support the artifact’s claims and reasoning. |
| C3 | Content selection | Relevance, priorities, information density, tone, and necessary qualifications for the intended purpose. |
| Organization | ||
| O1 | Overall structure | The overall explanatory, argumentative, decision, or action path. |
| Dimension | Criterion weights in order | Total weight |
|---|---|---|
| Content (C1–C3) | 9, 9, 7 | 25% |
| Organization (O1–O3) | 6, 5, 4 | 15% |
| Visuals (V1–V9) | 5, 8, 7, 5, 7, 6, 3, 3, 6 | 50% |
| Delivery (D1–D3) | 4, 3, 3 | 10% |
| Benchmark | Evidence and scope |
|---|---|
| StructBench [ Zhuo et al., 2026 ] | Atomic factual questions support content accuracy; recurring appearance and spatial-arrangement checks support visual presentation. |
| DrawAI-Bench [ Cao et al., 2026 ] | Fidelity criteria compare text content, formula relations, and connector semantics with references (Appendix A, Table 12), supporting content accuracy alongside visual and editability checks. |
| BizGenEval [ Li et al., 2026 ] | Text and knowledge checks assess textual and factual correctness across visual domains, alongside visual and semantic constraint checks for instruction compliance (Sec. 3.1–3.4). |
| VAO-Bench [ Guo et al., 2026 ] | Content preservation, dependent-update checks, and domain-validity ratings assess content consistency alongside visual and task-requirement checks (Appendix B.3). Its slide and webpage images count as G. |
| PresentBench [ Chen et al., 2026 ] | Material-dependent checklists assess factual and quantitative fidelity; shared rubrics assess visual design. Audience checks cover explanation difficulty and instructional goals across presentation contexts; medium-specific reading practices are not a routine separate target. |
| Code2Video [ Chen et al., 2025 ] | The Accuracy & Depth criterion assesses the correctness of presented knowledge alongside visual quality and temporal logic (Sec. 3.3). TeachQuiz measures knowledge transfer; this alone does not establish audience or medium adaptation. |
| Configuration | C | O | V | D | |||
|---|---|---|---|---|---|---|---|
| GPT-6 Astra | 49.51 | 21.01 | 70.52 | 21.61 | 22.75 | 20.14 | 21.22 |
| GPT-6.1 Sol | 49.40 | 19.98 | 69.37 | 21.30 | 22.23 | 18.63 | 20.00 |
| Claude Opus 5.5 | 48.44 | 16.59 | 65.03 | 16.33 | 19.61 | 15.31 | 19.14 |
| Kimi K3.1 Pro | 48.72 | 13.67 | 62.38 | 14.77 | 18.57 | 11.27 | 15.54 |
| DeepSeek-V4.1-Flash | 47.61 | 12.42 | 60.03 | 10.51 | 15.77 | 11.87 | 14.86 |
| Qwen3.8-Max | 46.07 | 5.87 | 51.95 | 4.58 | 9.53 | 4.36 | 11.19 |
| Configuration | Presentations (47) | Static graphics (72) | Animations/videos (32) | Web interfaces (60) |
|---|---|---|---|---|
| GPT-6 Astra | 71.73 | 72.10 | 71.41 | 67.20 |
| GPT-6.1 Sol | 70.92 | 70.96 | 69.98 | 65.94 |
| Claude Opus 5.5 | 66.51 | 69.53 | 67.71 | 57.05 |
| Kimi K3.1 Pro | 62.83 | 64.75 | 60.96 | 59.95 |
| DeepSeek-V4.1-Flash | 58.97 | 61.68 | 58.32 | 59.79 |
| Qwen3.8-Max | 52.15 | 55.12 | 52.26 | 47.81 |
| Configuration | Research (24) | Decision (16) | Concepts (56) | Guidance (20) | Promotion (21) | Public info. (14) |
|---|---|---|---|---|---|---|
| GPT-6 Astra | 71.89 | 72.52 | 71.96 | 71.76 | 70.49 | 72.62 |
| GPT-6.1 Sol | 71.40 | 71.04 | 70.88 | 69.55 | 69.80 | 71.78 |
| Claude Opus 5.5 | 65.20 | 67.92 | 68.53 | 68.72 | 69.21 | 70.09 |
| Kimi K3.1 Pro | 61.64 | 65.15 | 63.08 | 61.92 | 64.11 | 66.24 |
| DeepSeek-V4.1-Flash | 57.45 | 59.82 | 59.90 | 57.72 | 64.19 | 63.30 |
| Qwen3.8-Max | 51.43 | 53.09 | 53.85 | 50.59 | 56.03 | 57.46 |
| Configuration | Full protocol | Linear AP | Equal AP | Itemwise TF |
|---|---|---|---|---|
| GPT-6 Astra | 70.52 | 77.83 | 70.94 | 70.72 |
| GPT-6.1 Sol | 69.37 | 76.40 | 69.94 | 69.63 |
| Claude Opus 5.5 | 65.03 | 71.07 | 66.04 | 65.61 |
| Kimi K3.1 Pro | 62.38 | — | 63.75 | — |
| DeepSeek-V4.1-Flash | 60.03 | 64.51 | 60.87 | 61.21 |
| Qwen3.8-Max | 51.95 | 53.85 | 53.49 | 53.68 |
| Task IDs |
| P01, P02, P05, P07, P09, P10, P12, P17, P18, P20, P22, P23, P24, P26, P28, P29, P31, P33, P35, P36, P37, P112, P121, P124, P139, P140, P148, P152, P164, P183, P197, P199, P202, P207, P212, P215, P216, P219 |
| Criterion | Action and expected result | Observed result |
|---|---|---|
| FN01 | With a saved reading position, select a search result labeled as the article title; navigate to that title. | The interface restores the old reading position at scroll offset 1350, leaving the title above the viewport. |
| FN10 | After a storage-write failure, restore write access, add a bookmark, and reload; retain the new bookmark. | The interface confirms addition, but the persistent store retains only the earlier bookmarks; new entries disappear after reload. |
Generative AI Evaluation
Generative AI evaluation measures the quality, reliability, diversity, and real-world usefulness of generated outputs and AI-assisted workflows.
See all 335 papers