cs.AINov 16, 2026

VisualArtifactSpec 1.0: Evaluating Visual Artifacts under Task Specifications

Authors: Shuyi Ouyang, Xiaojun Tang, Zijian Zou, Xintong Zhang, Mingli Song, Guannan Zhang, Hongwei Xue

Organizations: Zhejiang University · Accio Team, Alibaba Group

Abstract

Visual artifact tasks test agents' ability to translate task specifications and source materials into outputs that meet practical needs. Evaluation across artifact families helps distinguish broadly applicable authoring capabilities from strengths confined to a particular medium. We introduce VisualArtifactSpec 1.0 (VAS), a benchmark of 211 creation and modification tasks spanning presentations, static graphics, animations and videos, and web interfaces. Each task specifies authoring objectives, source materials, constraints, and deliverables. Existing benchmarks provide limited support for systematically evaluating how authoring choices suit different audiences, purposes, and media across artifact families. VAS evaluates two aspects separately: Task Fulfillment verifies explicit requirements and applicable functionality, while Artifact Performance assesses Content, Organization, Visuals, and Delivery. Task-specific rating anchors instantiate shared criteria for the intended audience, purpose, and medium, with judgments grounded in source materials and artifact inspection records. We evaluate nine model-harness configurations on all 211 tasks. Overall scores range from 42.12 to 70.52, with Artifact Performance accounting for 73.4% of the gap between the highest- and lowest-scoring configurations.

Figures & tables

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Pixels or Code? Benchmarking Instruction-Guided Visual Artifact Operation

    Oct 12, 2026Shun Guo, Shuizhou Shi, Boao Wang +4Image Editing EvaluationText-Guided Image Editing

  2. BizGenEval: A Systematic Benchmark for Commercial Visual Content Generation

    Mar 26, 2026Yan Li, Zezi Zeng, Ziwei Zhou +13Image Generation EvaluationImage Generation

  3. AnnoBench: A Benchmark for Visualization Annotation Generation

    Jul 28, 2026Md Rahat-uz-Zaman, Md Dilshadur Rahman, Andrew McNutt +1Data VisualizationAutomated Evaluation