ImagenWorld: Stress-Testing Image Generation Models with Explainable Human Evaluation on Open-ended Real-World Tasks

Samin Mahdizadeh Sani; Max Ku; Nima Jamali; Matina Mahdizadeh Sani; Paria Khoshtab; Wei-Chieh Sun; Parnian Fazel; Zhi Rui Tam

cs.GRMar 29, 2026

ImagenWorld: Stress-Testing Image Generation Models with Explainable Human Evaluation on Open-ended Real-World Tasks

Authors: Samin Mahdizadeh Sani, Max Ku, Nima Jamali, Matina Mahdizadeh Sani, Paria Khoshtab, Wei-Chieh Sun, Parnian Fazel, Zhi Rui Tam, +18 more

Organizations: University of Waterloo · G-G-G · Independent · Comfy Org

Abstract

Advances in diffusion, autoregressive, and hybrid models have enabled high-quality image synthesis for tasks such as text-to-image, editing, and reference-guided composition. Yet, existing benchmarks remain limited, either focus on isolated tasks, cover only narrow domains, or provide opaque scores without explaining failure modes. We introduce ImagenWorld, a benchmark of 3.6K condition sets spanning six core tasks (generation and editing, with single or multiple references) and six topical domains (artworks, photorealistic images, information graphics, textual graphics, computer graphics, and screenshots). The benchmark is supported by 20K fine-grained human annotations and an explainable evaluation schema that tags localized object-level and segment-level errors, complementing automated VLM-based metrics. Our large-scale evaluation of 14 models yields several insights: (1) models typically struggle more in editing tasks than in generation tasks, especially in local edits. (2) models excel in artistic and photorealistic settings but struggle with symbolic and text-heavy domains such as screenshots and information graphics. (3) closed-source systems lead overall, while targeted data curation (e.g., Qwen-Image) narrows the gap in text-heavy cases. (4) modern VLM-based metrics achieve Kendall accuracies up to 0.79, approximating human ranking, but fall short of fine-grained, explainable error attribution. ImagenWorld provides both a rigorous benchmark and a diagnostic tool to advance robust image generation.

Figures & tables

Figure 1: Overview of our dataset and evaluation pipeline, covering six content categories and six generation/editing tasks. Model outputs are assessed by both human annotators (explainable schema) and vision–language models (scores only).

Method	Generation & Editing	Single Ref Guided	Multi Ref Guided	Human Rating	Topic Variety	Explainable Trace
ImagenHub ( Ku et al., 2024b )	✓	✓	✗	✓	✓	✗
GenAI-Arena ( Jiang et al., 2024 )	✓	✗	✗	✓	✗	✗
DreamBench++ ( Peng et al., 2025 )	✗	✓	✗	✗	✓	✗
I 2 EBench ( Ma et al., 2024 )	✗	✗	✗	✓	✓	✗
ICE-Bench ( Pan et al., 2025 )	✓	✓	✗	✗	✓	✗
MMIG-Bench ( Hua et al., 2025 )	✗	✓	✗	✓	✓	✓

Table 1: Comparison with different evaluation benchmarks on different properties.

Figure 2: Illustrative examples from our dataset, showing successful and failure cases for each task

Figure 3: Examples include object-level issues, where expected objects are missing or distorted, and segment-level issues, where SoM partitions highlight specific regions with visual inconsistencies that affect evaluation scores.

Model	Architecture	TIG	TIE	SRIG	SRIE	MRIG	MRIE
InstructPix2Pix ( Brooks et al., 2023b )	Diffusion	✗	✓	✗	✗	✗	✗
SDXL ( Podell et al., 2023 )	Diffusion	✓	✗	✗	✗	✗	✗
Infinity ( Han et al., 2025 )	AR	✓	✗	✗	✗	✗	✗
Janus Pro ( Chen et al., 2025b )	AR + Diffusion	✓	✗	✗	✗	✗	✗
GPT-Image-1 ( OpenAI, 2025 )	Unknown	✓	✓	✓	✓	✓	✓
UNO ( Wu et al., 2025c )	AR + Diffusion	✓	✗	✓	✗	✓	✗

Table 2: Task coverage and architectural categorization of the evaluated models are shown in chronological order. We classify multimodal large language models as a form of autoregressive model, and group flow matching within the diffusion family, following recent literature ( Xiong et al., 2025 ; Chang et al., 2025 ) . For closed-source models where architectural details are unavailable, we label them as Unknown.

	Human				Overalls		Agreement
Model	Prompt $\uparrow$ Relevance	Aesthetic $\uparrow$ Quality	Content $\uparrow$ Coherence	Artifacts $\uparrow$	$\bar{O}_{\text{Human}}$	$\bar{O}_{\text{VLM}}$	$\alpha$ Kripp.	$\rho_{s}$ Spear.
	Text-guided Image Generation (TIG)
GPT-Image-1	0.90 $\pm$ 0.14	0.92 $\pm$ 0.10	0.91 $\pm$ 0.13	0.92 $\pm$ 0.12	0.91 $\pm$ 0.10	0.92 $\pm$ 0.13	0.52	0.39
Qwen-Image	0.82 $\pm$ 0.22	0.92 $\pm$ 0.11	0.88 $\pm$ 0.17	0.85 $\pm$ 0.22	0.87 $\pm$ 0.15	0.86 $\pm$ 0.20	0.75	0.76
Flux.1-Krea-dev	0.77 $\pm$ 0.23	0.85 $\pm$ 0.16	0.84 $\pm$ 0.18	0.80 $\pm$ 0.22	0.82 $\pm$ 0.17	0.82 $\pm$ 0.23	0.67	0.79
Gemini-2.0-Flash	0.79 $\pm$ 0.21	0.78 $\pm$ 0.21	0.83 $\pm$ 0.19	0.79 $\pm$ 0.25	0.80 $\pm$ 0.19	0.82 $\pm$ 0.24	0.67	0.70

Table 3: All evaluated models across six core tasks with bolded maxima for the four human metrics and overall in each task.

\bar{O}_{\text{Human}}

denotes the human-annotated overall average;

\bar{O}_{\text{VLM}}

is the VLM-predicted overall.

\alpha

is Krippendorff’s alpha for inter-rater reliability and

\rho_{s}

is the Spearman’s rank correlation for human-vlm alignment.

Figure 4: Mean human evaluation scores across metrics by topic (left) and task (right).

Figure 5: Overall human rating by task and topic for the four unified models that support all six tasks.

Metric	Human–Human		VLM–Human
Metric	Spearman $\rho$	Kendall Acc.	Spearman $\rho$	Kendall Acc.	Bias
Prompt Relevance	0.67 $\pm$ 0.03	0.77 $\pm$ 0.01	0.70 $\pm$ 0.00	0.79 $\pm$ 0.00	$-0.07\pm 0.01$
Aesthetic Quality	0.55 $\pm$ 0.01	0.73 $\pm$ 0.00	0.62 $\pm$ 0.01	0.76 $\pm$ 0.00	$-0.01\pm 0.01$
Content Coherence	0.50 $\pm$ 0.01	0.71 $\pm$ 0.01	0.57 $\pm$ 0.01	0.74 $\pm$ 0.01	$-0.05\pm 0.01$
Artifact	0.64 $\pm$ 0.01	0.76 $\pm$ 0.01	0.59 $\pm$ 0.01	0.75 $\pm$ 0.00	$\;\;0.06\pm 0.00$
Overall	0.68 $\pm$ 0.02	0.76 $\pm$ 0.01	0.72 $\pm$ 0.00	0.78 $\pm$ 0.00	$-0.02\pm 0.00$

Table 4: Fold-averaged Spearman correlations, Kendall’s accuracy, and Bias (average signed difference between VLM and human ratings) comparing VLM-based judgments with human ratings across all criteria.

Figure 6: Percentage of cases where the model generating completely new image or simply return input in image editing tasks.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Label Studio interface: annotators read the prompt, inspect the image, rate four criteria (1–5), and flag VLM/SoM-suggested object- and segment-level issues.

Figure 8: Instruction-following problem: The model placed red and green crewmates instead of pink and yellow, and the yellow sign’s position does not match the request.

Figure 9: Examples of numerical inconsistencies.

Figure 11: Examples of generating a new image when the task is editing.

Figure 12: Examples of plots and diagram issues.

Figure 14: Distribution of error pixel proportions across the dataset. Histograms illustrate the fraction of error pixels relative to total mask pixels, excluding masks with none or all pixels marked as errors.

Model	Artworks	Computer Graphics	Information Graphics	Photorealistic Images	Screenshots	Textual Graphics
	Text-guided Image Generation
GPT-Image-1	0.95 $\pm$ 0.06	0.90 $\pm$ 0.09	0.81 $\pm$ 0.13	0.96 $\pm$ 0.06	0.93 $\pm$ 0.09	0.92 $\pm$ 0.08
Qwen-Image	0.97 $\pm$ 0.03	0.81 $\pm$ 0.14	0.79 $\pm$ 0.18	0.97 $\pm$ 0.07	0.73 $\pm$ 0.14	0.94 $\pm$ 0.11
Flux.1-Krea-dev	0.92 $\pm$ 0.06	0.75 $\pm$ 0.18	0.67 $\pm$ 0.22	0.94 $\pm$ 0.08	0.73 $\pm$ 0.12	0.88 $\pm$ 0.13
Gemini 2.0 Flash	0.87 $\pm$ 0.09	0.73 $\pm$ 0.20	0.61 $\pm$ 0.26	0.94 $\pm$ 0.09	0.75 $\pm$ 0.12	0.88 $\pm$ 0.14
UNO	0.89 $\pm$ 0.08	0.73 $\pm$ 0.16	0.62 $\pm$ 0.21	0.86 $\pm$ 0.13	0.64 $\pm$ 0.13	0.92 $\pm$ 0.09

Appendix

Table 5: Topic-wise performance across six tasks. Bolded entries are the per-topic best within each task.

Figure 15: Best-performing model per task×topic across all of our 14 baselines.

Figure 16: Best-performing model per task×topic for closed-source (left) and open-source (middle) models based on human overall scores; the $\Delta$ panel (right) shows $\Delta=\text{Closed}-\text{Open}$ .

Figure 17: Average human score per task×topic for closed-source (left) and open-source (middle) models; the $\Delta$ panel (right) shows $\Delta=\text{Closed}-\text{Open}$ .

Figure 18: Topic distribution for the entire dataset

Figure 19: Word cloud of subtopics across the entire dataset, with word size proportional to frequency

Task	Topic	Subtopics (Count)
TIG	A	Collage (26), Cultural/heritage art (9), Digital arts (15), Glitch Art (3), Pixel Art (5), Pop art (5), Stylized art (11), Traditional art (11), oil paintings (16)
	CG	3D renders (17), CAD models (13), Depth maps (11), Facial estimation and analysis (3), Game assets (8), Object detection visuals (11), Semantic or Instance segmentation (16), Texture Map (8), Texture maps (1), VFX and Particles (12)
	I	Diagrams, charts, and graphs (25), Icons and symbols (26), Maps (17), UI mockups (21), Visual instructions and manuals (11)
	P	Astronomical images (18), Long exposure (7), Medical Images (1), Medical images (21), Microscopic imagery (20), Minimalist Designs (5), Photography (28), Retro Futurism (2)
	S	Games (19), Receipts (35), Software interfaces (19), Websites (27)
	T	ASCII Art (6), Branding and visual identity (7), Calligraphy (6), Comics (13), Concrete Poetry (3), Graffiti (Text-based) (9), Illuminated Manuscripts (9), Memes (11), Posters (12), Textile Art with Text (6), Textual Tattoos (7), Typography (6), Word Clouds (5)

Appendix

Table 6: Subtopics across generation tasks (TIG, SRIG, MRIG).

Task	Topic	Subtopics (Count)
TIE	A	Animation (2), Blueprint (1), Ceramic art (1), Collage (4), Conceptual art (1), Cropping (1), Cubism (3), Cultural or Heritage art (4), Digital art (8), Digital arts (2), Expressionism (4), Fauvism (2), Futurism (4), Graffiti (1), Graffiti art (1), Hand drawn blueprint (1), Impressionism (4), Medieval art (1), Neoclassicism (2), Oil Painting (1), Painting (8), Paintings (3), Pixel art (4), Pixel arts (2), Pop art (3), Post-Impressionism (2), Renaissance (6), Renaissance arts (1), Romanticism (4), Sketches (2), Statue (5), Surrealism (3), Traditional art (4), Traditional arts (1), Vector art (3), Vector arts (1)
	CG	3D renders (10), CAD models (10), Depth maps (10), Facial estimation and analysis (10), Game assets (10), Mesh Processing (5), Object detection visuals (10), Physics Simulation (5), Semantic or Instance segmentation (10), Shader Programming (5), Skeletal Animation and Rigging (5), VFX and Particles (5), Volume Rendering (5)
	I	Diagrams, charts, and graphs (61), Exploded Views (5), Exploded views (2), Floor Plan (8), Icons and symbols (8), Maps (5), Tables (6), UI mockups (5)
	P	Astronomical images (21), Medical images (15), Microscopic imagery (20), Photography (45)
	S	Game Screenshots (1), Games (26), Mobile software interfaces (7), QR codes (11), Receipts (3), Software interfaces (50), Websites (14)
	T	Branding and visual identity (27), Comics (15), Memes (26), Posters (16), Typography (16)

Appendix

Table 7: Subtopics across editing tasks (TIE, SRIE, MRIE).

	LLM-based					Embedding & perceptual
Model	Prompt relevance	Aesthetic quality	Content Coherence	Artifact	Overall	CLIP	LPIPS
	Text-guided Image Generation
GPT-4o	0.89 $\pm$ 0.20	0.95 $\pm$ 0.12	0.91 $\pm$ 0.23	0.96 $\pm$ 0.13	0.93 $\pm$ 0.13	0.25 $\pm$ 0.12	N/A
Qwen-Image	0.79 $\pm$ 0.29	0.93 $\pm$ 0.17	0.86 $\pm$ 0.28	0.88 $\pm$ 0.24	0.87 $\pm$ 0.20	0.24 $\pm$ 0.12	N/A
Flux.1-Krea-dev	0.74 $\pm$ 0.30	0.89 $\pm$ 0.20	0.82 $\pm$ 0.33	0.89 $\pm$ 0.25	0.84 $\pm$ 0.23	0.24 $\pm$ 0.12	N/A
Janus Pro	0.40 $\pm$ 0.31	0.52 $\pm$ 0.34	0.60 $\pm$ 0.41	0.58 $\pm$ 0.38	0.52 $\pm$ 0.32	0.23 $\pm$ 0.10	N/A

Appendix

Table 8: Automatic evaluations on all entries in our dataset

	LLM-based					Embedding & perceptual
Model	Prompt relevance	Aesthetic quality	Content Coherence	Artifact	Overall	CLIP	LPIPS
	Text-guided Image Generation
GPT-Image-1	0.88 $\pm$ 0.22	0.95 $\pm$ 0.13	0.91 $\pm$ 0.23	0.96 $\pm$ 0.13	0.92 $\pm$ 0.13	0.24 $\pm$ 0.12	N/A
Qwen-Image	0.77 $\pm$ 0.31	0.93 $\pm$ 0.17	0.87 $\pm$ 0.28	0.89 $\pm$ 0.24	0.86 $\pm$ 0.20	0.23 $\pm$ 0.12	N/A
Flux.1-Krea-dev	0.72 $\pm$ 0.31	0.88 $\pm$ 0.21	0.80 $\pm$ 0.35	0.89 $\pm$ 0.26	0.82 $\pm$ 0.23	0.23 $\pm$ 0.12	N/A
Gemini 2.0 Flash	0.78 $\pm$ 0.28	0.85 $\pm$ 0.23	0.82 $\pm$ 0.31	0.83 $\pm$ 0.29	0.82 $\pm$ 0.24	0.24 $\pm$ 0.12	N/A

Appendix

Table 9: Automatic evaluations on our human-annotated subset

Explore similar work

Qwen-Image-Bench: From Generation to Creation in Text-to-Image Evaluation
May 27, 2026Niantong Li, Guangzheng Hu, Weixu Qiao +35Image Generation Multimodal Model Evaluation
Alibaba Inc.
WorldGenBench: A World-Knowledge-Integrated Benchmark for Reasoning-Driven Text-to-Image Generation
May 2, 2025Ruoshi Xu, Daoan Zhang, Che Jiang +7T2I Generation T2I Generation Evaluation
Southern University of Science and Technology · University of Rochester · New York University +2
Can Image Models Imagine Time? ImageTime: A Novel Benchmark for Probing Visual World Modeling Through Spatiotemporal Consistency
Jun 9, 2026Xinrui Wu, Lichen HuangWorld Model Learning Conditional Image Generation
University of Electronic Science and Technology of China

May 27, 2026J/K move · Enter open · S save

Qwen-Image-Bench: From Generation to Creation in Text-to-Image Evaluation

Niantong Li, Guangzheng Hu, Weixu Qiao, Ying Ba+34

Alibaba Inc.

Image Generation Multimodal Model Evaluation T2I Generation Evaluation

Open paper AskPDF arXiv

Text-to-Image generation has evolved from basic image synthesis into a frequently used core capability in professional creative workflows, where simple text-image alignment can no longer satisfy users' pressing demands for faithful real-world reconstruction and genuine creative expression. Existing benchmarks, however, remain anchored in these foundational criteria and do not yet capture the nuanced capabilities that matter in authentic artistic practice, making it difficult to reliably distinguish state-of-the-art T2I models. To address the gap, we introduce Qwen-Image-Bench, a creator-centric benchmark co-designed with professional artists and grounded in real-world creation scenarios. Qwen-Image-Bench enriches conventional evaluation with two application-driven dimensions: Real-world Fidelity and Creative Generation. Drawing on the staged reasoning inherent in professional artistic workflows, we organize these five pillars into a top-down hierarchical taxonomy that further decomposes into 23 second-level sub-capabilities and 56 third-level verifiable rubrics. To ensure broad coverage, we curate 1000 stratified prompts with each prompt jointly exercising more than four fine-grained facets across multiple pillars. We train a unified judge model Q-Judger based on Qwen3.6-27B, supervised by 80 professional annotators from global art academies under blind labeling and triple-review protocols, that scores every image across all 56 verifiable facets, producing fine-grained, rubric-grounded, and fully attributable diagnostics rather than a single opaque score. Empirically, Qwen-Image-Bench reliably distinguishes leading T2I models, achieving the greatest separation on the two application-driven dimensions of Real-world Fidelity and Creative Generation where existing benchmarks provide little insight, while also providing a trustworthy optimization signal for production-level T2I development.

Abstract

Explore similar work

Qwen-Image-Bench: From Generation to Creation in Text-to-Image Evaluation

WorldGenBench: A World-Knowledge-Integrated Benchmark for Reasoning-Driven Text-to-Image Generation

Can Image Models Imagine Time? ImageTime: A Novel Benchmark for Probing Visual World Modeling Through Spatiotemporal Consistency