Text-to-Image (T2I) models have recently achieved impressive visual fidelity, yet their evaluation remains constrained by benchmarks that are often difficult to interpret and insufficiently diagnostic. Existing skill-based evaluations tend to overlook critical failure modes that strongly impact usability but fall outside standard taxonomies, such as global incoherence arising from missing parts or physically implausible configurations (e.g., floating objects). In addition, prompt difficulty is typically controlled along a single dimension; either prompt length or the number of elements to generate. To address these limitations, we introduce Imag-Eval, a controlled benchmark designed to assess how T2I models ground compositional natural-language instructions into visual outputs. Unlike prior work that conflates surface linguistic complexity with compositional difficulty, Imag-Eval explicitly seeks to disentangles these factors by independently varying both the number of instances and the combination of constraints (rules), while avoiding error propagation. This design enables fine-grained and interpretable analysis of where cross-modal instruction following fails. Our benchmark comprises 1,140 prompts and 8,842 combined rules, and we evaluate it on several state-of-the-art models. Complementing this analysis with an additional study of over 2,000 prompts from a concurrent benchmark, our results suggest that, for structured skills, compositional difficulty is primarily governed by the number of grounded rules and their binding to instances,, rather than by prompt length alone.
Figures & tables
Figure 1: Cohesiveness . Dall-E positioned the elements as required, respected spatial relationships, generated the text with the required typo, but created a human instance with an extra arm.
Figure 3: Examples generated by various models.
Skill
Z-Image-Turbo
FLUX 1.0
Dall-E 3
Gemini-Flash
SDXL
SC
↑Counting
0.55
0.56
0.42
0.70
0.17
0.13
↑Spatial
0.79
0.69
0.59
0.73
0.34
<0.01
↑Size
0.84
0.62
0.70
0.85
0.15
0.20
↑Emotion
0.80(0.94)
0.67(0.92)
0.28(1.00)
0.98(0.98)
0.09(1.00)
0.10(1.00)
↑Color
0.98
0.88
0.69
0.96
0.16
0.36
↑Cohes.
0.50(0.90)
0.76 (0.90)
0.99(0.99)
0.68(0.98)
0.26(1.0)
0.35(1.0)
Table 1: Results for the "all skills" test, on manual annotation. FLUX 1.0=FLUX 1.0-dev, Gemini-Flash=Gemini-Flash-3.1-preview, SDXL=Stable Diffusion XL, SC=Stable Cascade. For the interpretative skills Emotion and Cohesiveness, values in parentheses report pairwise Cohen’s κ , averaged across annotator pairs. Detailed results are available in App. E . For most categories, a higher score means a better evaluation, except for WER.
Figure 4: Breakdown of evaluation metrics (avg) on multiple axis. More complete results available in App. F .
Criteria (avg)
Easy
Medium
Hard
Length
65.37
63.15
67.31
Entities
18.37
17.02
18.94
Relations
4.78
4.66
4.92
Parse depth
2.37
2.47
2.25
Prepositions
6.85
7.04
7.22
Table 2: Statistics on prompts of similar lengths, obtained using NLTK tokenizer. Entities=Nouns, Relations=Number of connectors (with, on, under…), Parse depth=Number of clauses (which, that…) and sentences.
Skill
Coefficients
P-values
Confidence intervals
VIF (L,CL)
Counting
(L) +0.0216 (CL) −0.3634
(L) 0.61 (CL) 9.8×10−17
(L) [−0.06;0.10] (CL) [−0.44;−0.27]
3.52
Spatial
(L) +0.0341 (CL) −0.1910
(L) 0.29 (CL) 4.89×10−9
(L) [−0.029;0.09] (CL) [−0.25;−0.12]
2.83
Size
(L) +0.0418 (CL) −0.1883
(L) 0.24 (CL) 2.13×10−7
(L) [−0.02;0.11] (CL) [−0.25;−0.11]
3.15
Emotion
(L) −0.1203 (CL) −0.0130
(L) 0.001 (CL) 0.73
(L) [−0.19;−0.04] (CL) [−0.08;0.06]
3.03
Colors
(L) −0.0074 (CL) −0.1296
(L) 0.82 (CL) 0.0001
(L) [−0.07;0.05] (CL) [−0.19;−0.06]
3.50
Text (WER)
(L) +0.0910 (CL) +0.1548
(L) 0.017 (CL) 0.000059
(L) [0.01;0.16] (CL) [0.07;0.23]
3.07
Table 3: Results of the multivariate regression and correlation analyses: skill accuracy vs (prompt length and compositional load). Abbreviations: CL=Compositional Load, L=Length, VIF = Variance Inflation Factor. All the coefficients were standardized prior to model fitting. Across most structured skills (Counting, Spatial, Size, Color, and Text), compositional load has a strong and statistically significant effect, whereas prompt length exhibits a weaker effect. In contrast, for interpretative skills (Emotion and Cohesiveness), this relationship is less pronounced.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Configuration parameters controlling difficulty granularity and the number of generated instances.
Table 4: Parameters for image generation, listed for reproducibility purposes. APL=Apache License. All models used were consistent with their intended use.
Figure 7: Our dataset (JSON prompt collections), annotation guidelines, and associated research assets will be released under the Creative Commons Attribution-NonCommercial 4.0 (CC BY-NC 4.0) license, while the accompanying source code will be released under the MIT License. These resources are intended solely for research, educational, and non-commercial R&D purposes. Textual statistics are extracted using the default tokenizer of NLTK Bird and Loper (2004) . We will release validated annotation samples; generated images will be released only when permitted by the corresponding model licenses and API terms.
Figure 8: A screenshot of the file containing instructions disclosed to the annotators. We included relevant image examples and how they will annotate relevant fields on the linked spreadsheet.
Figure 9: A screenshot containing an example of spreadsheet given to each annotator. This corresponds to the z-image distribution.
Skill
Z-Image-Turbo
FLUX 1.0
Dall-E 3
SDXL
SC
↑Counting
0.61
0.57
0.41
0.27
0.13
↑Spatial
0.82
0.71
0.52
0.28
<0.01
↑Size
0.83
0.77
0.72
0.32
0.20
↑Emotion
0.94
0.45
0.44
0.21
0.10
↑Color
0.92
0.86
0.75
0.52
0.35
↑Cohesiveness
0.90
0.90
0.71
0.11
0.35
Appendix
Table 5: Aggregate across all potential skill combinations, from 2 to 6-skill combinations. FLUX 1.0 = FLUX 1.0-dev, SDXL = Stable Diffusion XL, SC = Stable Cascade. Z-Image-Turbo outperforms other evaluated models in most metrics. Gemini was excluded due to content filter and budget issues leading to missing images.
Figure 10: Skill accuracy as a function of the number of rules (skill combinations). A clear and consistent trend emerges: performance degrades substantially as the number of enforced textual rules increases, across nearly all skills. When considered jointly with our findings on the effect of the number of instances to generate, these results motivate the need for a two-factor analysis of task complexity.
Figure 11: Heatmap comparing prompt length and compositional load. Cells report Pearson correlation coefficients. Across most skills, increases in compositional load exhibit stronger negative correlations with performance than prompt length. Notable exceptions are Emotion and Cohesiveness , for which correlation differences remain marginal.
Figure 12: Examples of confusing cases that lead us to manual annotation validation. Humans are mixed with cats, which leads to various reported accuracies, depending on if they decided to consider the annotation as human, cats, or both. The second figure was generated by stable Diffusion XL with the same prompt as the first one, but it generated an incoherent scene.
The rapid advancements of Text-to-Image (T2I) models have ushered in a new phase of AI-generated content, marked by their growing ability to interpret and follow user instructions. However, existing T2I model evaluation benchmarks fall short in limited prompt diversity and complexity, as well as coarse evaluation metrics, making it difficult to evaluate the fine-grained alignment performance between textual instructions and generated images. In this paper, we present TIIF-Bench Text-to-Image Instruction Following Benchmark), aiming to systematically assess T2I models' ability in interpreting and following intricate textual instructions. TIIF-Bench comprises 5,000 prompts organized along multiple dimensions and categorized into three levels of difficulty and complexity. To rigorously evaluate robustness to prompt length, each prompt is provided in both short and long versions with identical core semantics. We further propose a novel Global Normalized Edit Distance (GNED) metric for text rendering and provide aspect-ratio-diverse reference images for each prompt to assess style control. In addition, we collect 100 high-quality designer-level prompts covering diverse scenarios for comprehensive evaluation. To enable scalable and fine-grained evaluation, we explore the best paradigm for leveraging the world knowledge encoded in large Vision-Language Models (VLMs) as automated binary evaluators. Through extensive ablations, we develop a fully reproducible evaluator that provides interpretable reasoning and reliable verification, enabling our benchmark to discern subtle variations in T2I model outputs. Through comprehensive benchmarking of mainstream T2I models on TIIF-Bench, we analyze the strengths and weaknesses of current T2I systems and reveal the limitations of existing evaluation benchmarks. Project Page: https://a113n-w3i.github.io/TIIF_Bench/.
Xinyu Wei, Jinrui Zhang, Zeqing Wang +4
The Hong Kong Polytechnic University · OPPO Research Institute · Tsinghua University
Text-to-image (T2I) systems increasingly rely on upstream prompters, either humans or multimodal large language models (MLLMs), to translate user intent into detailed prompts. Yet current benchmarks fix the prompt and only evaluate T2I models, leaving the prompting proficiency of this upstream component entirely unmeasured. We introduce AtelierEval, the first unified benchmark that quantifies prompting proficiency across 360 expert-crafted tasks. Grounded in a cognitive view, it spans three task categories and instantiates tasks using a taxonomy of real-world challenges, with a dual interface for both humans and MLLMs. To enable scalable and reliable evaluation, we propose AtelierJudge, a skill-based, memory-augmented agentic evaluator. It produces subjective and objective scores for prompt-image pairs, achieving a Spearman correlation of 0.79 with human experts, approaching human performance. Extensive experiments benchmark 8 MLLMs against 48 human users across 4 T2I backends, validate AtelierEval as a robust diagnostic tool, and reveal the superiority of mimicry over planning, advocating for an image-augmented direction for future prompters. Our work is released to support future research.
Hanjun Luo, Zhimu Huang, Sylvia Chung +6
New York University Abu Dhabi · Zhejiang University · University of Electronic Science and Technology of China +2
The steady improvements of text-to-image (T2I) generative models lead to slow deprecation of automatic evaluation benchmarks that rely on static datasets, motivating researchers to seek alternative ways to evaluate T2I progress. We present Multimodal Text-to-Image Eval (MT2IE), an evaluation framework in which a single multimodal large language model (MLLM) acts as an evaluator agent, iteratively generating the evaluation prompts and scoring the resulting images. We show that MT2IE's image-text consistency scores have higher correlation with human judgment than metrics previously introduced in the literature. MT2IE generates prompts that are efficient at probing T2I model performance: closely recovering the official T2I model rankings of three structurally distinct benchmarks from just 20 generated evaluation prompts, 28-105x fewer than the benchmarks' own prompt sets. When compared to existing evaluation metrics such as CLIPScore, VIEScore, and VQAScore, MT2IE's T2I model rankings are more faithful and far more consistent across multiple evaluation seeds when using the same number of prompts. MT2IE can also adapt evaluation to the model being tested: rewriting each prompt based on the model's own measured performance to produce a bespoke per-model benchmark that still recovers the official rankings and keeps the evaluated model in an informative scoring range. We hope that these results will encourage the development of dynamic and interactive evaluation frameworks, and mitigate the deprecation of automatic evaluation benchmarks.