Text-to-Image (T2I) models have recently achieved impressive visual fidelity, yet their evaluation remains constrained by benchmarks that are often difficult to interpret and insufficiently diagnostic. Existing skill-based evaluations tend to overlook critical failure modes that strongly impact usability but fall outside standard taxonomies, such as global incoherence arising from missing parts or physically implausible configurations (e.g., floating objects). In addition, prompt difficulty is typically controlled along a single dimension; either prompt length or the number of elements to generate. To address these limitations, we introduce Imag-Eval, a controlled benchmark designed to assess how T2I models ground compositional natural-language instructions into visual outputs. Unlike prior work that conflates surface linguistic complexity with compositional difficulty, Imag-Eval explicitly seeks to disentangles these factors by independently varying both the number of instances and the combination of constraints (rules), while avoiding error propagation. This design enables fine-grained and interpretable analysis of where cross-modal instruction following fails. Our benchmark comprises 1,140 prompts and 8,842 combined rules, and we evaluate it on several state-of-the-art models. Complementing this analysis with an additional study of over 2,000 prompts from a concurrent benchmark, our results suggest that, for structured skills, compositional difficulty is primarily governed by the number of grounded rules and their binding to instances,, rather than by prompt length alone.
Figures & tables
Figure 1: Cohesiveness . Dall-E positioned the elements as required, respected spatial relationships, generated the text with the required typo, but created a human instance with an extra arm.
Figure 3: Examples generated by various models.
Skill
Z-Image-Turbo
FLUX 1.0
Dall-E 3
Gemini-Flash
SDXL
SC
↑Counting
0.55
0.56
0.42
0.70
0.17
0.13
↑Spatial
0.79
0.69
0.59
0.73
0.34
<0.01
↑Size
0.84
0.62
0.70
0.85
0.15
0.20
↑Emotion
0.80(0.94)
0.67(0.92)
0.28(1.00)
0.98(0.98)
0.09(1.00)
0.10(1.00)
↑Color
0.98
0.88
0.69
0.96
0.16
0.36
↑Cohes.
0.50(0.90)
0.76 (0.90)
0.99(0.99)
0.68(0.98)
0.26(1.0)
0.35(1.0)
Table 1: Results for the "all skills" test, on manual annotation. FLUX 1.0=FLUX 1.0-dev, Gemini-Flash=Gemini-Flash-3.1-preview, SDXL=Stable Diffusion XL, SC=Stable Cascade. For the interpretative skills Emotion and Cohesiveness, values in parentheses report pairwise Cohen’s κ , averaged across annotator pairs. Detailed results are available in App. E . For most categories, a higher score means a better evaluation, except for WER.
Figure 4: Breakdown of evaluation metrics (avg) on multiple axis. More complete results available in App. F .
Criteria (avg)
Easy
Medium
Hard
Length
65.37
63.15
67.31
Entities
18.37
17.02
18.94
Relations
4.78
4.66
4.92
Parse depth
2.37
2.47
2.25
Prepositions
6.85
7.04
7.22
Table 2: Statistics on prompts of similar lengths, obtained using NLTK tokenizer. Entities=Nouns, Relations=Number of connectors (with, on, under…), Parse depth=Number of clauses (which, that…) and sentences.
Skill
Coefficients
P-values
Confidence intervals
VIF (L,CL)
Counting
(L) +0.0216 (CL) −0.3634
(L) 0.61 (CL) 9.8×10−17
(L) [−0.06;0.10] (CL) [−0.44;−0.27]
3.52
Spatial
(L) +0.0341 (CL) −0.1910
(L) 0.29 (CL) 4.89×10−9
(L) [−0.029;0.09] (CL) [−0.25;−0.12]
2.83
Size
(L) +0.0418 (CL) −0.1883
(L) 0.24 (CL) 2.13×10−7
(L) [−0.02;0.11] (CL) [−0.25;−0.11]
3.15
Emotion
(L) −0.1203 (CL) −0.0130
(L) 0.001 (CL) 0.73
(L) [−0.19;−0.04] (CL) [−0.08;0.06]
3.03
Colors
(L) −0.0074 (CL) −0.1296
(L) 0.82 (CL) 0.0001
(L) [−0.07;0.05] (CL) [−0.19;−0.06]
3.50
Text (WER)
(L) +0.0910 (CL) +0.1548
(L) 0.017 (CL) 0.000059
(L) [0.01;0.16] (CL) [0.07;0.23]
3.07
Table 3: Results of the multivariate regression and correlation analyses: skill accuracy vs (prompt length and compositional load). Abbreviations: CL=Compositional Load, L=Length, VIF = Variance Inflation Factor. All the coefficients were standardized prior to model fitting. Across most structured skills (Counting, Spatial, Size, Color, and Text), compositional load has a strong and statistically significant effect, whereas prompt length exhibits a weaker effect. In contrast, for interpretative skills (Emotion and Cohesiveness), this relationship is less pronounced.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Configuration parameters controlling difficulty granularity and the number of generated instances.
Table 4: Parameters for image generation, listed for reproducibility purposes. APL=Apache License. All models used were consistent with their intended use.
Figure 7: Our dataset (JSON prompt collections), annotation guidelines, and associated research assets will be released under the Creative Commons Attribution-NonCommercial 4.0 (CC BY-NC 4.0) license, while the accompanying source code will be released under the MIT License. These resources are intended solely for research, educational, and non-commercial R&D purposes. Textual statistics are extracted using the default tokenizer of NLTK Bird and Loper (2004) . We will release validated annotation samples; generated images will be released only when permitted by the corresponding model licenses and API terms.
Figure 8: A screenshot of the file containing instructions disclosed to the annotators. We included relevant image examples and how they will annotate relevant fields on the linked spreadsheet.
Figure 9: A screenshot containing an example of spreadsheet given to each annotator. This corresponds to the z-image distribution.
Skill
Z-Image-Turbo
FLUX 1.0
Dall-E 3
SDXL
SC
↑Counting
0.61
0.57
0.41
0.27
0.13
↑Spatial
0.82
0.71
0.52
0.28
<0.01
↑Size
0.83
0.77
0.72
0.32
0.20
↑Emotion
0.94
0.45
0.44
0.21
0.10
↑Color
0.92
0.86
0.75
0.52
0.35
↑Cohesiveness
0.90
0.90
0.71
0.11
0.35
Appendix
Table 5: Aggregate across all potential skill combinations, from 2 to 6-skill combinations. FLUX 1.0 = FLUX 1.0-dev, SDXL = Stable Diffusion XL, SC = Stable Cascade. Z-Image-Turbo outperforms other evaluated models in most metrics. Gemini was excluded due to content filter and budget issues leading to missing images.
Figure 10: Skill accuracy as a function of the number of rules (skill combinations). A clear and consistent trend emerges: performance degrades substantially as the number of enforced textual rules increases, across nearly all skills. When considered jointly with our findings on the effect of the number of instances to generate, these results motivate the need for a two-factor analysis of task complexity.
Figure 11: Heatmap comparing prompt length and compositional load. Cells report Pearson correlation coefficients. Across most skills, increases in compositional load exhibit stronger negative correlations with performance than prompt length. Notable exceptions are Emotion and Cohesiveness , for which correlation differences remain marginal.
Figure 12: Examples of confusing cases that lead us to manual annotation validation. Humans are mixed with cats, which leads to various reported accuracies, depending on if they decided to consider the annotation as human, cats, or both. The second figure was generated by stable Diffusion XL with the same prompt as the first one, but it generated an incoherent scene.