Text-to-image (T2I) models struggle with compositional prompts requiring multiple objects, bound attributes, accurate counts, or spatial relations. Existing benchmarks face limits from fixed detector vocabularies, aggregated scores, and varying human-validation protocols. We introduce SANEval (Spatial, Attribute, and Numeracy Evaluation), an open-vocabulary compositional benchmark with two key contributions: (i) a public dataset of ~5,250 prompts paired with images from six state-of-the-art T2I models, split into algorithmically generated Simple and fully human-written Hard partitions across five compositional tasks, and (ii) a modular evaluation pipeline combining an LLM prompt parser, an open-vocabulary detector with LLM synonym expansion, and scorers for attribute binding, spatial relations, and numeracy. Each scorer emits numerical scores and diagnostic feedback for missing, extra, or mis-bound elements. Validated against human ratings (n = 500), SANEval achieves positive Pearson correlations across all categories (overall r = 0.222), outperforming CompBench++ (-0.016) and GenEval (-0.030). A manual audit of 24,012 detections shows 96.58% precision, with ablation confirming that synonym expansion is critical to pipeline performance. Dataset: https://huggingface.co/datasets/saneval/saneval-release. Pipeline: https://github.com/nielseni6/saneval.
Figures & tables
Figure 1: Block diagram of the SANEval benchmark.
Figure 2: Block diagram of the enhanced object detection (OD) pipeline. Given a set of expected objects from the Prompt Understanding Module, the pipeline first generates synonyms and hypernyms via an LLM (e.g., albatross → {albatross, seabird, large bird}), which are used as expanded class queries for the open-vocabulary detector (YOLO-E). The detector’s raw outputs are then passed to an LLM-based synonym mapper, which resolves detected labels back to the original expected objects (e.g., detected “bird” → expected “albatross”). This hybrid approach bridges the vocabulary gap between natural language prompts and detector class labels, expanding the detector query vocabulary.
Figure 3: Example of SANEval’s attribute-binding pipeline. Objects are detected and cropped, then a VLM reads their attributes. The upper example satisfies the prompt (“pink stop sign and orange bird”), yielding a score of 1.0 with no feedback. In the lower example (“yellow peach and a lime ceiling”), the ceiling is missing, yielding a score of 0.5 and corresponding diagnostic feedback.
Figure 4: Block diagram descriptions of the three different scorers for the SANEval benchmark.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Pipeline Component
Latency (Local CPU)
Cost (USD)
Spatial Evaluation
0.11 s
$0.02
Numeracy Evaluation
0.12 s
$0.03
Attribute Evaluation
0.26 s
$0.01
Appendix
Table 7: Cost and Latency Breakdown.
Model
Spatial
Numeracy
Attribute-Binding
Total
Color
Shape
Texture
Imagen 3.0
0.8160
0.7720
0.8400
0.5000
0.7240
0.7304
Imagen 4.0
0.7800
0.6380
0.7990
0.5170
0.7410
0.6950
Imagen 4.0 Ultra
0.9390
0.8870
0.9680
0.8110
0.9190
0.9048
Nano Banana
0.9220
0.8680
0.9050
0.6810
0.8150
0.8382
Seedream 3.0
0.9070
0.7430
0.9170
0.4310
0.6540
0.7304
Appendix
Table 11: Reported human-evaluation summary scores for spatial, numeracy, color, shape, and texture. Correlation targets use the original 1 – 5 ratings, rather than these summaries.
Figure 5: Human evaluation interface used to collect adherence responses for spatial dataset.
Figure 6: Human evaluation interface used to collect adherence responses for numeracy dataset.
Figure 7: Human evaluation interface used to collect adherence responses for color binding dataset.
Figure 8: Human evaluation interface used to collect adherence responses for shape binding dataset.
Figure 9: Human evaluation interface used to collect adherence responses for texture binding dataset.
Table 13: Qualitative sample prompts, generated images, scores, and diagnostic feedback from SANEval. Attribute entries show detected-object binding scores; other entries show prompt scores.
Figure 10: Examples from the SANEval-Hard prompt set: Spatial
Figure 11: Examples from the SANEval-Hard prompt set: Numeracy
Figure 12: Examples from the SANEval-Hard prompt set: Attribute Binding
Text-to-Image (T2I) models have recently achieved impressive visual fidelity, yet their evaluation remains constrained by benchmarks that are often difficult to interpret and insufficiently diagnostic. Existing skill-based evaluations tend to overlook critical failure modes that strongly impact usability but fall outside standard taxonomies, such as global incoherence arising from missing parts or physically implausible configurations (e.g., floating objects). In addition, prompt difficulty is typically controlled along a single dimension; either prompt length or the number of elements to generate. To address these limitations, we introduce Imag-Eval, a controlled benchmark designed to assess how T2I models ground compositional natural-language instructions into visual outputs. Unlike prior work that conflates surface linguistic complexity with compositional difficulty, Imag-Eval explicitly seeks to disentangles these factors by independently varying both the number of instances and the combination of constraints (rules), while avoiding error propagation. This design enables fine-grained and interpretable analysis of where cross-modal instruction following fails. Our benchmark comprises 1,140 prompts and 8,842 combined rules, and we evaluate it on several state-of-the-art models. Complementing this analysis with an additional study of over 2,000 prompts from a concurrent benchmark, our results suggest that, for structured skills, compositional difficulty is primarily governed by the number of grounded rules and their binding to instances,, rather than by prompt length alone.
Ibrahim Mohamed Serouis, David Jaramillo Duque
Talan Research and Innovation Center Toulouse, France · Talan Research and Innovation Center Paris, France
Modern text-to-image (T2I) models often have similar total scores but different strengths, making practical selection difficult. Fine-grained benchmarks decompose prompts into questions, yet often return them to prompt scores and fixed categories, weakening attribution and ignoring complexity. Related requirements are also scored separately or as one total, obscuring basic versus compositional failure. We present QC-T2I-Bench, a question-centric framework that converts open prompts into attributed atomic questions and organizes their dependencies with Davidsonian Scene Graphs (DSGs). We use hierarchy-constrained question aggregation to exclude downstream questions after a prerequisite fails and to prevent simple and complex prompts from receiving the same total weight. We then use the DSG structure to measure joint success within prompts and compare repeated entities across prompts, separating basic realization failures from failures under additional requirements. We evaluate multiple open-source T2I models on English and Chinese prompts. The resulting question-level evidence supports reliable ranking and fine-grained diagnosis: joint completion falls from 80.7% for components with two capabilities to 37.2% for those with seven or more. Finally, we reuse the same records for training-free routing; our cost-aware router matches ERNIE's 89.51-point estimate with 21.3% less GPU-s/MP.
Shaoan Zhao, Fang Zhao, Xueqiang Guo +9
Data Science & Artificial Intelligence Research Institute, China Unicom · Unicom Data Intelligence, China Unicom · China Unicom Group Co.,Ltd
Text-to-image (T2I) models often generate the wrong number of objects, yet existing benchmarks are too small or weakly controlled to explain why. We introduce NumBench, a benchmark of 640{,}000 prompts spanning 1{,}600 categories and counts from 1 to 100. Its factorial design varies object composition, spatial guidance, and appearance conditions while balancing counts and category exposure. We also develop a process model in which requested instances compete for a finite set of resolvable image regions. The model predicts a near-quadratic collision deficit at low occupancy and shows how coordinated placement reduces it. For scalable evaluation, we propose the Confidence-Weighted Numeric Precision Score (\cwnps), which aggregates three calibrated detectors and discounts uncertain proposals. Across five commercial systems, two open models, and two specialized counting methods, performance declines sharply with requested count; all evaluated methods are weak above 50 objects. Count range has the largest measured effect, followed by layout and composition. Grid guidance is strongest among guided layouts, consistent with the coordination prediction, although the analysis does not establish collision as the sole cause. A 14{,}400-image human study supports automated evaluation through count 50, while results on 243 natural-language prompts show transfer beyond NumBench templates.
Sandeep Wadhwa, Mayank Vatsa, Richa Singh +2
Indian Institute of Technology Jodhpur, India · Shiv Nadar University, India