SANEval: Open-Vocabulary Compositional Benchmarks with Failure-mode Diagnosis
Organizations: Stony Brook University, NY, USA · 2nd Set AI Corp. · Rowan University, NJ, USA
Abstract
Text-to-image (T2I) models struggle with compositional prompts requiring multiple objects, bound attributes, accurate counts, or spatial relations. Existing benchmarks face limits from fixed detector vocabularies, aggregated scores, and varying human-validation protocols. We introduce SANEval (Spatial, Attribute, and Numeracy Evaluation), an open-vocabulary compositional benchmark with two key contributions: (i) a public dataset of ~5,250 prompts paired with images from six state-of-the-art T2I models, split into algorithmically generated Simple and fully human-written Hard partitions across five compositional tasks, and (ii) a modular evaluation pipeline combining an LLM prompt parser, an open-vocabulary detector with LLM synonym expansion, and scorers for attribute binding, spatial relations, and numeracy. Each scorer emits numerical scores and diagnostic feedback for missing, extra, or mis-bound elements. Validated against human ratings (n = 500), SANEval achieves positive Pearson correlations across all categories (overall r = 0.222), outperforming CompBench++ (-0.016) and GenEval (-0.030). A manual audit of 24,012 detections shows 96.58% precision, with ablation confirming that synonym expansion is critical to pipeline performance. Dataset: https://huggingface.co/datasets/saneval/saneval-release. Pipeline: https://github.com/nielseni6/saneval.
Figures & tables
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Pipeline Component | Latency (Local CPU) | Cost (USD) |
| Spatial Evaluation | 0.11 s | $0.02 |
| Numeracy Evaluation | 0.12 s | $0.03 |
| Attribute Evaluation | 0.26 s | $0.01 |
| Model | Spatial | Numeracy | Attribute-Binding | Total | ||
| Color | Shape | Texture | ||||
| Imagen 3.0 | 0.8160 | 0.7720 | 0.8400 | 0.5000 | 0.7240 | 0.7304 |
| Imagen 4.0 | 0.7800 | 0.6380 | 0.7990 | 0.5170 | 0.7410 | 0.6950 |
| Imagen 4.0 Ultra | 0.9390 | 0.8870 | 0.9680 | 0.8110 | 0.9190 | 0.9048 |
| Nano Banana | 0.9220 | 0.8680 | 0.9050 | 0.6810 | 0.8150 | 0.8382 |
| Seedream 3.0 | 0.9070 | 0.7430 | 0.9170 | 0.4310 | 0.6540 | 0.7304 |
| Evaluation | Prompt | Image | Score | Conformity Feedback |
| Spatial | “a mouse on the bottom of a painting” | 0.0 | Missing object: painting | |
| Numeracy | “four trucks” | 0.5 | Two trucks missing | |
| Color Binding | “a brown shirt and a pink apple” | 0.2 | Missing object: shirt [brown]; Poor attribute binding for apple: expected [pink], score = 0.20 0.75 | |
| Shape Binding | “a pentagonal spoon” | 0.0 | Poor attribute binding for spoon: expected [pentagonal], score = 0.00 0.75 | |
| Texture Binding | “a rubber grass” | 0.3 | Poor attribute binding for grass: expected [rubber], score = 0.30 0.75 |
T2I Generation Evaluation
Text-to-image generation evaluation measures how accurately, safely, and consistently image models turn prompts into images.
See all 182 papers