cs.CVJan 30, 2026

SANEval: Open-Vocabulary Compositional Benchmarks with Failure-mode Diagnosis

Authors: Rishav Pramanik, Ian E. Nielsen, Jeff Smith, Saurav Pandit, Ravi P. Ramachandran, Zhaozheng Yin

Organizations: Stony Brook University, NY, USA · 2nd Set AI Corp. · Rowan University, NJ, USA

Abstract

Text-to-image (T2I) models struggle with compositional prompts requiring multiple objects, bound attributes, accurate counts, or spatial relations. Existing benchmarks face limits from fixed detector vocabularies, aggregated scores, and varying human-validation protocols. We introduce SANEval (Spatial, Attribute, and Numeracy Evaluation), an open-vocabulary compositional benchmark with two key contributions: (i) a public dataset of ~5,250 prompts paired with images from six state-of-the-art T2I models, split into algorithmically generated Simple and fully human-written Hard partitions across five compositional tasks, and (ii) a modular evaluation pipeline combining an LLM prompt parser, an open-vocabulary detector with LLM synonym expansion, and scorers for attribute binding, spatial relations, and numeracy. Each scorer emits numerical scores and diagnostic feedback for missing, extra, or mis-bound elements. Validated against human ratings (n = 500), SANEval achieves positive Pearson correlations across all categories (overall r = 0.222), outperforming CompBench++ (-0.016) and GenEval (-0.030). A manual audit of 24,012 detections shows 96.58% precision, with ablation confirming that synonym expansion is critical to pipeline performance. Dataset: https://huggingface.co/datasets/saneval/saneval-release. Pipeline: https://github.com/nielseni6/saneval.

Figures & tables

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Imag-Eval: a language-grounded framework for interpretable Text-to-Image instruction following evaluation

    Aug 29, 2026Ibrahim Mohamed Serouis, David Jaramillo DuqueT2I Generation EvaluationCompositional T2I Generation

  2. Scalable Question-Centric Text-to-Image Evaluation: Reliable Ranking, Fine-Grained Diagnosis, and Cost-Aware Routing

    Aug 25, 2026Shaoan Zhao, Fang Zhao, Xueqiang Guo +9T2I Generation EvaluationCompositional T2I Generation

  3. NumBench: Diagnosing Counting Failures in Text-to-Image Models

    Aug 28, 2026Sandeep Wadhwa, Mayank Vatsa, Richa Singh +2T2I GenerationT2I Generation Evaluation