NumBench: Diagnosing Counting Failures in Text-to-Image Models

Sandeep Wadhwa; Mayank Vatsa; Richa Singh; Parrva Chirag Shah; Prakhar Galriya

cs.CVAug 28, 2026

NumBench: Diagnosing Counting Failures in Text-to-Image Models

Authors: Sandeep Wadhwa, Mayank Vatsa, Richa Singh, Parrva Chirag Shah, Prakhar Galriya

Organizations: Indian Institute of Technology Jodhpur, India · Shiv Nadar University, India

Abstract

Text-to-image (T2I) models often generate the wrong number of objects, yet existing benchmarks are too small or weakly controlled to explain why. We introduce NumBench, a benchmark of 640{,}000 prompts spanning 1{,}600 categories and counts from 1 to 100. Its factorial design varies object composition, spatial guidance, and appearance conditions while balancing counts and category exposure. We also develop a process model in which requested instances compete for a finite set of resolvable image regions. The model predicts a near-quadratic collision deficit at low occupancy and shows how coordinated placement reduces it. For scalable evaluation, we propose the Confidence-Weighted Numeric Precision Score (\cwnps), which aggregates three calibrated detectors and discounts uncertain proposals. Across five commercial systems, two open models, and two specialized counting methods, performance declines sharply with requested count; all evaluated methods are weak above 50 objects. Count range has the largest measured effect, followed by layout and composition. Grid guidance is strongest among guided layouts, consistent with the coordination prediction, although the analysis does not establish collision as the sole cause. A 14{,}400-image human study supports automated evaluation through count 50, while results on 243 natural-language prompts show transfer beyond NumBench templates.

Figures & tables

Figure 1: Counting failures across five commercial T2I systems. High-count prompts produce missing or merged objects, category confusion, and repeated default patterns. System labels identify the evaluated interfaces; exact collection dates are recorded in the release metadata.

Benchmark	Prompts	Max Count	Categories	Count	Layout	Appear.	Comp.	Joint
DrawBench † ( Saharia et al. 2022 )	$\sim$ 200	$\sim$ 10	$<20$	$\times$	$\times$	$\times$	$\times$	$\times$
PaintSkills † ( Cho et al. 2023 )	7,330	$\sim$ 5	$\sim$ 80	$\times$	$\times$	$\times$	$\times$	$\times$
GenEval † ( Ghosh et al. 2023 )	553	4	80	$\checkmark$	$\times$	$\times$	$\times$	$\times$
T2I-CompBench † ( Huang et al. 2023 )	6,000	$\sim$ 5	$\sim$ 200	$\times$	$\times$	$\times$	$\times$	$\times$
HRS-Bench † ( Bakr et al. 2023 )	45,000	$\sim$ 10	700+	$\times$	$\times$	$\times$	$\times$	$\times$
GeckoNum ( Kajić et al. 2024 )	1,386	$\sim$ 10	$\sim$ 30	$\checkmark$	$\times$	$\times$	$\checkmark$	$\times$

Table 1: Comparison with counting benchmarks. † Counting is one component of a broader suite. Joint indicates that count, layout, appearance, and composition are controlled within one benchmark design.

Dimension	Levels	Allocation
Composition	2	80K single + 80K double
Count	100	Equal mass at each integer
Category	1,600	Marginally balanced assignment
Spatial layout	4	Free-form / grid / scene / random
Appearance	2	Clean + modified
Total		640,000 prompts

Table 2: NumBench construction. Every core prompt receives a free-form version, one guided counterpart, and clean and modified appearance forms.

Group	Descriptors	Interpretation
Visual quality	blurry, deformed, low quality, artifacts	Primarily changes fidelity and detector confidence.
Instance evidence	overlapping, merged, partial, cropped	Can make the requested cardinality visually ambiguous.

Table 3: The modified-appearance condition combines quality degradation with instance-separation difficulty; analyses keep these roles conceptually distinct.

		Single Category								Double Category
		1–25		26–50		51–75 ‡		76–100 ‡		1–25		26–50		51–75 ‡		76–100 ‡
Class	Method	b	cw	b	cw	b	cw	b	cw	b	cw	b	cw	b	cw	b	cw
Open	SDXL	.602	.553	.402	.360	.250	.222	.099	.076	.483	.445	.301	.271	.152	.125	.051	.038
Open	FLUX.2 [dev]	.608	.560	.409	.367	.259	.231	.105	.081	.489	.450	.309	.278	.157	.131	.056	.040
Specialized	CountGen	.722	.692	.520	.485	.345	.311	.160	.129	.642	.610	.443	.410	.255	.230	.111	.088
Specialized	Bounded Attention	.669	.623	.482	.442	.315	.284	.142	.110	.671	.641	.465	.435	.279	.251	.125	.100

Table 4: Basic (b) and confidence-weighted (cw) numeric precision on the full NumBench evaluation. Broad degradation and method ordering persist under both metrics. ‡ Values above count 50 are stress-test results because human agreement and detector–human correlation weaken in this range.

Figure 2: cw-NPS on the balanced 3,200-prompt subset for five commercial systems, with SDXL included as an open-model reference. S and M denote single-category and two-category prompts, respectively.

Figure 3: Factor analysis. Axis and legend labels use shorthand: “Prompt Noise” denotes the appearance condition, “Object Config.” denotes composition, and “Single/Double-Object” denotes single- and double-category prompts. (a) Count range has the largest effect (normalized values). (b) Specialized methods improve low-count performance but not the high-count regime. (c) Composition changes difficulty across count ranges.

Count range	Fleiss’ $\boldsymbol{\kappa}$	Detector–human $r$
1–25	.82	$>.80$
26–50	.58	$>.80$
76–100 ‡	.33	$\approx.40$
Overall	—	$>.75$

Table 5: Aggregate human validation for the reported strata. The source annotations do not provide separate

\kappa

or detector–human

r

estimates for 51–75. ‡ High-count stress test.

Figure 4: Human validation for “11 cats and 9 dogs.” Detector-based and human-count cw-NPS differ by at most .05 for each shown system. This example illustrates agreement in the validated count range; the aggregate correlation analysis, not a single image, is the main evidence.

Figure 5: Transfer to CoCoCount+. Left: generations for “3 ginger kittens and 2 grey kittens” expose exact-count, color-binding, missing-object, and overcount failures even in the low-count regime. Right: FID versus cw-NPS across the nine systems; perceptual quality does not predict numeric precision, and no evaluated system exceeds .67 cw-NPS on the 243 prompts.

Failure	Observable symptom	Most relevant axes	What NumBench measures	Recommended reporting
Cardinality	Too few or too many visible target instances	Count, layout	Category-wise absolute count error through b-NPS and cw-NPS	Scores by count range and composition
Binding	Correct total but wrong category or attribute allocation	Composition, category	Separate target counts for every requested category	Per-category counts on double-category prompts
Visibility	Merged, partial, cropped, or ambiguous instances	Appearance, layout	Detector confidence plus human ambiguity and agreement	Human validation and explicit stress-test labels

Table 6: NumBench separates three failure families that require different evidence and reporting. A single total-count score cannot distinguish them.

T2I Generation

Text-to-image generation creates images from natural-language prompts using generative models, with research focused on fidelity, control, efficiency, and safety.

See all 290 papers

Benchmarking Frontier Text-to-Image Models on Image-Description Prompts
Aug 15, 2026Sajjad Abdoli, Ghassan Al-Sumaidaee, Ahmed RashadText Rendering in T2I Generation T2I Generation
Perle
Scalable Question-Centric Text-to-Image Evaluation: Reliable Ranking, Fine-Grained Diagnosis, and Cost-Aware Routing
Aug 25, 2026Shaoan Zhao, Fang Zhao, Xueqiang Guo +9T2I Generation Evaluation Compositional T2I Generation
Data Science & Artificial Intelligence Research Institute, China Unicom · Unicom Data Intelligence, China Unicom · China Unicom Group Co.,Ltd
Imag-Eval: a language-grounded framework for interpretable Text-to-Image instruction following evaluation
Aug 29, 2026Ibrahim Mohamed Serouis, David Jaramillo DuqueT2I Generation Evaluation Compositional T2I Generation
Talan Research and Innovation Center Toulouse, France · Talan Research and Innovation Center Paris, France

Aug 15, 2026J/K move · Enter open · S save

Benchmarking Frontier Text-to-Image Models on Image-Description Prompts

Sajjad Abdoli, Ghassan Al-Sumaidaee, Ahmed Rashad

Perle

Text Rendering in T2I Generation T2I Generation T2I Generation Evaluation Compositional T2I Generation

Open paper AskPDF arXiv

Text-to-image models are typically reported on average-case prompts, which understates the gap between systems on compositionally demanding requests involving precise object counts, multi-object attribute binding, legible embedded text, and explicit spatial constraints. We evaluate four production text-to-image systems: Hunyuan 3.0, Gemini 3 Pro Image ("Nano Banana Pro"), Black Forest Labs FLUX.2, and Ideogram 3.0. The evaluation uses the 48 hardest prompts drawn from the DataSeeds.AI Sample Dataset (DSD), selected through an automated complexity-scoring pass over the full corpus. Every generated image is graded using an independent-judge rubric. GPT-5.4-Pro authors an atomic, weighted, mutually exclusive and collectively exhaustive (MECE) evaluation rubric, while Gemini 3.1 Pro Preview independently determines whether each criterion is satisfied. Gemini 3 Pro Image ranks first with a score of 84.8/100, narrowly ahead of FLUX.2 at 82.3/100. Ideogram 3.0 and Hunyuan 3.0 score 65.7/100 and 63.3/100, respectively. Failure analysis shows that the leading systems primarily lose points through object miscounting and geometric artifacts, whereas the trailing systems more frequently produce garbled text. Ideogram 3.0 also frequently omits requested elements. Full per-sample rubrics, scores, and failure annotations are available from the authors upon request.

Abstract

T2I Generation

Explore similar work

Benchmarking Frontier Text-to-Image Models on Image-Description Prompts

Scalable Question-Centric Text-to-Image Evaluation: Reliable Ranking, Fine-Grained Diagnosis, and Cost-Aware Routing

Imag-Eval: a language-grounded framework for interpretable Text-to-Image instruction following evaluation